[jira] [Assigned] (KAFKA-4479) Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator
[
https://issues.apache.org/jira/browse/KAFKA-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit Daga reassigned KAFKA-4479:
Assignee: (was: Amit Daga)
> Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator
> ---
>
> Key: KAFKA-4479
> URL: https://issues.apache.org/jira/browse/KAFKA-4479
> Project: Kafka
> Issue Type: Improvement
>Reporter: Ismael Juma
>Priority: Minor
> Labels: newbie, newbie++
> Attachments: test.zip
>
>
> If we pass `KafkaServer.time` to `GroupCoordinator`[1], some streams tests
> like QueryableStateIntegrationTest fail sem-regularly. [~damianguy] looked
> into it and described it as:
> {quote}
> Looking at the sequence of events, one thread is stopped, and hence leaves
> the group triggering a rebalance, but the other thread doesn’t seem to get
> the memo, tries to commit, fails, and then game-over.
> So.. the case that it fails the one alive thread is not getting a rebalance.
> This would happen during a `poll(..)` right? However i can see the thread is
> polling many times after the other thread has shutdown.
> It tries to commit every time around the loop, so:
> poll(..)
> process(..)
> maybeCommit(..)
> and there is like < 10ms between calls to `poll`.
> {quote}
> A theory was that the mock time was not advancing enough to trigger a
> rebalance in the group coordinator. However, the consumer is closed, so that
> should trigger a `LeaveGroup` request and it's unclear why a rebalance is not
> triggered for the live consumer.
> PR where this issue was first seen and discussed:
> https://github.com/apache/kafka/pull/2095
> [1]
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaServer.scala#L222
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KAFKA-4479) Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator
[
https://issues.apache.org/jira/browse/KAFKA-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit Daga updated KAFKA-4479:
-
Attachment: test.zip
> Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator
> ---
>
> Key: KAFKA-4479
> URL: https://issues.apache.org/jira/browse/KAFKA-4479
> Project: Kafka
> Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Amit Daga
>Priority: Minor
> Labels: newbie, newbie++
> Attachments: test.zip
>
>
> If we pass `KafkaServer.time` to `GroupCoordinator`[1], some streams tests
> like QueryableStateIntegrationTest fail sem-regularly. [~damianguy] looked
> into it and described it as:
> {quote}
> Looking at the sequence of events, one thread is stopped, and hence leaves
> the group triggering a rebalance, but the other thread doesn’t seem to get
> the memo, tries to commit, fails, and then game-over.
> So.. the case that it fails the one alive thread is not getting a rebalance.
> This would happen during a `poll(..)` right? However i can see the thread is
> polling many times after the other thread has shutdown.
> It tries to commit every time around the loop, so:
> poll(..)
> process(..)
> maybeCommit(..)
> and there is like < 10ms between calls to `poll`.
> {quote}
> A theory was that the mock time was not advancing enough to trigger a
> rebalance in the group coordinator. However, the consumer is closed, so that
> should trigger a `LeaveGroup` request and it's unclear why a rebalance is not
> triggered for the live consumer.
> PR where this issue was first seen and discussed:
> https://github.com/apache/kafka/pull/2095
> [1]
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaServer.scala#L222
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KAFKA-4479) Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator
[
https://issues.apache.org/jira/browse/KAFKA-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001370#comment-16001370
]
Amit Daga commented on KAFKA-4479:
--
[~ijuma] Initial findings: After trying to debug for a while, I found that the
test fails even before we try to close the stream runnables. Lines beyond [1]
are not executed.
Test report has been attached for your reference (test.zip). Please let me know
your inputs on this.
[1]
https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/QueryableStateIntegrationTest.java#L357
> Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator
> ---
>
> Key: KAFKA-4479
> URL: https://issues.apache.org/jira/browse/KAFKA-4479
> Project: Kafka
> Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Amit Daga
>Priority: Minor
> Labels: newbie, newbie++
> Attachments: test.zip
>
>
> If we pass `KafkaServer.time` to `GroupCoordinator`[1], some streams tests
> like QueryableStateIntegrationTest fail sem-regularly. [~damianguy] looked
> into it and described it as:
> {quote}
> Looking at the sequence of events, one thread is stopped, and hence leaves
> the group triggering a rebalance, but the other thread doesn’t seem to get
> the memo, tries to commit, fails, and then game-over.
> So.. the case that it fails the one alive thread is not getting a rebalance.
> This would happen during a `poll(..)` right? However i can see the thread is
> polling many times after the other thread has shutdown.
> It tries to commit every time around the loop, so:
> poll(..)
> process(..)
> maybeCommit(..)
> and there is like < 10ms between calls to `poll`.
> {quote}
> A theory was that the mock time was not advancing enough to trigger a
> rebalance in the group coordinator. However, the consumer is closed, so that
> should trigger a `LeaveGroup` request and it's unclear why a rebalance is not
> triggered for the live consumer.
> PR where this issue was first seen and discussed:
> https://github.com/apache/kafka/pull/2095
> [1]
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaServer.scala#L222
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (KAFKA-4479) Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator
[
https://issues.apache.org/jira/browse/KAFKA-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit Daga reassigned KAFKA-4479:
Assignee: Amit Daga
> Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator
> ---
>
> Key: KAFKA-4479
> URL: https://issues.apache.org/jira/browse/KAFKA-4479
> Project: Kafka
> Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Amit Daga
>Priority: Minor
> Labels: newbie, newbie++
>
> If we pass `KafkaServer.time` to `GroupCoordinator`[1], some streams tests
> like QueryableStateIntegrationTest fail sem-regularly. [~damianguy] looked
> into it and described it as:
> {quote}
> Looking at the sequence of events, one thread is stopped, and hence leaves
> the group triggering a rebalance, but the other thread doesn’t seem to get
> the memo, tries to commit, fails, and then game-over.
> So.. the case that it fails the one alive thread is not getting a rebalance.
> This would happen during a `poll(..)` right? However i can see the thread is
> polling many times after the other thread has shutdown.
> It tries to commit every time around the loop, so:
> poll(..)
> process(..)
> maybeCommit(..)
> and there is like < 10ms between calls to `poll`.
> {quote}
> A theory was that the mock time was not advancing enough to trigger a
> rebalance in the group coordinator. However, the consumer is closed, so that
> should trigger a `LeaveGroup` request and it's unclear why a rebalance is not
> triggered for the live consumer.
> PR where this issue was first seen and discussed:
> https://github.com/apache/kafka/pull/2095
> [1]
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaServer.scala#L222
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Work started] (KAFKA-4996) Fix findbugs multithreaded correctness warnings for streams
[
https://issues.apache.org/jira/browse/KAFKA-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on KAFKA-4996 started by Amit Daga.
> Fix findbugs multithreaded correctness warnings for streams
> ---
>
> Key: KAFKA-4996
> URL: https://issues.apache.org/jira/browse/KAFKA-4996
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Colin P. McCabe
> Assignee: Amit Daga
> Labels: newbie
>
> Fix findbugs multithreaded correctness warnings for streams
> {code}
> Multithreaded correctness Warnings
>
>
>
>
>
>
>
>Code Warning
>
>
>
>AT Sequence of calls to java.util.concurrent.ConcurrentHashMap may not
> be atomic in
> org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(long,
> ProcessorContext)
>
>IS Inconsistent synchronization of
> org.apache.kafka.streams.KafkaStreams.stateListener; locked 66% of time
>
>
>
>IS Inconsistent synchronization of
> org.apache.kafka.streams.processor.internals.StreamThread.stateListener;
> locked 66% of time
>
>
>IS Inconsistent synchronization of
> org.apache.kafka.streams.processor.TopologyBuilder.applicationId; locked 50%
> of time
>
>
>IS Inconsistent synchronization of
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.context; locked
> 66% of time
>
>
>IS Inconsistent synchronization of
> org.apache.kafka.streams.state.internals.CachingWindowStore.cache; locked 60%
> of time
>
>
>IS Inconsistent synchronization of
> org.apache.kafka.streams.state.internals.CachingWindowStore.context; locked
> 66% of time
>
>
>IS Inconsistent synchronization of
> org.apache.kafka.streams.state.internals.CachingWindowStore.name; locked 60%
> of time
>
>
>IS Inconsistent synchronization of
> org.apache.kafka.streams.state.internals.CachingWindowStore.serdes; locked
> 70% of time
>
>
>IS Inconsistent synchronization of
> org.apache.kafka.streams.state.internals.RocksDBStore.db; locked 63% of time
>
>
>
>IS Inconsistent synchronization of
> org.apache.kafka.streams.state.internals.RocksDBStore.serdes; locked 76% of
> time
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Re: Request to add to the contributor list
Thank you Guozhang. On Thu, May 4, 2017 at 2:00 AM, Guozhang Wang <wangg...@gmail.com> wrote: > Hi Amit, > > I have added you to the contributor list and assigned 4996 to you. Cheers. > > Will also review your PR soon. > > > Guozhang > > > On Tue, May 2, 2017 at 5:43 AM, Amit Daga <amitdaga...@gmail.com> wrote: > > > Hello, > > > > Wondering if you were able to look into it? > > > > Thanks, > > Amit Daga > > > > On Sun, Apr 30, 2017 at 1:29 PM, Amit Daga <amitdaga...@gmail.com> > wrote: > > > > > Hello Team, > > > > > > Hope you are doing well. > > > > > > My name is Amit Daga. This is to request you to add me to the > contributor > > > list. Also if KAFKA-4996 issue is still not assigned, I would like to > > work > > > on it. > > > > > > Thanks, > > > Amit > > > > > > > > > -- > -- Guozhang >
Re: Request to add to the contributor list
Hello, Wondering if you were able to look into it? Thanks, Amit Daga On Sun, Apr 30, 2017 at 1:29 PM, Amit Daga <amitdaga...@gmail.com> wrote: > Hello Team, > > Hope you are doing well. > > My name is Amit Daga. This is to request you to add me to the contributor > list. Also if KAFKA-4996 issue is still not assigned, I would like to work > on it. > > Thanks, > Amit >
Request to add to the contributor list
Hello Team, Hope you are doing well. My name is Amit Daga. This is to request you to add me to the contributor list. Also if KAFKA-4996 issue is still not assigned, I would like to work on it. Thanks, Amit
