Build failed in Jenkins: kafka-trunk-jdk8 #2802

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-6884; Consumer group command should use new admin client (#5032)

[github] MINOR: update release.py (#5374)

[github] KAFKA-4041: Update ZooKeeper to 3.4.13 (#5376)

[wangguoz] KAFKA-7103: Use bulkloading for RocksDBSegmentedBytesStore during 
init

[jkoshy] KAFKA-5098; KafkaProducer should reject sends to invalid topics

[jason] MINOR: Fix broken Javadoc on [AbstractIndex|OffsetIndex] (#5370)

[github] MINOR: web docs fixes on message header (#5381)

--
[...truncated 867.83 KB...]
kafka.admin.DeleteTopicTest > testResumeDeleteTopicWithRecoveredFollower PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted PASSED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic STARTED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic STARTED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic PASSED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion STARTED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion PASSED

kafka.admin.DeleteTopicTest > testDisableDeleteTopic STARTED

kafka.admin.DeleteTopicTest > testDisableDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic STARTED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition PASSED

kafka.admin.DeleteTopicTest > testDeletingPartiallyDeletedTopic STARTED

kafka.admin.DeleteTopicTest > testDeletingPartiallyDeletedTopic PASSED

kafka.admin.AdminTest > testBasicPreferredReplicaElection STARTED

kafka.admin.AdminTest > testBasicPreferredReplicaElection PASSED

kafka.admin.AdminTest > testPreferredReplicaJsonData STARTED

kafka.admin.AdminTest > testPreferredReplicaJsonData PASSED

kafka.admin.AdminTest > testControlledShutdown STARTED

kafka.admin.AdminTest > testControlledShutdown PASSED

kafka.admin.AdminTest > testReassigningNonExistingPartition STARTED

kafka.admin.AdminTest > testReassigningNonExistingPartition PASSED

kafka.admin.AdminTest > testGetBrokerMetadatas STARTED

kafka.admin.AdminTest > testGetBrokerMetadatas PASSED

kafka.admin.AdminTest > testBootstrapClientIdConfig STARTED

kafka.admin.AdminTest > testBootstrapClientIdConfig PASSED

kafka.admin.AdminTest > testPartitionReassignmentNonOverlappingReplicas STARTED

kafka.admin.AdminTest > testPartitionReassignmentNonOverlappingReplicas PASSED

kafka.admin.AdminTest > testReplicaAssignment STARTED

kafka.admin.AdminTest > testReplicaAssignment PASSED

kafka.admin.AdminTest > testPartitionReassignmentWithLeaderNotInNewReplicas 
STARTED

kafka.admin.AdminTest > testPartitionReassignmentWithLeaderNotInNewReplicas 
PASSED

kafka.admin.AdminTest > testTopicConfigChange STARTED

kafka.admin.AdminTest > testTopicConfigChange PASSED

kafka.admin.AdminTest > testResumePartitionReassignmentThatWasCompleted STARTED

kafka.admin.AdminTest > testResumePartitionReassignmentThatWasCompleted PASSED

kafka.admin.AdminTest > testManualReplicaAssignment STARTED

kafka.admin.AdminTest > testManualReplicaAssignment PASSED

kafka.admin.AdminTest > testConcurrentTopicCreation STARTED

kafka.admin.AdminTest > testConcurrentTopicCreation PASSED

kafka.admin.AdminTest > testPartitionReassignmentWithLeaderInNewReplicas STARTED

kafka.admin.AdminTest > testPartitionReassignmentWithLeaderInNewReplicas PASSED

kafka.admin.AdminTest > shouldPropagateDynamicBrokerConfigs STARTED

kafka.admin.AdminTest > shouldPropagateDynamicBrokerConfigs PASSED

kafka.admin.AdminTest > testTopicCreationWithCollision STARTED

kafka.admin.AdminTest > testTopicCreationWithCollision PASSED

kafka.admin.AdminTest > testTopicCreationInZK STARTED

kafka.admin.AdminTest > testTopicCreationInZK PASSED

kafka.admin.AclCommandTest > testInvalidAuthorizerProperty STARTED

kafka.admin.AclCommandTest > testInvalidAuthorizerProperty PASSED

kafka.admin.AclCommandTest > testAclsOnPrefixedResources STARTED

kafka.admin.AclCommandTest > testAclsOnPrefixedResources PASSED

kafka.admin.AclCommandTest > testAclCli STARTED

kafka.admin.AclCommandTest > testAclCli PASSED

kafka.admin.AclCommandTest > testProducerConsumerCli STARTED

kafka.admin.AclCommandTest > testProducerConsumerCli PASSED

kafka.admin.ConfigCommandTest > testScramCredentials STARTED

kafka.admin.ConfigCommandTest > testScramCredentials PASSED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForTopicsEntityType STARTED

kafka.admin.ConfigCommandTest > 

Re: KIP-327: Add describe all topics API to AdminClient

[Colin McCabe]

Thanks, Manikumar.

best,
Colin

On Tue, Jul 17, 2018, at 19:44, Manikumar wrote:
> Closing this KIP in favor of adding filtering support to the Metadata API
> and KIP-142. Will open a new KIP when ready.
> Thanks for your reviews.
> 
> On Mon, Jul 16, 2018 at 8:38 AM Colin McCabe  wrote:
> 
> > Thanks, Manikumar.  I've been meaning to bring up KIP-142 again.  It would
> > definitely be a nice improvement.
> >
> > best,
> > Colin
> >
> >
> > On Sat, Jul 14, 2018, at 08:51, Manikumar wrote:
> > > Hi Jason and Colin,
> > >
> > > Thanks for the feedback. I agree that having filtering support to the
> > > Metadata API would be useful and solves
> > > the scalability issues.
> > >
> > > But to implement specific use case of "describe all topics", regex
> > > support
> > > won't help. In any case user needs to
> > > call listTopics() to get topic list, and then make describeTopics()
> > > calls
> > > with a subset of the topics set.
> > > This leads to improving existing listTopics() API performance. Colin
> > > already raised a KIP for this: KIP-142
> > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-142%3A+Add+ListTopicsRequest+to+efficiently+list+all+the+topics+in+a+cluster
> > >
> > >  .
> > > May be we should consider implementing KIP-142.
> > >
> > > Since we have support wildcard ACLs, Initially, I can explore
> > > prefixed/wildcards patterns support to Metadata API.
> > > We can later extend support for regular expressions.
> > >
> > > Thanks
> > >
> > >
> > >
> > > On Sat, Jul 14, 2018 at 2:42 PM Ted Yu  wrote:
> > >
> > > > What if broker crashes before all the pages can be returned ?
> > > >
> > > > Cheers
> > > >
> > > > On Sat, Jul 14, 2018 at 1:07 AM Stephane Maarek <
> > > > steph...@simplemachines.com.au> wrote:
> > > >
> > > > > Why not paginate ? Then one can retrieve as many topics as desired ?
> > > > >
> > > > > On Sat., 14 Jul. 2018, 4:15 pm Colin McCabe, 
> > wrote:
> > > > >
> > > > > > Good point.  We should probably have a maximum number of results
> > like
> > > > > > 1000 or something.  That can go in the request RPC as well...
> > > > > > Cheers,
> > > > > > Colin
> > > > > >
> > > > > > On Fri, Jul 13, 2018, at 18:15, Ted Yu wrote:
> > > > > > > bq. describe topics by a regular expression on the server side
> > > > > > >
> > > > > > > Should caution be taken if the regex doesn't filter ("*") ?
> > > > > > >
> > > > > > > Cheers
> > > > > > >
> > > > > > > On Fri, Jul 13, 2018 at 6:02 PM Colin McCabe
> > > > > > >  wrote:>
> > > > > > > > As Jason wrote, this won't scale as the number of partitions
> > > > > > > > increases.> > We already have users who have tens of thousands
> > of
> > > > > > topics, or
> > > > > > > > more.  If> > you multiply that by 100x over the next few
> > years, you
> > > > > > end up with
> > > > > > > > this API> > returning full information about millions of
> > topics,
> > > > > which
> > > > > > clearly
> > > > > > > > doesn't> > work.
> > > > > > > >
> > > > > > > > We discussed this a lot in the original KIP-117 DISCUSS thread
> > > > > > > > which added> > the Java AdminClient.  ListTopics and
> > DescribeTopics
> > > > > > were
> > > > > > > > deliberately kept> > separate because we understood that
> > > > eventually a
> > > > > > single RPC would
> > > > > > > > not be> > able to return information about all the topics in
> > the
> > > > > > cluster.  So
> > > > > > > > I have> > to vote -1 for this proposal as it stands.
> > > > > > > >
> > > > > > > > I do agree that adding a way to describe topics by a regular
> > > > > > > > expression on> > the server side would be very useful.  This
> > would
> > > > > > also fix a major
> > > > > > > > scalability problem we have now, which is that when
> > > > > > > > subscribing via a> > regular expression, clients need to fetch
> > the
> > > > > > full list of all
> > > > > > > > topics in> > the cluster and filter locally.
> > > > > > > >
> > > > > > > > I think a regular expression library like re2 would be ideal
> > > > > > > > for this> > purpose.  re2 is standardized and language-agnostic
> > > > (it's
> > > > > > not tied
> > > > > > > > only to> > Java).  In contrast, Java regular expression change
> > with
> > > > > > different
> > > > > > > > releases> > of the JDK (there were some changes in java 8, for
> > > > > > example).
> > > > > > > > Also, re2> > regular expressions are linear time, never
> > exponential
> > > > > > time.  See
> > > > > > > > https://github.com/google/re2j
> > > > > > > >
> > > > > > > > regards,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 13, 2018, at 05:00, Andras Beni wrote:
> > > > > > > > > The KIP looks good to me.
> > > > > > > > > However, if there is willingness in the community to work on
> > > > > > > > > metadata> > > request with patterns, the feature proposed
> > here
> > > > and
> > > > > > filtering by
> > > > > > > > > '*' or> > > '.*' would be redundant.
> > > > > > > > >
> > > > > > > > > Andras
> > 

Build failed in Jenkins: kafka-2.0-jdk8 #73

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: update release.py (#5374)

[ismael] KAFKA-4041: Update ZooKeeper to 3.4.13 (#5376)

[ismael] MINOR: Update Jetty to 9.4.11 (#5377)

--
[...truncated 2.46 MB...]

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedReduceIfReducerIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedReduceIfReducerIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedAggregateIfInitializerIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedAggregateIfInitializerIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedAggregateIfMaterializedIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedAggregateIfMaterializedIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedAggregateIfMergerIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedAggregateIfMergerIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldMaterializeCount STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldMaterializeCount PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldMaterializeAggregated STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldMaterializeAggregated PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnCountIfMaterializedIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnCountIfMaterializedIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnAggregateIfAggregatorIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnAggregateIfAggregatorIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnReduceIfReducerIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnReduceIfReducerIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldAggregateSessionWindowed STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldAggregateSessionWindowed PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldCountSessionWindowed STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldCountSessionWindowed PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldReduceWindowed STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldReduceWindowed PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedAggregateIfAggregatorIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedAggregateIfAggregatorIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnAggregateIfInitializerIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnAggregateIfInitializerIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnAggregateIfMergerIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnAggregateIfMergerIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedReduceIfMaterializedIsNull STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldThrowNullPointerOnMaterializedReduceIfMaterializedIsNull PASSED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldMaterializeReduced STARTED

org.apache.kafka.streams.kstream.internals.SessionWindowedKStreamImplTest > 
shouldMaterializeReduced PASSED

org.apache.kafka.streams.kstream.internals.KStreamGlobalKTableLeftJoinTest > 
shouldClearGlobalTableEntryOnNullValueUpdates STARTED

org.apache.kafka.streams.kstream.internals.KStreamGlobalKTableLeftJoinTest > 

Re: KIP-327: Add describe all topics API to AdminClient

[Manikumar]

Closing this KIP in favor of adding filtering support to the Metadata API
and KIP-142. Will open a new KIP when ready.
Thanks for your reviews.

On Mon, Jul 16, 2018 at 8:38 AM Colin McCabe  wrote:

> Thanks, Manikumar.  I've been meaning to bring up KIP-142 again.  It would
> definitely be a nice improvement.
>
> best,
> Colin
>
>
> On Sat, Jul 14, 2018, at 08:51, Manikumar wrote:
> > Hi Jason and Colin,
> >
> > Thanks for the feedback. I agree that having filtering support to the
> > Metadata API would be useful and solves
> > the scalability issues.
> >
> > But to implement specific use case of "describe all topics", regex
> > support
> > won't help. In any case user needs to
> > call listTopics() to get topic list, and then make describeTopics()
> > calls
> > with a subset of the topics set.
> > This leads to improving existing listTopics() API performance. Colin
> > already raised a KIP for this: KIP-142
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-142%3A+Add+ListTopicsRequest+to+efficiently+list+all+the+topics+in+a+cluster
> >
> >  .
> > May be we should consider implementing KIP-142.
> >
> > Since we have support wildcard ACLs, Initially, I can explore
> > prefixed/wildcards patterns support to Metadata API.
> > We can later extend support for regular expressions.
> >
> > Thanks
> >
> >
> >
> > On Sat, Jul 14, 2018 at 2:42 PM Ted Yu  wrote:
> >
> > > What if broker crashes before all the pages can be returned ?
> > >
> > > Cheers
> > >
> > > On Sat, Jul 14, 2018 at 1:07 AM Stephane Maarek <
> > > steph...@simplemachines.com.au> wrote:
> > >
> > > > Why not paginate ? Then one can retrieve as many topics as desired ?
> > > >
> > > > On Sat., 14 Jul. 2018, 4:15 pm Colin McCabe, 
> wrote:
> > > >
> > > > > Good point.  We should probably have a maximum number of results
> like
> > > > > 1000 or something.  That can go in the request RPC as well...
> > > > > Cheers,
> > > > > Colin
> > > > >
> > > > > On Fri, Jul 13, 2018, at 18:15, Ted Yu wrote:
> > > > > > bq. describe topics by a regular expression on the server side
> > > > > >
> > > > > > Should caution be taken if the regex doesn't filter ("*") ?
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Fri, Jul 13, 2018 at 6:02 PM Colin McCabe
> > > > > >  wrote:>
> > > > > > > As Jason wrote, this won't scale as the number of partitions
> > > > > > > increases.> > We already have users who have tens of thousands
> of
> > > > > topics, or
> > > > > > > more.  If> > you multiply that by 100x over the next few
> years, you
> > > > > end up with
> > > > > > > this API> > returning full information about millions of
> topics,
> > > > which
> > > > > clearly
> > > > > > > doesn't> > work.
> > > > > > >
> > > > > > > We discussed this a lot in the original KIP-117 DISCUSS thread
> > > > > > > which added> > the Java AdminClient.  ListTopics and
> DescribeTopics
> > > > > were
> > > > > > > deliberately kept> > separate because we understood that
> > > eventually a
> > > > > single RPC would
> > > > > > > not be> > able to return information about all the topics in
> the
> > > > > cluster.  So
> > > > > > > I have> > to vote -1 for this proposal as it stands.
> > > > > > >
> > > > > > > I do agree that adding a way to describe topics by a regular
> > > > > > > expression on> > the server side would be very useful.  This
> would
> > > > > also fix a major
> > > > > > > scalability problem we have now, which is that when
> > > > > > > subscribing via a> > regular expression, clients need to fetch
> the
> > > > > full list of all
> > > > > > > topics in> > the cluster and filter locally.
> > > > > > >
> > > > > > > I think a regular expression library like re2 would be ideal
> > > > > > > for this> > purpose.  re2 is standardized and language-agnostic
> > > (it's
> > > > > not tied
> > > > > > > only to> > Java).  In contrast, Java regular expression change
> with
> > > > > different
> > > > > > > releases> > of the JDK (there were some changes in java 8, for
> > > > > example).
> > > > > > > Also, re2> > regular expressions are linear time, never
> exponential
> > > > > time.  See
> > > > > > > https://github.com/google/re2j
> > > > > > >
> > > > > > > regards,
> > > > > > > Colin
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 13, 2018, at 05:00, Andras Beni wrote:
> > > > > > > > The KIP looks good to me.
> > > > > > > > However, if there is willingness in the community to work on
> > > > > > > > metadata> > > request with patterns, the feature proposed
> here
> > > and
> > > > > filtering by
> > > > > > > > '*' or> > > '.*' would be redundant.
> > > > > > > >
> > > > > > > > Andras
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 13, 2018 at 12:38 AM Jason Gustafson
> > > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Manikumar,
> > > > > > > > >
> > > > > > > > > As Kafka begins to scale to larger and larger numbers of
> > > > > > > topics/partitions,
> > > > > > > > > I'm a little 

Re: [VOTE] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

[Gwen Shapira]

+1 (binding)

Thank you for the improvement!

On Mon, Jul 16, 2018 at 4:10 AM, Chia-Ping Tsai  wrote:

> hi folks,
>
> The discussion[1] of KIP-338[2] did not get any objection for last 6 days
> so it is time to start the voting thread.
>
> Thanks for your time!
>
> [1] https://lists.apache.org/thread.html/9bd4e61b73c901b51132ada49743b9
> b703d40b85fc4eeaa5c9099900@%3Cdev.kafka.apache.org%3E
>
> [2] https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 338+Support+to+exclude+the+internal+topics+in+kafka-topics.sh+command
>
> Cheers,
> chia-ping
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: [VOTE] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

[Chia-Ping Tsai]

hi Jason
> I think eventually we'd want --exclude-internal-topics to be the default,

(just a double check) the option name used in kip-338 is "--exclude-internal" 
rather than "--exclude-internal-topics". Do you prefer 
"--exclude-internal-topics"? 

Cheers,
Chia-Ping

On 2018/07/17 19:14:09, Jason Gustafson  wrote: 
> Hi, thanks for the KIP. +1 from me.
> 
> I think eventually we'd want --exclude-internal-topics to be the default,
> but it seems reasonable to keep the current behavior for compatibility.
> 
> -Jason
> 
> On Mon, Jul 16, 2018 at 2:23 PM, Dong Lin  wrote:
> 
> > +1
> >
> > Thanks!
> > Dong
> >
> > On Mon, Jul 16, 2018 at 9:17 AM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > +1 (non-binding)
> > >
> > > Thanks!
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From:   Chia-Ping Tsai 
> > > To: 
> > > Date:   07/16/2018 01:11 AM
> > > Subject:[VOTE] KIP-338 Support to exclude the internal topics in
> > > kafka-topics.sh command
> > >
> > >
> > >
> > > hi folks,
> > >
> > > The discussion[1] of KIP-338[2] did not get any objection for last 6 days
> > > so it is time to start the voting thread.
> > >
> > > Thanks for your time!
> > >
> > > [1]
> > > https://lists.apache.org/thread.html/9bd4e61b73c901b51132ada49743b9
> > > b703d40b85fc4eeaa5c9099900@%3Cdev.kafka.apache.org%3E
> > >
> > >
> > > [2]
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 338+Support+to+exclude+the+internal+topics+in+kafka-topics.sh+command
> > >
> > >
> > > Cheers,
> > > chia-ping
> > >
> > >
> > >
> > >
> > >
> > >
> >
> 

[jira] [Created] (KAFKA-7177) Update 2.0 documentation to reflect changed quota behaviors by KIP-219

[Jon Lee (JIRA)]

Jon Lee created KAFKA-7177:
--

 Summary: Update 2.0 documentation to reflect changed quota 
behaviors by KIP-219 
 Key: KAFKA-7177
 URL: https://issues.apache.org/jira/browse/KAFKA-7177
 Project: Kafka
  Issue Type: Task
  Components: documentation
Reporter: Jon Lee


KIP-219 changed the way quota violation is communicated between clients and 
brokers. Documentation should be updated accordingly. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7176) State store metrics for migrated tasks are not removed

[Sam Lendle (JIRA)]

Sam Lendle created KAFKA-7176:
-

 Summary: State store metrics for migrated tasks are not removed
 Key: KAFKA-7176
 URL: https://issues.apache.org/jira/browse/KAFKA-7176
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.1.0
Reporter: Sam Lendle


I observed that state store metrics for tasks that have been migrated to other 
instances are not removed and are still being updated with phantom values, 
(when viewed for example via jmx mbeans). 

For all tasks/threads on the same instance (including for migrated tasks), the 
values of state store metrics are all (nearly) the same. For the rate metrics 
at least, the value reported for each task is the rate I expect for all active 
tasks on that instance, so things are apparently being counted multiple times. 
Presumably, this is how migrated task metrics are being updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5098) KafkaProducer.send() blocks and generates TimeoutException if topic name has illegal char

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy resolved KAFKA-5098.
---
Resolution: Fixed

> KafkaProducer.send() blocks and generates TimeoutException if topic name has 
> illegal char
> -
>
> Key: KAFKA-5098
> URL: https://issues.apache.org/jira/browse/KAFKA-5098
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.2.0
> Environment: Java client running against server using 
> wurstmeister/kafka Docker image.
>Reporter: Jeff Larsen
>Assignee: Ahmed Al-Mehdi
>Priority: Major
> Fix For: 2.1.0
>
>
> The server is running with auto create enabled. If we try to publish to a 
> topic with a forward slash in the name, the call blocks and we get a 
> TimeoutException in the Callback. I would expect it to return immediately 
> with an InvalidTopicException.
> There are other blocking issues that have been reported which may be related 
> to some degree, but this particular cause seems unrelated.
> Sample code:
> {code}
> import org.apache.kafka.clients.producer.*;
> import java.util.*;
> public class KafkaProducerUnexpectedBlockingAndTimeoutException {
>   public static void main(String[] args) {
> Properties props = new Properties();
> props.put("bootstrap.servers", "kafka.example.com:9092");
> props.put("key.serializer", 
> "org.apache.kafka.common.serialization.StringSerializer");
> props.put("value.serializer", 
> "org.apache.kafka.common.serialization.StringSerializer");
> props.put("max.block.ms", 1); // 10 seconds should illustrate our 
> point
> String separator = "/";
> //String separator = "_";
> try (Producer producer = new KafkaProducer<>(props)) {
>   System.out.println("Calling KafkaProducer.send() at " + new Date());
>   producer.send(
>   new ProducerRecord("abc" + separator + 
> "someStreamName",
>   "Not expecting a TimeoutException here"),
>   new Callback() {
> @Override
> public void onCompletion(RecordMetadata metadata, Exception e) {
>   if (e != null) {
> System.out.println(e.toString());
>   }
> }
>   });
>   System.out.println("KafkaProducer.send() completed at " + new Date());
> }
>   }
> }
> {code}
> Switching to the underscore separator in the above example works as expected.
> Mea culpa: We neglected to research allowed chars in a topic name, but the 
> TimeoutException we encountered did not help point us in the right direction.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7175) Make version checking logic more flexible in streams_upgrade_test.py

[Ted Yu (JIRA)]

Ted Yu created KAFKA-7175:
-

 Summary: Make version checking logic more flexible in 
streams_upgrade_test.py
 Key: KAFKA-7175
 URL: https://issues.apache.org/jira/browse/KAFKA-7175
 Project: Kafka
  Issue Type: Improvement
Reporter: Ted Yu


During debugging of system test failure for KAFKA-5037, it was re-discovered 
that the version numbers inside version probing related messages are hard coded 
in streams_upgrade_test.py

This is in-flexible.

We should correlate latest version from Java class with the expected version 
numbers.

Matthias made the following suggestion:

We should also make this more generic and test upgrades from 3 -> 4, 3 -> 5 and 
4 -> 5. The current code does only go from latest version to future version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7174) Improve version probing of subscription info

[Ted Yu (JIRA)]

Ted Yu created KAFKA-7174:
-

 Summary: Improve version probing of subscription info
 Key: KAFKA-7174
 URL: https://issues.apache.org/jira/browse/KAFKA-7174
 Project: Kafka
  Issue Type: Improvement
Reporter: Ted Yu


During code review for KAFKA-5037, [~guozhang] made the following suggestion:

Currently the version probing works as the following:

when leader receives the subscription info encoded with a higher version that 
it can understand (e.g. the leader is on version 3, while one of the 
subscription received is encode with version 4), it will send back an empty 
assignment with the assignment encoded with version 3, and also 
latestSupportedVersion set to 3.

when the member receives the assignment, it checks if latestSupportedVersion is 
smaller than the version it used for encoding the sent subscription (i.e. the 
above logic). If it is smaller, then it means that leader cannot understand, in 
this case, version 4. It will then set the flag and then re-subscribe but with 
a down-graded encoding format of version 3.

NOW with PR #5322, we can let leader to clearly communicate this error via the 
error code, and upon receiving the assignment, if the error code is 
VERSION_PROBING, then the member can immediately know what happens, and hence 
can simplify the above logic. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] KIP-342 Add Customizable SASL extensions to OAuthBearer authentication

[Stanislav Kozlovski]

Hey group,

I just created a new KIP about adding customizable SASL extensions to the
OAuthBearer authentication mechanism. More details in the proposal

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-342%3A+Add+support+for+Custom+SASL+extensions+in+OAuthBearer+authentication
JIRA: KAFKA-7169 
PR: Pull request 

-- 
Best,
Stanislav

Build failed in Jenkins: kafka-trunk-jdk8 #2801

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Update Jetty to 9.4.11 (#5377)

[mjsax] MINOR: make the constructor of InMemoryKeyValueStore public so that it

--
[...truncated 881.86 KB...]
kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumVerifyOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumVerifyOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutTopicsOption PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithUnevenRacks 
STARTED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithUnevenRacks 
PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRackAware STARTED

kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRackAware PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithUnevenReplicas 
STARTED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithUnevenReplicas 
PASSED

kafka.admin.AdminRackAwareTest > testSkipBrokerWithReplicaAlreadyAssigned 
STARTED

kafka.admin.AdminRackAwareTest > testSkipBrokerWithReplicaAlreadyAssigned PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAware STARTED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAware PASSED

kafka.admin.AdminRackAwareTest > testRackAwareExpansion STARTED

kafka.admin.AdminRackAwareTest > testRackAwareExpansion PASSED

kafka.admin.AdminRackAwareTest > 
testAssignmentWith2ReplicasRackAwareWith6Partitions STARTED

kafka.admin.AdminRackAwareTest > 
testAssignmentWith2ReplicasRackAwareWith6Partitions PASSED

kafka.admin.AdminRackAwareTest > 
testAssignmentWith2ReplicasRackAwareWith6PartitionsAnd3Brokers STARTED

kafka.admin.AdminRackAwareTest > 
testAssignmentWith2ReplicasRackAwareWith6PartitionsAnd3Brokers PASSED

kafka.admin.AdminRackAwareTest > 
testGetRackAlternatedBrokerListAndAssignReplicasToBrokers STARTED

kafka.admin.AdminRackAwareTest > 
testGetRackAlternatedBrokerListAndAssignReplicasToBrokers PASSED

kafka.admin.AdminRackAwareTest > testMoreReplicasThanRacks STARTED

kafka.admin.AdminRackAwareTest > testMoreReplicasThanRacks PASSED

kafka.admin.AdminRackAwareTest > testSingleRack STARTED

kafka.admin.AdminRackAwareTest > testSingleRack PASSED

kafka.admin.AdminRackAwareTest > 
testAssignmentWithRackAwareWithRandomStartIndex STARTED

kafka.admin.AdminRackAwareTest > 
testAssignmentWithRackAwareWithRandomStartIndex PASSED

kafka.admin.AdminRackAwareTest > testLargeNumberPartitionsAssignment STARTED

kafka.admin.AdminRackAwareTest > testLargeNumberPartitionsAssignment PASSED

kafka.admin.AdminRackAwareTest > testLessReplicasThanRacks STARTED

kafka.admin.AdminRackAwareTest > testLessReplicasThanRacks PASSED

kafka.admin.ListConsumerGroupTest > classMethod STARTED

kafka.admin.ListConsumerGroupTest > classMethod FAILED
java.lang.AssertionError: Found unexpected threads 

Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol

[Vahid S Hashemian]

Hi Jason,

Thanks for the feedback. That sounds good to me.
I'll update the KIP accordingly and move the local generation option to 
"Rejected Alternatives".
I'll send a note when the revised KIP is ready.

--Vahid



From:   Jason Gustafson 
To: dev 
Date:   07/17/2018 02:22 PM
Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data 
Protocol



Hey Vahid,

I'm with Mike that it seems simpler to just use the consumer group
generation. Even if you can figure out how to reason about the local
generation, it still seems confusing to have something called "generation"
which is not the consumer group generation. There doesn't seem to be much
downside to using the actual generation as far as I can tell and it is 
much
easier to reason about. I was thinking we could expose it in
PartitionAssignor.Assignment as an additional field. I think this wouldn't
impact existing assignors, though it would allow for other stateful
assignors to use the field in a similar way.

It's a pity we didn't add a version to this schema, by the way. There's 
not
much we can do if we can only add fields at the end of the schema, but I
guess we're hoping this is the last change ;-)

Thanks,
Jason

On Fri, Jul 13, 2018 at 10:07 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Mike,
>
>
>
> Thanks a lot for reviewing the KIP and sharing your feedback.
>
> I agree that such an issue could surface with option 1, but the existing
>
> PR (that currently implements this option) checks for such duplicate
>
> assignments and ignores one in favor of the other. So at the end there
>
> will be valid (non-duplicate) assignments to consumers, but they might
>
> deviate a bit from the ideal assignment.
>
> If rare scenarios like this are deemed troublesome and we want to avoid
>
> them, option 2 would probably be the way to go. In that case and in my
>
> opinion, option 2 would a better solution compared to introducing 
another
>
> field (e.g. timestamp).
>
>
>
> Regards.
>
> --Vahid
>
>
>
>
>
>
>
> From:   Mike Freyberger 
>
> To: "dev@kafka.apache.org" 
>
> Date:   07/13/2018 08:42 PM
>
> Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User 
Data
>
> Protocol
>
>
>
>
>
>
>
> This is great!
>
>
>
> For the client side implementation, I think it’s still possible for 
there
>
> to be a duplication. I’ll try to walk through the example here.
>
>
>
> Let’s says there are 2 consumers, 1 topic with 2 partitions.
>
>
>
> After the initial rebalance, generation 0:
>
> Consumer A has partition 0
>
> Consumer B has partition 1
>
>
>
> Let’s say consumer B leaves the group (long debug, GC pause). This leads
>
> to another rebalance. This rebalance will be considered generation 1 and
>
> will result in:
>
>
>
> Generation 1, Consumer A owns partition 0,1
>
>
>
> Now let’s say Consumer B is still out of the group and then Consumer A
>
> leaves as well. While Consumer A is out of the group, Consumer B rejoins
>
> the group. During this rebalance, the only previous state would be the
>
> initial generation 0 assignment. So this assignment would be considered
>
> generation 1 as well and would result in:
>
>
>
> Generation 1, Consumer B owns partition 0,1
>
>
>
> When A rejoins the group, both consumers would claim ownership of both
>
> partitions and they would report the assignment was from generation 1.
>
> This gets us back into the same issue as before because the generation
>
> number cannot help at all. You could add a timestamp in addition to the
>
> generation marker, but that’d still be vulnerable to clock skew.
>
>
>
> Would hooking into the existing generation marker protect the assignor 
for
>
> this kind of situation? We need to make sure the selected implantation 
is
>
> protected against the kind of failure mentioned above.
>
>
>
> Also, I have been working on KIP-315, which is another Sticky Assignor,
>
> which also requires some kind of epoch/generation marker to be protected
>
> against zombies. So, I’d be in favor of a generic solution here that 
other
>
> assignors can leverage.
>
>
>
> Best,
>
>
>
> Mike Freyberger
>
>
>
> > On Jul 13, 2018, at 6:15 PM, Vahid S Hashemian
>
>  wrote:
>
> >
>
> > Hi all,
>
> >
>
> > I create a short KIP to address an issue in Sticky Assignor assignment
>
> > logic:
>
> >
>
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-

> 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol
>
>
>
> > Please take a look and share your feedback / comments.
>
> >
>
> > In particular, there is a Generation Marker section (
>
> >
>
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-

> 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol#KIP-
> 341:UpdateStickyAssignor'sUserDataProtocol-GenerationMarker
>
>
>
> > ) that provides two methods for implementing the improvement to the
>
> > protocol. I'd like to know which option is more popular.
>
> >
>
> > Thanks!
>
> > --Vahid
>
> >
>
> >
>
>
>
>
>
>
>
>
>
>
>
>

Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol

[Jason Gustafson]

Hey Vahid,

I'm with Mike that it seems simpler to just use the consumer group
generation. Even if you can figure out how to reason about the local
generation, it still seems confusing to have something called "generation"
which is not the consumer group generation. There doesn't seem to be much
downside to using the actual generation as far as I can tell and it is much
easier to reason about. I was thinking we could expose it in
PartitionAssignor.Assignment as an additional field. I think this wouldn't
impact existing assignors, though it would allow for other stateful
assignors to use the field in a similar way.

It's a pity we didn't add a version to this schema, by the way. There's not
much we can do if we can only add fields at the end of the schema, but I
guess we're hoping this is the last change ;-)

Thanks,
Jason

On Fri, Jul 13, 2018 at 10:07 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Mike,
>
>
>
> Thanks a lot for reviewing the KIP and sharing your feedback.
>
> I agree that such an issue could surface with option 1, but the existing
>
> PR (that currently implements this option) checks for such duplicate
>
> assignments and ignores one in favor of the other. So at the end there
>
> will be valid (non-duplicate) assignments to consumers, but they might
>
> deviate a bit from the ideal assignment.
>
> If rare scenarios like this are deemed troublesome and we want to avoid
>
> them, option 2 would probably be the way to go. In that case and in my
>
> opinion, option 2 would a better solution compared to introducing another
>
> field (e.g. timestamp).
>
>
>
> Regards.
>
> --Vahid
>
>
>
>
>
>
>
> From:   Mike Freyberger 
>
> To: "dev@kafka.apache.org" 
>
> Date:   07/13/2018 08:42 PM
>
> Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data
>
> Protocol
>
>
>
>
>
>
>
> This is great!
>
>
>
> For the client side implementation, I think it’s still possible for there
>
> to be a duplication. I’ll try to walk through the example here.
>
>
>
> Let’s says there are 2 consumers, 1 topic with 2 partitions.
>
>
>
> After the initial rebalance, generation 0:
>
> Consumer A has partition 0
>
> Consumer B has partition 1
>
>
>
> Let’s say consumer B leaves the group (long debug, GC pause). This leads
>
> to another rebalance. This rebalance will be considered generation 1 and
>
> will result in:
>
>
>
> Generation 1, Consumer A owns partition 0,1
>
>
>
> Now let’s say Consumer B is still out of the group and then Consumer A
>
> leaves as well. While Consumer A is out of the group, Consumer B rejoins
>
> the group. During this rebalance, the only previous state would be the
>
> initial generation 0 assignment. So this assignment would be considered
>
> generation 1 as well and would result in:
>
>
>
> Generation 1, Consumer B owns partition 0,1
>
>
>
> When A rejoins the group, both consumers would claim ownership of both
>
> partitions and they would report the assignment was from generation 1.
>
> This gets us back into the same issue as before because the generation
>
> number cannot help at all. You could add a timestamp in addition to the
>
> generation marker, but that’d still be vulnerable to clock skew.
>
>
>
> Would hooking into the existing generation marker protect the assignor for
>
> this kind of situation? We need to make sure the selected implantation is
>
> protected against the kind of failure mentioned above.
>
>
>
> Also, I have been working on KIP-315, which is another Sticky Assignor,
>
> which also requires some kind of epoch/generation marker to be protected
>
> against zombies. So, I’d be in favor of a generic solution here that other
>
> assignors can leverage.
>
>
>
> Best,
>
>
>
> Mike Freyberger
>
>
>
> > On Jul 13, 2018, at 6:15 PM, Vahid S Hashemian
>
>  wrote:
>
> >
>
> > Hi all,
>
> >
>
> > I create a short KIP to address an issue in Sticky Assignor assignment
>
> > logic:
>
> >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol
>
>
>
> > Please take a look and share your feedback / comments.
>
> >
>
> > In particular, there is a Generation Marker section (
>
> >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol#KIP-
> 341:UpdateStickyAssignor'sUserDataProtocol-GenerationMarker
>
>
>
> > ) that provides two methods for implementing the improvement to the
>
> > protocol. I'd like to know which option is more popular.
>
> >
>
> > Thanks!
>
> > --Vahid
>
> >
>
> >
>
>
>
>
>
>
>
>
>
>
>
>

[jira] [Resolved] (KAFKA-7103) Use bulkloading for RocksDBSegmentedBytesStore during init

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-7103.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

> Use bulkloading for RocksDBSegmentedBytesStore during init
> --
>
> Key: KAFKA-7103
> URL: https://issues.apache.org/jira/browse/KAFKA-7103
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.1.0
>Reporter: Liquan Pei
>Assignee: Liquan Pei
>Priority: Major
> Fix For: 2.1.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> We should use bulk loading for recovering RocksDBWindowStore, same as 
> RocksDBStore. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7170) NPE in ConsumerGroupCommand when describe consumer group

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-7170.

Resolution: Duplicate

Resolving as duplicate. Please continue discussion in KAFKA-7044 if needed.

> NPE in ConsumerGroupCommand when describe consumer group
> 
>
> Key: KAFKA-7170
> URL: https://issues.apache.org/jira/browse/KAFKA-7170
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 1.1.0
>Reporter: Echo Xu
>Priority: Major
>
>  
> Got the following error when run kafka-consumer-groups to describe a consumer 
> group.
> Command:
> {code:java}
> kafka-consumer-groups --bootstrap-server $BROKER_LIST --command-config 
> consumer.properties --describe --group $CONSUMER_GROUP_NAME{code}
>   Error (need change tools-log4j.properties log level to DEBUG): 
> {code:java}
> Error: Executing consumer group command failed due to null
> [2018-07-16 20:05:21,353] DEBUG Exception in consumer group command 
> (kafka.admin.ConsumerGroupCommand$)
> java.lang.NullPointerException
> at scala.Predef$.Long2long(Predef.scala:363)
> at 
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$getLogEndOffsets$2.apply(ConsumerGroupCommand.scala:612)
> at 
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$getLogEndOffsets$2.apply(ConsumerGroupCommand.scala:610)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.immutable.List.map(List.scala:296)
> at 
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.getLogEndOffsets(ConsumerGroupCommand.scala:610)
> at 
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describePartitions(ConsumerGroupCommand.scala:328)
> at 
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.collectConsumerAssignment(ConsumerGroupCommand.scala:308)
> at 
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.collectConsumerAssignment(ConsumerGroupCommand.scala:544)
> at 
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10$$anonfun$13.apply(ConsumerGroupCommand.scala:571)
> at 
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10$$anonfun$13.apply(ConsumerGroupCommand.scala:565)
> at scala.collection.immutable.List.flatMap(List.scala:338)
> at 
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10.apply(ConsumerGroupCommand.scala:565)
> at 
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10.apply(ConsumerGroupCommand.scala:558)
> at scala.Option.map(Option.scala:146)
> at 
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.collectGroupOffsets(ConsumerGroupCommand.scala:558)
> at 
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describeGroup(ConsumerGroupCommand.scala:271)
> at 
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:544)
> at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:77)
> at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Ted Yu]

+1

On Tue, Jul 17, 2018 at 1:40 PM Jason Gustafson  wrote:

> +1. This is useful (though the naming inconsistencies in the tools are
> vexing, as always).
>
> -Jason
>
> On Tue, Jul 17, 2018 at 12:24 PM, Dong Lin  wrote:
>
> > Hi all,
> >
> > It seems that there is no further concern with the KIP-304. I guess it is
> > time to start the voting thread.
> >
> > The KIP can be found at
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-340%
> > 3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-dirs.sh+
> > to+take+admin+client+property+file
> > . This KIP provides a way to allow kafka-reassign-partitions.sh and
> > kafka-log-dirs.sh to talk to broker over SSL.
> >
> > Cheers,
> > Dong
> >
>

Re: [VOTE] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Jason Gustafson]

+1. This is useful (though the naming inconsistencies in the tools are
vexing, as always).

-Jason

On Tue, Jul 17, 2018 at 12:24 PM, Dong Lin  wrote:

> Hi all,
>
> It seems that there is no further concern with the KIP-304. I guess it is
> time to start the voting thread.
>
> The KIP can be found at
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-340%
> 3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-dirs.sh+
> to+take+admin+client+property+file
> . This KIP provides a way to allow kafka-reassign-partitions.sh and
> kafka-log-dirs.sh to talk to broker over SSL.
>
> Cheers,
> Dong
>

[jira] [Resolved] (KAFKA-7173) Update BrokerApiVersionsCommand to use new AdminClient

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-7173.

Resolution: Duplicate

Duplicate of KAFKA-5723.

> Update BrokerApiVersionsCommand to use new AdminClient
> --
>
> Key: KAFKA-7173
> URL: https://issues.apache.org/jira/browse/KAFKA-7173
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Reporter: Jason Gustafson
>Assignee: Attila Sasvari
>Priority: Major
>  Labels: needs-kip
>
> This tool seems to be the last use of the deprecated scala AdminClient. This 
> may require a KIP to fix since the new AdminClient doesn't seem to expose 
> broker API versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] 1.1.1 RC3

[Jason Gustafson]

+1

- Did basic quickstarts on the 2.11 and 2.12 artifacts
- Checked upgrade notes and documentation

Thanks for managing the release, Dong!


-Jason

On Thu, Jul 12, 2018 at 2:36 PM, Harsha  wrote:

> +1.
> 1. Ran unit tests
> 2. Ran 3 node cluster to run few tests.
>
> Thanks,
> Harsha
>
> On Thu, Jul 12, 2018, at 7:29 AM, Manikumar wrote:
> > +1 (non-binding)  Ran tests,  Verified quick start,  producer/consumer
> perf
> > tests
> >
> >
> >
> > On Thu, Jul 12, 2018 at 11:06 AM Brett Rann 
> > wrote:
> >
> > > +1 (non binding)
> > > rolling upgrade of shared staging multitenacy (200+ consumer groups)
> > > cluster from 1.1.0 to 1.1.1-rc3 using the kafka_2.11-1.1.1.tgz
> artifact.
> > > cluster looks healthy after upgrade. Lack of burrow lag suggests
> consumers
> > > are still happy, and incoming messages remains the same.
> > >
> > > On Mon, Jul 9, 2018 at 8:36 AM Dong Lin  wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the fourth candidate for release of Apache Kafka 1.1.1.
> > > >
> > > > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was
> first
> > > > released with 1.1.0 about 3 months ago. We have fixed about 25 issues
> > > since
> > > > that release. A few of the more significant fixes include:
> > > >
> > > > KAFKA-6925  > > > > - Fix memory
> > > > leak in StreamsMetricsThreadImpl
> > > > KAFKA-6937  > > > > - In-sync
> > > > replica delayed during fetch if replica throttle is exceeded
> > > > KAFKA-6917  > > > > - Process txn
> > > > completion asynchronously to avoid deadlock
> > > > KAFKA-6893  > > > > - Create
> > > > processors before starting acceptor to avoid ArithmeticException
> > > > KAFKA-6870  > > > > -
> > > > Fix ConcurrentModificationException in SampledStat
> > > > KAFKA-6878  > > > > - Fix
> > > > NullPointerException when querying global state store
> > > > KAFKA-6879  > > > > - Invoke
> > > > session init callbacks outside lock to avoid Controller deadlock
> > > > KAFKA-6857  > > > > - Prevent
> > > > follower from truncating to the wrong offset if undefined leader
> epoch is
> > > > requested
> > > > KAFKA-6854  > > > > - Log cleaner
> > > > fails with transaction markers that are deleted during clean
> > > > KAFKA-6747  > > > > - Check
> > > > whether there is in-flight transaction before aborting transaction
> > > > KAFKA-6748  > > > > - Double
> > > > check before scheduling a new task after the punctuate call
> > > > KAFKA-6739  > > > > -
> > > > Fix IllegalArgumentException when down-converting from V2 to V0/V1
> > > > KAFKA-6728  > > > > -
> > > > Fix NullPointerException when instantiating the HeaderConverter
> > > >
> > > > Kafka 1.1.1 release plan:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
> > > >  Release+Plan+1.1.1>
> > > >
> > > > Release notes for the 1.1.1 release:
> > > > http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
> > > > 
> > > >
> > > > *** Please download, test and vote by Thursday, July 12, 12pm PT ***
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > http://kafka.apache.org/KEYS
> > > > 
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > http://home.apache.org/~lindong/kafka-1.1.1-rc3/
> > > > 
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/
> > > > 
> > > 

[VOTE] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Dong Lin]

Hi all,

It seems that there is no further concern with the KIP-304. I guess it is
time to start the voting thread.

The KIP can be found at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-340%3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-dirs.sh+to+take+admin+client+property+file
. This KIP provides a way to allow kafka-reassign-partitions.sh and
kafka-log-dirs.sh to talk to broker over SSL.

Cheers,
Dong

Re: [VOTE] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

[Jason Gustafson]

Hi, thanks for the KIP. +1 from me.

I think eventually we'd want --exclude-internal-topics to be the default,
but it seems reasonable to keep the current behavior for compatibility.

-Jason

On Mon, Jul 16, 2018 at 2:23 PM, Dong Lin  wrote:

> +1
>
> Thanks!
> Dong
>
> On Mon, Jul 16, 2018 at 9:17 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > +1 (non-binding)
> >
> > Thanks!
> > --Vahid
> >
> >
> >
> >
> > From:   Chia-Ping Tsai 
> > To: 
> > Date:   07/16/2018 01:11 AM
> > Subject:[VOTE] KIP-338 Support to exclude the internal topics in
> > kafka-topics.sh command
> >
> >
> >
> > hi folks,
> >
> > The discussion[1] of KIP-338[2] did not get any objection for last 6 days
> > so it is time to start the voting thread.
> >
> > Thanks for your time!
> >
> > [1]
> > https://lists.apache.org/thread.html/9bd4e61b73c901b51132ada49743b9
> > b703d40b85fc4eeaa5c9099900@%3Cdev.kafka.apache.org%3E
> >
> >
> > [2]
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 338+Support+to+exclude+the+internal+topics+in+kafka-topics.sh+command
> >
> >
> > Cheers,
> > chia-ping
> >
> >
> >
> >
> >
> >
>

[jira] [Resolved] (KAFKA-6717) TopicPartition Assined twice to a consumer group for 2 consumer instances

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian resolved KAFKA-6717.

Resolution: Duplicate

Marking it as duplicate to keep all the discussion in the other JIRA.

> TopicPartition Assined twice to a consumer group for 2 consumer instances 
> --
>
> Key: KAFKA-6717
> URL: https://issues.apache.org/jira/browse/KAFKA-6717
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.1
>Reporter: Yuancheng PENG
>Priority: Major
>
> I'm using \{{StickyAssignor}} for consuming more than 100 topics with certain 
> pattern.
> There are 10 consumers with the same group id.
> I expected that topic-partition to be assigned to only one consumer instance. 
> However some topic partitions are assigned twice in 2 different difference 
> instance, hence the consumer group process duplicate messages.
> {code:java}
> props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, 
> Collections.singletonList(StickyAssignor.class));
> KafkaConsumer c = new KafkaConsumer<>(props);
> c.subscribe(Pattern.compile(TOPIC_PATTERN), new 
> NoOpConsumerRebalanceListener());
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7173) Update BrokerApiVersionsCommand to use new AdminClient

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-7173:
--

 Summary: Update BrokerApiVersionsCommand to use new AdminClient
 Key: KAFKA-7173
 URL: https://issues.apache.org/jira/browse/KAFKA-7173
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Reporter: Jason Gustafson


This tool seems to be the last use of the deprecated scala AdminClient. This 
may require a KIP to fix since the new AdminClient doesn't seem to expose 
broker API versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

[Lucas Wang]

@Becket
1. Thanks for the comment. You are right that normally there should be just
one controller request because of muting,
and I had NOT intended to say there would be many enqueued controller
requests.
I went through the KIP again, and I'm not sure which part conveys that
info.
I'd be happy to revise if you point it out the section.

2. Though it should not happen in normal conditions, the current design
does not preclude multiple controllers running
at the same time, hence if we don't have the controller queue capacity
config and simply make its capacity to be 1,
network threads handling requests from different controllers will be
blocked during those troublesome times,
which is probably not what we want. On the other hand, adding the extra
config with a default value, say 20, guards us from issues in those
troublesome times, and IMO there isn't much downside of adding the extra
config.

@Mayuresh
Good catch, this sentence is an obsolete statement based on a previous
design. I've revised the wording in the KIP.

Thanks,
Lucas

On Tue, Jul 17, 2018 at 10:33 AM, Mayuresh Gharat <
gharatmayures...@gmail.com> wrote:

> Hi Lucas,
>
> Thanks for the KIP.
> I am trying to understand why you think "The memory consumption can rise
> given the total number of queued requests can go up to 2x" in the impact
> section. Normally the requests from controller to a Broker are not high
> volume, right ?
>
>
> Thanks,
>
> Mayuresh
>
> On Tue, Jul 17, 2018 at 5:06 AM Becket Qin  wrote:
>
> > Thanks for the KIP, Lucas. Separating the control plane from the data
> plane
> > makes a lot of sense.
> >
> > In the KIP you mentioned that the controller request queue may have many
> > requests in it. Will this be a common case? The controller requests still
> > goes through the SocketServer. The SocketServer will mute the channel
> once
> > a request is read and put into the request channel. So assuming there is
> > only one connection between controller and each broker, on the broker
> side,
> > there should be only one controller request in the controller request
> queue
> > at any given time. If that is the case, do we need a separate controller
> > request queue capacity config? The default value 20 means that we expect
> > there are 20 controller switches to happen in a short period of time. I
> am
> > not sure whether someone should increase the controller request queue
> > capacity to handle such case, as it seems indicating something very wrong
> > has happened.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > On Fri, Jul 13, 2018 at 1:10 PM, Dong Lin  wrote:
> >
> > > Thanks for the update Lucas.
> > >
> > > I think the motivation section is intuitive. It will be good to learn
> > more
> > > about the comments from other reviewers.
> > >
> > > On Thu, Jul 12, 2018 at 9:48 PM, Lucas Wang 
> > wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > I've updated the motivation section of the KIP by explaining the
> cases
> > > that
> > > > would have user impacts.
> > > > Please take a look at let me know your comments.
> > > >
> > > > Thanks,
> > > > Lucas
> > > >
> > > > On Mon, Jul 9, 2018 at 5:53 PM, Lucas Wang 
> > > wrote:
> > > >
> > > > > Hi Dong,
> > > > >
> > > > > The simulation of disk being slow is merely for me to easily
> > construct
> > > a
> > > > > testing scenario
> > > > > with a backlog of produce requests. In production, other than the
> > disk
> > > > > being slow, a backlog of
> > > > > produce requests may also be caused by high produce QPS.
> > > > > In that case, we may not want to kill the broker and that's when
> this
> > > KIP
> > > > > can be useful, both for JBOD
> > > > > and non-JBOD setup.
> > > > >
> > > > > Going back to your previous question about each ProduceRequest
> > covering
> > > > 20
> > > > > partitions that are randomly
> > > > > distributed, let's say a LeaderAndIsr request is enqueued that
> tries
> > to
> > > > > switch the current broker, say broker0, from leader to follower
> > > > > *for one of the partitions*, say *test-0*. For the sake of
> argument,
> > > > > let's also assume the other brokers, say broker1, have *stopped*
> > > fetching
> > > > > from
> > > > > the current broker, i.e. broker0.
> > > > > 1. If the enqueued produce requests have acks =  -1 (ALL)
> > > > >   1.1 without this KIP, the ProduceRequests ahead of LeaderAndISR
> > will
> > > be
> > > > > put into the purgatory,
> > > > > and since they'll never be replicated to other brokers
> > (because
> > > > of
> > > > > the assumption made above), they will
> > > > > be completed either when the LeaderAndISR request is
> > processed
> > > or
> > > > > when the timeout happens.
> > > > >   1.2 With this KIP, broker0 will immediately transition the
> > partition
> > > > > test-0 to become a follower,
> > > > > after the current broker sees the replication of the
> > remaining
> > > 19
> > > > > partitions, it can send a response indicating that
> > > > > it's no longer the 

Re: Restarting KIP-213 discussion

[Guozhang Wang]

Hello Adam,

Welcome back!

Please feel free to update the the existing KIP. To bookkeep Jan's changes,
I'd suggest adding a new section on the wiki.


Guozhang


On Tue, Jul 17, 2018 at 10:04 AM, Adam Bellemare 
wrote:

> Hey all
>
> I recently took a crack at implementing my own version of KIP-213 based on
> code written by the previous committee, Jan. I currently have it dev ready,
> but I want to restart some of the discussion based on some design decisions
> that I made, and I need some feedback on a few particular choices.
>
> Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 213+Support+non-key+joining+in+KTable
>
> My question is, I have a number of diagrams to add, some alternative
> designs, and various other comments. Do I simply update the above page, or
> should I create a new KIP? If I update the above page, should I overwrite
> Jan’s changes with my own, or just maintain a whole new section of my own?
>
> Thanks,
>
> Adam




-- 
-- Guozhang

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

[Mayuresh Gharat]

Hi Lucas,

Thanks for the KIP.
I am trying to understand why you think "The memory consumption can rise
given the total number of queued requests can go up to 2x" in the impact
section. Normally the requests from controller to a Broker are not high
volume, right ?


Thanks,

Mayuresh

On Tue, Jul 17, 2018 at 5:06 AM Becket Qin  wrote:

> Thanks for the KIP, Lucas. Separating the control plane from the data plane
> makes a lot of sense.
>
> In the KIP you mentioned that the controller request queue may have many
> requests in it. Will this be a common case? The controller requests still
> goes through the SocketServer. The SocketServer will mute the channel once
> a request is read and put into the request channel. So assuming there is
> only one connection between controller and each broker, on the broker side,
> there should be only one controller request in the controller request queue
> at any given time. If that is the case, do we need a separate controller
> request queue capacity config? The default value 20 means that we expect
> there are 20 controller switches to happen in a short period of time. I am
> not sure whether someone should increase the controller request queue
> capacity to handle such case, as it seems indicating something very wrong
> has happened.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Fri, Jul 13, 2018 at 1:10 PM, Dong Lin  wrote:
>
> > Thanks for the update Lucas.
> >
> > I think the motivation section is intuitive. It will be good to learn
> more
> > about the comments from other reviewers.
> >
> > On Thu, Jul 12, 2018 at 9:48 PM, Lucas Wang 
> wrote:
> >
> > > Hi Dong,
> > >
> > > I've updated the motivation section of the KIP by explaining the cases
> > that
> > > would have user impacts.
> > > Please take a look at let me know your comments.
> > >
> > > Thanks,
> > > Lucas
> > >
> > > On Mon, Jul 9, 2018 at 5:53 PM, Lucas Wang 
> > wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > The simulation of disk being slow is merely for me to easily
> construct
> > a
> > > > testing scenario
> > > > with a backlog of produce requests. In production, other than the
> disk
> > > > being slow, a backlog of
> > > > produce requests may also be caused by high produce QPS.
> > > > In that case, we may not want to kill the broker and that's when this
> > KIP
> > > > can be useful, both for JBOD
> > > > and non-JBOD setup.
> > > >
> > > > Going back to your previous question about each ProduceRequest
> covering
> > > 20
> > > > partitions that are randomly
> > > > distributed, let's say a LeaderAndIsr request is enqueued that tries
> to
> > > > switch the current broker, say broker0, from leader to follower
> > > > *for one of the partitions*, say *test-0*. For the sake of argument,
> > > > let's also assume the other brokers, say broker1, have *stopped*
> > fetching
> > > > from
> > > > the current broker, i.e. broker0.
> > > > 1. If the enqueued produce requests have acks =  -1 (ALL)
> > > >   1.1 without this KIP, the ProduceRequests ahead of LeaderAndISR
> will
> > be
> > > > put into the purgatory,
> > > > and since they'll never be replicated to other brokers
> (because
> > > of
> > > > the assumption made above), they will
> > > > be completed either when the LeaderAndISR request is
> processed
> > or
> > > > when the timeout happens.
> > > >   1.2 With this KIP, broker0 will immediately transition the
> partition
> > > > test-0 to become a follower,
> > > > after the current broker sees the replication of the
> remaining
> > 19
> > > > partitions, it can send a response indicating that
> > > > it's no longer the leader for the "test-0".
> > > >   To see the latency difference between 1.1 and 1.2, let's say there
> > are
> > > > 24K produce requests ahead of the LeaderAndISR, and there are 8 io
> > > threads,
> > > >   so each io thread will process approximately 3000 produce requests.
> > Now
> > > > let's investigate the io thread that finally processed the
> > LeaderAndISR.
> > > >   For the 3000 produce requests, if we model the time when their
> > > remaining
> > > > 19 partitions catch up as t0, t1, ...t2999, and the LeaderAndISR
> > request
> > > is
> > > > processed at time t3000.
> > > >   Without this KIP, the 1st produce request would have waited an
> extra
> > > > t3000 - t0 time in the purgatory, the 2nd an extra time of t3000 -
> t1,
> > > etc.
> > > >   Roughly speaking, the latency difference is bigger for the earlier
> > > > produce requests than for the later ones. For the same reason, the
> more
> > > > ProduceRequests queued
> > > >   before the LeaderAndISR, the bigger benefit we get (capped by the
> > > > produce timeout).
> > > > 2. If the enqueued produce requests have acks=0 or acks=1
> > > >   There will be no latency differences in this case, but
> > > >   2.1 without this KIP, the records of partition test-0 in the
> > > > ProduceRequests ahead of the LeaderAndISR will be appended to the
> local
> > > log,
> > > > 

Re: [VOTE] KIP-291: Have separate queues for control requests and data requests

[Joel Koshy]

+1 on the KIP.

(I'm not sure we actually necessary to introduce the condition variables
for the concern that Jun raised, but it's an implementation detail that we
can defer to a discussion in the PR.)

On Sat, Jul 14, 2018 at 10:45 PM, Lucas Wang  wrote:

> Hi Jun,
>
> I agree by using the conditional variables, there is no need to add such a
> new config.
> Also thanks for approving this KIP.
>
> Lucas
>

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

[Joel Koshy]

Hey Becket - good point. Lucas and I were talking about this offline last
week. It is true that there is only one request in flight for processing.
However, there may be more during a controller failover but it should not
be very high - basically the maximum number of controller failures that can
occur whilst handling any controller request.

This is in fact the more significant issue that may not have been fully
captured in the KIP motivation. i.e., right now, the server processes one
request at a time. Assuming there is moderate to heavy load on the broker,
while it is handling a controller request, it will accumulate a deluge of
regular client requests that will enter the request queue. After the sole
controller request is handled it will read the next controller request into
the request queue. So we end up with a single controller request, then a
potentially large number of regular requests, then a single controller
request, then a large number of regular requests and so on. This can become
especially problematic when you have many small controller requests (say if
you are steadily moving a few partitions at a time) spread over a short
span of time). With the prioritized queue this changes to: handle a
controller request, handle a vector of regular requests (where the vector
size is the number of request handler threads), handle the next controller
request, and so on. The maximum time between handling adjacent controller
requests will be within (*min(local time) of the vector of regular requests*).
So it helps significantly. We also considered the possibility of NOT
muting/unmuting the controller socket to help address this. This would also
mean we would need to pin the handling of all controller requests to one
specific request handler thread in order to ensure order. That change is
probably not worth the effort and we expect the current proposal to be
adequate.

Thanks,

Joel

On Tue, Jul 17, 2018 at 5:06 AM, Becket Qin  wrote:

> Thanks for the KIP, Lucas. Separating the control plane from the data plane
> makes a lot of sense.
>
> In the KIP you mentioned that the controller request queue may have many
> requests in it. Will this be a common case? The controller requests still
> goes through the SocketServer. The SocketServer will mute the channel once
> a request is read and put into the request channel. So assuming there is
> only one connection between controller and each broker, on the broker side,
> there should be only one controller request in the controller request queue
> at any given time. If that is the case, do we need a separate controller
> request queue capacity config? The default value 20 means that we expect
> there are 20 controller switches to happen in a short period of time. I am
> not sure whether someone should increase the controller request queue
> capacity to handle such case, as it seems indicating something very wrong
> has happened.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Fri, Jul 13, 2018 at 1:10 PM, Dong Lin  wrote:
>
> > Thanks for the update Lucas.
> >
> > I think the motivation section is intuitive. It will be good to learn
> more
> > about the comments from other reviewers.
> >
> > On Thu, Jul 12, 2018 at 9:48 PM, Lucas Wang 
> wrote:
> >
> > > Hi Dong,
> > >
> > > I've updated the motivation section of the KIP by explaining the cases
> > that
> > > would have user impacts.
> > > Please take a look at let me know your comments.
> > >
> > > Thanks,
> > > Lucas
> > >
> > > On Mon, Jul 9, 2018 at 5:53 PM, Lucas Wang 
> > wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > The simulation of disk being slow is merely for me to easily
> construct
> > a
> > > > testing scenario
> > > > with a backlog of produce requests. In production, other than the
> disk
> > > > being slow, a backlog of
> > > > produce requests may also be caused by high produce QPS.
> > > > In that case, we may not want to kill the broker and that's when this
> > KIP
> > > > can be useful, both for JBOD
> > > > and non-JBOD setup.
> > > >
> > > > Going back to your previous question about each ProduceRequest
> covering
> > > 20
> > > > partitions that are randomly
> > > > distributed, let's say a LeaderAndIsr request is enqueued that tries
> to
> > > > switch the current broker, say broker0, from leader to follower
> > > > *for one of the partitions*, say *test-0*. For the sake of argument,
> > > > let's also assume the other brokers, say broker1, have *stopped*
> > fetching
> > > > from
> > > > the current broker, i.e. broker0.
> > > > 1. If the enqueued produce requests have acks =  -1 (ALL)
> > > >   1.1 without this KIP, the ProduceRequests ahead of LeaderAndISR
> will
> > be
> > > > put into the purgatory,
> > > > and since they'll never be replicated to other brokers
> (because
> > > of
> > > > the assumption made above), they will
> > > > be completed either when the LeaderAndISR request is
> processed
> > or
> > > > when the timeout happens.
> > > >   1.2 With this KIP, 

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Anna Povzner]

Hi Jason,


I also like your proposal and agree that KafkaConsumer#seekToCommitted() is
more intuitive as a way to initialize both consumer's position and its
fetch state.


My understanding that KafkaConsumer#seekToCommitted() is purely for clients
who store their offsets externally, right? And we are still going to
add KafkaConsumer#findOffsets()
in this KIP as we discussed, so that the client can handle
LogTruncationException?


Thanks,

Anna


On Thu, Jul 12, 2018 at 3:57 PM Dong Lin  wrote:

> Hey Jason,
>
> It is a great summary. The solution sounds good. I might have minor
> comments regarding the method name. But we can discuss that minor points
> later after we reach consensus on the high level API.
>
> Thanks,
> Dong
>
>
> On Thu, Jul 12, 2018 at 11:41 AM, Jason Gustafson 
> wrote:
>
> > Hey Anna and Dong,
> >
> > Thanks a lot for the great discussion. I've been hanging back a bit
> because
> > honestly the best option hasn't seemed clear. I agree with Anna's general
> > observation that there is a distinction between the position of the
> > consumer and its fetch state up to that position. If you think about it,
> a
> > committed offset actually represents both of these. The metadata is used
> to
> > initialize the state of the consumer application and the offset
> initializes
> > the position. Additionally, we are extending the offset commit in this
> KIP
> > to also include the last epoch fetched by the consumer, which is used to
> > initialize the internal fetch state. Of course if you do an arbitrary
> > `seek` and immediately commit offsets, then there won't be a last epoch
> to
> > commit. This seems intuitive since there is no fetch state in this case.
> We
> > only commit fetch state when we have it.
> >
> > So if we think about a committed offset as initializing both the
> consumer's
> > position and its fetch state, then the gap in the API is evidently that
> we
> > don't have a way to initialize the consumer to a committed offset. We do
> it
> > implicitly of course for offsets stored in Kafka, but since external
> > storage is a use case we support, then we should have an explicit API as
> > well. Perhaps something like this:
> >
> > seekToCommitted(TopicPartition, OffsetAndMetadata)
> >
> > In this KIP, we are proposing to allow the `OffsetAndMetadata` object to
> > include the leader epoch, so I think this would have the same effect as
> > Anna's suggested `seekToRecord`. But perhaps it is a more natural fit
> given
> > the current API? Furthermore, if we find a need for additional metadata
> in
> > the offset commit API in the future, then we will just need to modify the
> > `OffsetAndMetadata` object and we will not need a new `seek` API.
> >
> > With this approach, I think then we can leave the `position` API as it
> is.
> > The position of the consumer is still just the next expected fetch
> offset.
> > If a user needs to record additional state based on previous fetch
> > progress, then they would use the result of the previous fetch to obtain
> > it. This makes the dependence on fetch progress explicit. I think we
> could
> > make this a little more convenience with a helper in the
> `ConsumerRecords`
> > object, but I think that's more of a nice-to-have.
> >
> > Thoughts?
> >
> > By the way, I have been iterating a little bit on the replica side of
> this
> > KIP. My initial proposal in fact did not have strong enough fencing to
> > protect all of the edge cases. I believe the current proposal fixes the
> > problems, but I am still verifying the model.
> >
> > Thanks,
> > Jason
> >
> >
> > On Wed, Jul 11, 2018 at 10:45 AM, Dong Lin  wrote:
> >
> > > Hey Anna,
> > >
> > > Thanks much for the explanation. Approach 1 also sounds good to me. I
> > think
> > > findOffsets() is useful for users who don't use automatic offset reset
> > > policy.
> > >
> > > Just one more question. Since users who store offsets externally need
> to
> > > provide leaderEpoch to findOffsets(...), do we need an extra API for
> user
> > > to get both offset and leaderEpoch, e.g. recordPosition()?
> > >
> > > Thanks,
> > > Dong
> > >
> > > On Wed, Jul 11, 2018 at 10:12 AM, Anna Povzner 
> > wrote:
> > >
> > > > Hi Dong,
> > > >
> > > >
> > > > What I called “not covering all use cases” is what you call
> best-effort
> > > > (not guaranteeing some corner cases). I think we are on the same page
> > > here.
> > > >
> > > >
> > > > I wanted to be clear in the API whether the consumer seeks to a
> > position
> > > > (offset) or to a record (offset, leader epoch). The only use-case of
> > > > seeking to a record is seeking to a committed offset for a user who
> > > stores
> > > > committed offsets externally. (Unless users find some other reason to
> > > seek
> > > > to a record.) I thought it was possible to provide this functionality
> > > with
> > > > findOffset(offset, leader epoch) followed by a seek(offset). However,
> > you
> > > > are right that this will not handle the race condition where
> > > 

Re: Contribution to Kafka

[Sayan Biswas]

Thanks a lot.

On Tue, 17 Jul 2018 at 22:25, Guozhang Wang  wrote:

> It's done. Cheers.
>
>
> Guozhang
>
> On Tue, Jul 17, 2018 at 9:45 AM, Sayan Biswas 
> wrote:
>
> > Hi,
> > I've signed up. My userid is: *Bitswazsky*
> >
> > On Tue, 17 Jul 2018 at 22:08, Guozhang Wang  wrote:
> >
> > > You just need to go do Apache JIRA and sign up with an id.
> > >
> > > https://issues.apache.org/jira/secure/Dashboard.jspa
> > >
> > >
> > > On Tue, Jul 17, 2018 at 9:22 AM, Sayan Biswas 
> > > wrote:
> > >
> > > > Hi,
> > > > I don't have an id yet. I had only subscribed to the dev mailing
> lists.
> > > Can
> > > > you please tell me how to get one?
> > > >
> > > > On Tue, 17 Jul 2018 at 21:41, Guozhang Wang 
> > wrote:
> > > >
> > > > > Hello Sayan,
> > > > >
> > > > > What's your apache id?
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Tue, Jul 17, 2018 at 8:14 AM, Sayan Biswas <
> > formal.sa...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Can I be added to the Kafka contributor's list?
> > > > > >
> > > > > > Regards,
> > > > > > Sayan
> > > > > >
> > > > > > --
> > > > > > Don't kill the dream - execute it.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > > > --
> > > > Don't kill the dream - execute it.
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> > --
> > Don't kill the dream - execute it.
> >
>
>
>
> --
> -- Guozhang
>


-- 
Don't kill the dream - execute it.

Restarting KIP-213 discussion

[Adam Bellemare]

Hey all

I recently took a crack at implementing my own version of KIP-213 based on code 
written by the previous committee, Jan. I currently have it dev ready, but I 
want to restart some of the discussion based on some design decisions that I 
made, and I need some feedback on a few particular choices.

Here is the KIP: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable

My question is, I have a number of diagrams to add, some alternative designs, 
and various other comments. Do I simply update the above page, or should I 
create a new KIP? If I update the above page, should I overwrite Jan’s changes 
with my own, or just maintain a whole new section of my own?

Thanks,

Adam 

Re: Contribution to Kafka

[Guozhang Wang]

It's done. Cheers.


Guozhang

On Tue, Jul 17, 2018 at 9:45 AM, Sayan Biswas 
wrote:

> Hi,
> I've signed up. My userid is: *Bitswazsky*
>
> On Tue, 17 Jul 2018 at 22:08, Guozhang Wang  wrote:
>
> > You just need to go do Apache JIRA and sign up with an id.
> >
> > https://issues.apache.org/jira/secure/Dashboard.jspa
> >
> >
> > On Tue, Jul 17, 2018 at 9:22 AM, Sayan Biswas 
> > wrote:
> >
> > > Hi,
> > > I don't have an id yet. I had only subscribed to the dev mailing lists.
> > Can
> > > you please tell me how to get one?
> > >
> > > On Tue, 17 Jul 2018 at 21:41, Guozhang Wang 
> wrote:
> > >
> > > > Hello Sayan,
> > > >
> > > > What's your apache id?
> > > >
> > > >
> > > > Guozhang
> > > >
> > > > On Tue, Jul 17, 2018 at 8:14 AM, Sayan Biswas <
> formal.sa...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Can I be added to the Kafka contributor's list?
> > > > >
> > > > > Regards,
> > > > > Sayan
> > > > >
> > > > > --
> > > > > Don't kill the dream - execute it.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > > --
> > > Don't kill the dream - execute it.
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
> --
> Don't kill the dream - execute it.
>



-- 
-- Guozhang

Re: Contribution to Kafka

[Sayan Biswas]

Hi,
I've signed up. My userid is: *Bitswazsky*

On Tue, 17 Jul 2018 at 22:08, Guozhang Wang  wrote:

> You just need to go do Apache JIRA and sign up with an id.
>
> https://issues.apache.org/jira/secure/Dashboard.jspa
>
>
> On Tue, Jul 17, 2018 at 9:22 AM, Sayan Biswas 
> wrote:
>
> > Hi,
> > I don't have an id yet. I had only subscribed to the dev mailing lists.
> Can
> > you please tell me how to get one?
> >
> > On Tue, 17 Jul 2018 at 21:41, Guozhang Wang  wrote:
> >
> > > Hello Sayan,
> > >
> > > What's your apache id?
> > >
> > >
> > > Guozhang
> > >
> > > On Tue, Jul 17, 2018 at 8:14 AM, Sayan Biswas 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Can I be added to the Kafka contributor's list?
> > > >
> > > > Regards,
> > > > Sayan
> > > >
> > > > --
> > > > Don't kill the dream - execute it.
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> > --
> > Don't kill the dream - execute it.
> >
>
>
>
> --
> -- Guozhang
>


-- 
Don't kill the dream - execute it.

Re: Contribution to Kafka

[Guozhang Wang]

You just need to go do Apache JIRA and sign up with an id.

https://issues.apache.org/jira/secure/Dashboard.jspa


On Tue, Jul 17, 2018 at 9:22 AM, Sayan Biswas 
wrote:

> Hi,
> I don't have an id yet. I had only subscribed to the dev mailing lists. Can
> you please tell me how to get one?
>
> On Tue, 17 Jul 2018 at 21:41, Guozhang Wang  wrote:
>
> > Hello Sayan,
> >
> > What's your apache id?
> >
> >
> > Guozhang
> >
> > On Tue, Jul 17, 2018 at 8:14 AM, Sayan Biswas 
> > wrote:
> >
> > > Hi all,
> > >
> > > Can I be added to the Kafka contributor's list?
> > >
> > > Regards,
> > > Sayan
> > >
> > > --
> > > Don't kill the dream - execute it.
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
> --
> Don't kill the dream - execute it.
>



-- 
-- Guozhang

Re: Contribution to Kafka

[Sayan Biswas]

Hi,
I don't have an id yet. I had only subscribed to the dev mailing lists. Can
you please tell me how to get one?

On Tue, 17 Jul 2018 at 21:41, Guozhang Wang  wrote:

> Hello Sayan,
>
> What's your apache id?
>
>
> Guozhang
>
> On Tue, Jul 17, 2018 at 8:14 AM, Sayan Biswas 
> wrote:
>
> > Hi all,
> >
> > Can I be added to the Kafka contributor's list?
> >
> > Regards,
> > Sayan
> >
> > --
> > Don't kill the dream - execute it.
> >
>
>
>
> --
> -- Guozhang
>


-- 
Don't kill the dream - execute it.

Re: Kafka Contributors

[Guozhang Wang]

Hello Austin,

What's your Apache id?


Guozhang


On Mon, Jul 16, 2018 at 4:23 PM, Austin Nobis 
wrote:

> Hi,
>
> Can I be added to the list of contributors for Apache Kafka so I can assign
> myself issues?
>
> Thanks,
> Austin
>



-- 
-- Guozhang

Re: Contribution to Kafka

[Guozhang Wang]

Hello Sayan,

What's your apache id?


Guozhang

On Tue, Jul 17, 2018 at 8:14 AM, Sayan Biswas 
wrote:

> Hi all,
>
> Can I be added to the Kafka contributor's list?
>
> Regards,
> Sayan
>
> --
> Don't kill the dream - execute it.
>



-- 
-- Guozhang

Contribution to Kafka

[Sayan Biswas]

Hi all,

Can I be added to the Kafka contributor's list?

Regards,
Sayan

-- 
Don't kill the dream - execute it.

[jira] [Created] (KAFKA-7172) Prometheus /metrics http endpoint for metrics monitoring integration

[Hari Sekhon (JIRA)]

Hari Sekhon created KAFKA-7172:
--

 Summary: Prometheus /metrics http endpoint for metrics monitoring 
integration
 Key: KAFKA-7172
 URL: https://issues.apache.org/jira/browse/KAFKA-7172
 Project: Kafka
  Issue Type: Improvement
  Components: metrics
Affects Versions: 1.1.2
Reporter: Hari Sekhon


Feature Request to add native http /metrics endpoint for Prometheus metrics 
monitoring integration:

[https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E]

See also ticket for /jmx http endpoint similar to what Hadoop and HBase have 
done for years in

https://issues.apache.org/jira/browse/KAFKA-3377

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Kafka Contributors

[Austin Nobis]

Hi,

Can I be added to the list of contributors for Apache Kafka so I can assign
myself issues?

Thanks,
Austin

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

[Becket Qin]

Thanks for the KIP, Lucas. Separating the control plane from the data plane
makes a lot of sense.

In the KIP you mentioned that the controller request queue may have many
requests in it. Will this be a common case? The controller requests still
goes through the SocketServer. The SocketServer will mute the channel once
a request is read and put into the request channel. So assuming there is
only one connection between controller and each broker, on the broker side,
there should be only one controller request in the controller request queue
at any given time. If that is the case, do we need a separate controller
request queue capacity config? The default value 20 means that we expect
there are 20 controller switches to happen in a short period of time. I am
not sure whether someone should increase the controller request queue
capacity to handle such case, as it seems indicating something very wrong
has happened.

Thanks,

Jiangjie (Becket) Qin


On Fri, Jul 13, 2018 at 1:10 PM, Dong Lin  wrote:

> Thanks for the update Lucas.
>
> I think the motivation section is intuitive. It will be good to learn more
> about the comments from other reviewers.
>
> On Thu, Jul 12, 2018 at 9:48 PM, Lucas Wang  wrote:
>
> > Hi Dong,
> >
> > I've updated the motivation section of the KIP by explaining the cases
> that
> > would have user impacts.
> > Please take a look at let me know your comments.
> >
> > Thanks,
> > Lucas
> >
> > On Mon, Jul 9, 2018 at 5:53 PM, Lucas Wang 
> wrote:
> >
> > > Hi Dong,
> > >
> > > The simulation of disk being slow is merely for me to easily construct
> a
> > > testing scenario
> > > with a backlog of produce requests. In production, other than the disk
> > > being slow, a backlog of
> > > produce requests may also be caused by high produce QPS.
> > > In that case, we may not want to kill the broker and that's when this
> KIP
> > > can be useful, both for JBOD
> > > and non-JBOD setup.
> > >
> > > Going back to your previous question about each ProduceRequest covering
> > 20
> > > partitions that are randomly
> > > distributed, let's say a LeaderAndIsr request is enqueued that tries to
> > > switch the current broker, say broker0, from leader to follower
> > > *for one of the partitions*, say *test-0*. For the sake of argument,
> > > let's also assume the other brokers, say broker1, have *stopped*
> fetching
> > > from
> > > the current broker, i.e. broker0.
> > > 1. If the enqueued produce requests have acks =  -1 (ALL)
> > >   1.1 without this KIP, the ProduceRequests ahead of LeaderAndISR will
> be
> > > put into the purgatory,
> > > and since they'll never be replicated to other brokers (because
> > of
> > > the assumption made above), they will
> > > be completed either when the LeaderAndISR request is processed
> or
> > > when the timeout happens.
> > >   1.2 With this KIP, broker0 will immediately transition the partition
> > > test-0 to become a follower,
> > > after the current broker sees the replication of the remaining
> 19
> > > partitions, it can send a response indicating that
> > > it's no longer the leader for the "test-0".
> > >   To see the latency difference between 1.1 and 1.2, let's say there
> are
> > > 24K produce requests ahead of the LeaderAndISR, and there are 8 io
> > threads,
> > >   so each io thread will process approximately 3000 produce requests.
> Now
> > > let's investigate the io thread that finally processed the
> LeaderAndISR.
> > >   For the 3000 produce requests, if we model the time when their
> > remaining
> > > 19 partitions catch up as t0, t1, ...t2999, and the LeaderAndISR
> request
> > is
> > > processed at time t3000.
> > >   Without this KIP, the 1st produce request would have waited an extra
> > > t3000 - t0 time in the purgatory, the 2nd an extra time of t3000 - t1,
> > etc.
> > >   Roughly speaking, the latency difference is bigger for the earlier
> > > produce requests than for the later ones. For the same reason, the more
> > > ProduceRequests queued
> > >   before the LeaderAndISR, the bigger benefit we get (capped by the
> > > produce timeout).
> > > 2. If the enqueued produce requests have acks=0 or acks=1
> > >   There will be no latency differences in this case, but
> > >   2.1 without this KIP, the records of partition test-0 in the
> > > ProduceRequests ahead of the LeaderAndISR will be appended to the local
> > log,
> > > and eventually be truncated after processing the LeaderAndISR.
> > > This is what's referred to as
> > > "some unofficial definition of data loss in terms of messages
> > > beyond the high watermark".
> > >   2.2 with this KIP, we can mitigate the effect since if the
> LeaderAndISR
> > > is immediately processed, the response to producers will have
> > > the NotLeaderForPartition error, causing producers to retry
> > >
> > > This explanation above is the benefit for reducing the latency of a
> > broker
> > > becoming the follower,
> > > closely related is 

[jira] [Created] (KAFKA-7171) KafkaPerformanceProducer crashes with same transaction id.

[kaushik srinivas (JIRA)]

kaushik srinivas created KAFKA-7171:
---

 Summary: KafkaPerformanceProducer crashes with same transaction id.
 Key: KAFKA-7171
 URL: https://issues.apache.org/jira/browse/KAFKA-7171
 Project: Kafka
  Issue Type: Bug
Affects Versions: 1.0.1
Reporter: kaushik srinivas


Running org.apache.kafka.tools.ProducerPerformance code to performance test the 
kafka cluster. As a trial cluster has only one broker and zookeeper with 12GB 
of heap space.

Running 6 producers on 3 machines with same transaction id (2 producers on each 
node).

Below are the settings of each producer,

kafka-run-class org.apache.kafka.tools.ProducerPerformance --print-metrics 
--topic perf1 --num-records 9223372036854 --throughput 25  --record-size 
200 --producer-props bootstrap.servers=localhost:9092 buffer.memory=524288000 
batch.size=524288

 

for 2 hours all producers run fine, then suddenly throughput of all producers 
increase 3 times and 4 producers on 2 nodes crashes with below exceptions,

[2018-07-16 14:00:18,744] ERROR Error executing user-provided callback on 
message for topic-partition perf1-6: 
(org.apache.kafka.clients.producer.internals.RecordBatch)
java.lang.ClassCastException: 
org.apache.kafka.clients.producer.internals.RecordAccumulator$RecordAppendResult
 cannot be cast to org.apache.kafka.clients.producer.internals.RecordBatch$Thunk
 at 
org.apache.kafka.clients.producer.internals.RecordBatch.done(RecordBatch.java:99)
 at 
org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:312)
 at 
org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:272)
 at 
org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:57)
 at 
org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:358)
 at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
 at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:229)
 at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
 at java.lang.Thread.run(Thread.java:748)

 

First machine (2 producers) run fine.

Need some pointers on this issue. 

Queires:

why the throughput is increasing 3 times after 2 hours of duration ?

why the other producers are crashing ?

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7163) Topic is not marked for deletion when delete.topic.enable is set false

[Shaobo Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaobo Liu resolved KAFKA-7163.
---
Resolution: Not A Problem

> Topic is not marked for deletion when delete.topic.enable is set false
> --
>
> Key: KAFKA-7163
> URL: https://issues.apache.org/jira/browse/KAFKA-7163
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.1
>Reporter: JieFang.He
>Priority: Major
>
> When I set delete.topic.enable to false and use the delete command to delete 
> it, the topic does not marked for deletion



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSSION] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Ismael Juma]

Hi Viktor,

The approach where all the methods have a default implementation and the
user chooses to override one of them seems the most appealing to me given
the current state. It doesn't seem like we give up much in that case, what
do you think?

Ismael

On Tue, Jul 10, 2018 at 7:15 AM Viktor Somogyi 
wrote:

> Hi Ismael,
>
> Well, yes. If we care about headers only then you'd need to add a dummy
> implementation for the 2 parameter method as well. Although it is not
> ideal, we're using the Serializer interface everywhere and convert it to
> extended with ensureExtended(serializer) and delegate to the 2 parameter
> method inside the wrapper which is returned in ensureExtended. Because of
> backward compatibility we have to keep delegating but I could perhaps add a
> dummy implementation for the 2 parameter too if you and others think that
> would be better. In this case though we'd have an interface where all
> methods are default (given the improvements of KIP-331
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
> >)
> and would have to rethink if this interface should be a
> @FunctionalInterface.
> I don't really have a context currently on how the 3 parameter method is
> used, most the code samples I found on github were using the 2 parameter
> method. I think I found one instance where the 3 parameter one was used but
> that delegated to the 2 param one :). Have to say though that this research
> is not representative.
> All in all I think it wouldn't hurt to provide a default implementation for
> the 2 param method too but then we have to give up the @FunctionalInterface
> annotation and we'll end up with an interface with no abstract methods but
> only defaults.
> What do you think?
>
> Cheers,
> Viktor
>
>
> On Mon, Jul 9, 2018 at 11:02 AM Ismael Juma  wrote:
>
> > Thanks for the KIP. It would be helpful to understand the user experience
> > for the case where the implementor uses the headers. It seems like it
> would
> > require overriding two methods?
> >
> > Ismael
> >
> > On Mon, Jul 9, 2018 at 1:50 AM Viktor Somogyi 
> > wrote:
> >
> > > Hi folks,
> > >
> > > I've published KIP-336 which is about consolidating the
> > > Serializer/Deserializer interfaces.
> > >
> > > Basically the story here is when ExtendedSerializer and
> > > ExtendedDeserializer were added we still supported Java 7 and therefore
> > had
> > > to use compatible constructs which now seem unnecessary since we
> dropped
> > > support for Java 7. Now in this KIP I propose a way to deprecate those
> > > patterns:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242
> > >
> > > I'd be happy to receive some feedback about the KIP I published.
> > >
> > > Cheers,
> > > Viktor
> > >
> >
>

Re: [DISCUSS] KIP-333 Consider a faster form of rebalancing

[Richard Yu]

 Hi Becket,
Thanks for reviewing this KIP. :)
I probably did not explicitly state what we were trying to avoid by introducing 
this mode. As mentioned in the KIP, there is a offset lag which could result 
after a crash. Our main goal is to avoid this lag (i.e. the latency in terms of 
time that results from the crash, not to reduce the number of records 
reprocessed).
I could provide a couple of diagrams with what I am envisioning because some 
points in my KIP might otherwise be hard to grasp (I will also include some 
diagrams to give you a better idea of an use case). As for your questions, I 
could provide a couple of answers:
1. Yes, the two consumers will in fact be processing in parallel. We do this 
because we want to accelerate the processing speed of the records to make up 
for the latency caused by the crash.
2. After the recovery point, records will not be processed twice. Let me 
describe the scenario I was envisioning: we would let the consumer that crashed 
seek to the end of the log using KafkaConsumer#seekToEnd. Meanwhile, a 
secondary consumer will start processing from the latest checkpointed offset 
and continue until it  has hit the place where the first consumer that crashed 
began processing after seekToEnd was first called. Since the consumer that 
crashed skipped from the recovery point to the end of the log, the intermediate 
offsets will be processed only by the secondary consumer. So it is important to 
note that the offset ranges which the two threads process will not overlap. 
(This is important as it prevents offsets from being processed more than once)

3. As for the committed offsets, the possibility of rewinding is not likely. If 
my understanding is correct, you are probably worried that after the crash, 
offsets that has already been previously committed will be committed again. The 
current design prevents that from happening, as the policy of where to start 
processing after a crash is universal across all Consumer instances -- we will 
begin processing from the latest offset committed. 

I hope that you at least got some of your questions answered. I will update the 
KIP soon, so please stay tuned.  

Thanks,Richard Yu
On Tuesday, July 17, 2018, 2:14:07 PM GMT+8, Becket Qin 
 wrote:  
 
 Hi Richard,

Thanks for the KIP. I am a little confused on what is proposed. The KIP
suggests that after recovery from a consumer crash, there will be two
consumers consuming from the same partition. One consumes starting from the
log end offset at the point of recovery, and another consumes starting from
the last committed offset and keeping consuming with the first consumer in
parallel? Does that mean the messages after the recovery point will be
consumed twice? If those two consumer commits offsets, does that mean the
committed offsets may rewind?

The proposal sounds a little hacky and introduce some non-deterministic
behavior. It would be useful to have a concrete use case example to explain
what is actually needed. If the goal is to reduce the number of records
that are reprocessed when consume crashes, maybe we can have an auto commit
interval based on number of messages. If the application just wants to read
from the end of the log after recovery from crash, would calling seekToEnd
explicitly work?

Thanks,

Jiangjie (Becket) Qin



On Thu, Jul 5, 2018 at 6:46 PM, Richard Yu 
wrote:

> Hi all,
>
> I would like to discuss KIP-333 (which proposes a faster mode of
> rebalancing).
> Here is the link for the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 333%3A+Add+faster+mode+of+rebalancing
>
> Thanks,
> Richard Yu
>
  

Re: [DISCUSSION] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Chia-Ping Tsai]

here is my two cents.

In order to keep the BC, SC, and correct deprecation cycle(deprecate 
Serializer/Deserializer in 2.0), the solution of this issue is to introduce a 
new class to replace the Serializer and Deserializer. The new class should be a 
subclass of Serializer/Deserializer and it has the method "byte[] 
serialize(String topic, Headers headers, T data)". Also, we have to add default 
implementation of the new method to Serializer/Deserializer for BC and SC, and 
Serializer/Deserializer should be marked as @deprecated.

The benefits are 1) we can keep the BC and SC in 2.0, 2) the 
Serializer/Deserializer can be removed in the future without breaking the 
compatibility rule since we have deprecated them, and 3) the new kafka user can 
use new Serializer/Deserializer in 2.0 and it still work in 3.0.

The side effect is that we have to update all Public APIs and docs referencing 
to the Serializer/Deserializer. However, IIRC the main usage of 
Serializer/Deserializer is in Producer/Consume so we can tolerate the pain.

On 2018/07/09 08:50:07, Viktor Somogyi  wrote: 
> Hi folks,
> 
> I've published KIP-336 which is about consolidating the
> Serializer/Deserializer interfaces.
> 
> Basically the story here is when ExtendedSerializer and
> ExtendedDeserializer were added we still supported Java 7 and therefore had
> to use compatible constructs which now seem unnecessary since we dropped
> support for Java 7. Now in this KIP I propose a way to deprecate those
> patterns:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242
> 
> I'd be happy to receive some feedback about the KIP I published.
> 
> Cheers,
> Viktor
> 

Jenkins build is back to normal : kafka-trunk-jdk8 #2800

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-333 Consider a faster form of rebalancing

[Becket Qin]

Hi Richard,

Thanks for the KIP. I am a little confused on what is proposed. The KIP
suggests that after recovery from a consumer crash, there will be two
consumers consuming from the same partition. One consumes starting from the
log end offset at the point of recovery, and another consumes starting from
the last committed offset and keeping consuming with the first consumer in
parallel? Does that mean the messages after the recovery point will be
consumed twice? If those two consumer commits offsets, does that mean the
committed offsets may rewind?

The proposal sounds a little hacky and introduce some non-deterministic
behavior. It would be useful to have a concrete use case example to explain
what is actually needed. If the goal is to reduce the number of records
that are reprocessed when consume crashes, maybe we can have an auto commit
interval based on number of messages. If the application just wants to read
from the end of the log after recovery from crash, would calling seekToEnd
explicitly work?

Thanks,

Jiangjie (Becket) Qin



On Thu, Jul 5, 2018 at 6:46 PM, Richard Yu 
wrote:

> Hi all,
>
> I would like to discuss KIP-333 (which proposes a faster mode of
> rebalancing).
> Here is the link for the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 333%3A+Add+faster+mode+of+rebalancing
>
> Thanks,
> Richard Yu
>

Build failed in Jenkins: kafka-0.10.2-jdk7 #227

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: add release script (#5328)

--
[...truncated 286.58 KB...]

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testAdvertisePortDefault STARTED

kafka.server.KafkaConfigTest > testAdvertisePortDefault PASSED

kafka.server.KafkaConfigTest > testVersionConfiguration STARTED

kafka.server.KafkaConfigTest > testVersionConfiguration PASSED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol PASSED

kafka.server.ServerGenerateBrokerIdTest > testGetSequenceIdMethod STARTED

kafka.server.ServerGenerateBrokerIdTest > testGetSequenceIdMethod PASSED

kafka.server.ServerGenerateBrokerIdTest > testBrokerMetadataOnIdCollision 
STARTED

kafka.server.ServerGenerateBrokerIdTest > testBrokerMetadataOnIdCollision PASSED

kafka.server.ServerGenerateBrokerIdTest > testAutoGenerateBrokerId STARTED

kafka.server.ServerGenerateBrokerIdTest > testAutoGenerateBrokerId PASSED

kafka.server.ServerGenerateBrokerIdTest > testMultipleLogDirsMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > testMultipleLogDirsMetaProps PASSED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId STARTED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId PASSED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
STARTED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
PASSED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps PASSED

kafka.server.LogOffsetTest > testFetchOffsetsBeforeWithChangingSegmentSize 
STARTED

kafka.server.LogOffsetTest > testFetchOffsetsBeforeWithChangingSegmentSize 
PASSED

kafka.server.LogOffsetTest > testGetOffsetsBeforeEarliestTime STARTED

kafka.server.LogOffsetTest > testGetOffsetsBeforeEarliestTime PASSED

kafka.server.LogOffsetTest > testGetOffsetsForUnknownTopic STARTED

kafka.server.LogOffsetTest > testGetOffsetsForUnknownTopic PASSED

kafka.server.LogOffsetTest > testEmptyLogsGetOffsets STARTED

kafka.server.LogOffsetTest > testEmptyLogsGetOffsets PASSED

kafka.server.LogOffsetTest > testFetchOffsetsBeforeWithChangingSegments STARTED

kafka.server.LogOffsetTest > testFetchOffsetsBeforeWithChangingSegments PASSED

kafka.server.LogOffsetTest > testGetOffsetsBeforeLatestTime STARTED

kafka.server.LogOffsetTest > testGetOffsetsBeforeLatestTime PASSED

kafka.server.LogOffsetTest > testGetOffsetsBeforeNow STARTED

kafka.server.LogOffsetTest > testGetOffsetsBeforeNow PASSED

kafka.server.ReplicationQuotaManagerTest > shouldThrottleOnlyDefinedReplicas 
STARTED

kafka.server.ReplicationQuotaManagerTest > shouldThrottleOnlyDefinedReplicas 
PASSED

kafka.server.ReplicationQuotaManagerTest > 
shouldSupportWildcardThrottledReplicas STARTED

kafka.server.ReplicationQuotaManagerTest > 
shouldSupportWildcardThrottledReplicas PASSED

kafka.server.ReplicationQuotaManagerTest > 
shouldExceedQuotaThenReturnBackBelowBoundAsTimePasses STARTED

kafka.server.ReplicationQuotaManagerTest > 
shouldExceedQuotaThenReturnBackBelowBoundAsTimePasses PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.DelayedOperationTest > testRequestPurge STARTED

kafka.server.DelayedOperationTest > testRequestPurge PASSED

kafka.server.DelayedOperationTest > testRequestExpiry STARTED

kafka.server.DelayedOperationTest > testRequestExpiry PASSED

kafka.server.DelayedOperationTest > testRequestSatisfaction STARTED

kafka.server.DelayedOperationTest > testRequestSatisfaction PASSED

kafka.server.DynamicConfigTest > shouldFailFollowerConfigsWithInvalidValues 
STARTED

kafka.server.DynamicConfigTest > shouldFailFollowerConfigsWithInvalidValues 
PASSED

kafka.server.DynamicConfigTest > shouldFailWhenChangingUserUnknownConfig STARTED

kafka.server.DynamicConfigTest > shouldFailWhenChangingUserUnknownConfig PASSED

kafka.server.DynamicConfigTest > shouldFailLeaderConfigsWithInvalidValues 
STARTED

kafka.server.DynamicConfigTest > shouldFailLeaderConfigsWithInvalidValues PASSED

kafka.server.DynamicConfigTest > shouldFailWhenChangingClientIdUnknownConfig 
STARTED

kafka.server.DynamicConfigTest > shouldFailWhenChangingClientIdUnknownConfig 
PASSED

kafka.server.DynamicConfigTest >