Re: [VOTE] 1.1.1 RC3

[Brett Rann]

+1 (non binding)
rolling upgrade of shared staging multitenacy (200+ consumer groups)
cluster from 1.1.0 to 1.1.1-rc3 using the kafka_2.11-1.1.1.tgz artifact.
cluster looks healthy after upgrade. Lack of burrow lag suggests consumers
are still happy, and incoming messages remains the same.

On Mon, Jul 9, 2018 at 8:36 AM Dong Lin  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the fourth candidate for release of Apache Kafka 1.1.1.
>
> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
> released with 1.1.0 about 3 months ago. We have fixed about 25 issues since
> that release. A few of the more significant fixes include:
>
> KAFKA-6925  > - Fix memory
> leak in StreamsMetricsThreadImpl
> KAFKA-6937  > - In-sync
> replica delayed during fetch if replica throttle is exceeded
> KAFKA-6917  > - Process txn
> completion asynchronously to avoid deadlock
> KAFKA-6893  > - Create
> processors before starting acceptor to avoid ArithmeticException
> KAFKA-6870  > -
> Fix ConcurrentModificationException in SampledStat
> KAFKA-6878  > - Fix
> NullPointerException when querying global state store
> KAFKA-6879  > - Invoke
> session init callbacks outside lock to avoid Controller deadlock
> KAFKA-6857  > - Prevent
> follower from truncating to the wrong offset if undefined leader epoch is
> requested
> KAFKA-6854  > - Log cleaner
> fails with transaction markers that are deleted during clean
> KAFKA-6747  > - Check
> whether there is in-flight transaction before aborting transaction
> KAFKA-6748  > - Double
> check before scheduling a new task after the punctuate call
> KAFKA-6739  > -
> Fix IllegalArgumentException when down-converting from V2 to V0/V1
> KAFKA-6728  > -
> Fix NullPointerException when instantiating the HeaderConverter
>
> Kafka 1.1.1 release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
> 
>
> Release notes for the 1.1.1 release:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
> 
>
> *** Please download, test and vote by Thursday, July 12, 12pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
> 
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/
> 
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
> 
>
> * Javadoc:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/javadoc/
> 
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc3 tag:
> https://github.com/apache/kafka/tree/1.1.1-rc3
> 
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
> 
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
> 
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-1.1-jdk7/162
> 
>  >*
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/1.1/156/
> 
>
> Please test and verify the release 

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

[Ted Yu]

bq. Map changes, Set removals,

Is it possible to combine the two parameters into one Map where null Config
value signifies removal of config ?
This way, the following wouldn't occur (reducing un-intended config
removal):

bq. If a configuration key is specified in both *changes* and *removals*

*Cheers*

On Wed, Jul 11, 2018 at 10:54 AM Colin McCabe  wrote:

> Hi all,
>
> Previously, we discussed some issues with alterConfigs here on the mailing
> list, and eventually came to the conclusion that the RPC as implemented
> can't be used for a shell command modifying configurations.
>
> I wrote up a small KIP to fix the issues with the RPC.  Please take a look
> at
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-339%3A+Create+a+new+ModifyConfigs+API
>
> best,
> Colin
>

Re: [DISCUSS] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Dong Lin]

wiki page is currently read-only and it is unavailable for write operation.
Will update it later.

On Wed, Jul 11, 2018 at 8:48 PM, Dong Lin  wrote:

> Ah I see. Thanks for the suggestion. It is updated now.
>
> On Wed, Jul 11, 2018 at 8:13 PM, Ted Yu  wrote:
>
>> bq. the same approach used by "--config-file" in ConfigCommand.
>>
>> I should have copied more from the KIP.
>> What I meant was that ConfigCommand doesn't use "--config-file" option. So
>> 'same approach' implies StreamsResetter class, not ConfigCommand.
>>
>> I didn't mean to change ConfigCommand w.r.t. name of the option.
>>
>> Cheers
>>
>> On Wed, Jul 11, 2018 at 8:06 PM Dong Lin  wrote:
>>
>> > Do you mean we should replace "--command-config" with "--config-file" in
>> > ConfigCommand? There is backward compatibility concern with the change.
>> I
>> > am not sure the benefit of this change is worth the effort to deprecate
>> the
>> > old config. Maybe we should do it separately if more people thing it is
>> > necessary?
>> >
>> > On Wed, Jul 11, 2018 at 8:01 PM, Ted Yu  wrote:
>> >
>> > > bq. "--config-file" in ConfigCommand.
>> > >
>> > > Please update the above - it should be StreamsResetter
>> > >
>> > >
>> > > On Wed, Jul 11, 2018 at 7:59 PM Dong Lin  wrote:
>> > >
>> > > > Hey Ted,
>> > > >
>> > > > Thanks much for the suggestion. Yeah "config-file" looks better than
>> > > > "command-config". I have updated the KIP as suggested.
>> > > >
>> > > > Thanks,
>> > > > Dong
>> > > >
>> > > > On Wed, Jul 11, 2018 at 5:57 PM, Ted Yu 
>> wrote:
>> > > >
>> > > > > Looking at StreamsResetter.java :
>> > > > >
>> > > > >commandConfigOption = optionParser.accepts("config-file",
>> > > > "Property
>> > > > > file containing configs to be passed to admin cl
>> > > > >
>> > > > > Not sure you have considered naming the option in the above
>> fashion.
>> > > > >
>> > > > > Probably add the above to Alternative section.
>> > > > >
>> > > > > Cheers
>> > > > >
>> > > > > On Wed, Jul 11, 2018 at 2:04 PM Dong Lin 
>> > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > I have created KIP-340: Allow kafka-reassign-partitions.sh and
>> > > > > > kafka-log-dirs.sh to take admin client property file. See
>> > > > > >
>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > 340%3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-
>> > > > > dirs.sh+to+take+admin+client+property+file
>> > > > > > .
>> > > > > >
>> > > > > > This KIP provides a way to allow kafka-reassign-partitions.sh
>> and
>> > > > > > kafka-log-dirs.sh to talk to broker over SSL. Please review the
>> KIP
>> > > if
>> > > > > you
>> > > > > > have time.
>> > > > > >
>> > > > > >
>> > > > > > Thanks!
>> > > > > > Dong
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: [DISCUSS] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Dong Lin]

Ah I see. Thanks for the suggestion. It is updated now.

On Wed, Jul 11, 2018 at 8:13 PM, Ted Yu  wrote:

> bq. the same approach used by "--config-file" in ConfigCommand.
>
> I should have copied more from the KIP.
> What I meant was that ConfigCommand doesn't use "--config-file" option. So
> 'same approach' implies StreamsResetter class, not ConfigCommand.
>
> I didn't mean to change ConfigCommand w.r.t. name of the option.
>
> Cheers
>
> On Wed, Jul 11, 2018 at 8:06 PM Dong Lin  wrote:
>
> > Do you mean we should replace "--command-config" with "--config-file" in
> > ConfigCommand? There is backward compatibility concern with the change. I
> > am not sure the benefit of this change is worth the effort to deprecate
> the
> > old config. Maybe we should do it separately if more people thing it is
> > necessary?
> >
> > On Wed, Jul 11, 2018 at 8:01 PM, Ted Yu  wrote:
> >
> > > bq. "--config-file" in ConfigCommand.
> > >
> > > Please update the above - it should be StreamsResetter
> > >
> > >
> > > On Wed, Jul 11, 2018 at 7:59 PM Dong Lin  wrote:
> > >
> > > > Hey Ted,
> > > >
> > > > Thanks much for the suggestion. Yeah "config-file" looks better than
> > > > "command-config". I have updated the KIP as suggested.
> > > >
> > > > Thanks,
> > > > Dong
> > > >
> > > > On Wed, Jul 11, 2018 at 5:57 PM, Ted Yu  wrote:
> > > >
> > > > > Looking at StreamsResetter.java :
> > > > >
> > > > >commandConfigOption = optionParser.accepts("config-file",
> > > > "Property
> > > > > file containing configs to be passed to admin cl
> > > > >
> > > > > Not sure you have considered naming the option in the above
> fashion.
> > > > >
> > > > > Probably add the above to Alternative section.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Wed, Jul 11, 2018 at 2:04 PM Dong Lin 
> > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have created KIP-340: Allow kafka-reassign-partitions.sh and
> > > > > > kafka-log-dirs.sh to take admin client property file. See
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 340%3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-
> > > > > dirs.sh+to+take+admin+client+property+file
> > > > > > .
> > > > > >
> > > > > > This KIP provides a way to allow kafka-reassign-partitions.sh and
> > > > > > kafka-log-dirs.sh to talk to broker over SSL. Please review the
> KIP
> > > if
> > > > > you
> > > > > > have time.
> > > > > >
> > > > > >
> > > > > > Thanks!
> > > > > > Dong
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Ted Yu]

bq. the same approach used by "--config-file" in ConfigCommand.

I should have copied more from the KIP.
What I meant was that ConfigCommand doesn't use "--config-file" option. So
'same approach' implies StreamsResetter class, not ConfigCommand.

I didn't mean to change ConfigCommand w.r.t. name of the option.

Cheers

On Wed, Jul 11, 2018 at 8:06 PM Dong Lin  wrote:

> Do you mean we should replace "--command-config" with "--config-file" in
> ConfigCommand? There is backward compatibility concern with the change. I
> am not sure the benefit of this change is worth the effort to deprecate the
> old config. Maybe we should do it separately if more people thing it is
> necessary?
>
> On Wed, Jul 11, 2018 at 8:01 PM, Ted Yu  wrote:
>
> > bq. "--config-file" in ConfigCommand.
> >
> > Please update the above - it should be StreamsResetter
> >
> >
> > On Wed, Jul 11, 2018 at 7:59 PM Dong Lin  wrote:
> >
> > > Hey Ted,
> > >
> > > Thanks much for the suggestion. Yeah "config-file" looks better than
> > > "command-config". I have updated the KIP as suggested.
> > >
> > > Thanks,
> > > Dong
> > >
> > > On Wed, Jul 11, 2018 at 5:57 PM, Ted Yu  wrote:
> > >
> > > > Looking at StreamsResetter.java :
> > > >
> > > >commandConfigOption = optionParser.accepts("config-file",
> > > "Property
> > > > file containing configs to be passed to admin cl
> > > >
> > > > Not sure you have considered naming the option in the above fashion.
> > > >
> > > > Probably add the above to Alternative section.
> > > >
> > > > Cheers
> > > >
> > > > On Wed, Jul 11, 2018 at 2:04 PM Dong Lin 
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have created KIP-340: Allow kafka-reassign-partitions.sh and
> > > > > kafka-log-dirs.sh to take admin client property file. See
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 340%3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-
> > > > dirs.sh+to+take+admin+client+property+file
> > > > > .
> > > > >
> > > > > This KIP provides a way to allow kafka-reassign-partitions.sh and
> > > > > kafka-log-dirs.sh to talk to broker over SSL. Please review the KIP
> > if
> > > > you
> > > > > have time.
> > > > >
> > > > >
> > > > > Thanks!
> > > > > Dong
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Dong Lin]

Do you mean we should replace "--command-config" with "--config-file" in
ConfigCommand? There is backward compatibility concern with the change. I
am not sure the benefit of this change is worth the effort to deprecate the
old config. Maybe we should do it separately if more people thing it is
necessary?

On Wed, Jul 11, 2018 at 8:01 PM, Ted Yu  wrote:

> bq. "--config-file" in ConfigCommand.
>
> Please update the above - it should be StreamsResetter
>
>
> On Wed, Jul 11, 2018 at 7:59 PM Dong Lin  wrote:
>
> > Hey Ted,
> >
> > Thanks much for the suggestion. Yeah "config-file" looks better than
> > "command-config". I have updated the KIP as suggested.
> >
> > Thanks,
> > Dong
> >
> > On Wed, Jul 11, 2018 at 5:57 PM, Ted Yu  wrote:
> >
> > > Looking at StreamsResetter.java :
> > >
> > >commandConfigOption = optionParser.accepts("config-file",
> > "Property
> > > file containing configs to be passed to admin cl
> > >
> > > Not sure you have considered naming the option in the above fashion.
> > >
> > > Probably add the above to Alternative section.
> > >
> > > Cheers
> > >
> > > On Wed, Jul 11, 2018 at 2:04 PM Dong Lin  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have created KIP-340: Allow kafka-reassign-partitions.sh and
> > > > kafka-log-dirs.sh to take admin client property file. See
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 340%3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-
> > > dirs.sh+to+take+admin+client+property+file
> > > > .
> > > >
> > > > This KIP provides a way to allow kafka-reassign-partitions.sh and
> > > > kafka-log-dirs.sh to talk to broker over SSL. Please review the KIP
> if
> > > you
> > > > have time.
> > > >
> > > >
> > > > Thanks!
> > > > Dong
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Ted Yu]

bq. "--config-file" in ConfigCommand.

Please update the above - it should be StreamsResetter


On Wed, Jul 11, 2018 at 7:59 PM Dong Lin  wrote:

> Hey Ted,
>
> Thanks much for the suggestion. Yeah "config-file" looks better than
> "command-config". I have updated the KIP as suggested.
>
> Thanks,
> Dong
>
> On Wed, Jul 11, 2018 at 5:57 PM, Ted Yu  wrote:
>
> > Looking at StreamsResetter.java :
> >
> >commandConfigOption = optionParser.accepts("config-file",
> "Property
> > file containing configs to be passed to admin cl
> >
> > Not sure you have considered naming the option in the above fashion.
> >
> > Probably add the above to Alternative section.
> >
> > Cheers
> >
> > On Wed, Jul 11, 2018 at 2:04 PM Dong Lin  wrote:
> >
> > > Hi all,
> > >
> > > I have created KIP-340: Allow kafka-reassign-partitions.sh and
> > > kafka-log-dirs.sh to take admin client property file. See
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 340%3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-
> > dirs.sh+to+take+admin+client+property+file
> > > .
> > >
> > > This KIP provides a way to allow kafka-reassign-partitions.sh and
> > > kafka-log-dirs.sh to talk to broker over SSL. Please review the KIP if
> > you
> > > have time.
> > >
> > >
> > > Thanks!
> > > Dong
> > >
> >
>

Re: [DISCUSS] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Dong Lin]

Hey Ted,

Thanks much for the suggestion. Yeah "config-file" looks better than
"command-config". I have updated the KIP as suggested.

Thanks,
Dong

On Wed, Jul 11, 2018 at 5:57 PM, Ted Yu  wrote:

> Looking at StreamsResetter.java :
>
>commandConfigOption = optionParser.accepts("config-file", "Property
> file containing configs to be passed to admin cl
>
> Not sure you have considered naming the option in the above fashion.
>
> Probably add the above to Alternative section.
>
> Cheers
>
> On Wed, Jul 11, 2018 at 2:04 PM Dong Lin  wrote:
>
> > Hi all,
> >
> > I have created KIP-340: Allow kafka-reassign-partitions.sh and
> > kafka-log-dirs.sh to take admin client property file. See
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 340%3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-
> dirs.sh+to+take+admin+client+property+file
> > .
> >
> > This KIP provides a way to allow kafka-reassign-partitions.sh and
> > kafka-log-dirs.sh to talk to broker over SSL. Please review the KIP if
> you
> > have time.
> >
> >
> > Thanks!
> > Dong
> >
>

Re: [DISCUSS] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Ted Yu]

Looking at StreamsResetter.java :

   commandConfigOption = optionParser.accepts("config-file", "Property
file containing configs to be passed to admin cl

Not sure you have considered naming the option in the above fashion.

Probably add the above to Alternative section.

Cheers

On Wed, Jul 11, 2018 at 2:04 PM Dong Lin  wrote:

> Hi all,
>
> I have created KIP-340: Allow kafka-reassign-partitions.sh and
> kafka-log-dirs.sh to take admin client property file. See
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-340%3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-dirs.sh+to+take+admin+client+property+file
> .
>
> This KIP provides a way to allow kafka-reassign-partitions.sh and
> kafka-log-dirs.sh to talk to broker over SSL. Please review the KIP if you
> have time.
>
>
> Thanks!
> Dong
>

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

[John Roesler]

Hi all,

I have updated KIP-328 with all the feedback I've gotten so far. Please
take another look and let me know what you think!

Thanks,
-John

On Wed, Jul 11, 2018 at 12:28 AM Guozhang Wang  wrote:

> That is a good point..
>
> I cannot think of a better option than documentation and warning, and also
> given that we'd probably better not reusing the function name `until` for
> close time.
>
>
> Guozhang
>
>
> On Tue, Jul 10, 2018 at 3:31 PM, John Roesler  wrote:
>
> > I had some opportunity to reflect on the default for close time today...
> >
> > Note that the current "close time" is equal to the retention time, and
> > therefore "close" today shares the default retention of 24h.
> >
> > It would definitely break any application that today specifies a
> retention
> > time to set close shorter than that time. It's also likely to break apps
> if
> > they *don't* set the retention time and rely on the 24h default. So it's
> > unfortunate, but I think if "close" isn't set, we should use the
> retention
> > time instead of a fixed default.
> >
> > When we ultimately remove the retention time parameter ("until"), we will
> > have to set "close" to a default of 24h.
> >
> > Of course, this has a negative impact on the user of "final results",
> since
> > they won't see any output at all for retentionTime/24h, and may find this
> > confusing. What can we do about this except document it well? Maybe log a
> > warning if we see that close wasn't explicitly set while using "final
> > results"?
> >
> > Thanks,
> > -John
> >
> > On Tue, Jul 10, 2018 at 10:46 AM John Roesler  wrote:
> >
> > > Hi Guozhang,
> > >
> > > That sounds good to me. I'll include that in the KIP.
> > >
> > > Thanks,
> > > -John
> > >
> > > On Mon, Jul 9, 2018 at 6:33 PM Guozhang Wang 
> wrote:
> > >
> > >> Let me clarify a bit on what I meant about moving `retentionPeriod` to
> > >> WindowStoreBuilder:
> > >>
> > >> In another discussion we had around KIP-319 / 330, that the "retention
> > >> period" should not really be a window spec, but only a window store
> > spec,
> > >> as it only affects how long to retain each window to be queryable
> along
> > >> with the storage cost.
> > >>
> > >> More specifically, today the "maintainMs" returned from Windows is
> used
> > in
> > >> three places:
> > >>
> > >> 1) for windowed aggregations, they are passed in directly into
> > >> `Stores.persistentWindows()` as the retention period parameters. For
> > this
> > >> use case we should just let the WindowStoreBuilder to specify this
> value
> > >> itself.
> > >>
> > >> NOTE: It is also returned in the KStreamWindowAggregate processor, to
> > >> determine if a received record should be dropped due to its lateness.
> We
> > >> may need to think of another way to get this value inside the
> processor
> > >>
> > >> 2) for windowed stream-stream join, it is used as the join range
> > parameter
> > >> but only to check that "windowSizeMs <= retentionPeriodMs". We can do
> > this
> > >> check at the store builder lever instead of at the processor level.
> > >>
> > >>
> > >> If we can remove its usage in both 1) and 2), then we should be able
> to
> > >> safely remove this from the `Windows` spec.
> > >>
> > >>
> > >> Guozhang
> > >>
> > >>
> > >> On Mon, Jul 9, 2018 at 3:53 PM, John Roesler 
> wrote:
> > >>
> > >> > Thanks for the reply, Guozhang,
> > >> >
> > >> > Good! I agree, that is also a good reason, and I actually made use
> of
> > >> that
> > >> > in my tests. I'll update the KIP.
> > >> >
> > >> > By the way, I chose "allowedLateness" as I was trying to pick a
> better
> > >> name
> > >> > than "close", but I think it's actually the wrong name. We don't
> want
> > to
> > >> > bound the lateness of events in general, only with respect to the
> end
> > of
> > >> > their window.
> > >> >
> > >> > If we have a window [0,10), with "allowedLateness" of 5, then if we
> > get
> > >> an
> > >> > event with timestamp 3 at time 9, the name implies we'd reject it,
> > which
> > >> > seems silly. Really, we'd only want to start rejecting that event at
> > >> stream
> > >> > time 15.
> > >> >
> > >> > What I meant was more like "allowedLatenessAfterWindowEnd", but
> > that's
> > >> too
> > >> > verbose. I think that "close" + some documentation about what it
> means
> > >> will
> > >> > be better.
> > >> >
> > >> > 1: "Close" would be measured from the end of the window, so a
> > reasonable
> > >> > default would be "0". Recall that "close" really only needs to be
> > >> specified
> > >> > for final results, and a default of 0 would produce the most
> intuitive
> > >> > results. If folks later discover that they are missing some late
> > events,
> > >> > they can adjust the parameter accordingly. IMHO, any other value
> would
> > >> just
> > >> > be a guess on our part.
> > >> >
> > >> > 2a:
> > >> > I think you're saying to re-use "until" instead of adding "close" to
> > the
> > >> > window.
> > >> >
> > >> > The downside here would be that the semantic 

Build failed in Jenkins: kafka-trunk-jdk8 #2797

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Tighten FileRecords size checks to prevent overflow (#5332)

--
[...truncated 869.89 KB...]

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testReassignPartitionLeaderElectionWithEmptyIsr PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElectionAllIsrSimultaneouslyShutdown 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElectionAllIsrSimultaneouslyShutdown PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionEnabled 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionEnabled 
PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaNotInIsrNotLive 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaNotInIsrNotLive 
PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionDisabled 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionDisabled 
PASSED

kafka.controller.PartitionStateMachineTest > 
testNonexistentPartitionToNewPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testNonexistentPartitionToNewPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionErrorCodeFromCreateStates STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionErrorCodeFromCreateStates PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToNonexistentPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToNonexistentPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOfflineTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOfflineTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOfflinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOfflinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOnlinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOnlinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOfflinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOfflinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionZkUtilsExceptionFromCreateStates 
STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionZkUtilsExceptionFromCreateStates 
PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidNewPartitionToNonexistentPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNewPartitionToNonexistentPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNewPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNewPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionErrorCodeFromStateLookup STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionErrorCodeFromStateLookup PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransitionForControlledShutdown STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransitionForControlledShutdown PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionZkUtilsExceptionFromStateLookup 
STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionZkUtilsExceptionFromStateLookup 
PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNonexistentPartitionTransition STARTED

Jenkins build is back to normal : kafka-trunk-jdk10 #282

[Apache Jenkins Server]

See 

[DISCUSS] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Dong Lin]

Hi all,

I have created KIP-340: Allow kafka-reassign-partitions.sh and
kafka-log-dirs.sh to take admin client property file. See
https://cwiki.apache.org/confluence/display/KAFKA/KIP-340%3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-dirs.sh+to+take+admin+client+property+file
.

This KIP provides a way to allow kafka-reassign-partitions.sh and
kafka-log-dirs.sh to talk to broker over SSL. Please review the KIP if you
have time.


Thanks!
Dong

Re: [VOTE] 1.1.1 RC3

[Jakub Scholz]

+1 (non-binbding) ... built from source, run tests and used it with several
of my applications without any problems.

Thanks & Regards
Jakub


On Mon, Jul 9, 2018 at 12:36 AM Dong Lin  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the fourth candidate for release of Apache Kafka 1.1.1.
>
> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
> released with 1.1.0 about 3 months ago. We have fixed about 25 issues since
> that release. A few of the more significant fixes include:
>
> KAFKA-6925  - Fix memory
> leak in StreamsMetricsThreadImpl
> KAFKA-6937  - In-sync
> replica delayed during fetch if replica throttle is exceeded
> KAFKA-6917  - Process
> txn
> completion asynchronously to avoid deadlock
> KAFKA-6893  - Create
> processors before starting acceptor to avoid ArithmeticException
> KAFKA-6870  -
> Fix ConcurrentModificationException in SampledStat
> KAFKA-6878  - Fix
> NullPointerException when querying global state store
> KAFKA-6879  - Invoke
> session init callbacks outside lock to avoid Controller deadlock
> KAFKA-6857  - Prevent
> follower from truncating to the wrong offset if undefined leader epoch is
> requested
> KAFKA-6854  - Log
> cleaner
> fails with transaction markers that are deleted during clean
> KAFKA-6747  - Check
> whether there is in-flight transaction before aborting transaction
> KAFKA-6748  - Double
> check before scheduling a new task after the punctuate call
> KAFKA-6739  -
> Fix IllegalArgumentException when down-converting from V2 to V0/V1
> KAFKA-6728  -
> Fix NullPointerException when instantiating the HeaderConverter
>
> Kafka 1.1.1 release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
>
> Release notes for the 1.1.1 release:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, July 12, 12pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/javadoc/
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc3 tag:
> https://github.com/apache/kafka/tree/1.1.1-rc3
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-1.1-jdk7/162
> *
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/1.1/156/
>
> Please test and verify the release artifacts and submit a vote for this RC,
> or report any issues so we can fix them and get a new RC out ASAP. Although
> this release vote requires PMC votes to pass, testing, votes, and bug
> reports are valuable and appreciated from everyone.
>
>
> Regards,
> Dong
>

Re: [VOTE] 2.0.0 RC2

[Jakub Scholz]

+1 (non-binbding) ... I built the RC2 from source, run tests and used it
with several of my applications without any problems.

Thanks & Regards
Jakub

On Tue, Jul 10, 2018 at 7:17 PM Rajini Sivaram 
wrote:

> Hello Kafka users, developers and client-developers,
>
>
> This is the third candidate for release of Apache Kafka 2.0.0.
>
>
> This is a major version release of Apache Kafka. It includes 40 new  KIPs
> and
>
> several critical bug fixes. Please see the 2.0.0 release plan for more
> details:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
>
>
> A few notable highlights:
>
>- Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for CreateTopics
>(KIP-277)
>- SASL/OAUTHBEARER implementation (KIP-255)
>- Improved quota communication and customization of quotas (KIP-219,
>KIP-257)
>- Efficient memory usage for down conversion (KIP-283)
>- Fix log divergence between leader and follower during fast leader
>failover (KIP-279)
>- Drop support for Java 7 and remove deprecated code including old scala
>clients
>- Connect REST extension plugin, support for externalizing secrets and
>improved error handling (KIP-285, KIP-297, KIP-298 etc.)
>- Scala API for Kafka Streams and other Streams API improvements
>(KIP-270, KIP-150, KIP-245, KIP-251 etc.)
>
>
> Release notes for the 2.0.0 release:
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/RELEASE_NOTES.html
>
>
> *** Please download, test and vote by Friday, July 13, 4pm PT
>
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
>
> http://kafka.apache.org/KEYS
>
>
> * Release artifacts to be voted upon (source and binary):
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/
>
>
> * Maven artifacts to be voted upon:
>
> https://repository.apache.org/content/groups/staging/
>
>
> * Javadoc:
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/javadoc/
>
>
> * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
>
> https://github.com/apache/kafka/tree/2.0.0-rc2
>
>
>
> * Documentation:
>
> http://kafka.apache.org/20/documentation.html
>
>
> * Protocol:
>
> http://kafka.apache.org/20/protocol.html
>
>
> * Successful Jenkins builds for the 2.0 branch:
>
> Unit/integration tests: https://builds.apache.org/job/kafka-2.0-jdk8/72/
>
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/2.0/27/
>
>
> /**
>
>
> Thanks,
>
>
> Rajini
>

[jira] [Created] (KAFKA-7152) replica should be in-sync if its LEO equals leader's LEO

[Dong Lin (JIRA)]

Dong Lin created KAFKA-7152:
---

 Summary: replica should be in-sync if its LEO equals leader's LEO
 Key: KAFKA-7152
 URL: https://issues.apache.org/jira/browse/KAFKA-7152
 Project: Kafka
  Issue Type: Improvement
Reporter: Dong Lin
Assignee: Dong Lin


Currently a replica will be moved out of ISR if follower has not fetched from 
leader for 10 sec (default replica.lag.time.max.ms). This cases problem in the 
following scenario:

Say follower's ReplicaFetchThread needs to fetch 2k partitions from the leader 
broker. Only 100 out of 2k partitions are actively being produced to and 
therefore the total bytes in rate for those 2k partitions are small. The 
following will happen:

 

1) The follower's ReplicaFetcherThread sends FetchRequest for those 2k 
partitions.

2) Because the total bytes-in-rate for those 2k partitions is very small, 
follower is able to catch up and leader broker adds these 2k partitions to ISR. 
Follower's lastCaughtUpTimeMs for all partitions are updated to the current 
time T0.

3) Since follower has caught up for all 2k partitions, leader updates 2k 
partition znodes to include the follower in the ISR. It may take 20 seconds to 
write 2k partition znodes if each znode write operation takes 10 ms.

4) At T0 + 15, maybeShrinkIsr() is invoked on leader broker. Since there is no 
FetchRequet from the follower for more than 10 seconds after T0, all those 2k 
partitions will be considered as out of syn and the follower will be removed 
from ISR.

5) The follower receives FetchResponse at least 20 seconds after T0. That means 
the next FetchRequest from follower to leader will be after T0 + 20.

The sequence of events described above will loop over time. There will be 
constant churn of URP in the cluster even if follower can catch up with 
leader's byte-in-rate. This reduces the cluster availability.

 

In order to address this problem, one simple approach is to keep follower in 
the ISR as long as follower's LEO equals leader's LEO regardless of follower's 
lastCaughtUpTimeMs. This is particularly useful if there are a lot of inactive 
partitions in the cluster.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7151) Broker running out of disk space may result in state where unclean leader election is required

[Anna Povzner (JIRA)]

Anna Povzner created KAFKA-7151:
---

 Summary: Broker running out of disk space may result in state 
where unclean leader election is required
 Key: KAFKA-7151
 URL: https://issues.apache.org/jira/browse/KAFKA-7151
 Project: Kafka
  Issue Type: Bug
Reporter: Anna Povzner


We have seen situations like the following:

1) Broker A is a leader for topic partition, and brokers B and C are the 
followers

2) Broker A is running out of disk space, shrinks ISR only to itself, and then 
sometime later gets disk errors, etc.

3) Broker A is stopped, disk space is reclaimed, and broker A is restarted

Result: Broker A becomes a leader, but followers cannot fetch because their log 
is ahead. The only way to continue is to enable unclean leader election.

 

There are several issues here:

-- if the machine is running out of disk space, we do not reliably get an error 
from a file system as soon as that happens. The broker could be in a state 
where some writes succeed (possibly if the write is not flushed to disk) and 
some writes fails, or maybe fail later. This may cause fetchers fetch records 
that are still in the leader's file system cache, and then the flush to disk 
failing on the leader, causes followers to be ahead of the leader.

-- I am not sure exactly why, but it seems like the leader broker (that is 
running out of disk space) may also stop servicing fetch requests making 
followers fall behind and kicked out of ISR.

Ideally, the broker should stop being a leader for any topic partition before 
accepting any records that may fail to be flushed to disk. One option is to 
automatically detect disk space usage and make a broker read-only for topic 
partitions if disk space gets to 80% or something. Maybe there is a better 
option.  

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] 2.0.0 RC2

[Vahid S Hashemian]

+1 (non-binding)

Built executables from source and ran quickstart (Ubuntu / Java 8)

Thanks!
--Vahid




From:   Brett Rann 
To: dev@kafka.apache.org
Cc: Users , kafka-clients 

Date:   07/10/2018 09:53 PM
Subject:Re: [VOTE] 2.0.0 RC2



+1 (non binding)
rolling upgrade of tiny shared staging multitenacy (200+ consumer groups)
cluster from 1.1 to 2.0.0-rc1 to 2.0.0-rc2. cluster looks healthy after
upgrade. Lack of burrow lag suggests consumers are still happy, and
incoming messages remains the same.  Will monitor.

On Wed, Jul 11, 2018 at 3:17 AM Rajini Sivaram 
wrote:

> Hello Kafka users, developers and client-developers,
>
>
> This is the third candidate for release of Apache Kafka 2.0.0.
>
>
> This is a major version release of Apache Kafka. It includes 40 new KIPs
> and
>
> several critical bug fixes. Please see the 2.0.0 release plan for more
> details:
>
> 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820

> <
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
>
>
>
> A few notable highlights:
>
> - Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for CreateTopics
> (KIP-277)
> - SASL/OAUTHBEARER implementation (KIP-255)
> - Improved quota communication and customization of quotas (KIP-219,
> KIP-257)
> - Efficient memory usage for down conversion (KIP-283)
> - Fix log divergence between leader and follower during fast leader
> failover (KIP-279)
> - Drop support for Java 7 and remove deprecated code including old scala
> clients
> - Connect REST extension plugin, support for externalizing secrets and
> improved error handling (KIP-285, KIP-297, KIP-298 etc.)
> - Scala API for Kafka Streams and other Streams API improvements
> (KIP-270, KIP-150, KIP-245, KIP-251 etc.)
>
>
> Release notes for the 2.0.0 release:
>
> 
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/RELEASE_NOTES.html

> <
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/RELEASE_NOTES.html
>
>
>
> *** Please download, test and vote by Friday, July 13, 4pm PT
>
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
>
> 
http://kafka.apache.org/KEYS

> <
http://kafka.apache.org/KEYS
>
>
>
> * Release artifacts to be voted upon (source and binary):
>
> 
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/

> <
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/
>
>
>
> * Maven artifacts to be voted upon:
>
> 
https://repository.apache.org/content/groups/staging/

> <
https://repository.apache.org/content/groups/staging/
>
>
>
> * Javadoc:
>
> 
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/javadoc/

> <
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/javadoc/
>
>
>
> * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
>
> 
https://github.com/apache/kafka/tree/2.0.0-rc2

> <
https://github.com/apache/kafka/tree/2.0.0-rc2
>
>
>
>
> * Documentation:
>
> 
http://kafka.apache.org/20/documentation.html

> <
http://kafka.apache.org/20/documentation.html
>
>
>
> * Protocol:
>
> 
http://kafka.apache.org/20/protocol.html

> <
http://kafka.apache.org/20/protocol.html
>
>
>
> * Successful Jenkins builds for the 2.0 branch:
>
> Unit/integration tests: 
https://builds.apache.org/job/kafka-2.0-jdk8/72/

> <
https://builds.apache.org/job/kafka-2.0-jdk8/72/
>
>
> System tests:
> 
https://jenkins.confluent.io/job/system-test-kafka/job/2.0/27/

> <
https://jenkins.confluent.io/job/system-test-kafka/job/2.0/27/
>
>
>
> /**
>
>
> Thanks,
>
>
> Rajini
>


-- 

Brett Rann

Senior DevOps Engineer


Zendesk International Ltd

395 Collins Street, Melbourne VIC 3000 Australia

Mobile: +61 (0) 418 826 017

[jira] [Created] (KAFKA-7150) Error in processing fetched data for one partition may stop follower fetching other partitions

[Anna Povzner (JIRA)]

Anna Povzner created KAFKA-7150:
---

 Summary: Error in processing fetched data for one partition may 
stop follower fetching other partitions
 Key: KAFKA-7150
 URL: https://issues.apache.org/jira/browse/KAFKA-7150
 Project: Kafka
  Issue Type: Bug
  Components: replication
Affects Versions: 1.1.0, 1.0.2, 0.11.0.3, 0.10.2.2
Reporter: Anna Povzner


If the followers fails to process data for one topic partitions, like out of 
order offsets error, the whole ReplicaFetcherThread is killed, which also stops 
fetching for other topic partitions serviced by this fetcher thread. This may 
result in un-necessary under-replicated partitions. I think it would be better 
to continue fetching for other topic partitions, and just remove the partition 
with an error from the responsibility of the fetcher thread.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

[Dong Lin]

Hey Jun,

Certainly. We can discuss later after KIP-320 settles.

Thanks!
Dong


On Wed, Jul 11, 2018 at 8:54 AM, Jun Rao  wrote:

> Hi, Dong,
>
> Sorry for the late response. Since KIP-320 is covering some of the similar
> problems described in this KIP, perhaps we can wait until KIP-320 settles
> and see what's still left uncovered in this KIP.
>
> Thanks,
>
> Jun
>
> On Mon, Jun 4, 2018 at 7:03 PM, Dong Lin  wrote:
>
> > Hey Jun,
> >
> > It seems that we have made considerable progress on the discussion of
> > KIP-253 since February. Do you think we should continue the discussion
> > there, or can we continue the voting for this KIP? I am happy to submit
> the
> > PR and move forward the progress for this KIP.
> >
> > Thanks!
> > Dong
> >
> >
> > On Wed, Feb 7, 2018 at 11:42 PM, Dong Lin  wrote:
> >
> > > Hey Jun,
> > >
> > > Sure, I will come up with a KIP this week. I think there is a way to
> > allow
> > > partition expansion to arbitrary number without introducing new
> concepts
> > > such as read-only partition or repartition epoch.
> > >
> > > Thanks,
> > > Dong
> > >
> > > On Wed, Feb 7, 2018 at 5:28 PM, Jun Rao  wrote:
> > >
> > >> Hi, Dong,
> > >>
> > >> Thanks for the reply. The general idea that you had for adding
> > partitions
> > >> is similar to what we had in mind. It would be useful to make this
> more
> > >> general, allowing adding an arbitrary number of partitions (instead of
> > >> just
> > >> doubling) and potentially removing partitions as well. The following
> is
> > >> the
> > >> high level idea from the discussion with Colin, Jason and Ismael.
> > >>
> > >> * To change the number of partitions from X to Y in a topic, the
> > >> controller
> > >> marks all existing X partitions as read-only and creates Y new
> > partitions.
> > >> The new partitions are writable and are tagged with a higher
> repartition
> > >> epoch (RE).
> > >>
> > >> * The controller propagates the new metadata to every broker. Once the
> > >> leader of a partition is marked as read-only, it rejects the produce
> > >> requests on this partition. The producer will then refresh the
> metadata
> > >> and
> > >> start publishing to the new writable partitions.
> > >>
> > >> * The consumers will then be consuming messages in RE order. The
> > consumer
> > >> coordinator will only assign partitions in the same RE to consumers.
> > Only
> > >> after all messages in an RE are consumed, will partitions in a higher
> RE
> > >> be
> > >> assigned to consumers.
> > >>
> > >> As Colin mentioned, if we do the above, we could potentially (1) use a
> > >> globally unique partition id, or (2) use a globally unique topic id to
> > >> distinguish recreated partitions due to topic deletion.
> > >>
> > >> So, perhaps we can sketch out the re-partitioning KIP a bit more and
> see
> > >> if
> > >> there is any overlap with KIP-232. Would you be interested in doing
> > that?
> > >> If not, we can do that next week.
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Tue, Feb 6, 2018 at 11:30 AM, Dong Lin 
> wrote:
> > >>
> > >> > Hey Jun,
> > >> >
> > >> > Interestingly I am also planning to sketch a KIP to allow partition
> > >> > expansion for keyed topics after this KIP. Since you are already
> doing
> > >> > that, I guess I will just share my high level idea here in case it
> is
> > >> > helpful.
> > >> >
> > >> > The motivation for the KIP is that we currently lose order guarantee
> > for
> > >> > messages with the same key if we expand partitions of keyed topic.
> > >> >
> > >> > The solution can probably be built upon the following ideas:
> > >> >
> > >> > - Partition number of the keyed topic should always be doubled (or
> > >> > multiplied by power of 2). Given that we select a partition based on
> > >> > hash(key) % partitionNum, this should help us ensure that, a message
> > >> > assigned to an existing partition will not be mapped to another
> > existing
> > >> > partition after partition expansion.
> > >> >
> > >> > - Producer includes in the ProduceRequest some information that
> helps
> > >> > ensure that messages produced ti a partition will monotonically
> > >> increase in
> > >> > the partitionNum of the topic. In other words, if broker receives a
> > >> > ProduceRequest and notices that the producer does not know the
> > partition
> > >> > number has increased, broker should reject this request. That
> > >> "information"
> > >> > maybe leaderEpoch, max partitionEpoch of the partitions of the
> topic,
> > or
> > >> > simply partitionNum of the topic. The benefit of this property is
> that
> > >> we
> > >> > can keep the new logic for in-order message consumption entirely in
> > how
> > >> > consumer leader determines the partition -> consumer mapping.
> > >> >
> > >> > - When consumer leader determines partition -> consumer mapping,
> > leader
> > >> > first reads the start position for each partition using
> > >> OffsetFetchRequest.
> > >> > If start position are all non-zero, then assignment can be done in
> its
> > >> > 

[DISCUSS]: KIP-339: Create a new ModifyConfigs API

[Colin McCabe]

Hi all,

Previously, we discussed some issues with alterConfigs here on the mailing 
list, and eventually came to the conclusion that the RPC as implemented can't 
be used for a shell command modifying configurations.

I wrote up a small KIP to fix the issues with the RPC.  Please take a look at 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-339%3A+Create+a+new+ModifyConfigs+API

best,
Colin

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Dong Lin]

Hey Anna,

Thanks much for the explanation. Approach 1 also sounds good to me. I think
findOffsets() is useful for users who don't use automatic offset reset
policy.

Just one more question. Since users who store offsets externally need to
provide leaderEpoch to findOffsets(...), do we need an extra API for user
to get both offset and leaderEpoch, e.g. recordPosition()?

Thanks,
Dong

On Wed, Jul 11, 2018 at 10:12 AM, Anna Povzner  wrote:

> Hi Dong,
>
>
> What I called “not covering all use cases” is what you call best-effort
> (not guaranteeing some corner cases). I think we are on the same page here.
>
>
> I wanted to be clear in the API whether the consumer seeks to a position
> (offset) or to a record (offset, leader epoch). The only use-case of
> seeking to a record is seeking to a committed offset for a user who stores
> committed offsets externally. (Unless users find some other reason to seek
> to a record.) I thought it was possible to provide this functionality with
> findOffset(offset, leader epoch) followed by a seek(offset). However, you
> are right that this will not handle the race condition where non-divergent
> offset found by findOffset() could change again before the consumer does
> the first fetch.
>
>
> Regarding position() — if we add position that returns (offset, leader
> epoch), this is specifically a position after a record that was actually
> consumed or position of a committed record. In which case, I still think
> it’s cleaner to get a record position of consumed message from a new helper
> method in ConsumerRecords() or from committed offsets.
>
>
> I think all the use-cases could be then covered with:
>
> (Approach 1)
>
> seekToRecord(offset, leaderEpoch) — this will just initialize/set the
> consumer state;
>
> findOffsets(offset, leaderEpoch) returns {offset, leaderEpoch}
>
>
> If we agree that the race condition is also a corner case, then I think we
> can cover use-cases with:
>
> (Approach 2)
>
> findOffsets(offset, leaderEpoch) returns offset — we still want leader
> epoch as a parameter for the users who store their committed offsets
> externally.
>
>
> I am actually now leaning more to approach 1, since it is more explicit,
> and maybe there are more use cases for it.
>
>
> Thanks,
>
> Anna
>
>
> On Tue, Jul 10, 2018 at 3:47 PM Dong Lin  wrote:
>
> > Hey Anna,
> >
> > Thanks for the comment. To answer your question, it seems that we can
> cover
> > all case in this KIP. As stated in "Consumer Handling" section, KIP-101
> > based approach will be used to derive the truncation offset from the
> > 2-tuple (offset, leaderEpoch). This approach is best effort and it is
> > inaccurate only in very rare scenarios (as described in KIP-279).
> >
> > By using seek(offset, leaderEpoch), consumer will still be able to follow
> > this best-effort approach to detect log truncation and determine the
> > truncation offset. On the other hand, if we use seek(offset), consumer
> will
> > not detect log truncation in some cases which weakens the guarantee of
> this
> > KIP. Does this make sense?
> >
> > Thanks,
> > Dong
> >
> > On Tue, Jul 10, 2018 at 3:14 PM, Anna Povzner  wrote:
> >
> > > Sorry, I hit "send" before finishing. Continuing...
> > >
> > >
> > > 2) Hiding most of the consumer handling log truncation logic with
> minimal
> > > exposure in KafkaConsumer API.  I was proposing this path.
> > >
> > >
> > > Before answering your specific questions… I want to answer to your
> > comment
> > > “In general, maybe we should discuss the final solution that covers all
> > > cases?”. With current KIP, we don’t cover all cases of consumer
> detecting
> > > log truncation because the KIP proposes a leader epoch cache in
> consumer
> > > that does not persist across restarts. Plus, we only store last
> committed
> > > offset (either internally or users can store externally). This has a
> > > limitation that the consumer will not always be able to find point of
> > > truncation just because we have a limited history (just one data
> point).
> > >
> > >
> > > So, maybe we should first agree on whether we accept that storing last
> > > committed offset/leader epoch has a limitation that the consumer will
> not
> > > be able to detect log truncation in all cases?
> > >
> > >
> > > Thanks,
> > >
> > > Anna
> > >
> > > On Tue, Jul 10, 2018 at 2:20 PM Anna Povzner 
> wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > Thanks for the follow up! I finally have much more clear
> understanding
> > of
> > > > where you are coming from.
> > > >
> > > > You are right. The success of findOffsets()/finding a point of
> > > > non-divergence depends on whether we have enough entries in the
> > > consumer's
> > > > leader epoch cache. However, I think this is a fundamental limitation
> > of
> > > > having a leader epoch cache that does not persist across consumer
> > > restarts.
> > > >
> > > > If we consider the general case where consumer may or may not have
> this
> > > > cache, then I see two paths:
> > > > 1) 

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Anna Povzner]

Hi Dong,


What I called “not covering all use cases” is what you call best-effort
(not guaranteeing some corner cases). I think we are on the same page here.


I wanted to be clear in the API whether the consumer seeks to a position
(offset) or to a record (offset, leader epoch). The only use-case of
seeking to a record is seeking to a committed offset for a user who stores
committed offsets externally. (Unless users find some other reason to seek
to a record.) I thought it was possible to provide this functionality with
findOffset(offset, leader epoch) followed by a seek(offset). However, you
are right that this will not handle the race condition where non-divergent
offset found by findOffset() could change again before the consumer does
the first fetch.


Regarding position() — if we add position that returns (offset, leader
epoch), this is specifically a position after a record that was actually
consumed or position of a committed record. In which case, I still think
it’s cleaner to get a record position of consumed message from a new helper
method in ConsumerRecords() or from committed offsets.


I think all the use-cases could be then covered with:

(Approach 1)

seekToRecord(offset, leaderEpoch) — this will just initialize/set the
consumer state;

findOffsets(offset, leaderEpoch) returns {offset, leaderEpoch}


If we agree that the race condition is also a corner case, then I think we
can cover use-cases with:

(Approach 2)

findOffsets(offset, leaderEpoch) returns offset — we still want leader
epoch as a parameter for the users who store their committed offsets
externally.


I am actually now leaning more to approach 1, since it is more explicit,
and maybe there are more use cases for it.


Thanks,

Anna


On Tue, Jul 10, 2018 at 3:47 PM Dong Lin  wrote:

> Hey Anna,
>
> Thanks for the comment. To answer your question, it seems that we can cover
> all case in this KIP. As stated in "Consumer Handling" section, KIP-101
> based approach will be used to derive the truncation offset from the
> 2-tuple (offset, leaderEpoch). This approach is best effort and it is
> inaccurate only in very rare scenarios (as described in KIP-279).
>
> By using seek(offset, leaderEpoch), consumer will still be able to follow
> this best-effort approach to detect log truncation and determine the
> truncation offset. On the other hand, if we use seek(offset), consumer will
> not detect log truncation in some cases which weakens the guarantee of this
> KIP. Does this make sense?
>
> Thanks,
> Dong
>
> On Tue, Jul 10, 2018 at 3:14 PM, Anna Povzner  wrote:
>
> > Sorry, I hit "send" before finishing. Continuing...
> >
> >
> > 2) Hiding most of the consumer handling log truncation logic with minimal
> > exposure in KafkaConsumer API.  I was proposing this path.
> >
> >
> > Before answering your specific questions… I want to answer to your
> comment
> > “In general, maybe we should discuss the final solution that covers all
> > cases?”. With current KIP, we don’t cover all cases of consumer detecting
> > log truncation because the KIP proposes a leader epoch cache in consumer
> > that does not persist across restarts. Plus, we only store last committed
> > offset (either internally or users can store externally). This has a
> > limitation that the consumer will not always be able to find point of
> > truncation just because we have a limited history (just one data point).
> >
> >
> > So, maybe we should first agree on whether we accept that storing last
> > committed offset/leader epoch has a limitation that the consumer will not
> > be able to detect log truncation in all cases?
> >
> >
> > Thanks,
> >
> > Anna
> >
> > On Tue, Jul 10, 2018 at 2:20 PM Anna Povzner  wrote:
> >
> > > Hi Dong,
> > >
> > > Thanks for the follow up! I finally have much more clear understanding
> of
> > > where you are coming from.
> > >
> > > You are right. The success of findOffsets()/finding a point of
> > > non-divergence depends on whether we have enough entries in the
> > consumer's
> > > leader epoch cache. However, I think this is a fundamental limitation
> of
> > > having a leader epoch cache that does not persist across consumer
> > restarts.
> > >
> > > If we consider the general case where consumer may or may not have this
> > > cache, then I see two paths:
> > > 1) Letting the user to track the leader epoch history externally, and
> > have
> > > more exposure to leader epoch and finding point of non-divergence in
> > > KafkaConsumer API. I understand this is the case you were talking
> about.
> > >
> > >
> > >
> > > On Tue, Jul 10, 2018 at 12:16 PM Dong Lin  wrote:
> > >
> > >> Hey Anna,
> > >>
> > >> Thanks much for your detailed explanation and example! It does help me
> > >> understand the difference between our understanding.
> > >>
> > >> So it seems that the solution based on findOffsets() currently focuses
> > >> mainly on the scenario that consumer has cached leaderEpoch -> offset
> > >> mapping whereas I was thinking about the 

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

[Jun Rao]

Hi, Dong,

Sorry for the late response. Since KIP-320 is covering some of the similar
problems described in this KIP, perhaps we can wait until KIP-320 settles
and see what's still left uncovered in this KIP.

Thanks,

Jun

On Mon, Jun 4, 2018 at 7:03 PM, Dong Lin  wrote:

> Hey Jun,
>
> It seems that we have made considerable progress on the discussion of
> KIP-253 since February. Do you think we should continue the discussion
> there, or can we continue the voting for this KIP? I am happy to submit the
> PR and move forward the progress for this KIP.
>
> Thanks!
> Dong
>
>
> On Wed, Feb 7, 2018 at 11:42 PM, Dong Lin  wrote:
>
> > Hey Jun,
> >
> > Sure, I will come up with a KIP this week. I think there is a way to
> allow
> > partition expansion to arbitrary number without introducing new concepts
> > such as read-only partition or repartition epoch.
> >
> > Thanks,
> > Dong
> >
> > On Wed, Feb 7, 2018 at 5:28 PM, Jun Rao  wrote:
> >
> >> Hi, Dong,
> >>
> >> Thanks for the reply. The general idea that you had for adding
> partitions
> >> is similar to what we had in mind. It would be useful to make this more
> >> general, allowing adding an arbitrary number of partitions (instead of
> >> just
> >> doubling) and potentially removing partitions as well. The following is
> >> the
> >> high level idea from the discussion with Colin, Jason and Ismael.
> >>
> >> * To change the number of partitions from X to Y in a topic, the
> >> controller
> >> marks all existing X partitions as read-only and creates Y new
> partitions.
> >> The new partitions are writable and are tagged with a higher repartition
> >> epoch (RE).
> >>
> >> * The controller propagates the new metadata to every broker. Once the
> >> leader of a partition is marked as read-only, it rejects the produce
> >> requests on this partition. The producer will then refresh the metadata
> >> and
> >> start publishing to the new writable partitions.
> >>
> >> * The consumers will then be consuming messages in RE order. The
> consumer
> >> coordinator will only assign partitions in the same RE to consumers.
> Only
> >> after all messages in an RE are consumed, will partitions in a higher RE
> >> be
> >> assigned to consumers.
> >>
> >> As Colin mentioned, if we do the above, we could potentially (1) use a
> >> globally unique partition id, or (2) use a globally unique topic id to
> >> distinguish recreated partitions due to topic deletion.
> >>
> >> So, perhaps we can sketch out the re-partitioning KIP a bit more and see
> >> if
> >> there is any overlap with KIP-232. Would you be interested in doing
> that?
> >> If not, we can do that next week.
> >>
> >> Jun
> >>
> >>
> >> On Tue, Feb 6, 2018 at 11:30 AM, Dong Lin  wrote:
> >>
> >> > Hey Jun,
> >> >
> >> > Interestingly I am also planning to sketch a KIP to allow partition
> >> > expansion for keyed topics after this KIP. Since you are already doing
> >> > that, I guess I will just share my high level idea here in case it is
> >> > helpful.
> >> >
> >> > The motivation for the KIP is that we currently lose order guarantee
> for
> >> > messages with the same key if we expand partitions of keyed topic.
> >> >
> >> > The solution can probably be built upon the following ideas:
> >> >
> >> > - Partition number of the keyed topic should always be doubled (or
> >> > multiplied by power of 2). Given that we select a partition based on
> >> > hash(key) % partitionNum, this should help us ensure that, a message
> >> > assigned to an existing partition will not be mapped to another
> existing
> >> > partition after partition expansion.
> >> >
> >> > - Producer includes in the ProduceRequest some information that helps
> >> > ensure that messages produced ti a partition will monotonically
> >> increase in
> >> > the partitionNum of the topic. In other words, if broker receives a
> >> > ProduceRequest and notices that the producer does not know the
> partition
> >> > number has increased, broker should reject this request. That
> >> "information"
> >> > maybe leaderEpoch, max partitionEpoch of the partitions of the topic,
> or
> >> > simply partitionNum of the topic. The benefit of this property is that
> >> we
> >> > can keep the new logic for in-order message consumption entirely in
> how
> >> > consumer leader determines the partition -> consumer mapping.
> >> >
> >> > - When consumer leader determines partition -> consumer mapping,
> leader
> >> > first reads the start position for each partition using
> >> OffsetFetchRequest.
> >> > If start position are all non-zero, then assignment can be done in its
> >> > current manner. The assumption is that, a message in the new partition
> >> > should only be consumed after all messages with the same key produced
> >> > before it has been consumed. Since some messages in the new partition
> >> has
> >> > been consumed, we should not worry about consuming messages
> >> out-of-order.
> >> > This benefit of this approach is that we can avoid unnecessary
> overhead
> >> in

Re: [VOTE] KIP-291: Have separate queues for control requests and data requests

[Jun Rao]

Hi, Lucas,

2. Good point about not knowing the request type in memory pool. Looking at
the implementation. It seems that queued.max.request.bytes is orthogonal to
queued.max.requests. So, this seems fine.

3. The implementation that you suggested sounds good. It would be useful
not to unnecessarily delay the processing of a request up to 300ms. I was
thinking that we could have RequestChannel manage a Lock and a couple of
Conditions and have sendRequest()/receiveRequest() coordinate on the lock
and the conditions (similar to how ArrayBlockingQueue is implemented). This
way, any new request can wake up the blocked request handling threads
immediately.

Thanks,

Jun


On Fri, Jun 29, 2018 at 4:53 PM, Lucas Wang  wrote:

> Hi Jun,
>
> Thanks for your comments.
> 1. I just replied in the discussion thread about the positive change this
> KIP can still bring
> if implemented on the latest trunk, which includes the async ZK operations
> for KAFKA-5642.
> The evaluation is done using an integration test.
> In production, we have not upgraded to Kafka 1.1 yet, and the code we are
> currently running does
> not include async ZK operations, therefore I don't have any real usage
> result.
>
> 2. Thanks for bringing this up. I haven't considered this setting, and the
> existing proposal in this KIP
> would make data requests and controller requests share a memory poll of
> size specified by the config
> queued.max.request.bytes. The downside is that if there is memory pressure,
> controller requests may be blocked
> from being read from a socket and does not get prioritized at the socket
> layer.
>
> If we have a separate bytes limit for the controller requests, I imagine
> there would be a separate memory pool
> dedicated to controller requests. Also it requires the processors to tell
> connections from a controller apart
> from connections from other brokers or clients, which would probably
> require a dedicated port for the controller?
> IMO, this change is mainly driven by the memory pressure, kind of an
> orthogonal issue, and we can address it with a separate KIP
> if desired. Please let me know what you think.
>
> 3. I plans to change the implementation of the method
> receiveRequest(timeout: Long) in the RequestChannel class as follows:
>
> val controllerRequest = controllerRequestQueue.poll()
> if (controllerRequest != null) {
>   controllerRequest
> } else {
>   dataRequestQueue.poll(timeout, TimeUnit.MILLISECONDS)
> }
>
> with this implementation, there is no need to explicitly choose a request
> handler thread to wake up depending on
> the types of request enqueued, and if a controller request arrives while
> some request handler threads are blocked on an empty data request queue,
> they will simply timeout and call the receiveRequest method again.
>
> In terms of performance, it means that in the worst case, for a controller
> request that just missed the receiveRequest call, it can be delayed for as
> long as
> the timeout parameter, which is hard coded to be 300 milliseconds. If there
> is just one request handler thread, the average delay is
> 150 milliseconds assuming the chance of a controller request arriving at
> any particular time is the same. With N request handler threads,
> the average delay is 150/N milliseconds, which does not seem to be a
> problem.
>
> We have considered waking up of request handler threads based on which
> queue the request handler threads are blocked,
> and that design was turned down because of its complexity. The design can
> be found at here
>  oller+request+queue+design>
> .
>
> If you mean a general purpose priority queue such as the
> java.util.PriorityQueue, we also have considered it and turned down the
> design because
> - The readily available class java.util.PriorityQueue is unbounded and
> we'll need to implement a bounded version
> - We would still like to have the FIFO semantics on both the controller
> request queue and data request queue, which conceptually does not fit very
> well
> with a general purpose priority queue, e.g. we would probably need to use
> the enqueue time to enforce FIFO semantics.
> - A typical operation on the priority queue is O(log n), whereas the sample
> implementation above gives O(1) performance regardless of the size of both
> queues.
>
> 4. For the two APIs sendRequest and receiveRequest, since we are only
> changing their implementation, not the API itself
> the two metrics will support two queues and the meaning of "Idle" still
> holds:
>
>
>
>
>
>
> *Before this KIPAfter this KIPNetworkProcessorAvgIdlePercentidle = blocked
> on selectnot idle includes being blocked on requestQueueidle = blocked on
> selectnot idle includes being blocked on either controller request queue or
> data request queueRequestHandlerAvgIdlePercentidle = blocked on reading
> from requestQueue idle = taking a request from the controller request
> queue, or blocked on reading from the 

Re: [DISCUSS] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

[Manikumar]

LGTM. Thanks for the KIP.

Thanks.

On Wed, Jul 11, 2018 at 3:31 PM Chia-Ping Tsai  wrote:

> hi Kafka,
>
> KIP-338 is trying to make us exclude internal topics easily when using the
> kafka-topics.sh. An new option will be added to kafka-topics.sh
> (TopicCommand) and user can use the option to exclude all internal topics
> when running the list or describe command.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-338+Support+to+exclude+the+internal+topics+in+kafka-topics.sh+command
>
> feedback is welcome
>
> Cheers,
> chia-ping
>

[DISCUSS] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

[Chia-Ping Tsai]

hi Kafka,

KIP-338 is trying to make us exclude internal topics easily when using the 
kafka-topics.sh. An new option will be added to kafka-topics.sh (TopicCommand) 
and user can use the option to exclude all internal topics when running the 
list or describe command.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-338+Support+to+exclude+the+internal+topics+in+kafka-topics.sh+command

feedback is welcome 

Cheers,
chia-ping

Re: Kafka Streams new release

[Ofir Manor]

>From the release plan:
  " While the target release date is fixed at ~2w after code freeze, RCs
will roll out as needed until the release vote passes"
   https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
You can follow the voting threads on this mailing list - the current thread
is "[VOTE] 2.0.0 RC2"
Once a vote for RC passes, that RC will be released as the 2.0.0 version.

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Wed, Jul 11, 2018 at 9:21 AM, Ayushi Sharma  wrote:

> When is the next Kafka Streams release (Kafka Streams 2.0.0) which was
> tentatively on 26June?
>

Re: Requesting Permission To Create KIP And Assign JIRAs

[Kevin Lu]

Hi,

I received access to assign JIRAs now, but I still cannot create KIPs.

When I hover over the "Create KIP" button, it says I do not have permission
to create content. When I click on the button, it goes a 404 Not Found page.

Can someone add me?

Thanks!

Regards,
Kevin

On Tue, Jun 26, 2018 at 10:37 PM Kevin Lu  wrote:

> Hi All,
>
> I would like to start contributing to Kafka but I do not have access to
> create KIPs or assign JIRA to myself.
>
> Can someone set it up for me?
>
> Confluence id: lu.kevin
> Jira username: lu.kevin
>
> Email: lu.ke...@berkeley.edu
>
> Thanks!
>
> Regards,
> Kevin
>


-- 
Kevin Li Lu
University of California, Berkeley | Class of 2017
College of Letters & Sciences | B.A. Computer Science
Cell: (408) 609-6238

Kafka Streams new release

[Ayushi Sharma]

When is the next Kafka Streams release (Kafka Streams 2.0.0) which was
tentatively on 26June?

[jira] [Reopened] (KAFKA-7141) kafka-consumer-group doesn't describe existing group

[huxihx (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huxihx reopened KAFKA-7141:
---
  Assignee: huxihx

> kafka-consumer-group doesn't describe existing group
> 
>
> Key: KAFKA-7141
> URL: https://issues.apache.org/jira/browse/KAFKA-7141
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.11.0.0, 1.0.1
>Reporter: Bohdana Panchenko
>Assignee: huxihx
>Priority: Major
>
> I am running two consumers: akka-stream-kafka consumer with standard config 
> section as described in the 
> [https://doc.akka.io/docs/akka-stream-kafka/current/consumer.html] and  
> kafka-console-consumer.
> akka-stream-kafka consumer configuration looks like this
> {color:#33}_akka.kafka.consumer{_{color}
> {color:#33}  _kafka-clients{_{color}
> {color:#33}    _group.id = "myakkastreamkafka-1"_{color}
> {color:#33}   _enable.auto.commit = false_{color}
> }
> {color:#33} }{color}
>  
>  I am able to see the both groups with the command
>  
>  *kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --list*
>  _Note: This will not show information about old Zookeeper-based consumers._
>  
>  _myakkastreamkafka-1_
>  _console-consumer-57171_
> {color:#33}I am able to view details about the console consumer 
> group{color}
> *kafka-consumer-groups --describe --bootstrap-server 127.0.0.1:9092 --group 
> console-consumer-57171*
>  _{color:#205081}Note: This will not show information about old 
> Zookeeper-based consumers.{color}_
> _{color:#205081}TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID 
> HOST CLIENT-ID{color}_
>  _{color:#205081}STREAM-TEST 0 0 0 0 
> consumer-1-6b928e07-196a-4322-9928-068681617878 /172.19.0.4 consumer-1{color}_
> {color:#33}But the command to describe my akka stream consumer gives me 
> empty output:{color}
> *kafka-consumer-groups --describe --bootstrap-server 127.0.0.1:9092 --group 
> myakkastreamkafka-1*
>  {color:#205081}_Note: This will not show information about old 
> Zookeeper-based consumers._{color}
> {color:#205081}_TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID 
> HOST CLIENT-ID_{color}
>  
> {color:#33}That is strange. Can you please check the issue?{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)