Build failed in Jenkins: kafka-trunk-jdk8 #4728

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10189: reset event queue time histogram when queue is empty


--
[...truncated 3.17 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 

Build failed in Jenkins: kafka-trunk-jdk14 #304

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10189: reset event queue time histogram when queue is empty


--
[...truncated 3.19 MB...]

org.apache.kafka.streams.scala.kstream.StreamJoinedTest > Create a StreamJoined 
should create a StreamJoined with Serdes and Store Suppliers STARTED

org.apache.kafka.streams.scala.kstream.StreamJoinedTest > Create a StreamJoined 
should create a StreamJoined with Serdes and Store Suppliers PASSED

org.apache.kafka.streams.scala.kstream.StreamJoinedTest > Create a StreamJoined 
should create a StreamJoined with Serdes and a State Store name STARTED

org.apache.kafka.streams.scala.kstream.StreamJoinedTest > Create a StreamJoined 
should create a StreamJoined with Serdes and a State Store name PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsJava PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaJoin STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaJoin PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaCogroup STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaCogroup PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaSimple STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaSimple PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaCogroupSimple STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaCogroupSimple PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaAggregate STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaAggregate PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaProperties STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaProperties PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaTransform STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaTransform PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > session windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > session windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilWindowCloses PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > non-windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > non-windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialized 
should create a Materialized with Serdes STARTED

Jenkins build is back to normal : kafka-trunk-jdk11 #1654

[Apache Jenkins Server]

See 

[GitHub] [kafka-site] vvcephei merged pull request #275: MINOR: update Streams docs for 2.6 and fix configs

[GitBox]

vvcephei merged pull request #275:
URL: https://github.com/apache/kafka-site/pull/275


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] 2.6.0 RC0

[Randall Hauch]

Thanks, Rajini. I've cut RC1, but after I pushed the commit I found a
discrepancy in the JavaDoc that mentions 2.7. Please see the "[VOTE] 2.6.0
RC1" discussion thread for details.

On Mon, Jul 20, 2020 at 4:59 AM Rajini Sivaram 
wrote:

> Thanks Randall. We didn't create a JIRA for the KIP-546 security fix since
> KAFKA-7740 that introduced the regression had a fix version of 2.5.0. A fix
> was applied without a JIRA to follow the security bug fix process. In the
> end, it turned out that KAFKA-7740 was only in 2.6, so the bug hadn't got
> into a release anyway.
>
> Regards,
>
> Rajini
>
>
> On Sun, Jul 19, 2020 at 2:46 PM Randall Hauch  wrote:
>
> > Thanks, Rajini. Is there a Jira issue for the fix related to KIP-546? If
> > so, please make sure the Fix Version(s) include `2.6.0`.
> >
> > I'm going to start RC1 later today and hope to get it published by
> Monday.
> > In the meantime, if anyone finds anything else in RC0, please raise it
> here
> > -- if it's after RC1 is published then we'll just cut another RC with any
> > fixes.
> >
> > We're down to just 5 system test failures [1], and folks are actively
> > working to address them. At least some are known to be flaky, but we
> still
> > want to get them fixed.
> >
> > Best regards,
> >
> > Randall
> >
> > On Sun, Jul 19, 2020 at 5:45 AM Rajini Sivaram 
> > wrote:
> >
> > > Hi Randall,
> > >
> > > Ron found an issue with the quota implementation added under KIP-546,
> > which
> > > is a blocking issue for 2.6.0 since it leaks SCRAM credentials in quota
> > > responses. A fix has been merged into 2.6 branch in the commit
> > >
> > >
> >
> https://github.com/apache/kafka/commit/dd71437de7675d92ad3e4ed01ac3ee11bf5da99d
> > > .
> > > We
> > > have also merged the fix for
> > > https://issues.apache.org/jira/browse/KAFKA-10223 into 2.6 branch
> since
> > it
> > > causes issues for non-Java clients during reassignments.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Wed, Jul 15, 2020 at 11:41 PM Randall Hauch 
> wrote:
> > >
> > > > Thanks, Levani.
> > > >
> > > > The content of
> > > >
> > > >
> > >
> >
> https://home.apache.org/~rhauch/kafka-2.6.0-rc0/kafka_2.12-2.6.0-site-docs.tgz
> > > > is the correct generated site. Somehow I messed coping that to the
> > > > https://github.com/apache/kafka-site/tree/asf-site/26 directory.
> I've
> > > > corrected the latter so that
> > https://kafka.apache.org/26/documentation/
> > > > now
> > > > exactly matches that documentation in RC0.
> > > >
> > > > Best regards,
> > > >
> > > > Randall
> > > >
> > > > On Wed, Jul 15, 2020 at 1:25 AM Levani Kokhreidze <
> > > levani.co...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Randall,
> > > > >
> > > > > Not sure if it’s intentional but, documentation for Kafka Streams
> > 2.6.0
> > > > > also contains “Streams API changes in 2.7.0”
> > > > > https://kafka.apache.org/26/documentation/streams/upgrade-guide <
> > > > > https://kafka.apache.org/26/documentation/streams/upgrade-guide>
> > > > >
> > > > > Also, there seems to be some formatting issue in 2.6.0 section.
> > > > >
> > > > > Levani
> > > > >
> > > > >
> > > > > > On Jul 15, 2020, at 1:48 AM, Randall Hauch 
> > wrote:
> > > > > >
> > > > > > Thanks for catching that, Gary. Apologies to all for announcing
> > this
> > > > > before
> > > > > > pushing the docs, but that's fixed and the following links are
> > > working
> > > > > > (along with the others in my email):
> > > > > >
> > > > > > * https://kafka.apache.org/26/documentation.html
> > > > > > * https://kafka.apache.org/26/protocol.html
> > > > > >
> > > > > > Randall
> > > > > >
> > > > > > On Tue, Jul 14, 2020 at 4:30 PM Gary Russell <
> gruss...@vmware.com>
> > > > > wrote:
> > > > > >
> > > > > >> Docs link [1] is broken.
> > > > > >>
> > > > > >> [1] https://kafka.apache.org/26/documentation.html
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

[VOTE] 2.6.0 RC1

[Randall Hauch]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 2.6.0. This is a
major release that includes many new features, including:

* TLSv1.3 has been enabled by default for Java 11 or newer.
* Smooth scaling out of Kafka Streams applications
* Kafka Streams support for emit on change
* New metrics for better operational insight
* Kafka Connect can automatically create topics for source connectors
* Improved error reporting options for sink connectors in Kafka Connect
* New Filter and conditional SMTs in Kafka Connect
* The default value for the `client.dns.lookup` configuration is
now `use_all_dns_ips`
* Upgrade Zookeeper to 3.5.8

This release also includes a few other features, 76 improvements, and 165
bug fixes.

Release notes for the 2.6.0 release:
https://home.apache.org/~rhauch/kafka-2.6.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Monday, July 20, 9am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~rhauch/kafka-2.6.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~rhauch/kafka-2.6.0-rc1/javadoc/

* Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
https://github.com/apache/kafka/releases/tag/2.6.0-rc1

* Documentation:
https://kafka.apache.org/26/documentation.html

* Protocol:
https://kafka.apache.org/26/protocol.html

* Successful Jenkins builds for the 2.6 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-2.6-jdk8/91/ (one
flaky test)
System tests: (link to follow)

Thanks,
Randall Hauch

Re: [VOTE] 2.6.0 RC0

[Randall Hauch]

All, I've pushed a new release candidate (RC1) and announced via the
"[VOTE] 2.6.0 RC1" discussion thread.

Randall

On Mon, Jul 20, 2020 at 9:51 PM Randall Hauch  wrote:

> Thanks, Rajini. I've cut RC1, but after I pushed the commit I found a
> discrepancy in the JavaDoc that mentions 2.7. Please see the "[VOTE] 2.6.0
> RC1" discussion thread for details.
>
> On Mon, Jul 20, 2020 at 4:59 AM Rajini Sivaram 
> wrote:
>
>> Thanks Randall. We didn't create a JIRA for the KIP-546 security fix since
>> KAFKA-7740 that introduced the regression had a fix version of 2.5.0. A
>> fix
>> was applied without a JIRA to follow the security bug fix process. In the
>> end, it turned out that KAFKA-7740 was only in 2.6, so the bug hadn't got
>> into a release anyway.
>>
>> Regards,
>>
>> Rajini
>>
>>
>> On Sun, Jul 19, 2020 at 2:46 PM Randall Hauch  wrote:
>>
>> > Thanks, Rajini. Is there a Jira issue for the fix related to KIP-546? If
>> > so, please make sure the Fix Version(s) include `2.6.0`.
>> >
>> > I'm going to start RC1 later today and hope to get it published by
>> Monday.
>> > In the meantime, if anyone finds anything else in RC0, please raise it
>> here
>> > -- if it's after RC1 is published then we'll just cut another RC with
>> any
>> > fixes.
>> >
>> > We're down to just 5 system test failures [1], and folks are actively
>> > working to address them. At least some are known to be flaky, but we
>> still
>> > want to get them fixed.
>> >
>> > Best regards,
>> >
>> > Randall
>> >
>> > On Sun, Jul 19, 2020 at 5:45 AM Rajini Sivaram > >
>> > wrote:
>> >
>> > > Hi Randall,
>> > >
>> > > Ron found an issue with the quota implementation added under KIP-546,
>> > which
>> > > is a blocking issue for 2.6.0 since it leaks SCRAM credentials in
>> quota
>> > > responses. A fix has been merged into 2.6 branch in the commit
>> > >
>> > >
>> >
>> https://github.com/apache/kafka/commit/dd71437de7675d92ad3e4ed01ac3ee11bf5da99d
>> > > .
>> > > We
>> > > have also merged the fix for
>> > > https://issues.apache.org/jira/browse/KAFKA-10223 into 2.6 branch
>> since
>> > it
>> > > causes issues for non-Java clients during reassignments.
>> > >
>> > > Regards,
>> > >
>> > > Rajini
>> > >
>> > >
>> > > On Wed, Jul 15, 2020 at 11:41 PM Randall Hauch 
>> wrote:
>> > >
>> > > > Thanks, Levani.
>> > > >
>> > > > The content of
>> > > >
>> > > >
>> > >
>> >
>> https://home.apache.org/~rhauch/kafka-2.6.0-rc0/kafka_2.12-2.6.0-site-docs.tgz
>> > > > is the correct generated site. Somehow I messed coping that to the
>> > > > https://github.com/apache/kafka-site/tree/asf-site/26 directory.
>> I've
>> > > > corrected the latter so that
>> > https://kafka.apache.org/26/documentation/
>> > > > now
>> > > > exactly matches that documentation in RC0.
>> > > >
>> > > > Best regards,
>> > > >
>> > > > Randall
>> > > >
>> > > > On Wed, Jul 15, 2020 at 1:25 AM Levani Kokhreidze <
>> > > levani.co...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Randall,
>> > > > >
>> > > > > Not sure if it’s intentional but, documentation for Kafka Streams
>> > 2.6.0
>> > > > > also contains “Streams API changes in 2.7.0”
>> > > > > https://kafka.apache.org/26/documentation/streams/upgrade-guide <
>> > > > > https://kafka.apache.org/26/documentation/streams/upgrade-guide>
>> > > > >
>> > > > > Also, there seems to be some formatting issue in 2.6.0 section.
>> > > > >
>> > > > > Levani
>> > > > >
>> > > > >
>> > > > > > On Jul 15, 2020, at 1:48 AM, Randall Hauch 
>> > wrote:
>> > > > > >
>> > > > > > Thanks for catching that, Gary. Apologies to all for announcing
>> > this
>> > > > > before
>> > > > > > pushing the docs, but that's fixed and the following links are
>> > > working
>> > > > > > (along with the others in my email):
>> > > > > >
>> > > > > > * https://kafka.apache.org/26/documentation.html
>> > > > > > * https://kafka.apache.org/26/protocol.html
>> > > > > >
>> > > > > > Randall
>> > > > > >
>> > > > > > On Tue, Jul 14, 2020 at 4:30 PM Gary Russell <
>> gruss...@vmware.com>
>> > > > > wrote:
>> > > > > >
>> > > > > >> Docs link [1] is broken.
>> > > > > >>
>> > > > > >> [1] https://kafka.apache.org/26/documentation.html
>> > > > > >>
>> > > > > >>
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [VOTE] 2.6.0 RC1

[Randall Hauch]

When I was checking the documentation for RC1 after the tag was pushed, I
noticed that the fix Rajini mentioned in the RC0 vote thread (
https://github.com/apache/kafka/pull/8979
)
and merged to the `2.6` branch includes the following comment about being
deprecated in 2.7:
https://github.com/apache/kafka/pull/8979/files#diff-369f0debebfcda6709beeaf11612b34bR20-R21
.

Rajini, can you please check the commits merged to the `2.6` do not have
the reference to 2.7? Since these are JavaDocs, I'm assuming that we'll
need to cut RC2.

But it'd be good for everyone else to double check this release.

Best regards,

Randall Hauch

On Mon, Jul 20, 2020 at 9:50 PM Randall Hauch  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 2.6.0. This is a
> major release that includes many new features, including:
>
> * TLSv1.3 has been enabled by default for Java 11 or newer.
> * Smooth scaling out of Kafka Streams applications
> * Kafka Streams support for emit on change
> * New metrics for better operational insight
> * Kafka Connect can automatically create topics for source connectors
> * Improved error reporting options for sink connectors in Kafka Connect
> * New Filter and conditional SMTs in Kafka Connect
> * The default value for the `client.dns.lookup` configuration is
> now `use_all_dns_ips`
> * Upgrade Zookeeper to 3.5.8
>
> This release also includes a few other features, 76 improvements, and 165
> bug fixes.
>
> Release notes for the 2.6.0 release:
> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, July 20, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/javadoc/
>
> * Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
> https://github.com/apache/kafka/releases/tag/2.6.0-rc1
>
> * Documentation:
> https://kafka.apache.org/26/documentation.html
>
> * Protocol:
> https://kafka.apache.org/26/protocol.html
>
> * Successful Jenkins builds for the 2.6 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-2.6-jdk8/91/ (one
> flaky test)
> System tests: (link to follow)
>
> Thanks,
> Randall Hauch
>

Jenkins build is back to normal : kafka-trunk-jdk8 #4727

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Jun Rao]

Hi, Satish, Ying, Harsha,

Thanks for the updated KIP. A few more comments below.

1000. Regarding Colin's question on querying the metadata directly in the
remote block store. One issue is that not all block stores offer the needed
api to query the metadata. For example, S3 only offers an api to list
objects under a prefix and this api has the eventual consistency semantic.

1001. The KIP described a few scenarios of unclean leader elections. This
is very useful, but I am wondering if this is the best approach. My
understanding of the proposed approach is to allow the new (unclean) leader
to take new messages immediately. While this increases availability, it
creates the problem that there could be multiple conflicting segments in
the remote store for the same offset range. This seems to make it harder
for RLMM to determine which archived log segments contain the correct data.
For example, an archived log segment could at one time be the correct data,
but be changed to incorrect data after an unclean leader election. An
alternative approach is to let the unclean leader use the archived data as
the source of truth. So, when the new (unclean) leader takes over, it first
reconciles the local data based on the archived data before taking new
messages. This makes the job of RLMM a bit easier since all archived data
are considered correct. This increases availability a bit. However, since
unclean leader elections are rare, this may be ok.

1002. RemoteStorageManager.
1002.1 There seems to be some inconsistencies in RemoteStorageManager. We
pass in RemoteLogSegmentId copyLogSegment(). For all other methods, we pass
in RemoteLogSegmentMetadata.
1002.2 Is endOffset in RemoteLogSegmentMetadata inclusive or exclusive?
1002.3 It seems that we need an api to get the leaderEpoch history for a
partition.
1002.4 Could you define the type of RemoteLogSegmentContext?

1003 RemoteLogMetadataManager
1003.1 I am not sure why we need both of the following methods
in RemoteLogMetadataManager. Could we combine them into one that takes in
offset and returns RemoteLogSegmentMetadata?
RemoteLogSegmentId getRemoteLogSegmentId(TopicPartition topicPartition,
long offset) throws IOException;
RemoteLogSegmentMetadata getRemoteLogSegmentMetadata(RemoteLogSegmentId
remoteLogSegmentId) throws IOException;
1003.2 There seems to be some inconsistencies in the methods below. I am
not sure why one takes RemoteLogSegmentMetadata and the other
takes RemoteLogSegmentId.
void putRemoteLogSegmentData(RemoteLogSegmentMetadata
remoteLogSegmentMetadata) throws IOException;
void deleteRemoteLogSegmentMetadata(RemoteLogSegmentId
remoteLogSegmentId) throws IOException;
1003.3 In void onServerStarted(final String serverEndpoint), what
is serverEndpoint used for?

1004. It would be useful to document how all the new APIs are being used.
For example, when is RemoteLogSegmentMetadata.markedForDeletion being set
and used? How are
RemoteLogMetadataManager.earliestLogOffset/highestLogOffset being used?

1005. Handling partition deletion: The KIP says "RLMM will eventually
delete these segments by using RemoteStorageManager." Which replica does
this logic?

1006. "If there are any failures in removing remote log segments then those
are stored in a specific topic (default as __remote_segments_to_be_deleted)
and user can consume the events(which contain remote-log-segment-id) from
that topic and clean them up from remote storage.  " Not sure if it's worth
the complexity of adding another topic. Could we just retry?

1007. RemoteFetchPurgatory: Could we just reuse the existing fetchPurgatory?

1008. Configurations:
1008.1 remote.log.retention.ms, remote.log.retention.minutes,
remote.log.retention.hours: It seems that we just need the ms one. Also,
are we changing the meaning of existing config log.retention.ms to mean the
local retention? For backward compatibility, it's better to not change the
meaning of existing configurations.
1008.2 Should remote.log.storage.enable be at the topic level?

1009. It would be useful to list all limitations in a separate section:
compacted topic, JBOD, etc. Also, is changing a topic from delete to
compact and vice versa allowed when tiering is enabled?

1010. Thanks for performance numbers. Are those with RocksDB as the cache?

Thanks,

Jun

On Wed, Jul 15, 2020 at 6:12 PM Harsha Ch  wrote:

> Hi Colin,
>Thats not what we said in the previous email. RLMM is
> pluggable storage and by running numbers even 1PB data you do not need more
> than 10GB local storage.
> If in future this becomes a blocker for any users we can revisit but this
> does not warrant another implementation at this point to push the data to
> remote storage.
> We can ofcourse implement another RLMM that is optional for users to
> configure to push to remote. But that doesn't need to be addressed in this
> KIP.
>
> Thanks,
> Harsha
>
> On Wed, Jul 15, 2020 at 5:50 PM Colin McCabe  wrote:
>
> > Hi Ying,
> >
> > Thanks for the 

Re: [DISCUSS] KIP-450: Sliding Windows

[Guozhang Wang]

Hi Leah,

Thanks for the updated KIP. I agree that extending SlidingWindows from
Windows is fine for the sake of not introducing more public APIs (and their
internal xxxImpl classes), and its cons is small enough to tolerate to me.


Guozhang


On Mon, Jul 20, 2020 at 1:49 PM Leah Thomas  wrote:

> Hi all,
>
> Thanks for the feedback on the KIP. I've updated the KIP page
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-450%3A+Sliding+Window+Aggregations+in+the+DSL
> >
> to address these points and have created a child page
> <
> https://cwiki.apache.org/confluence/display/KAFKA/Aggregation+details+for+KIP-450
> >
> to go more in depth on certain implementation details.
>
> *Grace Period:*
> I think Sophie raises a good point that the default grace period of 24
> hours is often too long and was chosen when retention time and grace period
> were the same. For SlidingWindows, I propose we make the grace period
> mandatory. To keep formatting consistent with other types of windows, grace
> period won't be an additional parameter in the #of method, but will still
> look like it does in other use cases:
> .windowedBy(SlidingWindows.of(twentySeconds).grace(fiftySeconds). If grace
> period isn't properly initialized, an error will be thrown through the
> process method.
>
> *Storage Layer + Aggregation:*
> SlidingWindows will use a WindowStore because computation can be done with
> the information stored in a WindowStore (window timestamp and value). Using
> the WindowStore also simplifies computation as SlidingWindows can leverage
> existing processes. Because we are using a WindowStore, the aggregation
> process will be similar to that of a hopping window. As records come in
> their value is added to the aggregation that already exists, following the
> same procedure as hopping windows. The aggregation difference between
> SlidingWindows and HoppingWindows comes in creating new windows for a
> SlidingWindow, where you need to find the existing records that belong to
> the new window. This computation is similar to the aggregation in
> SessionWindows and requires a scan to the WindowStore to find the window
> with the aggregation needed, which will always be pre-computed. The scan
> requires creating an iterator, but should have minimal performance effects
> as this strategy is already implemented in SessionWindows. More details on
> finding the aggregation that needs to be put in a new window can be found
> on the implementation page
> <
> https://cwiki.apache.org/confluence/display/KAFKA/Aggregation+details+for+KIP-450
> >
> .
>
> *Extending Windows, Windows or nothing*
> Because SlidingWindows are still defined by a windowSize (whereas
> SessionWindows are defined purely by data), I think it makes sense to
> leverage the existing Window processes instead of creating a new store type
> that would be very similar to the WindowStore. While it's true that the
> #windowsFor method isn't necessary for SlidingWindows, JoinWindows also
> extends Windows and throws an UnsupportedOperationException in the
> #windowsFor method, which is what SlidingWindows can do. The difference
> between extending Windows or Windows is minimal, as
> both are ways to pass window parameters. Extending Windows will
> give us more leverage in utilizing existing processes.
>
> *Emit Strategy*
> I would argue that emitting for every update is still the best way to go
> for SlidingWindows because it mimics the other types of windows, and
> suppression can be leveraged to limit what SlidingWindows emits. While some
> users may only want to see the last value, others may want to see more, and
> leaving the emit strategy to emit partial results allows both users to
> access what they want.
>
> *Additional Features*
> Supporting sliding windows inherently, and shifting inefficient hopping
> windows to sliding windows, is an interesting idea and could be built on
> top of SlidingWindows when they are finished, but right now seems out of
> scope for the needs of this KIP. Similarly, including a `subtraction`
> feature could have performance improvements, but doesn't seem necessary for
> the implementation of this KIP.
>
> Let me know what you think of the updates,
>
> Leah
>
> On Thu, Jul 16, 2020 at 11:57 AM John Roesler  wrote:
>
> > Hello all,
> >
> > Thanks for the KIP, Leah!
> >
> > Regarding (1): I'd go farther actually. Making Windows an abstract
> > class was a mistake from the beginning that led to us not being
> > able to fix a very confusing situation for users around retention times,
> > final results emitting, etc. Thus, I would not suggest extending
> > TimeWindows for sure, but would also not suggest extending Windows.
> >
> > The very simplest thing to do is follow the example of SessionWindows,
> > which is just a completely self-contained class. If we don't mess with
> > class inheritance, we won't ever have any of the problems related to
> > class inheritance. This is my preferred solution.
> >
> > Still, Sliding 

[GitHub] [kafka-site] ableegoldman commented on a change in pull request #275: MINOR: update Streams docs for 2.6 and fix configs

[GitBox]

ableegoldman commented on a change in pull request #275:
URL: https://github.com/apache/kafka-site/pull/275#discussion_r457761691



##
File path: 26/streams/upgrade-guide.html
##
@@ -95,7 +95,19 @@ Streams API
 Note that you need brokers with version 2.5 or newer to use this 
feature.
 
 
-As of 2.6.0 Kafka Streams deprecates KStream.through() if 
favor of the new KStream.repartition() operator
+For more highly available stateful applications, we've modified the 
task assignment algorithm to delay the movement of stateful active tasks to 
instances
+that aren't yet caught up with that task's state. Instead, to migrate 
a task from one instance to another (eg when scaling out),
+Streams will assign a warmup replica to the target instance so it can 
begin restoring the state while the active task stays available on an instance
+that already had the task. The instances warming up tasks will 
communicate their progress to the group so that, once ready, Streams can move 
active
+tasks to their new owners in the background. Check out https://cwiki.apache.org/confluence/x/0i4lBg;>KIP-441
+for full details, including several new configs for control over this 
new feature.
+
+
+New end-to-end latency metrics have been added. These task-level 
metrics will be logged at the INFO level and report the min and max end-to-end 
latency of a record at the beginning/source node(s)
+and end/terminal node(s) of a task. See https://cwiki.apache.org/confluence/x/gBkRCQ;>KIP-613 for more 
information.
+
+
+As of 2.6.0 Kafka Streams deprecates KStream.through() if 
favor of the new KStream.repartition() operator

Review comment:
   I didn't notice this in time to fix in AK, but we're missing the closing 
tag here so the code formatting is running loose. I'll add this on to an AK PR 
or something so it gets fixed for real





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Build failed in Jenkins: kafka-trunk-jdk14 #303

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9161: add docs for KIP-441 and KIP-613 and other configs that 
need


--
[...truncated 3.20 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue 

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

[Guozhang Wang]

Hello Bruno,

Thanks for the updated KIP. I made a pass and here are some comments:

1) What's the motivation of keeping it as INFO while KIP-471 metrics are
defined in DEBUG?

2) Some namings are a bit inconsistent with others and with KIP-471, for
example:

2.a) KIP-471 uses "memtable" while in this KIP we use "mem-table", also the
"memtable" is prefixed and then the metric name. I'd suggest we keep them
consistent. e.g. "num-immutable-mem-table" => "immutable-memtable-count",
"cur-size-active-mem-table" => "active-memable-bytes"

2.b) "immutable" are abbreviated as "imm" in some names but not in others,
I'd suggest we do not use abbreviations across all names,
e.g. "num-entries-imm-mem-tables" => "immutable-memtable-num-entries".

2.c) "-size" "-num" semantics is usually a bit unclear, and I'd suggest we
just more concrete terms, e.g. "total-sst-files-size" =>
"total-sst-files-bytes", "num-live-versions" => "live-versions-count",
"background-errors" => "background-errors-count".

3) Some metrics are a bit confusing, e.g.

3.a) What's the difference between "cur-size-all-mem-tables" and
"size-all-mem-tables"?

3.b) And the explanation of "estimate-table-readers-mem" does not read very
clear to me either, does it refer to "estimate-sst-file-read-buffer-bytes"?

3.c) How does "estimate-oldest-key-time" help with memory usage debugging?

4) For my own education, does "estimate-pending-compaction-bytes" capture
all the memory usage for compaction buffers?

5) This is just of a nit comment to help readers better understand rocksDB:
maybe we can explain in the wiki doc which part of rocksDB uses memory
(block cache, OS cache, memtable, compaction buffer, read buffer), and
which of them are on-heap and wich of them are off-heap, which can be hard
bounded and which can only be soft bounded and which cannot be bounded at
all, etc.


Guozhang


On Mon, Jul 20, 2020 at 11:00 AM Bruno Cadonna  wrote:

> Hi,
>
> During the implementation of this KIP and after some discussion about
> RocksDB metrics, I decided to make some major modifications to this KIP
> and kick off discussion again.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Report+Properties+of+RocksDB
>
> Best,
> Bruno
>
> On 15.05.20 17:11, Bill Bejeck wrote:
> > Thanks for the KIP, Bruno. Having sensible, easy to access RocksDB memory
> > reporting will be a welcomed addition.
> >
> > FWIW I also agree to have the metrics reported on a store level. I'm glad
> > you changed the KIP to that effect.
> >
> > -Bill
> >
> >
> >
> > On Wed, May 13, 2020 at 6:24 PM Guozhang Wang 
> wrote:
> >
> >> Hi Bruno,
> >>
> >> Sounds good to me.
> >>
> >> I think I'm just a bit more curious to see its impact on performance: as
> >> long as we have one INFO level rocksDB metrics, then we'd have to turn
> on
> >> the scheduled rocksdb metrics recorder whereas previously, we can
> decide to
> >> not turn on the recorder at all if all are set as DEBUG and we
> configure at
> >> INFO level in production. But this is an implementation detail anyways
> and
> >> maybe the impact is negligible after all. We can check and re-discuss
> this
> >> afterwards :)
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Wed, May 13, 2020 at 9:34 AM Sophie Blee-Goldman <
> sop...@confluent.io>
> >> wrote:
> >>
> >>> Thanks Bruno! I took a look at the revised KIP and it looks good to me.
> >>>
> >>> Sophie
> >>>
> >>> On Wed, May 13, 2020 at 6:59 AM Bruno Cadonna 
> >> wrote:
> >>>
>  Hi John,
> 
>  Thank you for the feedback!
> 
>  I agree and I will change the KIP as I stated in my previous e-mail to
>  Guozhang.
> 
>  Best,
>  Bruno
> 
>  On Tue, May 12, 2020 at 3:07 AM John Roesler 
> >>> wrote:
> >
> > Thanks, all.
> >
> > If you don’t mind, I’ll pitch in a few cents’ worth.
> >
> > In my life I’ve generally found more granular metrics to be more
> >>> useful,
>  as long as there’s a sane way to roll them up. It does seem nice to
> see
> >>> it
>  on the per-store level. For roll-up purposes, the task and thread tags
>  should be sufficient.
> >
> > I think the only reason we make some metrics Debug is that
> >> _recording_
>  them can be expensive. If there’s no added expense, I think we can
> just
>  register store-level metrics at Info level.
> >
> > Thanks for the KIP, Bruno!
> > -John
> >
> > On Mon, May 11, 2020, at 17:32, Guozhang Wang wrote:
> >> Hello Sophie / Bruno,
> >>
> >> I've also thought about the leveling question, and one motivation I
>  had for
> >> setting it in instance-level is that we want to expose it in INFO
>  level:
> >> today our report leveling is not very finer grained --- which I
> >> think
>  is
> >> sth. worth itself --- such that one have to either turn on all
> >> DEBUG
> >> metrics recording or none of them. If we can allow users to e.g.
>  specify
> >> 

Re: [VOTE] KIP-572: Improve timeouts and retires in Kafka Streams

[Matthias J. Sax]

While working on the PR, we realized that the command line tool

  bin/kafka-console-producer.sh

has a flag `--message-send-max-retries` to set the producer's `retries`
config. We also need to deprecate this flag.

I updated the KIP accordingly. Please let us know if there are any
concerns to this minor change to the KIP.

Thanks.


-Matthias

On 6/10/20 11:16 AM, Matthias J. Sax wrote:
> Thanks!
> 
> +1 (binding) from myself.
> 
> 
> I am closing the vote as accepted with 3 binding and 3 non-binding votes.
> 
> binding:
>  - John
>  - Guozhang
>  - Matthias
> 
> non-binding:
>  - Sophie
>  - Boyang
>  - Bruno
> 
> 
> 
> -Matthias
> 
> On 6/4/20 5:26 PM, Matthias J. Sax wrote:
>> Guozhang,
>>
>> what you propose makes sense, but this seems to get into implementation
>> detail territory already?
>>
>> Thus, if there are nor further change requests to the KIP wiki page
>> itself, I would like to proceed with the VOTE.
>>
>>
>> -Matthias
>>
>> On 5/20/20 12:30 PM, Guozhang Wang wrote:
>>> Thanks Matthias,
>>>
>>> I agree with you on all the bullet points above. Regarding the admin-client
>>> outer-loop retries inside partition assignor, I think we should treat error
>>> codes differently from those two blocking calls:
>>>
>>> Describe-topic:
>>> * unknown-topic (3): add this topic to the to-be-created topic list.
>>> * leader-not-available (5): do not try to create, retry in the outer loop.
>>> * request-timeout: break the current loop and retry in the outer loop.
>>> * others: fatal error.
>>>
>>> Create-topic:
>>> * topic-already-exists: retry in the outer loop to validate the
>>> num.partitions match expectation.
>>> * request-timeout: break the current loop and retry in the outer loop.
>>> * others: fatal error.
>>>
>>> And in the outer-loop, I think we can have a global timer for the whole
>>> "assign()" function, not only for the internal-topic-manager, and the timer
>>> can be hard-coded with, e.g. half of the rebalance.timeout to get rid of
>>> the `retries`; if we cannot complete the assignment before the timeout runs
>>> out, we can return just the partial assignment (e.g. if there are two
>>> tasks, but we can only get the topic metadata for one of them, then just do
>>> the assignment for that one only) while encoding in the error-code field to
>>> request for another rebalance.
>>>
>>> Guozhang
>>>
>>>
>>>
>>> On Mon, May 18, 2020 at 7:26 PM Matthias J. Sax  wrote:
>>>
 No worries Guozhang, any feedback is always very welcome! My reply is
 going to be a little longer... Sorry.



> 1) There are some inconsistent statements in the proposal regarding what
 to
> deprecated:

 The proposal of the KIP is to deprecate `retries` for producer, admin,
 and Streams. Maybe the confusion is about the dependency of those
 settings within Streams and that we handle the deprecation somewhat
 different for plain clients vs Streams:

 For plain producer/admin the default `retries` is set to MAX_VALUE. The
 config will be deprecated but still be respected.

 For Streams, the default `retries` is set to zero, however, this default
 retry does _not_ affect the embedded producer/admin clients -- both
 clients stay on their own default of MAX_VALUES.

 Currently, this introduces the issue, that if a user wants to increase
 Streams retries, she might by accident reduce the embedded client
 retries, too. To avoid this issue, she would need to set

 retries=100
 producer.retires=MAX_VALUE
 admin.retries=MAX_VALUE

 This KIP will fix this issue only in the long term though, ie, when
 `retries` is finally removed. Short term, using `retries` in
 StreamsConfig would still affect the embedded clients, but Streams, as
 well as both client would log a WARN message. This preserves backward
 compatibility.

 Withing Streams `retries` is ignored and the new `task.timeout.ms` is
 used instead. This increase the default resilience of Kafka Streams
 itself. We could also achieve this by still respecting `retries` and to
 change it's default value. However, because we deprecate `retries` it
 seems better to just ignore it and switch to the new config directly.

 I updated the KIPs with some more details.



> 2) We should also document the related behavior change in
 PartitionAssignor
> that uses AdminClient.

 This is actually a good point. Originally, I looked into this only
 briefly, but it raised an interesting question on how to handle it.

 Note that `TimeoutExceptions` are currently not handled in this retry
 loop. Also note that the default retries value for other errors would be
 MAX_VALUE be default (inherited from `AdminClient#retries` as mentioned
 already by Guozhang).

 Applying the new `task.timeout.ms` config does not seem to be
 appropriate because the AdminClient is used during a 

[jira] [Created] (KAFKA-10297) Don't use deprecated producer config `retries`

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-10297:
---

 Summary: Don't use deprecated producer config `retries`
 Key: KAFKA-10297
 URL: https://issues.apache.org/jira/browse/KAFKA-10297
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 2.7.0
Reporter: Matthias J. Sax
 Fix For: 2.7.0


In 2.7.0 release, producer config `retries` gets deprecated via KIP-572.

Connect is still using this config what needs to be fixed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-10253) Kafka Connect gets into an infinite rebalance loop

[Konstantin Lalafaryan (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Lalafaryan resolved KAFKA-10253.
---
Resolution: Not A Problem

Seems like we have found a problem, and it is related to the 
`{{config.storage.topic}}` topic which has been created with 3 partitions but 
it should be created *always with single partition.*

 

*_config.storage.topic_*
 The name of the topic where connector and task configuration data are stored. 
This must be the same for all Workers with the same group.id. Kafka Connect 
will upon startup attempt to automatically create this topic with a 
single-partition and compacted cleanup policy to avoid losing data, but it will 
simply use the topic if it already exists. If you choose to create this topic 
manually, always create it as a compacted topic with a single partition and a 
high replication factor (3x or more).

> Kafka Connect gets into an infinite rebalance loop
> --
>
> Key: KAFKA-10253
> URL: https://issues.apache.org/jira/browse/KAFKA-10253
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 2.5.0
>Reporter: Konstantin Lalafaryan
>Priority: Blocker
>
> Hello everyone!
>  
> We are running kafka-connect cluster  (3 workers) and very often it gets into 
> an infinite rebalance loop.
>  
> {code:java}
> 2020-07-09 08:51:25,731 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] Rebalance started 
> (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,731 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] (Re-)joining group 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,733 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] Was selected to perform assignments, but do not have latest 
> config found in sync request. Returning an empty configuration to trigger 
> re-sync. 
> (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] Successfully joined group with generation 305655831 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] Joined group at generation 305655831 with protocol version 2 
> and got assignment: Assignment{error=1, 
> leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', 
> leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], 
> taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with 
> rebalance delay: 0 
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] Rebalance started 
> (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] (Re-)joining group 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,736 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] Was selected to perform assignments, but do not have latest 
> config found in sync request. Returning an empty configuration to trigger 
> re-sync. 
> (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] Successfully joined group with generation 305655832 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] Joined group at generation 305655832 with protocol version 2 
> and got assignment: Assignment{error=1, 
> leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', 
> leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], 
> taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with 
> rebalance delay: 0 
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] Rebalance started 
> (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) 
> [DistributedHerder-connect-1-1]
> 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= 
> kafka-connect] (Re-)joining group 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) 
> 

Re: [DISCUSS] KIP-450: Sliding Windows

[Leah Thomas]

Hi all,

Thanks for the feedback on the KIP. I've updated the KIP page

to address these points and have created a child page

to go more in depth on certain implementation details.

*Grace Period:*
I think Sophie raises a good point that the default grace period of 24
hours is often too long and was chosen when retention time and grace period
were the same. For SlidingWindows, I propose we make the grace period
mandatory. To keep formatting consistent with other types of windows, grace
period won't be an additional parameter in the #of method, but will still
look like it does in other use cases:
.windowedBy(SlidingWindows.of(twentySeconds).grace(fiftySeconds). If grace
period isn't properly initialized, an error will be thrown through the
process method.

*Storage Layer + Aggregation:*
SlidingWindows will use a WindowStore because computation can be done with
the information stored in a WindowStore (window timestamp and value). Using
the WindowStore also simplifies computation as SlidingWindows can leverage
existing processes. Because we are using a WindowStore, the aggregation
process will be similar to that of a hopping window. As records come in
their value is added to the aggregation that already exists, following the
same procedure as hopping windows. The aggregation difference between
SlidingWindows and HoppingWindows comes in creating new windows for a
SlidingWindow, where you need to find the existing records that belong to
the new window. This computation is similar to the aggregation in
SessionWindows and requires a scan to the WindowStore to find the window
with the aggregation needed, which will always be pre-computed. The scan
requires creating an iterator, but should have minimal performance effects
as this strategy is already implemented in SessionWindows. More details on
finding the aggregation that needs to be put in a new window can be found
on the implementation page

.

*Extending Windows, Windows or nothing*
Because SlidingWindows are still defined by a windowSize (whereas
SessionWindows are defined purely by data), I think it makes sense to
leverage the existing Window processes instead of creating a new store type
that would be very similar to the WindowStore. While it's true that the
#windowsFor method isn't necessary for SlidingWindows, JoinWindows also
extends Windows and throws an UnsupportedOperationException in the
#windowsFor method, which is what SlidingWindows can do. The difference
between extending Windows or Windows is minimal, as
both are ways to pass window parameters. Extending Windows will
give us more leverage in utilizing existing processes.

*Emit Strategy*
I would argue that emitting for every update is still the best way to go
for SlidingWindows because it mimics the other types of windows, and
suppression can be leveraged to limit what SlidingWindows emits. While some
users may only want to see the last value, others may want to see more, and
leaving the emit strategy to emit partial results allows both users to
access what they want.

*Additional Features*
Supporting sliding windows inherently, and shifting inefficient hopping
windows to sliding windows, is an interesting idea and could be built on
top of SlidingWindows when they are finished, but right now seems out of
scope for the needs of this KIP. Similarly, including a `subtraction`
feature could have performance improvements, but doesn't seem necessary for
the implementation of this KIP.

Let me know what you think of the updates,

Leah

On Thu, Jul 16, 2020 at 11:57 AM John Roesler  wrote:

> Hello all,
>
> Thanks for the KIP, Leah!
>
> Regarding (1): I'd go farther actually. Making Windows an abstract
> class was a mistake from the beginning that led to us not being
> able to fix a very confusing situation for users around retention times,
> final results emitting, etc. Thus, I would not suggest extending
> TimeWindows for sure, but would also not suggest extending Windows.
>
> The very simplest thing to do is follow the example of SessionWindows,
> which is just a completely self-contained class. If we don't mess with
> class inheritance, we won't ever have any of the problems related to
> class inheritance. This is my preferred solution.
>
> Still, Sliding windows has a lot in common with TimeWindows and other
> fixed-size windows, namely that the windows are fixed in size. If we want
> to preserve the current two-part windowing API in which you can window
> by either "fixed" or "data driven" modes, I'd suggest we avoid increasing
> the blast radius of Windows by taking the opportunity to replace it with
> a proper interface and implement that interface instead.
>
> For example:
> https://github.com/apache/kafka/pull/9031
>
> Then, 

Re: [DISCUSS] Kafka 3.0

[Matthias J. Sax]

Did we reach any conclusion on the subject?

It seems we are aiming for 2.7 after 2.6 and plan the major version bump
to 3.0 after 2.7 (assuming we make progress on ZK removal as planned?)


-Matthias


On 5/18/20 1:11 PM, Boyang Chen wrote:
> One more thing I would like to see deprecated (hopefully no one mentioned
> before) is the zk based consumer offset support.
> 
> On Mon, May 11, 2020 at 2:15 PM Colin McCabe  wrote:
> 
>> Hi Michael,
>>
>> It would be better to discuss the background behind KIP-500 in a separate
>> thread, since this thread is about the Kafka 3.0 release.  As others have
>> said, your questions are answered in the KIP.  For example, "what is the
>> actual goal?" is addressed in the motivation section.
>>
>> I agree that Kafka's usage of Apache ZooKeeper could be optimized.  But
>> there are fundamental limitations to this approach compared to storing our
>> metadata internally.  For example, having to contact a remote server to
>> reload all your metadata on a controller failover simply doesn't scale past
>> a certain point.
>>
>> Apache Curator is a nice API, and if we were starting again today we would
>> certainly consider using it.  But it doesn't allow us to do anything more
>> efficiently than ZooKeeper could already do it.
>>
>> Finally, Kafka's core competence is logs.  While our replication protocol
>> is not Raft, it shares many similarities with that protocol.  So I think
>> it's a bit unfair to say that it is "catastrophic hubris" to believe we can
>> implement the protocol.
>>
>> best,
>> Colin
>>
>>
>> On Sun, May 10, 2020, at 11:02, Michael K. Edwards wrote:
>>> Yes, I've read the KIP.  But all it really says to me is "we have never
>>> gotten around to using ZooKeeper properly."  To the extent that any of
>> the
>>> distributed-state-maintenance problems discussed in "Metadata as an Event
>>> Log" can be solved — and some of them intrinsically can't, because CAP
>>> theorem — most of them are already implemented very effectively in
>> Curator
>>> recipes.  (For instance, Curator's Tree Cache
>>> https://curator.apache.org/curator-recipes/tree-cache.html is a good
>> fit to
>>> some of the state-maintenance needs.)
>>>
>>> Kafka does have some usage patterns that don't map neatly onto existing
>>> Curator recipes.  For instance, neither LeaderSelector nor LeaderLatch
>>> implements leader preference in the way that the existing Kafka partition
>>> leadership election procedure does.  But why not handle that by improving
>>> and extending Curator?  That way, other Curator users benefit, and we get
>>> additional highly experienced reviewers' eyes on the distributed
>>> algorithms, which are very very tricky to get right.
>>>
>>>
>>> On Sun, May 10, 2020 at 10:47 AM Ron Dagostino 
>> wrote:
>>>
 Hi Michael.  This is discussed in the KIP.



>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation

 Ron

> On May 10, 2020, at 1:35 PM, Michael K. Edwards <
>> m.k.edwa...@gmail.com>
 wrote:
>
> What is the actual goal of removing the ZooKeeper dependency?  In my
> experience, if ZooKeeper is properly provisioned and deployed, it's
 largely
> trouble-free.  (You do need to know how to use observers properly.)
 There
> are some subtleties about timeouts and leadership changes, but
>> they're
> pretty small stuff.  Why go to all the trouble of building a new
> distributed-consensus system that's going to have catastrophic bugs
>> for
> years to come?  It seems like such an act of hubris to me, as well
>> as a
> massive waste of engineering effort.  What is there to be gained?
>
>> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax 
 wrote:
>>
>> Sure, we can compile a list for Kafka Streams. But the KIP would be
>> for
>> 3.0, so I don't think it's urgent to do it now?
>>
>>
>> -Matthias
>>
>>> On 5/8/20 3:47 PM, Colin McCabe wrote:
>>> Thanks, Guozhang-- sounds like a good plan.
>>>
>>> I think it would be good to have a list of deprecated streams APIs
>> that
>> we want to remove in 3.0.  Maybe it's easiest to do that as its own
>> KIP?
>>>
>>> For MirrorMaker 1, we should have a KIP to deprecate its use in
>> 2.6 if
>> we want to remove it in 3.0.  I don't have a good sense of how
 practical it
>> is to deprecate this now, so I will defer to others here.  But the
>> KIP
>> freeze for 2.6 is coming soon, so if we want to make the case, now
>> is
 the
>> time.
>>>
>>> best,
>>> Colin
>>>
>>>
 On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote:
 Hey folks,

 Sorry for stating that the bridge release would not break any
>> compatibility
 before, which is incorrect and confused many people.

 I think one way 

Jenkins build is back to normal : kafka-2.3-jdk8 #219

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk11 #1653

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10286: Connect system tests should wait for workers to join group

[github] KAFKA-10295: Wait for connector recovery in test_bounce (#9043)


--
[...truncated 6.40 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > 

Build failed in Jenkins: kafka-trunk-jdk14 #302

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10286: Connect system tests should wait for workers to join group

[github] KAFKA-10295: Wait for connector recovery in test_bounce (#9043)


--
[...truncated 6.40 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

Build failed in Jenkins: kafka-trunk-jdk8 #4726

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10286: Connect system tests should wait for workers to join group

[github] KAFKA-10295: Wait for connector recovery in test_bounce (#9043)


--
[...truncated 6.34 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

Build failed in Jenkins: kafka-2.4-jdk8 #234

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-10286: Connect system tests should wait for workers to join group

[rhauch] KAFKA-10295: Wait for connector recovery in test_bounce (#9043)


--
[...truncated 2.76 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldTerminateWhenUsingTaskIdling[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldTerminateWhenUsingTaskIdling[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos 

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

[Bruno Cadonna]

Hi,

During the implementation of this KIP and after some discussion about 
RocksDB metrics, I decided to make some major modifications to this KIP 
and kick off discussion again.


https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Report+Properties+of+RocksDB

Best,
Bruno

On 15.05.20 17:11, Bill Bejeck wrote:

Thanks for the KIP, Bruno. Having sensible, easy to access RocksDB memory
reporting will be a welcomed addition.

FWIW I also agree to have the metrics reported on a store level. I'm glad
you changed the KIP to that effect.

-Bill



On Wed, May 13, 2020 at 6:24 PM Guozhang Wang  wrote:


Hi Bruno,

Sounds good to me.

I think I'm just a bit more curious to see its impact on performance: as
long as we have one INFO level rocksDB metrics, then we'd have to turn on
the scheduled rocksdb metrics recorder whereas previously, we can decide to
not turn on the recorder at all if all are set as DEBUG and we configure at
INFO level in production. But this is an implementation detail anyways and
maybe the impact is negligible after all. We can check and re-discuss this
afterwards :)


Guozhang


On Wed, May 13, 2020 at 9:34 AM Sophie Blee-Goldman 
wrote:


Thanks Bruno! I took a look at the revised KIP and it looks good to me.

Sophie

On Wed, May 13, 2020 at 6:59 AM Bruno Cadonna 

wrote:



Hi John,

Thank you for the feedback!

I agree and I will change the KIP as I stated in my previous e-mail to
Guozhang.

Best,
Bruno

On Tue, May 12, 2020 at 3:07 AM John Roesler 

wrote:


Thanks, all.

If you don’t mind, I’ll pitch in a few cents’ worth.

In my life I’ve generally found more granular metrics to be more

useful,

as long as there’s a sane way to roll them up. It does seem nice to see

it

on the per-store level. For roll-up purposes, the task and thread tags
should be sufficient.


I think the only reason we make some metrics Debug is that

_recording_

them can be expensive. If there’s no added expense, I think we can just
register store-level metrics at Info level.


Thanks for the KIP, Bruno!
-John

On Mon, May 11, 2020, at 17:32, Guozhang Wang wrote:

Hello Sophie / Bruno,

I've also thought about the leveling question, and one motivation I

had for

setting it in instance-level is that we want to expose it in INFO

level:

today our report leveling is not very finer grained --- which I

think

is

sth. worth itself --- such that one have to either turn on all

DEBUG

metrics recording or none of them. If we can allow users to e.g.

specify

"turn on streams-metrics and stream-state-metrics, but not others"

and

then

I think it should be just at store-level. However, right now if we

want to

set it as store-level then it would be DEBUG and not exposed by

default.


So it seems we have several options in addition to the proposed

one:


a) we set it at store-level as INFO; but then one can argue why

this

is

INFO while others (bytes-written, etc) are DEBUG.
b) we set it at store-level as DEBUG, believing that we do not

usually

need

to turn it on.
c) maybe, we can set it at task-level (? I'm not so sure myself

about

this.. :P) as INFO.


Guozhang




On Mon, May 11, 2020 at 12:29 PM Sophie Blee-Goldman <

sop...@confluent.io>

wrote:


Hey Bruno,

Thanks for the KIP! I have one high-level concern, which is that

we

should

consider
reporting these metrics on the per-store level rather than

instance-wide. I

know I was
the one who first proposed making it instance-wide, so bear with

me:


While I would still argue that the instance-wide memory usage is

probably

the most *useful*,
exposing them at the store-level does not prevent users from

monitoring the

instance-wide
memory. They should be able to roll up all the store-level

metrics

on an

instance to
compute the total off-heap memory. But rolling it up for the

users

does

prevent them from
using this to debug rare cases where one store may be using

significantly

more memory than
expected.

It's also worth considering that some users may be using the

bounded

memory

config setter
to put a cap on the off-heap memory of the entire process, in

which

case

the memory usage
metric for any one store should reflect the memory usage of the

entire

instance. In that case
any effort to roll up the memory usages ourselves would just be

wasted.


Sorry for the reversal, but after a second thought I'm pretty

strongly in

favor of reporting these
at the store level.

Best,
Sophie

On Wed, May 6, 2020 at 8:41 AM Bruno Cadonna 


wrote:



Hi all,

I'd like to discuss KIP-607 that aims to add RocksDB memory

usage

metrics to Kafka Streams.










https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Record+the+Memory+Used+by+RocksDB+to+Kafka+Streams


Best,
Bruno






--
-- Guozhang








--
-- Guozhang

Build failed in Jenkins: kafka-2.6-jdk8 #91

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-10286: Connect system tests should wait for workers to join group

[rhauch] KAFKA-10295: Wait for connector recovery in test_bounce (#9043)


--
[...truncated 3.15 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo PASSED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
STARTED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex STARTED

Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

[Matthias J. Sax]

Thanks Badai. LGTM.

On 7/19/20 4:26 PM, Badai Aqrandista wrote:
> Hi all
> 
> I have made a small change to KIP-431 to make it clearer which one is
> "Partition" and "Offset". Also I have moved key field to the back,
> before the value:
> 
> $ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
> test --from-beginning --property print.partition=true --property
> print.key=true --property print.timestamp=true --property
> print.offset=true --property print.headers=true --property
> key.separator='|'
> 
> CreateTime:1592475472398|Partition:0|Offset:3|h1:v1,h2:v2|key1|value1
> CreateTime:1592475472456|Partition:0|Offset:4|NO_HEADERS|key2|value2
> 
> Regards
> Badai
> 
> On Sun, Jun 21, 2020 at 11:39 PM Badai Aqrandista  wrote:
>>
>> Excellent.
>>
>> Would like to hear more feedback from others.
>>
>> On Sat, Jun 20, 2020 at 1:27 AM David Jacot  wrote:
>>>
>>> Hi Badai,
>>>
>>> Thanks for your reply.
>>>
>>> 2. Yes, that makes sense.
>>>
>>> Best,
>>> David
>>>
>>> On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista  wrote:
>>>
 David

 Thank you for replying

 1. It seems that `print.partition` is already implemented. Do you confirm?
 BADAI: Yes, you are correct. I have removed it from the KIP.

 2. Will `null.literal` be only used when the value of the message
 is NULL or for any fields? Also, it seems that we print out "null"
 today when the key or the value is empty. Shall we use "null" as
 a default instead of ""?
 BADAI: For any fields. Do you think this is useful?

 3. Could we add a small example of the output in the KIP?
 BADAI: Yes, I have updated the KIP to add a couple of example.

 4. When there are no headers, are we going to print something
 to indicate it to the user? For instance, we print out NO_TIMESTAMP
 where there is no timestamp.
 BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS.

 Thanks
 Badai


 On Thu, Jun 18, 2020 at 7:25 PM David Jacot  wrote:
>
> Hi Badai,
>
> Thanks for resuming this. I have few small comments:
>
> 1. It seems that `print.partition` is already implemented. Do you
 confirm?
>
> 2. Will `null.literal` be only used when the value of the message
> is NULL or for any fields? Also, it seems that we print out "null"
> today when the key or the value is empty. Shall we use "null" as
> a default instead of ""?
>
> 3. Could we add a small example of the output in the KIP?
>
> 4. When there are no headers, are we going to print something
> to indicate it to the user? For instance, we print out NO_TIMESTAMP
> where there is no timestamp.
>
> Best,
> David
>
> On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista 
 wrote:
>
>> Hi all,
>>
>> I have contacted Mateusz separately and he is ok for me to take over
>> KIP-431:
>>
>>
>>
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter
>>
>> I have updated it a bit. Can anyone give a quick look at it again and
>> give me some feedback?
>>
>> This feature will be very helpful for people supporting Kafka in
>> operations.
>>
>> If it is ready for a vote, please let me know.
>>
>> Thanks
>> Badai
>>
>> On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista 
>> wrote:
>>>
>>> Mateusz
>>>
>>> This KIP would be very useful for debugging. But the last discussion
>>> is in Feb 2019.
>>>
>>> Are you ok if I take over this KIP?
>>>
>>> --
>>> Thanks,
>>> Badai
>>
>>
>>
>> --
>> Thanks,
>> Badai
>>



 --
 Thanks,
 Badai

>>
>>
>>
>> --
>> Thanks,
>> Badai
> 
> 
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk11 #1652

[Apache Jenkins Server]

See 


Changes:

[manikumar.reddy] MINOR: Enable broker/client compatibility tests for 2.5.0 
release


--
[...truncated 3.20 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

Build failed in Jenkins: kafka-2.6-jdk8 #90

[Apache Jenkins Server]

See 


Changes:

[manikumar.reddy] MINOR: Enable broker/client compatibility tests for 2.5.0 
release


--
[...truncated 3.15 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo PASSED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
STARTED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureApplicationAndRecordMetadata STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureApplicationAndRecordMetadata PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureRecordsOutputToChildByName STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureRecordsOutputToChildByName PASSED

org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
STARTED

org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 

Build failed in Jenkins: kafka-trunk-jdk8 #4725

[Apache Jenkins Server]

See 


Changes:

[manikumar.reddy] MINOR: Enable broker/client compatibility tests for 2.5.0 
release


--
[...truncated 3.17 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 

Build failed in Jenkins: kafka-trunk-jdk14 #301

[Apache Jenkins Server]

See 


Changes:

[manikumar.reddy] MINOR: Enable broker/client compatibility tests for 2.5.0 
release


--
[...truncated 3.20 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

[jira] [Resolved] (KAFKA-10295) ConnectDistributedTest.test_bounce should wait for graceful stop

[Randall Hauch (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch resolved KAFKA-10295.
---
  Reviewer: Randall Hauch
Resolution: Fixed

> ConnectDistributedTest.test_bounce should wait for graceful stop
> 
>
> Key: KAFKA-10295
> URL: https://issues.apache.org/jira/browse/KAFKA-10295
> Project: Kafka
>  Issue Type: Test
>  Components: KafkaConnect
>Affects Versions: 2.3.1, 2.5.0, 2.4.1, 2.6.0
>Reporter: Greg Harris
>Assignee: Greg Harris
>Priority: Minor
> Fix For: 2.3.2, 2.6.0, 2.4.2, 2.5.1, 2.7.0
>
>
> In ConnectDistributedTest.test_bounce, there are flakey failures that appear 
> to follow this pattern:
>  # The test is parameterized for hard bounces, and with Incremental 
> Cooperative Rebalancing enabled (does not appear for protocol=eager)
>  # A source task is on a worker that will experience a hard bounce
>  # The source task has written records which it has not yet committed in 
> source offsets
>  # The worker is hard-bounced, and the source task is lost
>  # Incremental Cooperative Rebalance starts it's 
> scheduled.rebalance.max.delay.ms delay before recovering the task
>  # The test ends, connectors and Connect are stopped
>  # The test verifies that the sink connector has only written records that 
> have been committed by the source connector
>  # This verification fails because the source offsets are stale, and there 
> are un-committed records in the topic, and the sink connector has written at 
> least one of them.
> This can be addressed by ensuring that the test waits for the rebalance delay 
> to expire, and for the lost task to recover and commit offsets past the 
> progress it made before the bounce.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-10286) Connect system tests should wait for workers to join group

[Randall Hauch (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch resolved KAFKA-10286.
---
  Reviewer: Randall Hauch
Resolution: Fixed

> Connect system tests should wait for workers to join group
> --
>
> Key: KAFKA-10286
> URL: https://issues.apache.org/jira/browse/KAFKA-10286
> Project: Kafka
>  Issue Type: Test
>  Components: KafkaConnect
>Affects Versions: 2.6.0
>Reporter: Greg Harris
>Assignee: Greg Harris
>Priority: Minor
> Fix For: 2.3.2, 2.6.0, 2.4.2, 2.5.1, 2.7.0
>
>
> There are a few flakey test failures for {{connect_distributed_test}} in 
> which one of the workers does not join the group quickly, and the test fails 
> in the following manner:
>  # The test starts each of the connect workers, and waits for their REST APIs 
> to become available
>  # All workers start up, complete plugin scanning, and start their REST API
>  # At least one worker kicks off an asynchronous job to join the group that 
> hangs for a yet unknown reason (30s timeout)
>  # The test continues without all of the members joined
>  # The test makes a call to the REST api that it expects to succeed, and gets 
> an error
>  # The test fails without the worker ever joining the group
> Instead of allowing the test to fail in this manner, we could wait for each 
> worker to join the group with the existing 60s startup timeout. This change 
> would go into effect for all system tests using the 
> {{ConnectDistributedService}}, currently just {{connect_distributed_test}} 
> and {{connect_rest_test}}. 
> Alternatively we could retry the operation that failed, or ensure that we use 
> a known-good worker to continue the test, but these would require more 
> involved code changes. The existing wait-for-startup logic is the most 
> natural place to fix this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] 2.6.0 RC0

[Rajini Sivaram]

Thanks Randall. We didn't create a JIRA for the KIP-546 security fix since
KAFKA-7740 that introduced the regression had a fix version of 2.5.0. A fix
was applied without a JIRA to follow the security bug fix process. In the
end, it turned out that KAFKA-7740 was only in 2.6, so the bug hadn't got
into a release anyway.

Regards,

Rajini


On Sun, Jul 19, 2020 at 2:46 PM Randall Hauch  wrote:

> Thanks, Rajini. Is there a Jira issue for the fix related to KIP-546? If
> so, please make sure the Fix Version(s) include `2.6.0`.
>
> I'm going to start RC1 later today and hope to get it published by Monday.
> In the meantime, if anyone finds anything else in RC0, please raise it here
> -- if it's after RC1 is published then we'll just cut another RC with any
> fixes.
>
> We're down to just 5 system test failures [1], and folks are actively
> working to address them. At least some are known to be flaky, but we still
> want to get them fixed.
>
> Best regards,
>
> Randall
>
> On Sun, Jul 19, 2020 at 5:45 AM Rajini Sivaram 
> wrote:
>
> > Hi Randall,
> >
> > Ron found an issue with the quota implementation added under KIP-546,
> which
> > is a blocking issue for 2.6.0 since it leaks SCRAM credentials in quota
> > responses. A fix has been merged into 2.6 branch in the commit
> >
> >
> https://github.com/apache/kafka/commit/dd71437de7675d92ad3e4ed01ac3ee11bf5da99d
> > .
> > We
> > have also merged the fix for
> > https://issues.apache.org/jira/browse/KAFKA-10223 into 2.6 branch since
> it
> > causes issues for non-Java clients during reassignments.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Wed, Jul 15, 2020 at 11:41 PM Randall Hauch  wrote:
> >
> > > Thanks, Levani.
> > >
> > > The content of
> > >
> > >
> >
> https://home.apache.org/~rhauch/kafka-2.6.0-rc0/kafka_2.12-2.6.0-site-docs.tgz
> > > is the correct generated site. Somehow I messed coping that to the
> > > https://github.com/apache/kafka-site/tree/asf-site/26 directory. I've
> > > corrected the latter so that
> https://kafka.apache.org/26/documentation/
> > > now
> > > exactly matches that documentation in RC0.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Wed, Jul 15, 2020 at 1:25 AM Levani Kokhreidze <
> > levani.co...@gmail.com>
> > > wrote:
> > >
> > > > Hi Randall,
> > > >
> > > > Not sure if it’s intentional but, documentation for Kafka Streams
> 2.6.0
> > > > also contains “Streams API changes in 2.7.0”
> > > > https://kafka.apache.org/26/documentation/streams/upgrade-guide <
> > > > https://kafka.apache.org/26/documentation/streams/upgrade-guide>
> > > >
> > > > Also, there seems to be some formatting issue in 2.6.0 section.
> > > >
> > > > Levani
> > > >
> > > >
> > > > > On Jul 15, 2020, at 1:48 AM, Randall Hauch 
> wrote:
> > > > >
> > > > > Thanks for catching that, Gary. Apologies to all for announcing
> this
> > > > before
> > > > > pushing the docs, but that's fixed and the following links are
> > working
> > > > > (along with the others in my email):
> > > > >
> > > > > * https://kafka.apache.org/26/documentation.html
> > > > > * https://kafka.apache.org/26/protocol.html
> > > > >
> > > > > Randall
> > > > >
> > > > > On Tue, Jul 14, 2020 at 4:30 PM Gary Russell 
> > > > wrote:
> > > > >
> > > > >> Docs link [1] is broken.
> > > > >>
> > > > >> [1] https://kafka.apache.org/26/documentation.html
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-10296) Connector task reported RUNNING after hard bounce of worker

[Greg Harris (Jira)]

Greg Harris created KAFKA-10296:
---

 Summary: Connector task reported RUNNING after hard bounce of 
worker
 Key: KAFKA-10296
 URL: https://issues.apache.org/jira/browse/KAFKA-10296
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.4.1, 2.5.0, 2.3.1
Reporter: Greg Harris


While fixing flakiness for ConnectDistributedTest.test_bounce in KAFKA-10295, I 
observed that the status of connectors on zombie/offline workers was 
inconsistent during Incremental Cooperative Rebalancing's 
scheduled.delay.max.interval.ms. This is the reproduction case observed there:
 # A task is running on worker A, with another worker B in the same distributed 
cluster
 # Observe that on worker B's REST API, the task is initially correctly 
"RUNNING"
 # Worker A is hard-stopped, and goes offline
 # Observe that on worker B's REST API, the task is still "RUNNING"
 # The group rebalances without worker A in the group, and begins the delay
 # Observe that on worker B's REST API, the task is still "RUNNING"
 # Worker A recovers and joins the group, before the delay expires
 # Observe that on worker B's REST API, the task is still "RUNNING"
 # The rebalance delay expires, and the task is assigned and started
 # Observe that on worker B's REST API, the task is now correctly "RUNNING"

 * In the first state (4), after the worker goes offline, but before the other 
workers learn that they have gone offline, it is acceptable that the task is 
still reported as running. We can't expect that the other workers know that 
worker A has gone offline until the group membership protocol informs them.
 * In the second state (6), when a rebalance occurs and the worker is first 
known to be unhealthy, the state of the task is ambiguous, since it may be down 
completely, or running on a zombie worker. I'm not sure how best to capture 
this state under the existing enum's options, but it's probably closest to 
"UNASSIGNED" since the leader doesn't think that any worker is currently 
running that task.
 * In the third state (8), when the bounced worker returns, the task is 
reported RUNNING on a worker which does not have the task assigned. This is the 
most inaccurate state reported, since the cluster has reached consensus, and 
yet the REST API still reports the wrong state of the task.

State (6) could be assigned a new state "UNKNOWN", introduced by a KIP. 
However, this is a large investment in process and time for what amounts to an 
almost cosmetic change, and we could either leave this state as "RUNNING", or 
change it to "UNASSIGNED"
State (8) could be described by "UNASSIGNED", and would be a tangible 
improvement for test_bounce, which currently needs to intentionally ignore the 
REST API's result here because it is inaccurate.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)