date:20200706

[jira] [Created] (KAFKA-10242) Adding metrics to track the total count of idempotent producers that Broker need to track

2020-07-06 Thread Ming Liu (Jira)

Ming Liu created KAFKA-10242:


 Summary: Adding metrics to track the total count of idempotent 
producers that Broker need to track
 Key: KAFKA-10242
 URL: https://issues.apache.org/jira/browse/KAFKA-10242
 Project: Kafka
  Issue Type: Improvement
  Components: producer 
Affects Versions: 2.5.0
Reporter: Ming Liu
 Fix For: 2.6.0


We found it is very useful to track the total number of idempotent producers 
that broker is tracking.
In our production environment, we have many idempotent producers for a cluster 
and sometimes that number increased to very high number which requires some 
attention to mitigate.
This is especially true for client (< 2.4) where the client retry might 
generate too many different idempotent producers which can trigger broker GC.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (KAFKA-6453) Reconsider timestamp propagation semantics

2020-07-06 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-6453:


IMHO, the PR only covers part of this ticket. We should also document how 
timestamps are computed for output records of aggregations and joins.

> Reconsider timestamp propagation semantics
> --
>
> Key: KAFKA-6453
> URL: https://issues.apache.org/jira/browse/KAFKA-6453
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Victoria Bialas
>Priority: Major
>  Labels: needs-kip
>
> Atm, Kafka Streams only has a defined "contract" about timestamp propagation 
> at the Processor API level: all processor within a sub-topology, see the 
> timestamp from the input topic record and this timestamp will be used for all 
> result record when writing them to an topic, too.
> The DSL, inherits this "contract" atm.
> From a DSL point of view, it would be desirable to provide a different 
> contract to the user. To allow this, we need to do the following:
>  - extend Processor API to allow manipulation timestamps (ie, a Processor can 
> set a new timestamp for downstream records)
>  - define a DSL "contract" for timestamp propagation for each DSL operator
>  - document the DSL "contract"
>  - implement the DSL "contract" using the new/extended Processor API



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1623

2020-07-06 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: prune the metadata upgrade test matrix (#8971)

[github] KAFKA-10166: checkpoint recycled standbys and ignore empty rocksdb base

[github] MINOR: document timestamped state stores (#8920)


--
[...truncated 3.19 MB...]
org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0101:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:test
> Task :streams:upgrade-system-tests-0102:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0102:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0102:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0102:compileTestJava
> Task :streams:upgrade-system-tests-0102:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:testClasses
> Task :streams:upgrade-system-tests-0102:checkstyleTest
> Task :streams:upgrade-system-tests-0102:spotbugsMain NO-SOURCE
>

[jira] [Resolved] (KAFKA-10239) The groupInstanceId field in DescribeGroup response should be ignorable

2020-07-06 Thread Boyang Chen (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-10239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-10239.
-
Resolution: Fixed

> The groupInstanceId field in DescribeGroup response should be ignorable
> ---
>
> Key: KAFKA-10239
> URL: https://issues.apache.org/jira/browse/KAFKA-10239
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.5.0, 2.4.1
>Reporter: Jason Gustafson
>Assignee: Boyang Chen
>Priority: Critical
> Fix For: 2.6.0, 2.4.2, 2.5.1
>
>
> We noticed the following error in the logs in the handling of a DescribeGroup 
> request:
> ```
> org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to 
> write a non-default groupInstanceId at version 3
> ```
> The problem is that the field is not marked as ignorable. So if the user is 
> relying on static membership and uses an older AdminClient, they will see 
> this error.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk14 #272

2020-07-06 Thread Apache Jenkins Server

See

Untrimmed Index Files resulting in data loss

2020-07-06 Thread John Malizia

Hi there, about a week ago I submitted an issue report and an associated PR

https://issues.apache.org/jira/browse/KAFKA-10207
https://github.com/apache/kafka/pull/8936

Is there any further information I can provide to help this along? Sorry to
bug everyone about this, but after identifying the issue it seems like
something that should be handled more gracefully.

[jira] [Created] (KAFKA-10241) Pursue a better way to cover ignorable RPC fields

2020-07-06 Thread Boyang Chen (Jira)

Boyang Chen created KAFKA-10241:
---

 Summary: Pursue a better way to cover ignorable RPC fields 
 Key: KAFKA-10241
 URL: https://issues.apache.org/jira/browse/KAFKA-10241
 Project: Kafka
  Issue Type: Improvement
  Components: clients, core
Reporter: Boyang Chen


We have hit case such as https://issues.apache.org/jira/browse/KAFKA-10239 
where we accidentally include a non-ignorable field into the returned response, 
and eventually crash older clients who doesn't support this field. It would be 
good to add a generic test suite to cover all existing and new RPC changes to 
ensure that we don't have a chance to put a non-ignorable field for older 
version of clients.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-2.6-jdk8 #71

2020-07-06 Thread Apache Jenkins Server

See

Build failed in Jenkins: kafka-trunk-jdk11 #1622

2020-07-06 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest (#8963)


--
[...truncated 6.37 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest >

Untrimmed index Files resulting in data loss

2020-07-06 Thread John Malizia

Hi there, about a week ago I submitted an issue report and an associated PR

https://issues.apache.org/jira/browse/KAFKA-10207
https://github.com/apache/kafka/pull/8936

Is there any further information I can provide to help this along? Sorry to
bug everyone about this, but after identifying the issue it seems like
something that can be handled more gracefully.

Jenkins build is back to normal : kafka-trunk-jdk8 #4696

2020-07-06 Thread Apache Jenkins Server

See

Build failed in Jenkins: kafka-trunk-jdk14 #271

2020-07-06 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-10017: fix flaky EosBetaUpgradeIntegrationTest (#8963)


--
[...truncated 3.18 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest >

Build failed in Jenkins: kafka-trunk-jdk11 #1621

2020-07-06 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: Improve logging around initial log loading (#8970)

[github] KAFKA-9769: ReplicaManager Partition.makeFollower Increases LeaderEpoch

[github] KAFKA-8398: Prevent NPE in `forceUnmap` (#8983)


--
[...truncated 3.19 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest >

Re: [DISCUSS] KIP-639 Move nodeLevelSensor and storeLevelSensor methods from StreamsMetricsImpl to StreamsMetrics

2020-07-06 Thread Mohamed Chebbi


Thank Bruno for your review.

Changes was added as you sugested.

Le 06/07/2020 à 14:57, Bruno Cadonna a écrit :

Hi Mohamed,

Thank you for the KIP.

Comments regarding the KIP wiki:

1. In section "Public Interface", you should state what you want to
change in interface StreamsMetrics. In your case, you want to add two
methods. You can find a good example how to describe this in KIP-444
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams).

2. In section "Compatibility, Deprecation, and Migration Plan" you
should state if anything is needed to keep backward compatibility.
Since you just want to add two methods to the interface, nothing is
needed. You should describe that under that section.

Regarding the KIP content, I left some comments on the corresponding
Jira ticket.

Best,
Bruno


On Sun, Jul 5, 2020 at 3:48 AM Mohamed Chebbi  wrote:


https://cwiki.apache.org/confluence/display/KAFKA/KIP-639%3A+Move+nodeLevelSensor+and+storeLevelSensor+methods+from+StreamsMetricsImpl+to+StreamsMetrics

Build failed in Jenkins: kafka-trunk-jdk8 #4695

2020-07-06 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: Improve logging around initial log loading (#8970)

[github] KAFKA-9769: ReplicaManager Partition.makeFollower Increases LeaderEpoch

[github] KAFKA-8398: Prevent NPE in `forceUnmap` (#8983)


--
[...truncated 3.16 MB...]

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest >

Re: Potential Improvement for Kafka Producer

2020-07-06 Thread Arvin Zheng

Thanks to Matthias for the response, I've created a KIP and started another
email thread to discuss this, check following.
[DISCUSS] Include min.insync.replicas in MetadataResponse to make Producer
smarter in partitioning events

Matthias J. Sax  于2020年7月6日周一 下午12:30写道：

> Arvin,
>
> thanks for your email. This is definitely the right channel. I am
> personally not familiar enough with the producer code, but what you say
> makes sense to me from a high level.
>
> Maybe it would be best if you would file a Jira to improve the producer
> accordingly? I guess, this change would require a KIP.
>
> Of course, if you are interested, feel free to pick it up yourself.
>
>
> -Matthias
>
>
> On 6/28/20 8:53 AM, Arvin Zheng wrote:
> > Hi All,
> >
> > Not sure if this is the right channel and thread to ask, but would like
> to
> > discuss a potential improvement to Java Kafka Producer.
> >
> > ```
> > Currently the Kafka Producer is able to identify unavailable partitions
> and
> > avoid routing messages to them, but the definition of an unavailable
> > partitions is - the leader of the partition is not available.
> > From Producer point of view, acks for sending messages can be [all, -1,
> 0,
> > 1]
> > 1. When acks is set to either 0 or 1, leader availability is good enough
> to
> > determine whether we should route messages to that partition.
> > 2. When acks is set to -1 or all, leader available doesn't mean we are
> able
> > to persist messages to that partition successfully, instead, we need to
> > make sure
> > a. leader is available.
> > b. at least min.insync.replicas number of replicas are available
> > ```
> >
> > To achieve 2, what we need is to carry min.insync.replicas information
> of a
> > topic to the metadata response, so that Producer is able to determine if
> it
> > should route messages to that partition when there's no enough replicas
> > available and it's acks is set to -1 or all.
> >
> > Advantages that I can think of
> > 1. Avoid exhausting the entire Producer cache when a partition is not
> > available for a long time and
> > a. retries is set to a large value
> > b. acks is set to all
> > 2. Avoid unnecessary network tries
> >
> > Not sure if this is a valid case but would like to hear any opinions.
> >
> > Br,
> > Arvin
> >
>
>

Build failed in Jenkins: kafka-2.6-jdk8 #70

2020-07-06 Thread Apache Jenkins Server

See 


Changes:

[jason] MINOR: Improve logging around initial log loading (#8970)


--
[...truncated 3.15 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest >

Build failed in Jenkins: kafka-trunk-jdk14 #270

2020-07-06 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: Improve logging around initial log loading (#8970)

[github] KAFKA-9769: ReplicaManager Partition.makeFollower Increases LeaderEpoch

[github] KAFKA-8398: Prevent NPE in `forceUnmap` (#8983)


--
[...truncated 3.19 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos

[jira] [Created] (KAFKA-10240) Sink tasks should not throw WakeupException on shutdown

2020-07-06 Thread Chris Egerton (Jira)

Chris Egerton created KAFKA-10240:
-

 Summary: Sink tasks should not throw WakeupException on shutdown
 Key: KAFKA-10240
 URL: https://issues.apache.org/jira/browse/KAFKA-10240
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.4.1, 2.5.0, 2.3.1, 2.4.0, 2.2.2, 2.2.1, 2.3.0, 2.1.1, 
2.2.0, 2.1.0, 2.0.1, 2.0.0, 2.6.0
Reporter: Chris Egerton
Assignee: Chris Egerton


* When a task is scheduled for shutdown, the framework [wakes up the 
consumer|https://github.com/apache/kafka/blob/8a24da376b801c6eb6522ad4861b83f5beb5826c/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L159]
 for that task.

 * As is noted in the [Javadocs for that 
method|https://github.com/apache/kafka/blob/8a24da376b801c6eb6522ad4861b83f5beb5826c/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L2348],
 “If no thread is blocking in a method which can throw 
{{org.apache.kafka.common.errors.WakeupException}}, the next call to such a 
method will raise it instead.”

 * It just so happens that, if the framework isn’t in the middle of a call to 
the consumer and then the task gets stopped, the next call the framework will 
make on the consumer may be to commit offsets, which will immediately throw a 
{{WakeupException}}.

 * Currently, the framework handles this by [immediately retrying the offset 
commit|https://github.com/apache/kafka/blob/8a24da376b801c6eb6522ad4861b83f5beb5826c/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L337-L339]
 until it either throws a different exception or succeeds, and then throwing 
the original {{WakeupException}}. Since this synchronous commit of offsets only 
occurs during task shutdown, it's unnecessary to throw the {{WakeupException}} 
back to the caller, and can cause alarming {{ERROR}}-level messages to get 
logged by the worker.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10239) The groupInstanceId field in DescribeGroup response should be ignorable

2020-07-06 Thread Jason Gustafson (Jira)

Jason Gustafson created KAFKA-10239:
---

 Summary: The groupInstanceId field in DescribeGroup response 
should be ignorable
 Key: KAFKA-10239
 URL: https://issues.apache.org/jira/browse/KAFKA-10239
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
 Fix For: 2.6.0


We noticed the following error in the logs in the handling of a DescribeGroup 
request:
```
org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write 
a non-default groupInstanceId at version 3
```
The problem is that the field is not marked as ignorable. So if the user is 
relying on static membership and uses an older AdminClient, they will see this 
error.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10238) kafka-console-producer null value

2020-07-06 Thread Ryan (Jira)

Ryan created KAFKA-10238:


 Summary: kafka-console-producer null value
 Key: KAFKA-10238
 URL: https://issues.apache.org/jira/browse/KAFKA-10238
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 2.5.0
Reporter: Ryan


It would be convenient if users could use the kafka-console-producer to create 
messages with a null value (tombstone messages).  Currently users must install 
additional software such as a Python API or Kafkacat.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Potential Improvement for Kafka Producer

2020-07-06 Thread Matthias J. Sax

Arvin,

thanks for your email. This is definitely the right channel. I am
personally not familiar enough with the producer code, but what you say
makes sense to me from a high level.

Maybe it would be best if you would file a Jira to improve the producer
accordingly? I guess, this change would require a KIP.

Of course, if you are interested, feel free to pick it up yourself.


-Matthias


On 6/28/20 8:53 AM, Arvin Zheng wrote:
> Hi All,
> 
> Not sure if this is the right channel and thread to ask, but would like to
> discuss a potential improvement to Java Kafka Producer.
> 
> ```
> Currently the Kafka Producer is able to identify unavailable partitions and
> avoid routing messages to them, but the definition of an unavailable
> partitions is - the leader of the partition is not available.
> From Producer point of view, acks for sending messages can be [all, -1, 0,
> 1]
> 1. When acks is set to either 0 or 1, leader availability is good enough to
> determine whether we should route messages to that partition.
> 2. When acks is set to -1 or all, leader available doesn't mean we are able
> to persist messages to that partition successfully, instead, we need to
> make sure
> a. leader is available.
> b. at least min.insync.replicas number of replicas are available
> ```
> 
> To achieve 2, what we need is to carry min.insync.replicas information of a
> topic to the metadata response, so that Producer is able to determine if it
> should route messages to that partition when there's no enough replicas
> available and it's acks is set to -1 or all.
> 
> Advantages that I can think of
> 1. Avoid exhausting the entire Producer cache when a partition is not
> available for a long time and
> a. retries is set to a large value
> b. acks is set to all
> 2. Avoid unnecessary network tries
> 
> Not sure if this is a valid case but would like to hear any opinions.
> 
> Br,
> Arvin
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk11 #1620

2020-07-06 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: rename class `RecordTimeDefintion` to `RecordTimeDefinition`


--
[...truncated 3.19 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

Re: [DISCUSS] KIP-406: GlobalStreamThread should honor custom reset policy

2020-07-06 Thread Matthias J. Sax

Atm, the config should be ignored and the global-consumer should use
"none" in a hard-coded way.

However, if am still wondering if we actually want/need to allow users
to specify the reset policy? It might be worth to consider, to just
change the behavior: catch the exception, log an ERROR (for information
purpose), wipe the store, seekToBeginning(), and recreate the store?

Btw: if we want to allow users to set the reset policy, this should be
possible via the config, or via overwriting the config in the method
itself. Thus, we would need to add the new overloaded method to
`Topology` and `StreamsBuilder`.

Another question to ask: what about GlobalKTables? Should they behave
the same? An alternative design could be, to allow users to specify a
flexible reset policy for global-stores, but not for GlobalKTables and
use the strategy suggested above for this case.

Thoughts?


-Matthias


On 7/2/20 2:14 PM, John Roesler wrote:
> Hi Navinder,
> 
> Thanks for the response. I’m sorry if I’m being dense... You said we are not 
> currently using the config, but I thought we would pass the config through to 
> the client.  Can you confirm whether or not the existing config works for 
> your use case?
> 
> Thanks,
> John
> 
> On Sun, Jun 28, 2020, at 14:09, Navinder Brar wrote:
>> Sorry my bad. Found it.
>>
>>
>>
>> Prefix used to override {@link KafkaConsumer consumer} configs for the 
>> global consumer client from
>>
>> * the general consumer client configs. The override precedence is the 
>> following (from highest to lowest precedence):
>> * 1. global.consumer.[config-name]..
>> public static final String GLOBAL_CONSUMER_PREFIX = "global.consumer.";
>>
>>
>>
>> So, that's great. We already have a config exposed to reset offsets for 
>> global topics via global.consumer.auto.offset.reset just that we are 
>> not actually using it inside GlobalStreamThread to reset.
>>
>> -Navinder
>> On Monday, 29 June, 2020, 12:24:21 am IST, Navinder Brar 
>>  wrote:  
>>  
>>  Hi John,
>>
>> Thanks for your feedback. 
>> 1. I think there is some confusion on my first point, the enum I am 
>> sure we can use the same one but the external config which controls the 
>> resetting in global stream thread either we can the same one which 
>> users use for source topics(StreamThread) or we can provide a new one 
>> which specifically controls global topics. For e.g. currently if I get 
>> an InvalidOffsetException in any of my source topics, I can choose 
>> whether to reset from Earliest or Latest(with auto.offset.reset). Now 
>> either we can use the same option and say if I get the same exception 
>> for global topics I will follow same resetting. Or some users might 
>> want to have totally different setting for both source and global 
>> topics, like for source topic I want resetting from Latest but for 
>> global topics I want resetting from Earliest so in that case adding a 
>> new config might be better.
>>
>> 2. I couldn't find this config currently 
>> "global.consumer.auto.offset.reset". Infact in GlobalStreamThread.java 
>> we are throwing a StreamsException for InvalidOffsetException and there 
>> is a test as 
>> well GlobalStreamThreadTest#shouldDieOnInvalidOffsetException(), so I 
>> think this is the config we are trying to introduce with this KIP.
>>
>> -Navinder  On Saturday, 27 June, 2020, 07:03:04 pm IST, John Roesler 
>>  wrote:  
>>  
>>  Hi Navinder,
>>
>> Thanks for this proposal!
>>
>> Regarding your question about whether to use the same policy
>> enum or not, the underlying mechanism is the same, so I think
>> we can just use the same AutoOffsetReset enum.
>>
>> Can you confirm whether setting the reset policy config on the
>> global consumer currently works or not? Based on my reading
>> of StreamsConfig, it looks like it would be:
>> "global.consumer.auto.offset.reset".
>>
>> If that does work, would you still propose to augment the
>> Java API?
>>
>> Thanks,
>> -John
>>
>> On Fri, Jun 26, 2020, at 23:52, Navinder Brar wrote:
>>> Hi,
>>>
>>> KIP: 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-406%3A+GlobalStreamThread+should+honor+custom+reset+policy
>>>
>>> I have taken over this KIP since it has been dormant for a long time 
>>> and this looks important for use-cases that have large global data, so 
>>> rebuilding global stores from scratch might seem overkill in case of 
>>> InvalidOffsetExecption.
>>>
>>> We want to give users the control to use reset policy(as we do in 
>>> StreamThread) in case they hit invalid offsets. I have still not 
>>> decided whether to restrict this option to the same reset policy being 
>>> used by StreamThread(using auto.offset.reset config) or add another 
>>> reset config specifically for global stores 
>>> "global.auto.offset.reset" which gives users more control to choose 
>>> separate policies for global and stream threads.
>>>
>>> I would like to hear your opinions on the KIP.
>>>
>>>
>>> -Navinder



signature.asc
Description: OpenPGP digital

[jira] [Resolved] (KAFKA-10173) BufferUnderflowException during Kafka Streams Upgrade

2020-07-06 Thread John Roesler (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-10173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Roesler resolved KAFKA-10173.
--
Fix Version/s: (was: 2.5.1)
   (was: 2.4.2)
   Resolution: Fixed

> BufferUnderflowException during Kafka Streams Upgrade
> -
>
> Key: KAFKA-10173
> URL: https://issues.apache.org/jira/browse/KAFKA-10173
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 2.5.0
>Reporter: Karsten Schnitter
>Assignee: John Roesler
>Priority: Blocker
>  Labels: suppress
> Fix For: 2.6.0
>
>
> I migrated a Kafka Streams application from version 2.3.1 to 2.5.0. I 
> followed the steps described in the upgrade guide and set the property 
> {{migrate.from=2.3}}. On my dev system with just one running instance I got 
> the following exception:
> {noformat}
> stream-thread [0-StreamThread-2] Encountered the following error during 
> processing:
> java.nio.BufferUnderflowException: null
>   at java.base/java.nio.HeapByteBuffer.get(Unknown Source)
>   at java.base/java.nio.ByteBuffer.get(Unknown Source)
>   at 
> org.apache.kafka.streams.state.internals.BufferValue.extractValue(BufferValue.java:94)
>   at 
> org.apache.kafka.streams.state.internals.BufferValue.deserialize(BufferValue.java:83)
>   at 
> org.apache.kafka.streams.state.internals.InMemoryTimeOrderedKeyValueBuffer.restoreBatch(InMemoryTimeOrderedKeyValueBuffer.java:368)
>   at 
> org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89)
>   at 
> org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:92)
>   at 
> org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:350)
>   at 
> org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:94)
>   at 
> org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:401)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:779)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670)
> {noformat}
> I figured out, that this problem only occurs for stores, where I use the 
> suppress feature. If I rename the changelog topics during the migration, the 
> problem will not occur. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

2020-07-06 Thread Matthias J. Sax

William,

thanks for the KIP. LGMT. Feel free to start a vote.


-Matthias


On 7/4/20 10:14 AM, John Roesler wrote:
> Hi Richard,
> 
> It’s good to hear from you!
> 
> Thanks for bringing up the wall-clock suppression feature. IIRC, someone 
> actually started a KIP discussion for it already, but I don’t think it went 
> to a vote. I don’t recall any technical impediment, just the lack of 
> availability to finish it up. Although there is some association, it would be 
> good to keep the KIPs separate.
> 
> Thanks,
> John
> 
> On Sat, Jul 4, 2020, at 10:05, Richard Yu wrote:
>> Hi all,
>>
>> This reminds me of a previous issue I think that we were discussing.
>> @John Roesler  I think you should remember this 
>> one.
>>
>> A while back, we were talking about having suppress operator emit 
>> records by wall-clock time instead of stream time.
>> If we are adding this, wouldn't that make it more feasible for us to 
>> implement that feature for suppression?
>>
>> If I recall correctly, there actually had been quite a bit of user 
>> demand for such a feature.
>> Might be good to include it in this KIP (or maybe use one of the prior 
>> KIPs for this feature).
>>
>> Best,
>> Richard
>>
>> On Sat, Jul 4, 2020 at 6:58 AM John Roesler  wrote:
>>> Hi all,
>>>
>>>  1. Thanks, Boyang, it is nice to see usage examples in KIPs like this. It 
>>> helps during the discussion, and it’s also good documentation later on. 
>>>
>>>  2. Yeah, this is a subtle point. The motivation mentions being able to 
>>> control the time during tests, but to be able to make it work, the 
>>> processor implementation needs a public method on ProcessorContext to get 
>>> the time. Otherwise, processors would have to check the type of the context 
>>> and cast, depending on whether they’re running inside a test or not. In 
>>> retrospect, if we’d had a usage example, this probably would have been 
>>> clear. 
>>>
>>>  3. I don’t think we expect people to have their own implementations of 
>>> ProcessorContext. Since all implementations are internal, it’s really an 
>>> implementation detail whether we use a default method, abstract methods, or 
>>> concrete methods. I can’t think of an implementation that really wants to 
>>> just look up the system time. In the production code path, we cache the 
>>> time for performance, and in testing, we use a mock time. 
>>>
>>>  Thanks,
>>>  John
>>>
>>>
>>>  On Fri, Jul 3, 2020, at 06:41, Piotr Smoliński wrote:
>>>  > 1. Makes sense; let me propose something
>>>  > 
>>>  > 2. That's not testing-only. The goal is to use the same API to access 
>>>  > the time
>>>  > in deployment and testing environments. The major driver is 
>>>  > System.currentTimeMillis(),
>>>  > which a) cannot be used in tests b) could go in specific cases back c) 
>>>  > is not compatible
>>>  > with punctuator call. The idea is that we could access clock using 
>>>  > uniform API. 
>>>  > For completness we should have same API for system and stream time.
>>>  > 
>>>  > 3. There aren't that many subclasses. Two important ones are 
>>>  > ProcessorContextImpl and 
>>>  > MockProcessorContext (and third one: 
>>>  > ForwardingDisableProcessorContext). If given
>>>  > implementation does not support schedule() call, there is no reason to 
>>>  > support clock access. 
>>>  > The default implementation should just throw 
>>>  > UnsupportedOperationException just to prevent
>>>  > from compilation errors in possible subclasses.
>>>  > 
>>>  > On 2020/07/01 02:24:43, Boyang Chen  wrote: 
>>>  > > Thanks Will for the KIP. A couple questions and suggestions:
>>>  > > 
>>>  > > 1. I think for new APIs to make most sense, we should add a minimal 
>>> example
>>>  > > demonstrating how it could be useful to structure unit tests w/o the 
>>> new
>>>  > > APIs.
>>>  > > 2. If this is a testing-only feature, could we only add it
>>>  > > to MockProcessorContext?
>>>  > > 3. Regarding the API, since this will be added to the ProcessorContext 
>>> with
>>>  > > many subclasses, does it make sense to provide default implementations 
>>> as
>>>  > > well?
>>>  > > 
>>>  > > Boyang
>>>  > > 
>>>  > > On Tue, Jun 30, 2020 at 6:56 PM William Bottrell 
>>>  > > wrote:
>>>  > > 
>>>  > > > Thanks, John! I made the change. How much longer should I let there 
>>> be
>>>  > > > discussion before starting a VOTE?
>>>  > > >
>>>  > > > On Sat, Jun 27, 2020 at 6:50 AM John Roesler  
>>> wrote:
>>>  > > >
>>>  > > > > Thanks, Will,
>>>  > > > >
>>>  > > > > That looks good to me. I would only add "cached" or something
>>>  > > > > to indicate that it wouldn't just transparently look up the current
>>>  > > > > System.currentTimeMillis every time.
>>>  > > > >
>>>  > > > > For example:
>>>  > > > > /**
>>>  > > > > * Returns current cached wall-clock system timestamp in 
>>> milliseconds.
>>>  > > > > *
>>>  > > > > * @return the current cached wall-clock system timestamp in 
>>> milliseconds
>>>  > > > > */
>>>  > > > > long

[jira] [Created] (KAFKA-10237) Properly handle in-memory stores OOM

2020-07-06 Thread Boyang Chen (Jira)

Boyang Chen created KAFKA-10237:
---

 Summary: Properly handle in-memory stores OOM
 Key: KAFKA-10237
 URL: https://issues.apache.org/jira/browse/KAFKA-10237
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Boyang Chen


We have seen the in-memory store buffered too much data and eventually get OOM. 
Generally speaking, OOM has no real indication of the underlying problem and 
increases the difficulty for user debugging, since the failed thread may not be 
the actual culprit which causes the explosion. If we could get better 
protection to avoid hitting memory limit, or at least giving out a clear guide, 
the end user debugging would be much simpler. 

To make it work, we need to enforce a certain memory limit below heap size and 
take actions  when hitting it. The first question would be, whether we set a 
numeric limit, such as 100MB or 500MB, or a percentile limit, such as 60% or 
80% of total memory.

The second question is about the action itself. One approach would be crashing 
the store immediately and inform the user to increase their application 
capacity. The second approach would be opening up an on-disk store 
spontaneously and offload the data to it.

Personally I'm in favor of approach #2 because it has minimum impact to the 
on-going application. However it is more complex and potentially requires 
significant works to define the proper behavior such as the default store 
configuration, how to manage its lifecycle, etc.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #4694

2020-07-06 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: rename class `RecordTimeDefintion` to `RecordTimeDefinition`


--
[...truncated 3.16 MB...]
org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord

Re: [VOTE] KIP-418: A method-chaining way to branch KStream

2020-07-06 Thread Matthias J. Sax

I am late, but I am also +1 (binding).

-Matthias

On 7/6/20 2:16 AM, Ivan Ponomarev wrote:
> Wow!
> 
> So excited to hear that!
> 
> Thanks for your collaboration, now it's my turn to write a PR.
> 
> Regards,
> 
> Ivan
> 
> 04.07.2020 20:16, John Roesler пишет:
>> Hi Ivan,
>>
>> Congratulations! It looks like you have 3 binding and 2 non-binding
>> votes, so you can announce this KIP as accepted and follow up with a PR.
>>
>> Thanks,
>> John
>>
>> On Mon, Jun 29, 2020, at 20:46, Bill Bejeck wrote:
>>> Thanks for the KIP Ivan, +1 (binding).
>>>
>>> -Bill
>>>
>>> On Mon, Jun 29, 2020 at 7:22 PM Guozhang Wang 
>>> wrote:
>>>
 +1 (binding). Thanks Ivan!


 Guozhang

 On Mon, Jun 29, 2020 at 3:55 AM Jorge Esteban Quilcate Otoya <
 quilcate.jo...@gmail.com> wrote:

> This will be a great addition. Thanks Ivan!
>
> +1 (non-binding)
>
> On Fri, Jun 26, 2020 at 7:07 PM John Roesler 
 wrote:
>
>> Thanks, Ivan!
>>
>> I’m +1 (binding)
>>
>> -John
>>
>> On Thu, May 28, 2020, at 17:24, Ivan Ponomarev wrote:
>>> Hello all!
>>>
>>> I'd like to start the vote for KIP-418 which proposes deprecation of
>>> current `branch` method and provides a method-chaining based API for
>>> branching.
>>>
>>>
>>
>
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-418%3A+A+method-chaining+way+to+branch+KStream

>>>
>>> Regards,
>>>
>>> Ivan
>>>
>>
>


 -- 
 -- Guozhang

>>>
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk14 #269

2020-07-06 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: rename class `RecordTimeDefintion` to `RecordTimeDefinition`


--
[...truncated 3.19 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest >

[jira] [Resolved] (KAFKA-9769) ReplicaManager Partition.makeFollower Increases LeaderEpoch when ZooKeeper disconnect occurs

2020-07-06 Thread Jun Rao (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-9769.

Fix Version/s: 2.7.0
 Assignee: Andrew Choi
   Resolution: Fixed

merged the PR to trunk

> ReplicaManager Partition.makeFollower Increases LeaderEpoch when ZooKeeper 
> disconnect occurs
> 
>
> Key: KAFKA-9769
> URL: https://issues.apache.org/jira/browse/KAFKA-9769
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Reporter: Andrew Choi
>Assignee: Andrew Choi
>Priority: Minor
>  Labels: kafka, replica, replication
> Fix For: 2.7.0
>
>
> The ZooKeeper Session once expired and got disconnected and the broker 
> received the 1st LeaderAndIsr request simultaneously. As the broker was 
> processing the 1st LeaderAndIsr Request, the ZooKeeper session has not been 
> reestablished just yet.
> Within the makeFollowers method, _partition.getOrCreateReplica_ is called 
> before the fetcher begins. _partition.getOrCreateReplica_ needs to fetch 
> information from ZooKeeper but an exception is thrown when calling the 
> ZooKeeper client because the session is invalid, rendering the fetcher start 
> to be skipped.
>  
> In Partition class's getOrCreateReplica method calls AdminZkClient's 
> fetchEntityConfig(..) which throws an exception if the ZooKeeper session is 
> invalid. 
>  
> {code:java}
> val props = adminZkClient.fetchEntityConfig(ConfigType.Topic, topic){code}
>  
> When this occurs, the leader epoch should not have been incremented due to 
> ZooKeeper being invalid because once the second LeaderAndIsr request comes 
> in, the leader epoch could be the same between the brokers. 
> Few options I can think of for a fix. I think third route could be feasible:
> 1 - Make LeaderEpoch update and fetch update atomic.
> 2 - Wait until all individual partitions are successful without problems then 
> process fetch.
> 3 - Catch the ZooKeeper exception in the caller code block 
> (ReplicaManager.makeFollowers) and simply do not touch the remaining 
> partitions to ensure that the batch of successful partitions up to that point 
> are updated and processed (fetch).
> 4 - Or make LeaderAndIsr request never arrive at the broker in case of 
> ZooKeeper disconnect, then that would be safe because it is already possible 
> for some replicas to receive the LeaderAndIsr later than the others. However, 
> in that case, the code need to make sure the controller will retry.
>  
> {code:java}
>  else if (requestLeaderEpoch > currentLeaderEpoch) {
>  // If the leader epoch is valid record the epoch of the controller that made 
> the leadership decision. 
> // This is useful while updating the isr to maintain the decision maker 
> controller's epoch in the zookeeper path 
> if (stateInfo.basePartitionState.replicas.contains(localBrokerId))   
> partitionState.put(partition, stateInfo)
> else 
>  
> def getOrCreateReplica(replicaId: Int, isNew: Boolean = false): Replica = {   
>allReplicasMap.getAndMaybePut(replicaId, { 
> if (isReplicaLocal(replicaId)) {
> val adminZkClient = new AdminZkClient(zkClient) val props = 
> adminZkClient.fetchEntityConfig(ConfigType.Topic, topic)
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-614: Add Prefix Scan support for State Stores

2020-07-06 Thread Sagar

Hi John,

Thanks, I have updated the KIP.

Thanks!
Sagar.

On Mon, Jul 6, 2020 at 12:00 AM John Roesler  wrote:

> Hi Sagar,
>
> Sorry for the ambiguity. You could just mention it in the Public
> Interfaces section. Or, if you want to be more specific, you can show it in
> the method definition snippet. I don’t think it matters, as long as it’s
> clearly stated, since it affects backward compatibility with existing store
> implementations.
>
> Thanks,
> John
>
> On Sun, Jul 5, 2020, at 11:25, Sagar wrote:
> > Hi John,
> >
> > Thank you! Question on the comment, where should I add the default
> > implementation? I guess that needs to be added in the Proposal Section of
> > the kIP.
> >
> > Thanks!
> > Sagar.
> >
> > On Sat, Jul 4, 2020 at 11:46 PM John Roesler 
> wrote:
> >
> > > Thanks Sagar,
> > >
> > > That looks good to me! The only minor comment I’d make is that I think
> the
> > > method declaration should have a default implementation that throws an
> > > UnsupportedOperationException, for source compatibility with existing
> state
> > > stores.
> > >
> > > Otherwise, as far as I’m concerned, I’m ready to vote.
> > >
> > > Thanks,
> > > John
> > >
> > > On Sat, Jul 4, 2020, at 12:19, Sagar wrote:
> > > > Hi John,
> > > >
> > > > I have updated the KIP with all the new changes we discussed in this
> > > > discussion thread.
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-614%3A+Add+Prefix+Scan+support+for+State+Stores
> > > >
> > > > Request you to go through the same.
> > > >
> > > > Thanks!
> > > > Sagar.
> > > >
> > > > On Tue, Jun 30, 2020 at 8:09 AM John Roesler 
> > > wrote:
> > > >
> > > > > Hi Sagar,
> > > > >
> > > > > That’s a good observation; yes, it should go in the
> > > ReadOnlyKeyValueStore
> > > > > interface.
> > > > >
> > > > > Thanks again for the great work,
> > > > > John
> > > > >
> > > > > On Sun, Jun 28, 2020, at 23:54, Sagar wrote:
> > > > > > Hi John,
> > > > > >
> > > > > > Thank you for the positive feedback! The meaningful discussions
> we
> > > had on
> > > > > > the mailing list helped me understand what needed to be done.
> > > > > >
> > > > > > I am definitely open to any further suggestions on this.
> > > > > >
> > > > > > Before I updated the KIP, I just had one question, is it fine to
> > > have it
> > > > > > for KeyValueStore or should I move it to ReadOnlyKeyValueStore
> where
> > > even
> > > > > > the range query resides?
> > > > > >
> > > > > > Regarding the 2 notes on UnsupportedOperationException and
> changing
> > > the
> > > > > > name to prefixScan, i will incorporate both of them into the KIP.
> > > > > >
> > > > > > Thanks!
> > > > > > Sagar.
> > > > > >
> > > > > > On Sun, Jun 28, 2020 at 11:55 PM John Roesler <
> vvcep...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Woah, this is great, Sagar!
> > > > > > >
> > > > > > > I think this API looks really good. I'm curious if anyone else
> has
> > > > > > > any concern. For my part, I think this will work just fine.
> People
> > > > > > > might face tricky bugs getting their key serde and their prefix
> > > > > > > serde "aligned", but I think the API makes it pretty obvious
> what
> > > > > > > has to happen to make this work. As long as the API isn't going
> > > > > > > to "trick" anyone by trying to abstract away things that can't
> be
> > > > > > > abstracted, this is the best we can do. In other words, I think
> > > > > > > your approach is ideal here.
> > > > > > >
> > > > > > > I also really appreciate that you took the time to do a full
> POC
> > > > > > > with end-to-end tests to show that the proposal is actually
> > > > > > > going to work.
> > > > > > >
> > > > > > > A couple of notes as you update the KIP:
> > > > > > >
> > > > > > > 1. I think that for "optional" state store features like this,
> we
> > > > > > > should add a default implementation to the interface that
> > > > > > > throws UnsupportedOperationException. That way,
> > > > > > > any existing store implementations won't fail to compile
> > > > > > > on the new version. And any store that just can't support
> > > > > > > a prefix scan would simply not override the method.
> > > > > > >
> > > > > > > 2. I think you meant "prefixScan", not "prefixSeek", since
> > > > > > > we're actually getting an iterator that only returns prefix-
> > > > > > > matching keys, as opposed to just seeking to that prefix.
> > > > > > >
> > > > > > > Thanks again for the work you've put into this. I look
> > > > > > > forward to reviewing the updated KIP.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -John
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Jun 28, 2020, at 12:17, Sagar wrote:
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > I took some time out and as we discussed, looked to implement
> > > these
> > > > > > > > changes. Most of these changes are for demonstrative purposes
> > > but I
> > > > > > > thought
> > > > > > > > I will share.
> > > > > > > >
> > > > > > > > I added the

[DISCUSS] Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events

2020-07-06 Thread Arvin Zheng

Hi everyone,

I would like to start the discussion for KIP-637
https://cwiki.apache.org/confluence/display/KAFKA/KIP-637%3A+Include+min.insync.replicas+in+MetadataResponse+to+make+Producer+smarter+in+partitioning+events

Looking forward to your feedback.

Thanks,
Arvin

Re: Feedback: Print schemaId using bin/kafka-dump-log.sh

2020-07-06 Thread Mohanraj Nagasamy

Do anyone have feedback on this? ☺

From: Mohanraj Nagasamy 
Date: Wednesday, July 1, 2020 at 6:29 PM
To: "dev@kafka.apache.org" 
Subject: Feedback: Print schemaId using bin/kafka-dump-log.sh

When I try to dump kafka logs for diagnosing or debugging a problem, It's handy 
to see if the kafka log message schemaId or not. If it has got, print the 
schemaId.

I'm soliciting feedback as to whether it is worth making this change to 
kafka-core codebase.

I'm new to the kafka-community - forgive me if this wasn't part of the process.

This is the change I made:

```
 core/src/main/scala/kafka/tools/DumpLogSegments.scala | 21 
+++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/core/src/main/scala/kafka/tools/DumpLogSegments.scala 
b/core/src/main/scala/kafka/tools/DumpLogSegments.scala
index 9e9546a92..a8750ac3d 100755
--- a/core/src/main/scala/kafka/tools/DumpLogSegments.scala
+++ b/core/src/main/scala/kafka/tools/DumpLogSegments.scala
@@ -35,6 +35,7 @@ object DumpLogSegments {

   // visible for testing
   private[tools] val RecordIndent = "|"
+  private val MAGIC_BYTE = 0x0

   def main(args: Array[String]): Unit = {
 val opts = new DumpLogSegmentsOptions(args)
@@ -277,8 +278,24 @@ object DumpLogSegments {
   }
 } else if (printContents) {
   val (key, payload) = parser.parse(record)
-  key.foreach(key => print(s" key: $key"))
-  payload.foreach(payload => print(s" payload: $payload"))
+  key.foreach(key => {
+val keyBuffer = record.key()
+if (keyBuffer.get() == MAGIC_BYTE) {
+  print(s" keySchemaId: ${keyBuffer.getInt} key: $key")
+}
+else {
+  print(s" key: $key")
+}
+  })
+
+  payload.foreach(payload => {
+val valueBuffer = record.value()
+if (valueBuffer.get() == MAGIC_BYTE) {
+  print(s" payloadSchemaId: ${valueBuffer.getInt} payload: 
$payload")
+} else {
+  print(s" payload: $payload")
+}
+  })
 }
 println()
   }
(END)
```

And this is how the output looks like:

```
$ bin/kafka-dump-log.sh --files 
data/kafka/logdir/avro_topic_test-0/.log --print-data-log

| offset: 50 CreateTime: 1593570942959 keysize: -1 valuesize: 16 sequence: -1 
headerKeys: [] payloadSchemaId: 1 payload:
TracRowe
baseOffset: 51 lastOffset: 51 count: 1 baseSequence: -1 lastSequence: -1 
producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false 
isControl: false position: 2918 CreateTime: 1593570958044 size: 101 magic: 2 
compresscodec: NONE crc: 1913155179 isvalid: true
| offset: 51 CreateTime: 1593570958044 keysize: 3 valuesize: 30 sequence: -1 
headerKeys: [] key: ... payloadSchemaId: 2 payload: .iRKoMVeoVVnTmQEuqwSTHZQ
baseOffset: 52 lastOffset: 52 count: 1 baseSequence: -1 lastSequence: -1 
producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false 
isControl: false position: 3019 CreateTime: 1593570969001 size: 84 magic: 2 
compresscodec: NONE crc: 2188466405 isvalid: true
```

-Mohan

Re: KIP-560 Discuss

2020-07-06 Thread Sang wn Lee

I'm sorry.
I just modified the KIP!

On 2020/03/07 20:09:47, "Matthias J. Sax"  wrote: 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> Thanks for the KIP Sang!
> 
> I have a couple of more comments about the wiki page:
> 
> (1) The "Public Interface" section should only list the new stuff. This
> KIP does not change anything with regard to the existing options
> `--input-topic` or `--intermediate-topic` and thus it's just "noise" to
> have them in this section. Only list the new option
> `allInputTopicsOption`.
> 
> (2) Don't post code, ie, the implementation of private methods. KIPs
> should only describe public interface changes.
> 
> (3) The KIP should describe that we intend to use
> `describeConsumerGroups` calls to discover the topic names -- atm, it's
> unclear from the KIP how the new feature actually works.
> 
> (4) If the new flag is used, we will discover input and intermediate
> topics. Hence, the name is miss leading. We could call it
> `--all-user-topics` and explain in the description that "user topics"
> are input and intermediate topics for this case (in general, also output
> topics are "user topics" but there is nothing to be done for output
> topics). Thoughts?
> 
> 
> - -Matthias
> 
> 
> On 1/27/20 6:35 AM, Sang wn Lee wrote:
> > thank you John Roesle
> >
> > It is a good idea "—all-input-topics"
> >
> > I agree with you
> >
> > I'll update right away
> >
> >
> > On 2020/01/24 14:14:17, "John Roesler" 
> > wrote:
> >
> >> Hi all, thanks for the explanation. I was also not sure how the
> >> kip would be possible to implement.
> >>
> >> No that it does seem plausible, my only feedback is that the
> >> command line option could align better with the existing one.
> >> That is, the existing option is called “—input-topics”, so it
> >> seems like the new one should be called “—all-input-topics”.
> >>
> >> Thanks, John
> >>
> >> On Fri, Jan 24, 2020, at 01:42, Boyang Chen wrote:
> >>> Thanks Sophie for the explanation! I read Sang's PR and
> >>> basically he did exactly what you proposed (check it here
> >>>  in case I'm
> >>> wrong).
> >>>
> >>> I think Sophie's response answers Gwen's question already,
> >>> while in the meantime for a KIP itself we are not required to
> >>> mention all the internal details about how to make the changes
> >>> happen (like how to actually get the external topics),
> >>> considering the change scope is pretty small as well. But
> >>> again, it would do no harm if we mention it inside Proposed
> >>> Change session specifically so that people won't get confused
> >>> about how.
> >>>
> >>>
> >>> On Thu, Jan 23, 2020 at 8:26 PM Sophie Blee-Goldman
> >>>  wrote:
> >>>
>  Hi all,
> 
>  I think what Gwen is trying to ask (correct me if I'm wrong)
>  is how we can infer which topics are associated with Streams
>  from the admin client's topic list. I agree that this
>  doesn't seem possible, since as she pointed out the topics
>  list (or even description) lacks the specific information we
>  need.
> 
>  What we could do instead is use the admin client's
>  `describeConsumerGroups` API to get the information on the
>  Streams app's consumer group specifically -- note that the
>  Streams application.id config is also used as the consumer
>  group id, so each app forms a group to read from the input
>  topics. We could compile a list of these topics just by
>  looking at each member's assignment (and even check for a
>  StreamsPartitionAssignor to verify that this is indeed a
>  Streams app group, if we're being paranoid).
> 
>  The reset tool actually already gets the consumer group
>  description, in order to validate there are no active
>  consumers in the group. We may as well grab the list of
>  topics from it while it's there. Or did you have something
>  else in mind?
> 
>  On Sat, Jan 18, 2020 at 6:17 PM Sang wn Lee
>   wrote:
> 
> > Thank you
> >
> > I understand you
> >
> > 1. admin client has topic list 2. applicationId can only
> > have one stream, so It won't be a problem! 3. For example,
> > --input-topic [reg] Allowing reg solves some inconvenience
> >
> >
> > On 2020/01/18 18:15:23, Gwen Shapira 
> > wrote:
> >> I am not sure I follow. Afaik:
> >>
> >> 1. Topics don't include client ID information 2. Even if
> >> you did, the same ID could be used for topics that are
> >> not
> > Kafka
> >> Streams input
> >>
> >> The regex idea sounds doable, but I'm not sure it solves
> >> much?
> >>
> >>
> >> On Sat, Jan 18, 2020, 7:12 AM Sang wn Lee
> >> 
>  wrote:
> >>
> >>> Thank you Gwen Shapira! We'll add a flag to clear all
> >>> topics by clientId It is ‘reset-all-external-topics’
> >>>
> >>> I also want to use regex on the input topic flag to
> >>> clear all
>

Re: KIP-560 Discuss

2020-07-06 Thread Sang wn Lee

I'm sorry.
I just modified the KIP.

On 2020/05/26 20:11:46, Boyang Chen  wrote: 
> Hey Sang, unfortunately we couldn't make it in 2.6. Do you still plan to
> work on this KIP?
> 
> On Thu, May 14, 2020 at 6:49 PM Boyang Chen 
> wrote:
> 
> > Hey Sang, seems this thread has been quiet, are you still working on this
> > KIP?
> >
> > On Sat, Mar 7, 2020 at 3:54 PM Matthias J. Sax 
> > wrote:
> >
> >> Thanks for the KIP Sang!
> >>
> >> I have a couple of more comments about the wiki page:
> >>
> >> (1) The "Public Interface" section should only list the new stuff. This
> >> KIP does not change anything with regard to the existing options
> >> `--input-topic` or `--intermediate-topic` and thus it's just "noise" to
> >> have them in this section. Only list the new option
> >> `allInputTopicsOption`.
> >>
> >> (2) Don't post code, ie, the implementation of private methods. KIPs
> >> should only describe public interface changes.
> >>
> >> (3) The KIP should describe that we intend to use
> >> `describeConsumerGroups` calls to discover the topic names -- atm, it's
> >> unclear from the KIP how the new feature actually works.
> >>
> >> (4) If the new flag is used, we will discover input and intermediate
> >> topics. Hence, the name is miss leading. We could call it
> >> `--all-user-topics` and explain in the description that "user topics"
> >> are input and intermediate topics for this case (in general, also output
> >> topics are "user topics" but there is nothing to be done for output
> >> topics). Thoughts?
> >>
> >>
> >> -Matthias
> >>
> >> On 1/27/20 6:35 AM, Sang wn Lee wrote:
> >> > thank you John Roesle
> >> >
> >> > It is a good idea
> >> > "—all-input-topics"
> >> >
> >> > I agree with you
> >> >
> >> > I'll update right away
> >> >
> >> >
> >> > On 2020/01/24 14:14:17, "John Roesler"  wrote:
> >> >> Hi all, thanks for the explanation. I was also not sure how the kip
> >> would be possible to implement.
> >> >>
> >> >> No that it does seem plausible, my only feedback is that the command
> >> line option could align better with the existing one. That is, the existing
> >> option is called “—input-topics”, so it seems like the new one should be
> >> called “—all-input-topics”.
> >> >>
> >> >> Thanks,
> >> >> John
> >> >>
> >> >> On Fri, Jan 24, 2020, at 01:42, Boyang Chen wrote:
> >> >>> Thanks Sophie for the explanation! I read Sang's PR and basically he
> >> did
> >> >>> exactly what you proposed (check it here
> >> >>>  in case I'm wrong).
> >> >>>
> >> >>> I think Sophie's response answers Gwen's question already, while in
> >> the
> >> >>> meantime for a KIP itself we are not required to mention all the
> >> internal
> >> >>> details about how to make the changes happen (like how to actually
> >> get the
> >> >>> external topics), considering the change scope is pretty small as
> >> well. But
> >> >>> again, it would do no harm if we mention it inside Proposed Change
> >> session
> >> >>> specifically so that people won't get confused about how.
> >> >>>
> >> >>>
> >> >>> On Thu, Jan 23, 2020 at 8:26 PM Sophie Blee-Goldman <
> >> sop...@confluent.io>
> >> >>> wrote:
> >> >>>
> >>  Hi all,
> >> 
> >>  I think what Gwen is trying to ask (correct me if I'm wrong) is how
> >> we can
> >>  infer which topics are associated with
> >>  Streams from the admin client's topic list. I agree that this
> >> doesn't seem
> >>  possible, since as she pointed out the
> >>  topics list (or even description) lacks the specific information we
> >> need.
> >> 
> >>  What we could do instead is use the admin client's
> >>  `describeConsumerGroups` API to get the information
> >>  on the Streams app's consumer group specifically -- note that the
> >> Streams
> >>  application.id config is also used
> >>  as the consumer group id, so each app forms a group to read from the
> >> input
> >>  topics. We could compile a list
> >>  of these topics just by looking at each member's assignment (and
> >> even check
> >>  for a StreamsPartitionAssignor
> >>  to verify that this is indeed a Streams app group, if we're being
> >>  paranoid).
> >> 
> >>  The reset tool actually already gets the consumer group description,
> >> in
> >>  order to validate there are no active
> >>  consumers in the group. We may as well grab the list of topics from
> >> it
> >>  while it's there. Or did you have something
> >>  else in mind?
> >> 
> >>  On Sat, Jan 18, 2020 at 6:17 PM Sang wn Lee 
> >> wrote:
> >> 
> >> > Thank you
> >> >
> >> > I understand you
> >> >
> >> > 1. admin client has topic list
> >> > 2. applicationId can only have one stream, so It won't be a problem!
> >> > 3. For example, --input-topic [reg]
> >> > Allowing reg solves some inconvenience
> >> >
> >> >
> >> > On 2020/01/18 18:15:23, Gwen Shapira  wrote:
> >> >> I am not sure

Re: [DISCUSS] KIP-639 Move nodeLevelSensor and storeLevelSensor methods from StreamsMetricsImpl to StreamsMetrics

2020-07-06 Thread Bruno Cadonna

Hi Mohamed,

Thank you for the KIP.

Comments regarding the KIP wiki:

1. In section "Public Interface", you should state what you want to
change in interface StreamsMetrics. In your case, you want to add two
methods. You can find a good example how to describe this in KIP-444
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams).

2. In section "Compatibility, Deprecation, and Migration Plan" you
should state if anything is needed to keep backward compatibility.
Since you just want to add two methods to the interface, nothing is
needed. You should describe that under that section.

Regarding the KIP content, I left some comments on the corresponding
Jira ticket.

Best,
Bruno

On Sun, Jul 5, 2020 at 3:48 AM Mohamed Chebbi  wrote:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-639%3A+Move+nodeLevelSensor+and+storeLevelSensor+methods+from+StreamsMetricsImpl+to+StreamsMetrics
>

[jira] [Created] (KAFKA-10236) Kafka Streams | onCommit interceptor with EOS enabled

2020-07-06 Thread Vinodhini (Jira)

Vinodhini created KAFKA-10236:
-

 Summary: Kafka Streams | onCommit interceptor with EOS enabled 
 Key: KAFKA-10236
 URL: https://issues.apache.org/jira/browse/KAFKA-10236
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.5.0
Reporter: Vinodhini


Coming from 
[https://stackoverflow.com/questions/62700075/is-there-a-way-to-get-committed-offset-in-eos-kafka-stream|https://stackoverflow.com/questions/62700075/is-there-a-way-to-get-committed-offset-in-eos-kafka-stream?]

 

*Background :*

Setting consumer interceptor to StreamsConfig will ensure that the 
interceptor(s) are called when messages are consumed/committed. Snippet from 
{{org.apache.kafka.clients.consumer.internals.ConsumerCoordinator#commitOffsetsSync}}

 

{{}}
{code:java}
 if (future.succeeded()) {
if (interceptors != null)
interceptors.onCommit(offsets);
return true;
}{code}
 

But the {{consumerInterceptor.onCommit()}} was never called even though I saw 
the offsets being committed at the source topic.

*Issue:*

I figured that it was because I was using kstreams with Exactly once processing 
guarantee enabled.

This was the logic at 
{{org.apache.kafka.streams.processor.internals.StreamTask#commit}}

 
{code:java}
if (this.eosEnabled) {
this.producer.sendOffsetsToTransaction(consumedOffsetsAndMetadata,
  this.applicationId);
this.producer.commitTransaction();
if (startNewTransaction) {
this.producer.beginTransaction();
}
} else {
this.consumer.commitSync(consumedOffsetsAndMetadata);
}
{code}
As you can see, {{consumer.commitSync}} which in turns calls the 
{{consumerCoordinator.commit}} which calls the {{interceptor.onCommit}}, never 
gets called. Because with eos enabled, it is the transaction api that gets 
invoked.

 

*Request* 

Provide a way to get committed offset from Interceptors for EOS enabled also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-418: A method-chaining way to branch KStream

2020-07-06 Thread Ivan Ponomarev


Wow!

So excited to hear that!

Thanks for your collaboration, now it's my turn to write a PR.

Regards,

Ivan

04.07.2020 20:16, John Roesler пишет:

Hi Ivan,

Congratulations! It looks like you have 3 binding and 2 non-binding votes, so 
you can announce this KIP as accepted and follow up with a PR.

Thanks,
John

On Mon, Jun 29, 2020, at 20:46, Bill Bejeck wrote:

Thanks for the KIP Ivan, +1 (binding).

-Bill

On Mon, Jun 29, 2020 at 7:22 PM Guozhang Wang  wrote:


+1 (binding). Thanks Ivan!


Guozhang

On Mon, Jun 29, 2020 at 3:55 AM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:


This will be a great addition. Thanks Ivan!

+1 (non-binding)

On Fri, Jun 26, 2020 at 7:07 PM John Roesler 

wrote:



Thanks, Ivan!

I’m +1 (binding)

-John

On Thu, May 28, 2020, at 17:24, Ivan Ponomarev wrote:

Hello all!

I'd like to start the vote for KIP-418 which proposes deprecation of
current `branch` method and provides a method-chaining based API for
branching.







https://cwiki.apache.org/confluence/display/KAFKA/KIP-418%3A+A+method-chaining+way+to+branch+KStream


Regards,

Ivan








--
-- Guozhang

[VOTE] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

2020-07-06 Thread Tom Bentley

Hi,

I'd like to start a vote on KIP-621 which is about deprecating methods in
DescribeLogDirsResult which leak internal classes, replacing them with
public API classes.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158862109

Thanks,

Tom

Re: Contribution Permission

2020-07-06 Thread Boyang Chen

Hey Jonas,

I have added you the permission in both JIRA and wiki. Happy contributing!

On Sun, Jul 5, 2020 at 11:12 PM Jonas Amslinger  wrote:

> Hi,
>
> I would like to start contributing as a developer and kindly ask to be
> added to the contributor list.
>
> Jira and Wiki username: puhlerblet
>
> Thanks and regards,
>
> Jonas

Contribution Permission

2020-07-06 Thread Jonas Amslinger

Hi,

I would like to start contributing as a developer and kindly ask to be added to 
the contributor list.

Jira and Wiki username: puhlerblet

Thanks and regards,

Jonas

42 matches

Mail list logo