[jira] [Created] (KAFKA-10513) Newly added partitions are not assigned to running static consumer

[Marlon Ou (Jira)]

Marlon Ou created KAFKA-10513:
-

 Summary: Newly added partitions are not assigned to running static 
consumer
 Key: KAFKA-10513
 URL: https://issues.apache.org/jira/browse/KAFKA-10513
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 2.6.0
Reporter: Marlon Ou


If consumers are polling messages from a certain topic with static membership 
and we add new partitions to this topic while the consumers are running, no 
partition reassignment is ever triggered (and hence messages published into the 
new partitions are never consumed). 

To reproduce, simply set group instance IDs on the consumers: 
{code:java}
props.setProperty("group.instance.id", instanceId);
{code}
And then while the static consumers are running, use Kafka's admin client to 
add more partitions to the topic:
{code:java}
adminClient.createPartitions(...)
{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

[Jun Rao]

Hi, Jose,

Thanks for the updated KIP. A few more comments below.

20. "Metadata Cache: The component that generates snapshots, reads
snapshots and reads logs for observer replicas of the topic partition
__cluster_metadata." It seems this is needed on every broker, not just
observers?

21. Our current convention is to use exclusive offset for naming
checkpoint files. For example, a producer snapshot file of 1234.snapshot
means that the file includes the producer state up to, but not including
offset 1234. So, we probably want to follow the same convention for the new
checkpoint file.

22. Snapshot Format: KIP-631 only defines the format for individual
records. It seems that we need to define the container format here. For
example, we need to store the length of each record. Also, does the
snapshot file need a CRC field?

23. Could we provide the default value for the new
configs controller.snapshot.minimum.records and max.replication.lag.ms.
Also, max.replication.lag.ms seems to just control the snapshot frequency
by time and not directly relate to replication. So, maybe it should be
called sth like controller.snapshot.minimum.interval.ms?

24. "Kafka allows the clients to delete records that are less than a given
offset by using the DeleteRecords RPC . Those requests will be validated
using the same logic enumerated above." Hmm, should we allow deleteRecord
on the metadata topic? If we do, does it trim the snapshot accordingly?

25. "The followers of the __cluster_metadata topic partition will
concurrently fetch the snapshot and replicated log. This means that
candidates with incomplete snapshots will send a vote request with a
LastOffsetEpoch of -1 and a LastOffset of -1 no matter the LEO of the
replicated log." My understanding is that a follower will either fetch from
the snapshot or the log, but not both at the same time. Could you explain
how the concurrent part works? Also, what's an incomplete snapshot?

26. FetchRequest:
26.1 Handling Fetch Request: I agree with Jason that SnapshotOffsetAndEpoch
already tells us the next offset to fetch. So, we don't need to
set NextOffsetAndEpoch in the response.
26.2 Is there a reason to rename LogStartOffset to LogBeginOffset? I am not
sure if they are truly identical semantically. For example, currently, the
follower moves it's logStartOffset based on the leader's. Will we do the
same thing with LogBeginOffset?

27. FetchSnapshotRequest: It seems that SnapshotOffsetAndEpoch shouldn't be
optional. Also, its version number 12 is incorrect.

28. FetchSnapshotResponse: Do we need the position field? It seems it's the
same as in the request.

29. "OFFSET_OUT_OF_RANGE - when the fetch snapshot request’s offset is
greater than the size of the snapshot." By offset, do you mean position?

30. It's possible for a broker to die while copying the snapshot file from
the leader or saving its locally generated snapshot. On restart, how does
the broker know whether a local snapshot file is complete or not?

Thanks,

Jun

On Fri, Sep 18, 2020 at 1:38 PM Jason Gustafson  wrote:

> Hi Jose,
>
> A few comments/questions below:
>
> 1. There is a comment in the proposal which suggests that we will maintain
> multiple snapshots:
>
> > Having multiple snapshots is useful for minimizing re-fetching of the
> snapshot when a new snapshot is generated.
>
> However, the document later says that snapshots get deleted as the LBO
> advances. Just wanted to clarify the intent. Will we generally only have
> one snapshot?
>
> 2. The proposal says the following:
>
> > During leader election, followers with incomplete or missing snapshot
> will send a vote request and response as if they had an empty log.
>
> Maybe you can help me understand the scenario we're talking about since I'm
> not sure I understand the point of this. If the intent is to not allow such
> a follower to become leader, why would it ever become a candidate? On the
> other hand, if the intent is to still allow it to become leader in some
> disaster scenario, then why would it not use its latest log state? For
> inbound Vote requests, I think it should definitely still consider its
> latest log state when deciding whether to grant a vote.
>
> 3. Are we overloading `replica.fetch.max.bytes` for snapshot fetches as
> well? It looks like we are specifying this at the partition level, but it
> might be more useful to track the maximum bytes at the request level. On a
> related note, it might be useful to think through heuristics for balancing
> between the requests in a partition. Unlike fetches, it seems like we'd
> want to complete snapshot loading partition by partition. I wonder if it
> would be simpler for FetchSnapshot to handle just one partition.
>
> 4. It would help if the document motivated the need to track the snapshot
> epoch. Since we are only snapshotting below the high watermark, are you
> thinking about recovering from data loss scenarios?
>
> 5. Might need to fix the following:
>
> > Otherwise, the leader will 

[jira] [Created] (KAFKA-10512) JmxTool Can Crash on Unmarshall Error

[Matthew Wong (Jira)]

Matthew Wong created KAFKA-10512:


 Summary: JmxTool Can Crash on Unmarshall Error
 Key: KAFKA-10512
 URL: https://issues.apache.org/jira/browse/KAFKA-10512
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 2.6.0
Reporter: Matthew Wong


JmxTool can potentially crash from errors when querying for MBean objects. The 
errors can be caused by MBean objects that have attributes which can't be 
serialized.

When querying for all metrics, if the tool encounters such nonserializable 
MBean attributes, the tool will crash without outputting any metrics. Instead, 
the tool should print an error message and filter out the problematic objects, 
proceeding to print all other metrics.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: ApacheCon Bug Bash

[Sohil Shah]

Hi Tom,

I would love to participate. You can scan the AIPlatform here
https://github.com/bugsbunnyshah/AIPlatform

Thanks
Sohil


On Tue, Sep 22, 2020 at 6:11 PM Tom DuBuisson  wrote:

> Kafka Developers,
>
>
>
>
>
>
>
> As part of our sponsorship of ApacheCon, our company MuseDev is doing a Bug
>
> Bash for select Apache projects. We'll bring members of the ApacheCon
>
> community together to find and fix a range of security and performance bugs
>
> during the conference, and gameify the experience with teams, a
>
> leaderboard, and prizes. The bash is open to everyone whether attending the
>
> conference or not, and our whole dev team will also be participating to
>
> help fix as many bugs as we can.
>
>
>
>
>
>
>
> We're seeding the bug list with results from Muse, our code analysis
>
> platform, which runs as a Github App and comments on possible bugs as part
>
> of the pull request workflow.  Here's an example of what it looks like:
>
>
>
> https://github.com/curl/curl/pull/5971#discussion_r490252196
>
> 
>
>
>
>
>
>
>
> We explored a number of Apache projects and are reaching out because our
>
> analysis through Muse found some interesting bugs that could be fixed
>
> during the Bash.
>
>
>
>
>
>
>
> We're writing to see if you'd be interested in having your project included
>
> in the Bash. Everything is set up on our end, and if you're interested, we
>
> would need you to say yes on this listserv, and we’ll work with the Apache
>
> Infrastructure team to grant Muse access to your Github mirror. We'll then
>
> make sure it's all set-up and ready for the Bash. And of course, everyone
>
> on the project is most welcome to join the Bash and help us smash some
> bugs.
>
>
>
>
>
> -Tom
>
>

Jenkins build is back to normal : Kafka » kafka-2.6-jdk8 #17

[Apache Jenkins Server]

See 

ApacheCon Bug Bash

[Tom DuBuisson]

Kafka Developers,



As part of our sponsorship of ApacheCon, our company MuseDev is doing a Bug
Bash for select Apache projects. We'll bring members of the ApacheCon
community together to find and fix a range of security and performance bugs
during the conference, and gameify the experience with teams, a
leaderboard, and prizes. The bash is open to everyone whether attending the
conference or not, and our whole dev team will also be participating to
help fix as many bugs as we can.



We're seeding the bug list with results from Muse, our code analysis
platform, which runs as a Github App and comments on possible bugs as part
of the pull request workflow.  Here's an example of what it looks like:

https://github.com/curl/curl/pull/5971#discussion_r490252196




We explored a number of Apache projects and are reaching out because our
analysis through Muse found some interesting bugs that could be fixed
during the Bash.



We're writing to see if you'd be interested in having your project included
in the Bash. Everything is set up on our end, and if you're interested, we
would need you to say yes on this listserv, and we’ll work with the Apache
Infrastructure team to grant Muse access to your Github mirror. We'll then
make sure it's all set-up and ready for the Bash. And of course, everyone
on the project is most welcome to join the Bash and help us smash some bugs.


-Tom

Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #80

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-10511) Fix minor behavior difference in `MockLog`

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-10511:
---

 Summary: Fix minor behavior difference in `MockLog`
 Key: KAFKA-10511
 URL: https://issues.apache.org/jira/browse/KAFKA-10511
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jason Gustafson
Assignee: Jason Gustafson


Fix minor difference in the implementation of the epoch cache in MockLog. In 
`LeaderEpochFileCache`, we ensure new entries increase both start offset and 
epoch monotonically. We also do not allow duplicates.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #80

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Fix log message when tasks directory is cleaned manually (#9262)

[github] KAFKA-10492; Core Kafka Raft Implementation (KIP-595) (#9130)


--
[...truncated 3.34 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #79

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Fix log message when tasks directory is cleaned manually (#9262)

[github] KAFKA-10492; Core Kafka Raft Implementation (KIP-595) (#9130)


--
[...truncated 3.32 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

Jenkins build is back to normal : Kafka » kafka-trunk-jdk15 #79

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-10492) Core Raft implementation

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-10492.
-
Resolution: Fixed

> Core Raft implementation
> 
>
> Key: KAFKA-10492
> URL: https://issues.apache.org/jira/browse/KAFKA-10492
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
>
> This issue tracks the core implementation of the Raft protocol specified in 
> KIP-595: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : Kafka » kafka-trunk-jdk8 #78

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #79

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10068: add task assignment performance tests (#8892)


--
[...truncated 6.59 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher 

Re: [DISCUSS] Apache Kafka 2.7.0 release

[Bill Bejeck]

Boyan,

Done. Thanks for the heads up.

-Bill

On Mon, Sep 21, 2020 at 6:36 PM Boyang Chen 
wrote:

> Hey Bill,
>
> unfortunately KIP-590 will not be in 2.7 release, could you move it to
> postponed KIPs?
>
> Best,
> Boyang
>
> On Thu, Sep 10, 2020 at 2:41 PM Bill Bejeck  wrote:
>
> > Hi Gary,
> >
> > It's been added.
> >
> > Regards,
> > Bill
> >
> > On Thu, Sep 10, 2020 at 4:14 PM Gary Russell 
> wrote:
> >
> > > Can someone add a link to the release plan page [1] to the Future
> > Releases
> > > page [2]?
> > >
> > > I have the latter bookmarked.
> > >
> > > Thanks.
> > >
> > > [1]:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158872629
> > > [2]:
> > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
> > > 
> > > From: Bill Bejeck 
> > > Sent: Wednesday, September 9, 2020 4:35 PM
> > > To: dev 
> > > Subject: Re: [DISCUSS] Apache Kafka 2.7.0 release
> > >
> > > Hi Dongjin,
> > >
> > > I've moved both KIPs to the release plan.
> > >
> > > Keep in mind the cutoff for KIP acceptance is September 30th. If the
> KIP
> > > discussions are completed, I'd recommend starting a vote for them.
> > >
> > > Regards,
> > > Bill
> > >
> > > On Wed, Sep 9, 2020 at 8:39 AM Dongjin Lee  wrote:
> > >
> > > > Hi Bill,
> > > >
> > > > Could you add the following KIPs to the plan?
> > > >
> > > > - KIP-508: Make Suppression State Queriable
> > > > <
> > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-508%253A%2BMake%2BSuppression%2BState%2BQueriabledata=02%7C01%7Cgrussell%40vmware.com%7Cf9f4193557084c2f746508d854ffe7d2%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637352805378806436sdata=CkJill9%2FuBqp2HdVQrIjElj2z1nMgQXRaUyWrvY94dk%3Dreserved=0
> > > > >
> > > > - KIP-653: Upgrade log4j to log4j2
> > > > <
> > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-653%253A%2BUpgrade%2Blog4j%2Bto%2Blog4j2data=02%7C01%7Cgrussell%40vmware.com%7Cf9f4193557084c2f746508d854ffe7d2%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637352805378806436sdata=nHbw6WiQpkWT3KgPfanEtDCh3sWcL0O%2By8Fu0Bl4ivc%3Dreserved=0
> > > > >
> > > >
> > > > Both KIPs are completely implemented with passing all tests, but not
> > got
> > > > reviewed by the committers. Could anyone have a look?
> > > >
> > > > Thanks,
> > > > Dongjin
> > > >
> > > > On Wed, Sep 9, 2020 at 8:38 AM Leah Thomas 
> > wrote:
> > > >
> > > > > Hi Bill,
> > > > >
> > > > > Could you also add KIP-450 to the release plan? It's been merged.
> > > > >
> > > > >
> > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-450%253A%2BSliding%2BWindow%2BAggregations%2Bin%2Bthe%2BDSLdata=02%7C01%7Cgrussell%40vmware.com%7Cf9f4193557084c2f746508d854ffe7d2%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637352805378806436sdata=1KbAPyL7NKWQSZWBKItUTpAJF5SY6%2FMCj8Rn%2Fw2qO20%3Dreserved=0
> > > > >
> > > > > Cheers,
> > > > > Leah
> > > > >
> > > > > On Tue, Sep 8, 2020 at 9:32 AM Bill Bejeck 
> > wrote:
> > > > >
> > > > > > Hi Bruno,
> > > > > >
> > > > > > Thanks for letting me know, I've added KIP-662 to the release
> plan.
> > > > > >
> > > > > > -Bill
> > > > > >
> > > > > > On Mon, Sep 7, 2020 at 11:33 AM Bruno Cadonna <
> br...@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Bill,
> > > > > > >
> > > > > > > Could you add KIP-662 [1] to the release plan. The KIP has been
> > > > already
> > > > > > > implemented.
> > > > > > >
> > > > > > > Best,
> > > > > > > Bruno
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-662%253A%2BThrow%2BException%2Bwhen%2BSource%2BTopics%2Bof%2Ba%2BStreams%2BApp%2Bare%2BDeleteddata=02%7C01%7Cgrussell%40vmware.com%7Cf9f4193557084c2f746508d854ffe7d2%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637352805378806436sdata=cxbyU9BJuJkM2JJ6yqfr5dHXrg7Mfr1%2BOKxCy%2FJQiCw%3Dreserved=0
> > > > > > >
> > > > > > > On 26.08.20 16:54, Bill Bejeck wrote:
> > > > > > > > Greetings All!
> > > > > > > >
> > > > > > > > I've published a release plan at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D158872629data=02%7C01%7Cgrussell%40vmware.com%7Cf9f4193557084c2f746508d854ffe7d2%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637352805378806436sdata=iXp0nfkEK5gcuIJcIaZQthfnPR9cJ%2F8x2vBpQR942zA%3Dreserved=0
> > > > > > > .
> > > > > > > > I have included all of the KIPs that are currently approved,
> > but
> > > > I'm
> > > > > > > happy
> > > > > > > > to make any adjustments as necessary.
> > > > > > > >

Re: [DISCUSS] KIP-653: Upgrade log4j to log4j2

[Dongjin Lee]

Hi devs,

I updated the KIP with the migration plan I discussed with Ismael.

I think 2.7.0 is the perfect time for starting migration into log4j2. If we
miss this opportunity, the migration would be much harder. So please have a
look at this proposal.

I also opened a voting thread for this.

Thanks,
Dongjin

On Thu, Sep 17, 2020 at 2:29 AM Dongjin Lee  wrote:

> Hi Ismael,
>
> > Have we considered switching to the log4j2 logging config format by
> default and providing a mechanism to use the old format?
>
> As of present, the proposal leaves the default config format switching to
> sometime in the future. However, I think it is not a difficult task and is
> up to the community's decision. The draft implementation already includes
> log4j2 counterparts for all existing 1.x format (template) configs.
> Although it still uses traditional log4j format as a default for backward
> compatibility, the users who prefer the log4j2 configs can use it by
> setting `export
> KAFKA_LOG4J_OPTS="-Dlog4j.configurationFile={log4j2-config-file-path}"`.
> Whenever we change the default logging format, we must don't forget to
> switch this functionality to the reverse, i.e., making log4j 1.x format
> available as an opt-in.
>
> I am so concerned about the community's opinion when would be adequate to
> make the log4j2 config as default.
>
> Thanks,
> Dongjin
>
> +1. As a note, I have an almost-completed implementation of log4j2
> equivalent for the log4j-appender. I think it would be great if this
> feature can be provided with changing the default logging config format.
>
> On Wed, Sep 16, 2020 at 11:49 PM Ismael Juma  wrote:
>
>> Thanks for the KIP, Dongjin. Have we considered switching to the log4j2
>> logging config format by default and providing a mechanism to use the old
>> format? It is likely that we will release 3.0 as the release after 2.7, so
>> it would provide a good opportunity to move on from the legacy config
>> format. The other option is to stick with the old format for 3.0 and
>> migrate to the new format in 4.0.
>>
>> Ismael
>>
>> On Wed, Aug 5, 2020 at 7:45 AM Dongjin Lee  wrote:
>>
>> > Hi, Kafka dev,
>> >
>> > I hope to initiate the discussion of KIP-653, upgrading log4j to log4j2.
>> >
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
>> >
>> > All kinds of feedbacks are greatly appreciated!
>> >
>> > Best,
>> > Dongjin
>> >
>> > --
>> > *Dongjin Lee*
>> >
>> > *A hitchhiker in the mathematical world.*
>> >
>> >
>> >
>> >
>> > *github:  github.com/dongjinleekr
>> > keybase:
>> https://keybase.io/dongjinleekr
>> > linkedin:
>> kr.linkedin.com/in/dongjinleekr
>> > speakerdeck:
>> > speakerdeck.com/dongjin
>> > *
>> >
>>
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
> *
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*




*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

[VOTE] KIP-653: Upgrade log4j to log4j2

[Dongjin Lee]

Hi devs,

Here I open the vote for KIP-653: Upgrade log4j to log4j2. It replaces the
obsolete log4j logging library into the current standard, log4j2, with
maintaining backward-compatibility.

Thanks,
Dongjin

-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*




*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

[jira] [Resolved] (KAFKA-10507) Limit the set of APIs returned in pre-authentication ApiVersions

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-10507.
-
Resolution: Won't Do

> Limit the set of APIs returned in pre-authentication ApiVersions 
> -
>
> Key: KAFKA-10507
> URL: https://issues.apache.org/jira/browse/KAFKA-10507
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: David Jacot
>Priority: Major
>
> We use the ApiVersions RPC to check whether the SaslHandshake and 
> SaslAuthenticate APIs are supported before authenticating with the broker. 
> Currently the response contains all APIs supported by the broker. It seems 
> like a good idea to reduce the set of APIs returned at this level to only 
> those which are supported prior to authentication. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] KIP-516: Topic Identifiers

[Justine Olshan]

Hi all,

I'd like to call a vote on KIP-516: Topic Identifiers. Here is the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers

The discussion thread is here:
https://lists.apache.org/thread.html/7efa8cd169cadc7dc9cf86a7c0dbbab1836ddb5024d310fcebacf80c@%3Cdev.kafka.apache.org%3E

Please take a look and vote if you have a chance.

Thank you,
Justine

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #77

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10459: Document IQ APIs where order does not hold between stores 
(#9251)


--
[...truncated 3.26 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowNoSuchElementExceptionForUnusedOutputTopicWithDynamicRouting[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowNoSuchElementExceptionForUnusedOutputTopicWithDynamicRouting[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 

Re: [VOTE] KIP-584: Versioning scheme for features

[Kowshik Prakasam]

Hi all,

I wanted to let you know that I have made the following changes to the
KIP-584 write up. The purpose is to ensure the design is correct for a few
things which came up during implementation:

1. Per FeatureUpdate error code: The UPDATE_FEATURES controller API is no
longer transactional. Going forward, we allow for individual FeatureUpdate
to succeed/fail in the request. As a result, the response schema now
contains an error code per FeatureUpdate as well as a top-level error code.
Overall this is a better design because it better represents the nature of
the API: each FeatureUpdate in the request is independent of the other
updates, and the controller can process/apply these independently to ZK.
When an UPDATE_FEATURES request fails, this new design provides better
clarity to the caller on which FeatureUpdate could not be applied (via the
individual error codes). In the previous design, we were unable to achieve
such an increased level of clarity in communicating the error codes.

2. Due to #1, there were some minor changes required to the proposed Admin
APIs (describeFeatures and updateFeatures). A few unnecessary public APIs
have been removed, and couple essential ones have been added. The latest
changes now represent the latest design.

3. The timeoutMs field has been removed from the the UPDATE_FEATURES API
request, since it was not found to be required during implementation.

4. Previously we handled the incompatible broker lifetime race condition in
the controller by skipping sending of UpdateMetadataRequest to the
incompatible broker. But this had a few edge cases. Instead, now we handle
it by refusing to register the incompatible broker in the controller. This
is a better design because if we already acted on an incompatible broker
registration, then some damage may already be done to the cluster. This is
because the UpdatateMetadataRequest will still be sent to other brokers and
its metadata will be available to the clients. Therefore we would like to
avoid this problem with the new design where the controller would not keep
track of an incompatible broker because the broker will eventually shutdown
automatically (when reacting to the incompatibility).

Please let me know if you have any questions.


Cheers,
Kowshik


On Mon, Jun 8, 2020 at 3:32 AM Kowshik Prakasam 
wrote:

> Hi all,
>
> I wanted to let you know that I have made the following minor changes to
> the KIP-584  write up. The
> purpose is to ensure the design is correct for a few things which came up
> during implementation:
>
> 1. Feature version data type has been made to be int16 (instead of int64).
> The reason is two fold:
> a. Usage of int64 felt overkill. Feature version bumps are infrequent
> (since these bumps represent breaking changes that are generally
> infrequent). Therefore int16 is big enough to support version bumps of a
> particular feature.
> b. The int16 data type aligns well with existing API versions data
> type. Please see the file
> '/clients/src/main/resources/common/message/ApiVersionsResponse.json'.
>
> 2. Finalized feature version epoch data type has been made to be int32
> (instead of int64). The reason is that the epoch value is the value of ZK
> node version, whose data type is int32.
>
> 3. Introduced a new 'status' field in the '/features' ZK node schema. The
> purpose is to implement Colin's earlier point for the strategy for
> transitioning from not having a /features znode to having one. An
> explanation has been provided in the following section of the KIP detailing
> the different cases:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP584:Versioningschemeforfeatures-FeatureZKnodestatus
> .
>
> Please let me know if you have any questions or concerns.
>
>
> Cheers,
> Kowshik
>
>
>
> Cheers,
> Kowshik
>
> On Tue, Apr 28, 2020 at 11:24 PM Kowshik Prakasam 
> wrote:
>
>> Hi all,
>>
>> This KIP vote has been open for ~12 days. The summary of the votes is
>> that we have 3 binding votes (Colin, Guozhang, Jun), and 3 non-binding
>> votes (David, Dhruvil, Boyang). Therefore, the KIP vote passes. I'll mark
>> KIP as accepted and start working on the implementation.
>>
>> Thanks a lot!
>>
>>
>> Cheers,
>> Kowshik
>>
>> On Mon, Apr 27, 2020 at 12:15 PM Colin McCabe  wrote:
>>
>>> Thanks, Kowshik.  +1 (binding)
>>>
>>> best,
>>> Colin
>>>
>>> On Sat, Apr 25, 2020, at 13:20, Kowshik Prakasam wrote:
>>> > Hi Colin,
>>> >
>>> > Thanks for the explanation! I agree with you, and I have updated the
>>> > KIP.
>>> > Here is a link to relevant section:
>>> >
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP-584:Versioningschemeforfeatures-Controller:ZKnodebootstrapwithdefaultvalues
>>> >
>>> >
>>> > Cheers,
>>> > Kowshik
>>> >
>>> > On Fri, Apr 24, 2020 at 8:50 PM Colin McCabe 
>>> wrote:
>>> >
>>> > > On Fri, Apr 24, 2020, at 00:01, Kowshik Prakasam wrote:

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

[Bruno Cadonna]

Walker,

I am sorry, but I still have a comment on the KIP although you have 
already started voting.


What happens when a consumer of the group skips the rebalancing that 
propagates the shutdown request? Do you give a guarantee that all Kafka 
Streams clients are shutdown or is it best effort? If it is best effort, 
I guess the proposed method might not be used in critical cases where 
stopping record consumption may prevent or limit damage. I am not saying 
that it must be a guarantee, but this question should be answered in the 
KIP, IMO.


Best,
Bruno

On 22.09.20 01:14, Walker Carlson wrote:

The error code right now is the assignor error, 2 is coded for shutdown
but it could be expanded to encode the causes or for other errors that need
to be communicated. For example we can add error code 3 to close the thread
but leave the client in an error state if we choose to do so in the future.

On Mon, Sep 21, 2020 at 3:43 PM Boyang Chen 
wrote:


Thanks for the KIP Walker.

In the KIP we mentioned "In order to communicate the shutdown request from
one client to the others we propose to update the SubcriptionInfoData to
include a short field which will encode an error code.", is there a
dedicated error code that we should define here, or it is case-by-case?

On Mon, Sep 21, 2020 at 1:38 PM Walker Carlson 
wrote:


I am changing the name to "Add method to Shutdown entire Streams
Application" since we are no longer using an Exception, it seems more
appropriate.

Also it looks like the discussion is pretty much finished so I will be
calling it to a vote.

thanks,
Walker

On Sat, Sep 19, 2020 at 7:49 PM Guozhang Wang 

wrote:



Sounds good to me. I also feel that this call should be non-blocking

but

I

guess I was confused from the discussion thread that the API is

designed

in

a blocking fashion which contradicts with my perspective and hence I

asked

for clarification :)

Guozhang


On Wed, Sep 16, 2020 at 12:32 PM Walker Carlson 


wrote:


Hello Guozhang,

As for the logging I plan on having three logs. First, the client log

that

it is requesting an application shutdown, second, the leader log

processId

of the invoker, third, then the StreamRebalanceListener it logs that

it

is

closing because of an `stream.appShutdown`. Hopefully this will be

enough

to make the cause of the close clear.

I see what you mean about the name being dependent on the behavior of

the

method so I will try to clarify.  This is how I currently envision

the

call

working.

It is not an option to directly initiate a shutdown through a

StreamThread

object from a KafkaStreams object because "KafkaConsumer is not safe

for

multi-threaded access". So how it works is that the method in

KafkaStreams

finds the first alive thread and sets a flag in the StreamThread. The
StreamThread will receive the flag in its runloop then set the error

code

and trigger a rebalance, afterwards it will stop processing. After

the

KafkaStreams has set the flag it will return true and continue

running.

If

there are no alive threads the shutdown will fail and return false.

What do you think the blocking behavior should be? I think that the
StreamThread should definitely stop to prevent any of the corruption

we

are

trying to avoid by shutting down, but I don't see any advantage of

the

KafkaStreams call blocking.

You are correct to be concerned about the uncaught exception handler.

If

there are no live StreamThreads the rebalance will not be started at

all

and this would be a problem. However the user should be aware of this
because of the return of false and react appropriately. This would

also

be

fixed if we implemented our own handler so we can rebalance before

the

StreamThread closes.

With that in mind I believe that `initiateClosingAllClients` would be

an

appropriate name. WDYT?

Walker


On Wed, Sep 16, 2020 at 11:43 AM Guozhang Wang 

wrote:



Hello Walker,

Thanks for the updated KIP. Previously I'm also a bit hesitant on

the

newly

added public exception to communicate user-requested whole app

shutdown,

but the reason I did not bring this up is that I feel there's

still a

need

from operational aspects that we can differentiate the scenario

where

an

instance is closed because of a) local `streams.close()` triggered,

or

b) a

remote instance's `stream.shutdownApp` triggered. So if we are

going

to

remove that exception (which I'm also in favor), we should at least
differentiate from the log4j levels.

Regarding the semantics that "It should wait to receive the

shutdown

request in the rebalance it triggers." I'm not sure I fully

understand,

since this may be triggered from the stream thread's uncaught

exception

handler, if that thread is already dead then maybe a rebalance

listener

would not even be fired at all. Although I know this is some

implementation

details that you probably abstract away from the proposal, I'd like

to

make

sure that we are on the same page regarding its 

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #78

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10459: Document IQ APIs where order does not hold between stores 
(#9251)


--
[...truncated 3.30 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

[Dániel Urbán]

An example I had in mind was the ProduceResponse - the auditor might need
access to the new end offset of the partitions.
The event-based approach sounds good - new events and fields can be added
on-demand. Do we need the same versioning strategy we use with the
requests/responses?

Daniel

Viktor Somogyi-Vass  ezt írta (időpont: 2020.
szept. 21., H, 14:08):

> Hi Daniel,
>
> > If the auditor needs access to the details of the action, one could argue
> that even the response should be passed down to the auditor.
> At this point I don't think we need to include responses into the interface
> but if you have a use-case we can consider doing that.
>
> > Is it feasible to convert the Java requests and responses to public API?
> Well I think that in this case we would need to actually transform a lot of
> classes and that might be a bit too invasive. Although since the protocol
> itself *is* a public API it might make sense to have some kind of Java
> representation as a public API as well.
>
> > If not, do we have another option to access this info in the auditor?
> I think one option would be to do what the original KIP-567 was
> implemented. Basically we could have an AuditEvent interface that would
> contain request specific data. Its obvious drawback is that it has to be
> implemented for most of the 40 something protocols but on the upside these
> classes shouldn't be complicated. I can try to do a PoC with this to see
> how it looks like and whether it solves the problem. To be honest I think
> it would be better than publishing the request classes as an API because
> here we're restricting access to only what is necessary.
>
> Thanks,
> Viktor
>
>
>
> On Fri, Sep 18, 2020 at 8:37 AM Dániel Urbán 
> wrote:
>
> > Hi,
> >
> > Thanks for the KIP.
> >
> > If the auditor needs access to the details of the action, one could argue
> > that even the response should be passed down to the auditor.
> > Is it feasible to convert the Java requests and responses to public API?
> > If not, do we have another option to access this info in the auditor?
> > I know that the auditor could just send proper requests through the API
> to
> > the brokers, but that seems like an awful lot of overhead, and could
> > introduce timing issues as well.
> >
> > Daniel
> >
> >
> > Viktor Somogyi-Vass  ezt írta (időpont: 2020.
> > szept. 16., Sze, 17:17):
> >
> > > One more after-thought on your second point (AbstractRequest): the
> > reason I
> > > introduced it in the first place was that this way implementers can
> > access
> > > request data. A use case can be if they want to audit a change in
> > > configuration or client quotas but not just acknowledge the fact that
> > such
> > > an event happened but also capture the change itself by peeking into
> the
> > > request. Sometimes it can be useful especially when people want to
> trace
> > > back the order of events and what happened when and not just
> acknowledge
> > > that there was an event of a certain kind. I also recognize that this
> > might
> > > be a very loose interpretation of auditing as it's not strictly related
> > to
> > > authorization but rather a way of tracing the admin actions within the
> > > cluster. It even could be a different API therefore but because of the
> > > variety of the Kafka APIs it's very hard to give a method that fits
> all,
> > so
> > > it's easier to pass down the AbstractRequest and the implementation can
> > do
> > > the extraction of valuable info. So that's why I added this in the
> first
> > > place and I'm interested in your thoughts.
> > >
> > > On Wed, Sep 16, 2020 at 4:41 PM Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com>
> > > wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for reviewing the KIP.
> > > >
> > > > 1.) I just wanted to follow the conventions used with the Authorizer
> as
> > > it
> > > > is built in a similar fashion, although it's true that in KafkaServer
> > we
> > > > call the configure() method and the start() in the next line. This
> > would
> > > be
> > > > the same in Auditor and even simpler as there aren't any parameters
> to
> > > > start(), so I can remove it. If it turns out there is a need for it,
> we
> > > can
> > > > add it later.
> > > >
> > > > 2.) Yes, this is a very good point, I will remove it, however in this
> > > case
> > > > I don't think we need to add the ApiKey as it is already available in
> > > > AuthorizableRequestContext.requestType(). One less parameter :).
> > > >
> > > > 3.) I'll add it. It will simply log important changes in the cluster
> > like
> > > > topic events (create, update, delete, partition or replication factor
> > > > change), ACL events, config changes, reassignment, altering log dirs,
> > > > offset delete, group delete with the authorization info like who
> > > initiated
> > > > the call, was it authorized, were there any errors. Let me know if
> you
> > > > think there are other APIs I should include.
> > > >
> > > > 4.) The builder is there mostly for