Re: [DISCUSS] Brokers disconnect intermittently with TLS1.3

[Luke Chen]

Hi Shylaja,
Thanks for reporting the issue.
> Given that TLS1.3 does not support renegotiation, can I make it
applicable just for TLS1.2?
Are you saying you're trying to make Kafka default supports to TLS1.2,
instead of TLS1.3?
If so, I don't think it's a good idea to fall back to an older and weaker
security protocol just because of a bug.
Instead, I think we should try to investigate it and fix it from the root.

So, are you sure this is a issue that `renegotiation` is not supported by
TLSv1.3?
Could we fix it?

Thank you.
Luke

On Tue, Nov 16, 2021 at 4:05 AM Kokoori, Shylaja 
wrote:

> Hi all,
>
> Using TLS1.3 (with JDK11) is causing an intermittent increase in
> inter-broker p99 latency, as mentioned by Yiming in Kafka-9320<
> https://issues.apache.org/jira/browse/KAFKA-9320?focusedCommentId=17401818=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17401818>.
> We tested this with Kafka 2.8.
> The issue seems to be because of a renegotiation exception being thrown by
>
> read(ByteBuffer dst)
>
> &
>
> write(ByteBuffer src)
>
> in
>
> clients/src/main/java/org/apache/kafka/common/network/SslTransportLayer.java
>
> This exception is causing the connection to close between the brokers
> before read/write is completed.
>
> In our internal experiments we have seen the p99 latency stabilize when we
> remove this exception.
>
> Given that TLS1.3 does not support renegotiation, can I make it applicable
> just for TLS1.2?
>
> I have also created a ticket<
> https://issues.apache.org/jira/browse/KAFKA-13418>
>
> Any feedback is welcome.
>
> Thank you,
>
> Shylaja
>
>
>
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #571

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 501119 lines...]
[2021-11-16T06:06:54.525Z] LeaderElectionTest > 
testLeaderElectionWithStaleControllerEpoch() STARTED
[2021-11-16T06:06:57.833Z] 
[2021-11-16T06:06:57.833Z] LeaderElectionTest > 
testLeaderElectionWithStaleControllerEpoch() PASSED
[2021-11-16T06:06:57.833Z] 
[2021-11-16T06:06:57.833Z] LeaderElectionTest > testLeaderElectionAndEpoch() 
STARTED
[2021-11-16T06:07:06.364Z] 
[2021-11-16T06:07:06.364Z] LeaderElectionTest > testLeaderElectionAndEpoch() 
PASSED
[2021-11-16T06:07:06.364Z] 
[2021-11-16T06:07:06.364Z] KafkaMetricReporterClusterIdTest > 
testClusterIdPresent() STARTED
[2021-11-16T06:07:08.345Z] 
[2021-11-16T06:07:08.345Z] KafkaMetricReporterClusterIdTest > 
testClusterIdPresent() PASSED
[2021-11-16T06:07:09.610Z] 
[2021-11-16T06:07:09.610Z] LogRecoveryTest > 
testHWCheckpointNoFailuresMultipleLogSegments() STARTED
[2021-11-16T06:07:14.986Z] 
[2021-11-16T06:07:14.986Z] LogRecoveryTest > 
testHWCheckpointNoFailuresMultipleLogSegments() PASSED
[2021-11-16T06:07:14.986Z] 
[2021-11-16T06:07:14.986Z] LogRecoveryTest > 
testHWCheckpointWithFailuresMultipleLogSegments() STARTED
[2021-11-16T06:07:28.873Z] 
[2021-11-16T06:07:28.874Z] LogRecoveryTest > 
testHWCheckpointWithFailuresMultipleLogSegments() PASSED
[2021-11-16T06:07:28.874Z] 
[2021-11-16T06:07:28.874Z] LogRecoveryTest > 
testHWCheckpointNoFailuresSingleLogSegment() STARTED
[2021-11-16T06:07:34.586Z] 
[2021-11-16T06:07:34.586Z] LogRecoveryTest > 
testHWCheckpointNoFailuresSingleLogSegment() PASSED
[2021-11-16T06:07:34.586Z] 
[2021-11-16T06:07:34.586Z] LogRecoveryTest > 
testHWCheckpointWithFailuresSingleLogSegment() STARTED
[2021-11-16T06:07:47.222Z] 
[2021-11-16T06:07:47.222Z] LogRecoveryTest > 
testHWCheckpointWithFailuresSingleLogSegment() PASSED
[2021-11-16T06:07:47.222Z] 
[2021-11-16T06:07:47.222Z] ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithLeaderThrottle() STARTED
[2021-11-16T06:08:05.133Z] 
[2021-11-16T06:08:05.133Z] ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithLeaderThrottle() PASSED
[2021-11-16T06:08:05.133Z] 
[2021-11-16T06:08:05.133Z] ReplicationQuotasTest > shouldThrottleOldSegments() 
STARTED
[2021-11-16T06:08:15.891Z] 
[2021-11-16T06:08:15.891Z] ReplicationQuotasTest > shouldThrottleOldSegments() 
PASSED
[2021-11-16T06:08:15.891Z] 
[2021-11-16T06:08:15.891Z] ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle() STARTED
[2021-11-16T06:08:38.545Z] 
[2021-11-16T06:08:38.545Z] ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle() PASSED
[2021-11-16T06:08:38.545Z] 
[2021-11-16T06:08:38.545Z] DeleteTopicsRequestWithDeletionDisabledTest > 
testDeleteRecordsRequest() STARTED
[2021-11-16T06:08:39.659Z] 
[2021-11-16T06:08:39.659Z] DeleteTopicsRequestWithDeletionDisabledTest > 
testDeleteRecordsRequest() PASSED
[2021-11-16T06:08:39.659Z] 
[2021-11-16T06:08:39.659Z] FetchRequestWithLegacyMessageFormatTest > 
testFetchRequestV2WithOversizedMessage() STARTED
[2021-11-16T06:08:43.947Z] 
[2021-11-16T06:08:43.947Z] FetchRequestWithLegacyMessageFormatTest > 
testFetchRequestV2WithOversizedMessage() PASSED
[2021-11-16T06:08:43.947Z] 
[2021-11-16T06:08:43.947Z] DynamicConfigChangeTest > 
testMessageFormatVersionChange() STARTED
[2021-11-16T06:08:46.044Z] 
[2021-11-16T06:08:46.044Z] DynamicConfigChangeTest > 
testMessageFormatVersionChange() PASSED
[2021-11-16T06:08:46.044Z] 
[2021-11-16T06:08:46.044Z] DynamicConfigChangeTest > testProcessNotification() 
STARTED
[2021-11-16T06:08:48.309Z] 
[2021-11-16T06:08:48.309Z] DynamicConfigChangeTest > testProcessNotification() 
PASSED
[2021-11-16T06:08:48.309Z] 
[2021-11-16T06:08:48.309Z] DynamicConfigChangeTest > 
shouldParseWildcardReplicationQuotaProperties() STARTED
[2021-11-16T06:08:50.664Z] 
[2021-11-16T06:08:50.664Z] DynamicConfigChangeTest > 
shouldParseWildcardReplicationQuotaProperties() PASSED
[2021-11-16T06:08:50.664Z] 
[2021-11-16T06:08:50.664Z] DynamicConfigChangeTest > 
testDefaultClientIdQuotaConfigChange() STARTED
[2021-11-16T06:08:52.647Z] 
[2021-11-16T06:08:52.647Z] DynamicConfigChangeTest > 
testDefaultClientIdQuotaConfigChange() PASSED
[2021-11-16T06:08:52.647Z] 
[2021-11-16T06:08:52.647Z] DynamicConfigChangeTest > testQuotaInitialization() 
STARTED
[2021-11-16T06:08:57.256Z] 
[2021-11-16T06:08:57.256Z] DynamicConfigChangeTest > testQuotaInitialization() 
PASSED
[2021-11-16T06:08:57.256Z] 
[2021-11-16T06:08:57.256Z] DynamicConfigChangeTest > 
testUserQuotaConfigChange() STARTED
[2021-11-16T06:08:59.330Z] 
[2021-11-16T06:08:59.330Z] DynamicConfigChangeTest > 
testUserQuotaConfigChange() PASSED
[2021-11-16T06:08:59.330Z] 
[2021-11-16T06:08:59.330Z] DynamicConfigChangeTest > 
testClientIdQuotaConfigChange() STARTED
[2021-11-16T06:09:02.406Z] 
[2021-11-16T06:09:02.406Z] DynamicConfigChangeTest > 
testClientIdQuotaConfigChange() PASSED
[2021-11-16T06:09:02.406Z] 

Re: [DISCUSS] KIP-796: Interactive Query v2

[John Roesler]

Thanks for the review, Guozhang!

1. This is a great point. I fell into the age-old trap of
only considering the simplest store type and forgot about
this extra wrinkle of the "key schema" that we use in
Windowed and Session stores.

Depending on how we want to forge forward with our provided
queries, I think it can still work out ok. The simplest
solution is just to have windowed versions of our queries
for use on the windowed stores. That should work naively
because we're basically just preserving the existing
interactions. It might not be ideal in the long run, but at
least it lets us make IQv2 orthogonal from other efforts to
simplify the stores themselves.

If we do that, then it would actually be correct to go ahead
and just return the serdes that are present in the Metered
stores today. For example, if I have a Windowed store with
Integer keys, then the key serde I get from serdesForStore
would just be the IntegerSerde. The query I'd use the
serialized key with would be a RawWindowedKeyQuery, which
takes a byte[] key and a timestamp. Then, the low-level
store (the segmented store in this case) would have to take
the next step to use its schema before making that last-mile
query. Note, this is precisely how fetch is implemented
today in RocksDBWindowStore:

public byte[] fetch(final Bytes key, final long timestamp) {
  return wrapped().get(WindowKeySchema.toStoreKeyBinary(key,
timestamp, seqnum));
}

In other words, if we set up our provided Query types to
stick close to the current store query methods, then
everything "should work out" (tm).

I think where things start to get more complicated would be
if we wanted to expose the actual, raw, on-disk binary key
to the user, along with a serde that can interpret it. Then,
we would have to pack up the serde and the schema. If we go
down that road, then knowing which one (the key serde or the
windowed schema + the key serde) the user wants when they
ask for "the serde" would be a challenge.

I'm actually thinking maybe we don't need to include the
serdesForStore method in this proposal, but instead leave it
for follow-on work (if desired) to add it along with raw
queries, since it's really only needed if you want raw
queries and (as you mentioned later) there may be better
ways to present the serdes, which is always easier to figure
out once there's a use case.


2. Hmm, if I understand what you mean by the "formatted"
layer, is that the one supplied by the
WindowedBytesStoreSupplier or SessionBytesStoreSupplier in
Materialized? I think the basic idea of this proposal is to
let whatever store gets supplied there be the "last stop"
for the query.

For the case of our default windowed store, this is the
segmented RocksDB store. It's true that this store "wraps" a
bunch of segments, but it would be the segmented store's
responsibility to handle the query, not defer to the
segments. This might mean different things for different
queries, but naively, I think it can just invoke to the
current implementation of each of its methods.

There might be future queries that require more
sophisticated responses, but we should be able to add new
queries for those, which have no restrictions on their
response types. For example, we could choose to respond to a
scan with a list of iterators, one for each segment.


3. I agree the large switch (or if/else) (or Map) for query
dispatch is a concern. That's the thing I'm most worried
will become cumbersome. I think your idea is neat, though,
because a lot of our surface area is providing a bunch of
those different combinations of query attributes. I think if
we get a little meta, we can actually fold it into the
existing KIP.

Rather than making Query any more restrictive, what we could
do is to choose to follow your idea for the provided queries
we ship with Streams. Although I had been thinking we would
ship a KeyQuery, RangeQuery, etc., we could absolutely
compactify those queries as much as possible so that there
are only a few queries with those dimensions you listed.

That way we can avoid blowing up the query space with our
own provided queries, but we can still keep the framework as
general as possible.

4. I'm not sure, actually! I just thought it would be neat
to have. I know I've spent my fair share of adding println
statements to Streams or stepping though the debugger when
something like that proposal would have done the job.

So, I guess the answer is yes, I was just thinking of it as
a debugging/informational tool. I also think that if we want
to make it more structured in the future, we should be able
to evolve that part of the API without and major problems.


5. That's another great point, and it's a miss on my part.
The short answer is that we'd simply throw whatever runtime
exceptions are appropriate, but I should and will document
what they will be.


6. I do think those APIs need some attention, but I was
actually hoping to treat that as a separate scope for design
work later. I think that there 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #90

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #570

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 498005 lines...]
[2021-11-16T03:58:29.848Z] [INFO] Parameter: packageInPathFormat, Value: myapps
[2021-11-16T03:58:29.848Z] [INFO] Parameter: package, Value: myapps
[2021-11-16T03:58:29.848Z] [INFO] Parameter: version, Value: 0.1
[2021-11-16T03:58:29.848Z] [INFO] Parameter: groupId, Value: streams.examples
[2021-11-16T03:58:29.848Z] [INFO] Parameter: artifactId, Value: streams.examples
[2021-11-16T03:58:29.848Z] [INFO] Project created from Archetype in dir: 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples
[2021-11-16T03:58:29.848Z] [INFO] 

[2021-11-16T03:58:29.848Z] [INFO] BUILD SUCCESS
[2021-11-16T03:58:29.848Z] [INFO] 

[2021-11-16T03:58:29.848Z] [INFO] Total time:  1.368 s
[2021-11-16T03:58:29.848Z] [INFO] Finished at: 2021-11-16T03:58:29Z
[2021-11-16T03:58:29.848Z] [INFO] 

[Pipeline] dir
[2021-11-16T03:58:30.373Z] Running in 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples
[Pipeline] {
[Pipeline] sh
[2021-11-16T03:58:33.350Z] + mvn compile
[2021-11-16T03:58:34.395Z] [INFO] Scanning for projects...
[2021-11-16T03:58:34.395Z] [INFO] 
[2021-11-16T03:58:34.395Z] [INFO] -< 
streams.examples:streams.examples >--
[2021-11-16T03:58:34.395Z] [INFO] Building Kafka Streams Quickstart :: Java 0.1
[2021-11-16T03:58:34.395Z] [INFO] [ jar 
]-
[2021-11-16T03:58:34.395Z] [INFO] 
[2021-11-16T03:58:34.395Z] [INFO] --- maven-resources-plugin:2.6:resources 
(default-resources) @ streams.examples ---
[2021-11-16T03:58:34.395Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-11-16T03:58:34.395Z] [INFO] Copying 1 resource
[2021-11-16T03:58:34.395Z] [INFO] 
[2021-11-16T03:58:34.395Z] [INFO] --- maven-compiler-plugin:3.1:compile 
(default-compile) @ streams.examples ---
[2021-11-16T03:58:34.720Z] 
[2021-11-16T03:58:34.720Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsExportImportPlan() PASSED
[2021-11-16T03:58:34.720Z] 
[2021-11-16T03:58:34.720Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsToSpecificOffset() STARTED
[2021-11-16T03:58:35.439Z] [INFO] Changes detected - recompiling the module!
[2021-11-16T03:58:35.439Z] [INFO] Compiling 3 source files to 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples/target/classes
[2021-11-16T03:58:35.962Z] [INFO] 

[2021-11-16T03:58:35.962Z] [INFO] BUILD SUCCESS
[2021-11-16T03:58:35.962Z] [INFO] 

[2021-11-16T03:58:35.962Z] [INFO] Total time:  1.887 s
[2021-11-16T03:58:35.962Z] [INFO] Finished at: 2021-11-16T03:58:35Z
[2021-11-16T03:58:35.962Z] [INFO] 

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2021-11-16T03:58:45.036Z] 
[2021-11-16T03:58:45.036Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsToSpecificOffset() PASSED
[2021-11-16T03:58:45.036Z] 
[2021-11-16T03:58:45.036Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftPlus() STARTED
[2021-11-16T03:58:56.756Z] 
[2021-11-16T03:58:56.756Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftPlus() PASSED
[2021-11-16T03:58:56.756Z] 
[2021-11-16T03:58:56.756Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsToLatest() STARTED
[2021-11-16T03:59:06.422Z] 
[2021-11-16T03:59:06.422Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsToLatest() PASSED
[2021-11-16T03:59:06.422Z] 
[2021-11-16T03:59:06.422Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest() STARTED
[2021-11-16T03:59:14.605Z] 
[2021-11-16T03:59:14.606Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest() PASSED
[2021-11-16T03:59:14.606Z] 
[2021-11-16T03:59:14.606Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsExistingTopicAllGroups() STARTED
[2021-11-16T03:59:35.190Z] 
[2021-11-16T03:59:35.190Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsExistingTopicAllGroups() PASSED
[2021-11-16T03:59:35.190Z] 
[2021-11-16T03:59:35.190Z] ResetConsumerGroupOffsetTest > 
testResetOffsetsByDuration() STARTED
[2021-11-16T03:59:43.984Z] 
[2021-11-16T03:59:43.984Z] 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #569

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 500377 lines...]
[2021-11-16T01:51:40.353Z] 
[2021-11-16T01:51:40.353Z] PlaintextConsumerTest > 
testConsumeMessagesWithCreateTime() PASSED
[2021-11-16T01:51:40.353Z] 
[2021-11-16T01:51:40.353Z] PlaintextConsumerTest > testAsyncCommit() STARTED
[2021-11-16T01:51:44.623Z] 
[2021-11-16T01:51:44.623Z] PlaintextConsumerTest > testAsyncCommit() PASSED
[2021-11-16T01:51:44.623Z] 
[2021-11-16T01:51:44.623Z] PlaintextConsumerTest > 
testLowMaxFetchSizeForRequestAndPartition() STARTED
[2021-11-16T01:52:18.180Z] 
[2021-11-16T01:52:18.180Z] PlaintextConsumerTest > 
testLowMaxFetchSizeForRequestAndPartition() PASSED
[2021-11-16T01:52:18.180Z] 
[2021-11-16T01:52:18.180Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnStopPolling() STARTED
[2021-11-16T01:52:37.033Z] 
[2021-11-16T01:52:37.033Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnStopPolling() PASSED
[2021-11-16T01:52:37.033Z] 
[2021-11-16T01:52:37.033Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInRevocation() STARTED
[2021-11-16T01:52:42.773Z] 
[2021-11-16T01:52:42.773Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInRevocation() PASSED
[2021-11-16T01:52:42.773Z] 
[2021-11-16T01:52:42.773Z] PlaintextConsumerTest > 
testPerPartitionLagMetricsCleanUpWithAssign() STARTED
[2021-11-16T01:52:48.542Z] 
[2021-11-16T01:52:48.542Z] PlaintextConsumerTest > 
testPerPartitionLagMetricsCleanUpWithAssign() PASSED
[2021-11-16T01:52:48.542Z] 
[2021-11-16T01:52:48.543Z] PlaintextConsumerTest > 
testPartitionsForInvalidTopic() STARTED
[2021-11-16T01:52:51.935Z] 
[2021-11-16T01:52:51.935Z] PlaintextConsumerTest > 
testPartitionsForInvalidTopic() PASSED
[2021-11-16T01:52:51.935Z] 
[2021-11-16T01:52:51.935Z] PlaintextConsumerTest > 
testPauseStateNotPreservedByRebalance() STARTED
[2021-11-16T01:52:56.398Z] 
[2021-11-16T01:52:56.398Z] PlaintextConsumerTest > 
testPauseStateNotPreservedByRebalance() PASSED
[2021-11-16T01:52:56.398Z] 
[2021-11-16T01:52:56.398Z] PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst() STARTED
[2021-11-16T01:53:02.169Z] 
[2021-11-16T01:53:02.169Z] PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst() PASSED
[2021-11-16T01:53:02.169Z] 
[2021-11-16T01:53:02.169Z] PlaintextConsumerTest > testSeek() STARTED
[2021-11-16T01:53:09.570Z] 
[2021-11-16T01:53:09.570Z] PlaintextConsumerTest > testSeek() PASSED
[2021-11-16T01:53:09.570Z] 
[2021-11-16T01:53:09.570Z] PlaintextConsumerTest > 
testConsumingWithNullGroupId() STARTED
[2021-11-16T01:53:18.129Z] 
[2021-11-16T01:53:18.129Z] PlaintextConsumerTest > 
testConsumingWithNullGroupId() PASSED
[2021-11-16T01:53:18.129Z] 
[2021-11-16T01:53:18.129Z] PlaintextConsumerTest > testPositionAndCommit() 
STARTED
[2021-11-16T01:53:23.606Z] 
[2021-11-16T01:53:23.606Z] PlaintextConsumerTest > testPositionAndCommit() 
PASSED
[2021-11-16T01:53:23.606Z] 
[2021-11-16T01:53:23.606Z] PlaintextConsumerTest > 
testFetchRecordLargerThanMaxPartitionFetchBytes() STARTED
[2021-11-16T01:53:27.838Z] 
[2021-11-16T01:53:27.838Z] PlaintextConsumerTest > 
testFetchRecordLargerThanMaxPartitionFetchBytes() PASSED
[2021-11-16T01:53:27.838Z] 
[2021-11-16T01:53:27.838Z] PlaintextConsumerTest > testUnsubscribeTopic() 
STARTED
[2021-11-16T01:53:33.574Z] 
[2021-11-16T01:53:33.574Z] PlaintextConsumerTest > testUnsubscribeTopic() PASSED
[2021-11-16T01:53:33.574Z] 
[2021-11-16T01:53:33.574Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnClose() STARTED
[2021-11-16T01:53:45.672Z] 
[2021-11-16T01:53:45.672Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnClose() PASSED
[2021-11-16T01:53:45.672Z] 
[2021-11-16T01:53:45.672Z] PlaintextConsumerTest > 
testMultiConsumerStickyAssignor() STARTED
[2021-11-16T01:54:19.622Z] 
[2021-11-16T01:54:19.622Z] PlaintextConsumerTest > 
testMultiConsumerStickyAssignor() PASSED
[2021-11-16T01:54:19.622Z] 
[2021-11-16T01:54:19.622Z] PlaintextConsumerTest > 
testFetchRecordLargerThanFetchMaxBytes() STARTED
[2021-11-16T01:54:20.765Z] 
[2021-11-16T01:54:20.765Z] PlaintextConsumerTest > 
testFetchRecordLargerThanFetchMaxBytes() PASSED
[2021-11-16T01:54:20.765Z] 
[2021-11-16T01:54:20.765Z] PlaintextConsumerTest > testAutoCommitOnClose() 
STARTED
[2021-11-16T01:54:26.610Z] 
[2021-11-16T01:54:26.610Z] PlaintextConsumerTest > testAutoCommitOnClose() 
PASSED
[2021-11-16T01:54:26.610Z] 
[2021-11-16T01:54:26.610Z] PlaintextConsumerTest > testListTopics() STARTED
[2021-11-16T01:54:30.967Z] 
[2021-11-16T01:54:30.967Z] PlaintextConsumerTest > testListTopics() PASSED
[2021-11-16T01:54:30.967Z] 
[2021-11-16T01:54:30.967Z] PlaintextConsumerTest > 
testExpandingTopicSubscriptions() STARTED
[2021-11-16T01:54:37.299Z] 
[2021-11-16T01:54:37.299Z] PlaintextConsumerTest > 
testExpandingTopicSubscriptions() PASSED
[2021-11-16T01:54:37.299Z] 
[2021-11-16T01:54:37.299Z] PlaintextConsumerTest > 

Re: [DISCUSS] KIP-796: Interactive Query v2

[Guozhang Wang]

Hello John,

Great, great, great writeup! :) And thank you for bringing this up finally.
I have made a pass on the KIP as well as the POC PR of it, here are some
initial thoughts:

First are some meta ones:

1. Today the serdes do not only happen at the metered-store layer,
unfortunately. For windowed / sessioned stores, and also some newly added
ones for stream-stream joins that are optimized for time-based range
queries, for example, the serdes are actually composite at multiple layers.
And the queries on the outer interface are also translated with serde
wrapped / stripped along the way in layers. To be more specific, today our
store hierarchy is like this:

metered * -> cached -> logged * -> formatted * (e.g. segmenged,
list-valued) -> inner (rocksdb, in-memory)

and serdes today could happen on the layers with * above, where each layer
is stuffing a bit more as prefix/suffix into the query bytes. This is not
really by design or ideal, but a result of history accumulated tech debts..
There's a related JIRA ticket for it:
https://issues.apache.org/jira/browse/KAFKA-13286. I guess my point is that
we need to be a bit careful regarding how to implement the
`KafkaStreams#serdesForStore(storeName)`, as we may expect some bumpy roads
moving forward.

2. Related to 1 above, I think we cannot always delegate the `query()`
implementation to the `inner` store layer, since some serde, or even some
computation logic happens at the outer, especially the `formatted` layer.
For example, besides the cached layer, the `formatted` layer also needs to
make sure the `query` object is being appropriately translated before
handing it off downstreams to the inner store, and also need to translate
the `queryResult` a bit while handing it upwards in the hierarchy.

3. As we add more query types in the IQv2, the inner store's `query`
instantiation may be getting very clumsy with a large "switch" condition on
all the possible query types. Although custom stores could consider only
supporting a few, having the `default` case to ignore all others, built-in
stores may still need to exhaust all possible types. I'm wondering if it's
a good trade-off to make `Query` be more restricted on extensibility to
have less exploding query type space, e.g. if a Query can only be extended
with some predefined dimensions like:

* query-field: key, non-key (some field extractor from the value bytes need
to be provided)
* query-scope: single, range
* query-match-type (only be useful for a range scope): prefix-match (e.g.
for a range key query, the provided is only a prefix, and all keys
containing this prefix should be returned), full-match
* query-value-type: object, raw-bytes

4. What's the expected usage for the execution info? Is it only for logging
purposes? If yes then I think not enforcing any string format is fine, that
the store layers can just attach any string information that they feel
useful.

5. I do not find any specific proposals for exception handling, what would
that look like? E.g. besides all the expected error cases like non-active,
how would we communicate other unexpected error cases such as store closed,
IO error, bad query parameters, etc?

6. Since we do not deprecate any existing APIs in this KIP, it's a bit hard
for readers to understand what is eventually going to be covered by IQv2.
For example, we know that eventually `KafkaStreams#store` would be gone,
but what about `KafkaStreams#queryMetadataForKey`, and
`#streamsMetadataForStore`, and also `allLocalStorePartitionLags`? I think
it would be great to mention the end world state with IQv2 even if the KIP
itself would not deprecate anything yet.

7. It seems people are still a bit confused about the
"Position/PositionBound" topics, and personally I think it's okay to
exclude them in this KIP just to keep its (already large) scope smaller.
Also after we started implementing the KIP in full, we may have learned new
things while fighting the details in the weeds, and that would be a better
timing for us to consider new parameters such as bounds, but also caching
bypassing, and other potential features as well.

Some minor ones:

8. What about just naming the new classes as `StateQueryRequest/Result`, or
`StoreQueryRequest/Result`? The word "interactive" is for describing its
semantics in docs, but I feel for class names we can use a more meaningful
prefix.

9. Should the RawKeyQuery be extending `KeyQuery`, or directly
implementing `Query?

10. Why do we need the new class "InteractiveQuerySerdes" along with
existing classes? In your PR it seems just using `StateSerdes` directly.

11. Why do we have a new template type "R" in the QueryResult class in
addition to ""? Should R always be equal to V?

12. Related to 10/11 above, what about letting the QueryResult to always be
returning values in raw bytes, along with the serdes? And then it's up to
the callers whether they want the bytes to be immediately deserialized or
want them to be written somewhere and deserialized 

[jira] [Created] (KAFKA-13457) socketChannel in Acceptor#accept is not closed upon IOException

[Haoze Wu (Jira)]

Haoze Wu created KAFKA-13457:


 Summary: socketChannel in Acceptor#accept is not closed upon 
IOException
 Key: KAFKA-13457
 URL: https://issues.apache.org/jira/browse/KAFKA-13457
 Project: Kafka
  Issue Type: Bug
  Components: network
Affects Versions: 2.8.0
Reporter: Haoze Wu


When the kafka.network.Acceptor in SocketServer.scala accepts a new connection 
in the `accept` function, it handles the `TooManyConnectionsException` and 
`ConnectionThrottledException`. However, line 717 or the socketChannel 
operations within the try block may potentially throw an IOException as well, 
which is not handled.

 
{code:java}
//core/src/main/scala/kafka/network/SocketServer.scala
// Acceptor class
  private def accept(key: SelectionKey): Option[SocketChannel] = {
    val serverSocketChannel = key.channel().asInstanceOf[ServerSocketChannel]
    val socketChannel = serverSocketChannel.accept()     // line 717
    try {
      connectionQuotas.inc(endPoint.listenerName, 
socketChannel.socket.getInetAddress, blockedPercentMeter)
      socketChannel.configureBlocking(false)             
      socketChannel.socket().setTcpNoDelay(true)         
      socketChannel.socket().setKeepAlive(true)          
      if (sendBufferSize != Selectable.USE_DEFAULT_BUFFER_SIZE)
        socketChannel.socket().setSendBufferSize(sendBufferSize)
      Some(socketChannel)
    } catch {
      case e: TooManyConnectionsException =>       
        info(s"Rejected connection from ${e.ip}, address already has the 
configured maximum of ${e.count} connections.")
        close(endPoint.listenerName, socketChannel)
        None
      case e: ConnectionThrottledException => 
        val ip = socketChannel.socket.getInetAddress
        debug(s"Delaying closing of connection from $ip for ${e.throttleTimeMs} 
ms")
        val endThrottleTimeMs = e.startThrottleTimeMs + e.throttleTimeMs
        throttledSockets += DelayedCloseSocket(socketChannel, endThrottleTimeMs)
        None
    }
  }
{code}
This thrown IOException is caught in the caller `acceptNewConnections` in line 
706, which only prints an error message. The socketChannel that throws this 
IOException is not closed.

 
{code:java}
//core/src/main/scala/kafka/network/SocketServer.scala
  private def acceptNewConnections(): Unit = {
    val ready = nioSelector.select(500)
    if (ready > 0) {
      val keys = nioSelector.selectedKeys()
      val iter = keys.iterator()
      while (iter.hasNext && isRunning) {
        try {
          val key = iter.next
          iter.remove()          if (key.isAcceptable) {
            accept(key).foreach { socketChannel => 
                ...
              } while (!assignNewConnection(socketChannel, processor, 
retriesLeft == 0))
            }
          } else
            throw new IllegalStateException("Unrecognized key state for 
acceptor thread.")
        } catch {
          case e: Throwable => error("Error while accepting connection", e)   
// line 706
        }
      }
    }
  }
{code}
We found during testing this would cause our Kafka clients to experience errors 
(InvalidReplicationFactorException) for 40+ seconds when creating new topics. 
After 40 seconds, the clients would be able to create new topics successfully.

We check that after adding the socketChannel.close() upon IOException, the 
symptoms will disappear, so the clients do not need to wait for 40s to be 
working again.

 

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-796: Interactive Query v2

[John Roesler]

Hi Patrick and Sagar,

Thanks for the feedback! I'll just break out the questions
and address them one at a time.

Patrick 1.
The default bound that I'm proposing is only to let active
tasks answer queries (which is also the default with IQ
today). Therefore, calling getPositionBound() would return a
PositionBound for which isLatest() is true.

Patrick 2.
I might have missed something in revision, but I'm not sure
what you're referring to exactly when you say they are
different. The IQRequest only has a PositionBound, and the
IQResponse only has a (concrete) Position, so I think they
are named accordingly (getPositionBound and getPosition). Am
I overlooking what you are talking about?

Sagar 1.
I think you're talking about the KeyValueStore#get(key)
method? This is a really good question. I went ahead and
dropped in an addendum to the KeyQuery example to show how
you would run the query in today's API. Here's a caracature
of the two APIS:

current:
  KeyValueStore store = kafkaStreams.store(
"mystore",
keyValueStore())
  int value = store.get(key);

proposed:
  int value = kafkaStreams.query(
"mystore",
KeyQuery.withKey(key));

So, today we first get the store interface and then we
invoke the method, and under the proposal, we would instead
just ask KafkaStreams to execute the query on the store. In
addition to all the other stuff I said in the motivation,
one thing I think is neat about this API is that it means we
can re-use queries across stores. So, for example, we could
also use KeyQuery on WindowStores, even though there's no
common interface between WindowStore and KeyValueStore.

In other words, stores can support any queries that make
sense and _not_ support any queries that don't make sense.
This gets into your second question...

Sagar 2.
Very good question. Your experience with your KIP-614
contribution was one of the things that made me want to
revise IQ to begin with. It seems like there's a really
stark gap between how straightforward the proposal is to add
a new store operation, and then how hard it is to actually
implement a new operation, due to all those intervening
wrappers.

There are two categories of wrappers to worry about:
- Facades: These only exist to disallow access to write
APIs, which are exposed through IQ today but shouldn't be
called. These are simply unnecessary under IQv2, since we
only run queries instead of returning the whole store.
- Store Layers: This is what you provided examples of. We
have store layers that let us compose features like
de/serialization and metering, changelogging, caching, etc.
A nice thing about this design is that we mostly don't have
to worry at all about those wrapper layers at all. Each of
these stores would simply delegate any query to lower layers
unless there is something they need to do. In my POC, I
simply added a delegating implementation to
WrappedStateStore, which meant that I didn't need to touch
most of the wrappers when I added a new query.

Here's what I think future contributors will have to worry
about:
1. The basic query execution in the base byte stores
(RocksDB and InMemory)
2. The Caching stores IF they want the query to be served
from the cache
3. The Metered stores IF some serialization needs to be done
for the query

And that's it! We should be able to add new queries without
touching any other store layer besides those, and each one
of those is involved because it has some specific reason to
be.


Thanks again, Patrick and Sagar! Please let me know if I
failed to address your questions, or if you have any more.

Thanks,
-John

On Mon, 2021-11-15 at 22:37 +0530, Sagar wrote:
> Hi John,
> 
> Thanks for the great writeup! Couple of things I wanted to bring up(may or
> mayn't be relevant):
> 
> 1) The sample implementation that you have presented for KeyQuery is very
> helpful. One thing which may be added to it is how it connects to the
> KeyValue.get(key) method. That's something that atleast I couldn't totally
> figure out-not sure about others though. I understand that it is out of
> scope of th KIP to explain for every query that IQ supports but one
> implementation just to get a sense of how the changes would feel like.
> 2) The other thing that I wanted to know is that StateStore on it's own has
> a lot of implementations and some of them are wrappers, So at what levels
> do users need to implement the query methods? Like a MeteredKeyValueStore
> wraps RocksDbStore and calls it internally through a wrapped call. As per
> the new changes, how would the scheme of things look like for the same
> KeyQuery?
> 
> Thanks!
> Sagar.
> 
> 
> On Mon, Nov 15, 2021 at 6:20 PM Patrick Stuedi 
> wrote:
> 
> > Hi John,
> > 
> > Thanks for submitting the KIP! One question I have is, assuming one
> > instantiates InteractiveQueryRequest via withQuery, and then later calls
> > getPositionBound, what will the result be? Also I noticed the Position
> > returning method is in InteractiveQueryRequest and 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #11

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 500572 lines...]
[2021-11-15T21:48:40.257Z] > Task :raft:testClasses UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :connect:json:testJar
[2021-11-15T21:48:40.257Z] > Task :connect:json:testSrcJar
[2021-11-15T21:48:40.257Z] > Task :metadata:compileTestJava UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :metadata:testClasses UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :core:compileScala UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :core:classes UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :core:compileTestJava NO-SOURCE
[2021-11-15T21:48:40.257Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-11-15T21:48:40.257Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2021-11-15T21:48:40.257Z] 
[2021-11-15T21:48:40.257Z] > Task :streams:processMessages
[2021-11-15T21:48:40.257Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2021-11-15T21:48:40.257Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_3.1/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-11-15T21:48:40.257Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2021-11-15T21:48:40.257Z] 
[2021-11-15T21:48:40.257Z] > Task :core:compileTestScala UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :core:testClasses UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :streams:compileJava UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :streams:classes UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :streams:copyDependantLibs UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :streams:jar UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-11-15T21:48:40.257Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-11-15T21:48:44.028Z] > Task :connect:api:javadoc
[2021-11-15T21:48:44.028Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-11-15T21:48:44.028Z] > Task :connect:api:jar UP-TO-DATE
[2021-11-15T21:48:44.028Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-11-15T21:48:44.028Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-11-15T21:48:44.028Z] > Task :connect:json:jar UP-TO-DATE
[2021-11-15T21:48:44.028Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-11-15T21:48:44.028Z] > Task :connect:api:javadocJar
[2021-11-15T21:48:44.028Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-11-15T21:48:44.028Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-11-15T21:48:44.028Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-11-15T21:48:44.028Z] > Task :connect:json:publishToMavenLocal
[2021-11-15T21:48:44.028Z] > Task :connect:api:testJar
[2021-11-15T21:48:44.028Z] > Task :connect:api:testSrcJar
[2021-11-15T21:48:44.028Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-11-15T21:48:44.028Z] > Task :connect:api:publishToMavenLocal
[2021-11-15T21:48:45.806Z] > Task :streams:javadoc
[2021-11-15T21:48:46.749Z] > Task :streams:javadocJar
[2021-11-15T21:48:46.749Z] > Task :streams:compileTestJava UP-TO-DATE
[2021-11-15T21:48:46.749Z] > Task :streams:testClasses UP-TO-DATE
[2021-11-15T21:48:46.749Z] > Task :streams:testJar
[2021-11-15T21:48:46.749Z] > Task :streams:testSrcJar
[2021-11-15T21:48:46.749Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2021-11-15T21:48:46.749Z] > Task :streams:publishToMavenLocal
[2021-11-15T21:48:48.678Z] > Task :clients:javadoc
[2021-11-15T21:48:48.678Z] > Task :clients:javadocJar
[2021-11-15T21:48:48.678Z] 
[2021-11-15T21:48:48.678Z] > Task :clients:srcJar
[2021-11-15T21:48:48.678Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2021-11-15T21:48:48.678Z]   - Gradle detected a problem with the following 
location: '/home/jenkins/workspace/Kafka_kafka_3.1/clients/src/generated/java'. 
Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-11-15T21:48:49.621Z] 
[2021-11-15T21:48:49.621Z] > Task :clients:testJar
[2021-11-15T21:48:49.621Z] > Task :clients:testSrcJar

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #568

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 499215 lines...]
[2021-11-15T21:16:52.297Z] > Task :raft:testClasses UP-TO-DATE
[2021-11-15T21:16:52.297Z] > Task :connect:json:testJar
[2021-11-15T21:16:52.297Z] > Task :connect:json:testSrcJar
[2021-11-15T21:16:52.297Z] > Task :metadata:compileTestJava UP-TO-DATE
[2021-11-15T21:16:52.297Z] > Task :metadata:testClasses UP-TO-DATE
[2021-11-15T21:16:52.297Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-11-15T21:16:52.297Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2021-11-15T21:16:52.297Z] > Task :core:compileScala UP-TO-DATE
[2021-11-15T21:16:52.297Z] > Task :core:classes UP-TO-DATE
[2021-11-15T21:16:52.297Z] > Task :core:compileTestJava NO-SOURCE
[2021-11-15T21:16:52.297Z] 
[2021-11-15T21:16:52.297Z] > Task :streams:processMessages
[2021-11-15T21:16:52.297Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2021-11-15T21:16:52.297Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-11-15T21:16:52.297Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2021-11-15T21:16:53.342Z] 
[2021-11-15T21:16:53.342Z] > Task :streams:compileJava UP-TO-DATE
[2021-11-15T21:16:53.342Z] > Task :streams:classes UP-TO-DATE
[2021-11-15T21:16:53.342Z] > Task :streams:copyDependantLibs UP-TO-DATE
[2021-11-15T21:16:53.342Z] > Task :streams:jar UP-TO-DATE
[2021-11-15T21:16:53.342Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-11-15T21:16:53.342Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-11-15T21:16:53.342Z] > Task :core:compileTestScala UP-TO-DATE
[2021-11-15T21:16:53.342Z] > Task :core:testClasses UP-TO-DATE
[2021-11-15T21:16:56.502Z] > Task :connect:api:javadoc
[2021-11-15T21:16:56.502Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-11-15T21:16:56.502Z] > Task :connect:api:jar UP-TO-DATE
[2021-11-15T21:16:56.502Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-11-15T21:16:56.502Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-11-15T21:16:56.502Z] > Task :connect:json:jar UP-TO-DATE
[2021-11-15T21:16:56.502Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-11-15T21:16:56.502Z] > Task :connect:api:javadocJar
[2021-11-15T21:16:56.502Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-11-15T21:16:56.502Z] > Task :connect:json:publishToMavenLocal
[2021-11-15T21:16:56.502Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-11-15T21:16:56.502Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-11-15T21:16:56.502Z] > Task :connect:api:testJar
[2021-11-15T21:16:56.502Z] > Task :connect:api:testSrcJar
[2021-11-15T21:16:56.502Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-11-15T21:16:56.502Z] > Task :connect:api:publishToMavenLocal
[2021-11-15T21:16:59.966Z] > Task :streams:javadoc
[2021-11-15T21:16:59.966Z] > Task :streams:javadocJar
[2021-11-15T21:16:59.966Z] > Task :streams:compileTestJava UP-TO-DATE
[2021-11-15T21:16:59.966Z] > Task :streams:testClasses UP-TO-DATE
[2021-11-15T21:16:59.966Z] > Task :streams:testJar
[2021-11-15T21:17:01.278Z] > Task :streams:testSrcJar
[2021-11-15T21:17:01.278Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2021-11-15T21:17:01.278Z] > Task :streams:publishToMavenLocal
[2021-11-15T21:17:01.278Z] > Task :clients:javadoc
[2021-11-15T21:17:01.278Z] > Task :clients:javadocJar
[2021-11-15T21:17:02.532Z] 
[2021-11-15T21:17:02.532Z] > Task :clients:srcJar
[2021-11-15T21:17:02.532Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2021-11-15T21:17:02.532Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/generated/java'. Reason: 
Task ':clients:srcJar' uses this output of task ':clients:processMessages' 
without declaring an explicit or implicit dependency. This can lead to 
incorrect results being produced, depending on what order the tasks are 
executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-11-15T21:17:03.480Z] 
[2021-11-15T21:17:03.480Z] > Task :clients:testJar
[2021-11-15T21:17:03.480Z] > Task :clients:testSrcJar

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #89

[Apache Jenkins Server]

See 

[DISCUSS] Brokers disconnect intermittently with TLS1.3

[Kokoori, Shylaja]

Hi all,

Using TLS1.3 (with JDK11) is causing an intermittent increase in inter-broker 
p99 latency, as mentioned by Yiming in 
Kafka-9320.
 We tested this with Kafka 2.8.
The issue seems to be because of a renegotiation exception being thrown by

read(ByteBuffer dst)

&

write(ByteBuffer src)

in
clients/src/main/java/org/apache/kafka/common/network/SslTransportLayer.java

This exception is causing the connection to close between the brokers before 
read/write is completed.

In our internal experiments we have seen the p99 latency stabilize when we 
remove this exception.

Given that TLS1.3 does not support renegotiation, can I make it applicable just 
for TLS1.2?

I have also created a ticket

Any feedback is welcome.

Thank you,

Shylaja

Re: [DISCUSS] KIP-779: Allow Source Tasks to Handle Producer Exceptions

[Knowles Atchison Jr]

Chris,

Thank you for the feedback. I can certainly update the KIP to state that
once exactly one support is in place, the task would be failed even if
error.tolerance were set to all. Programmatically it would still require
PRs to be merged to build on top of. I also liked my original
implementation of the hook as it gave the connector writers the most
flexibility in handling producer errors. I changed the original
implementation as the progression/changes still supported my use case and I
thought it would move this process along faster.

Knowles

On Thu, Nov 11, 2021 at 3:43 PM Chris Egerton 
wrote:

> Hi Knowles,
>
> I think this looks good for the most part but I'd still like to see an
> explicit mention in the KIP (and proposed doc/Javadoc changes) that states
> that, with exactly-once support enabled, producer exceptions that result
> from failures related to exactly-once support (including but not limited to
> ProducerFencedExcecption instances (
>
> https://kafka.apache.org/30/javadoc/org/apache/kafka/common/errors/ProducerFencedException.html
> ))
> will not be skipped even with "errors.tolerance" set to "all", and will
> instead unconditionally cause the task to fail. Your proposal that
> "WorkerSourceTask could check the configuration before handing off the
> records and exception to this function" seems great as long as we update
> "handing off the records and exceptions to this function" to the
> newly-proposed behavior of "logging the exception and continuing to poll
> the task for data".
>
> I'm also a little bit wary of updating the existing "errors.tolerance"
> configuration to have new behavior that users can't opt out of without also
> opting out of the current behavior they get with "errors.tolerance" set to
> "all", but I think I've found a decent argument in favor of it. One thought
> that came to mind is whether this use case was originally considered when
> KIP-298 was being discussed. However, it appears that KAFKA-8586 (
> https://issues.apache.org/jira/browse/KAFKA-8586), the fix for which
> caused
> tasks to fail on non-retriable, asynchronous producer exceptions instead of
> logging them and continuing, was discovered over a full year after the
> changes for KIP-298 (https://github.com/apache/kafka/pull/5065) were
> merged. I suspect that the current proposal aligns nicely with the original
> design intent of KIP-298, and that if KAFKA-8586 were discovered before or
> during discussion for KIP-298, non-retriable, asynchronous producer
> exceptions would have been included in its scope. With that in mind,
> although it may cause issues for some niche use cases, I think that this is
> a valid change and would be worth the tradeoff of potentially complicating
> life for a small number of users. I'd be interested in Arjun's thoughts on
> this though (as he designed and implemented KIP-298), and if this analysis
> is agreeable, we may want to document that information in the KIP as well
> to strengthen our case for not introducing a new configuration property and
> instead making this behavior tied to the existing "errors.tolerance"
> property with no opt-out besides using a new value for that property.
>
> My last thought is that, although it may be outside the scope of this KIP,
> I believe your original proposal of giving tasks a hook to handle
> downstream exceptions is actually quite valid. The DLQ feature for sink
> connectors is an extremely valuable one as it prevents data loss when
> "errors.tolerance" is set to "all" by allowing users to reprocess
> problematic records at a later date without stopping the flow of data in
> their connector entirely. As others have noted, it's difficult if not
> outright impossible to provide a Kafka DLQ topic for source connectors with
> the same guarantees, and so allowing source connectors the option of
> storing problematic records back in the system that they came from seems
> like a reasonable alternative. I think we're probably past the point of
> making that happen in this KIP, but I don't believe the changes you've
> proposed make that any harder in the future than it is now (which is
> great!), and I wanted to voice my general support for a mechanism like this
> in case you or someone following along think it'd be worth it to pursue at
> a later date.
>
> Thanks for your KIP and thanks for your patience with the process!
>
> Cheers,
>
> Chris
>
> On Fri, Nov 5, 2021 at 8:26 AM Knowles Atchison Jr 
> wrote:
>
> > Good morning,
> >
> > If there is no additional feedback, I am going to call a vote for this
> KIP
> > on Monday.
> >
> > Knowles
> >
> > On Tue, Nov 2, 2021 at 10:00 AM Knowles Atchison Jr <
> katchiso...@gmail.com
> > >
> > wrote:
> >
> > > Third time's the charm.
> > >
> > > I've added a getter for the RetryWithToleranceOperator to get the
> > > ToleranceType. I've updated WorkerSourceTask to check this setting to
> see
> > > if it is ToleranceType.ALL.
> > >
> > > Setting "errors.tolerance" to "all" solves both 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.1 #10

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-10104) Remove deprecated --zookeeper flags as specified in KIP-604

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-10104.
-
Resolution: Duplicate

> Remove deprecated --zookeeper flags as specified in KIP-604
> ---
>
> Key: KAFKA-10104
> URL: https://issues.apache.org/jira/browse/KAFKA-10104
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin McCabe
>Priority: Major
>
> Remove deprecated --zookeeper flags as specified in KIP-604



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-799 Align behaviour for producer callbacks with documented behaviour

[Jun Rao]

Hi, Séamus,

Thanks for the KIP. We definitely want the input semantic for the producer
callback to be consistent. It's probably better to make the input semantic
consistent between the producer callback and the interceptor.

If producer.send() throws an ApiException, we may not know the partition
for the record. In that case, we probably just want to set the partition to
-1 as the interceptor does.

Jun



On Thu, Nov 11, 2021 at 11:51 AM John Roesler  wrote:

> Thanks for the reply, Séamus,
>
> Ah, I missed that the actual value of the placeholder is
> that otherwise, you wouldn't know the topic/partition of the
> error.
>
> I guess, on balance, it doesn't feel like this situation
> really justifies moving to a new callback interface (to pass
> back the topic/partition separately from the other
> metadata), even though that might have been nicer in a
> vacuum. So, if you think it's nice to have those metadata,
> then I think the fact that it's been the de-facto behavior
> for many callback situations for a while now, and the fact
> that it's the established pattern of the interceptor
> indicates that it's probably fine to do as you propose and
> just standardize on the placeholder in error cases.
>
> Thanks!
> -John
>
> On Thu, 2021-11-11 at 17:08 +, Séamus Ó Ceanainn wrote:
> > Hey John,
> >
> > > did you consider just going back to the original behavior?
> >
> > I hadn't considered going back to the exact original behaviour as I think
> > there's a valid point made in discussions around KAFKA-7412 (I
> > forget whether in a JIRA or PR comment) that returning the topic
> partition
> > when available can be useful for users. Something I did consider is to
> > include the topic partition separately to the metadata value when
> > exceptions occur so that metadata could still be null in those cases
> while
> > still having topic partition data available.
> >
> > My opinion is that this other behaviour would be nicer (where returned
> > metadata is null but topic partition information is still available),
> > however it would not be consistent with the implementation of
> > ProducerInterceptor.onAcknowledgement method. I would tend to favour
> > consistency in this case (as both methods are handled very similarly in
> > code), and I don't think there's a strong argument to make a breaking
> > change to ProducerInterceptor when there is nothing currently broken in
> > that implementation (like there currently is with Callback).
> >
> > Of course if the general consensus is that consistency between the
> > behaviour of the two methods (ProducerInterceptor.onAcknowledgement and
> > Callback.onCompletion) does not matter, or that a change in the behaviour
> > of ProducerInterceptor.onAcknowledgement should also be included in the
> > scope of this KIP, I'm open to updating the KIP to reflect that.
> >
> > > Although it’s technically an implementation detail (which doesn’t need
> to
> > be in a KIP), I like the fact that you’re planning to refactor the code
> to
> > enforce consistent handling of the callbacks.
> >
> > I wasn't entirely sure how to deal with changes to the interfaces within
> > the 'clients.producer.internals' package, so I thought it was best to err
> > on the side of including too much in the KIP.  I'll remove the
> unnecessary
> > detail to ensure the discussion doesn't get derailed, for anyone
> interested
> > in implementation details there is a draft PR linked in the KIP with that
> > refactoring done, so any discussion on that topic can take place in
> Github
> > / JIRA.
> >
> > Regards,
> > Séamus.
> >
> > On Thu, 11 Nov 2021 at 14:33, John Roesler  wrote:
> >
> > > Thanks for the KIP, Séamus!
> > >
> > > I agree that the current situation you’re describing doesn’t seem
> ideal,
> > > and it’s probably worth a slight behavior change to fix it.
> > >
> > > It’s too bad that we introduced that placeholder record, since it seems
> > > less error prone for users if we have the invariant that exactly one
> > > argument is non-null. I’m concerned (as reported in KAFKA-7412) that
> people
> > > could mistake the placeholder for a successful ack. Since we’re
> considering
> > > some breakage to fix this inconsistency, did you consider just going
> back
> > > to the original behavior?
> > >
> > > Although it’s technically an implementation detail (which doesn’t need
> to
> > > be in a KIP), I like the fact that you’re planning to refactor the
> code to
> > > enforce consistent handling of the callbacks.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Nov 11, 2021, at 07:25, Séamus Ó Ceanainn wrote:
> > > > Hi,
> > > >
> > > > As outlined in KIP-799: Align behaviour for producer callbacks with
> > > > documented behaviour
> > > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-799%3A+Align+behaviour+for+producer+callbacks+with+documented+behaviour
> > > > ,
> > > > there is an inconsistency between the documented behaviour and
> > > > implementation of producer callbacks for 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #567

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 499316 lines...]
[2021-11-15T18:10:15.320Z] PlaintextConsumerTest > 
testFetchRecordLargerThanFetchMaxBytes() PASSED
[2021-11-15T18:10:15.320Z] 
[2021-11-15T18:10:15.320Z] PlaintextConsumerTest > testAutoCommitOnClose() 
STARTED
[2021-11-15T18:10:20.063Z] 
[2021-11-15T18:10:20.063Z] PlaintextConsumerTest > testFetchInvalidOffset() 
PASSED
[2021-11-15T18:10:20.063Z] 
[2021-11-15T18:10:20.063Z] PlaintextConsumerTest > testAutoCommitIntercept() 
STARTED
[2021-11-15T18:10:20.235Z] 
[2021-11-15T18:10:20.235Z] PlaintextConsumerTest > testAutoCommitOnClose() 
PASSED
[2021-11-15T18:10:20.235Z] 
[2021-11-15T18:10:20.235Z] PlaintextConsumerTest > testListTopics() STARTED
[2021-11-15T18:10:23.728Z] 
[2021-11-15T18:10:23.728Z] PlaintextConsumerTest > testListTopics() PASSED
[2021-11-15T18:10:23.728Z] 
[2021-11-15T18:10:23.728Z] PlaintextConsumerTest > 
testExpandingTopicSubscriptions() STARTED
[2021-11-15T18:10:25.354Z] 
[2021-11-15T18:10:25.354Z] PlaintextConsumerTest > testAutoCommitIntercept() 
PASSED
[2021-11-15T18:10:25.354Z] 
[2021-11-15T18:10:25.354Z] PlaintextConsumerTest > 
testFetchHonoursMaxPartitionFetchBytesIfLargeRecordNotFirst() STARTED
[2021-11-15T18:10:30.235Z] 
[2021-11-15T18:10:30.235Z] PlaintextConsumerTest > 
testExpandingTopicSubscriptions() PASSED
[2021-11-15T18:10:30.235Z] 
[2021-11-15T18:10:30.235Z] PlaintextConsumerTest > 
testMultiConsumerDefaultAssignor() STARTED
[2021-11-15T18:10:30.406Z] 
[2021-11-15T18:10:30.406Z] PlaintextConsumerTest > 
testFetchHonoursMaxPartitionFetchBytesIfLargeRecordNotFirst() PASSED
[2021-11-15T18:10:30.406Z] 
[2021-11-15T18:10:30.406Z] PlaintextConsumerTest > testCommitSpecifiedOffsets() 
STARTED
[2021-11-15T18:10:36.474Z] 
[2021-11-15T18:10:36.474Z] PlaintextConsumerTest > testCommitSpecifiedOffsets() 
PASSED
[2021-11-15T18:10:36.474Z] 
[2021-11-15T18:10:36.474Z] PlaintextConsumerTest > 
testPerPartitionLeadMetricsCleanUpWithSubscribe() STARTED
[2021-11-15T18:10:41.865Z] 
[2021-11-15T18:10:41.865Z] PlaintextConsumerTest > 
testPerPartitionLeadMetricsCleanUpWithSubscribe() PASSED
[2021-11-15T18:10:41.865Z] 
[2021-11-15T18:10:41.865Z] PlaintextConsumerTest > testCommitMetadata() STARTED
[2021-11-15T18:10:42.936Z] 
[2021-11-15T18:10:42.936Z] PlaintextConsumerTest > 
testMultiConsumerDefaultAssignor() PASSED
[2021-11-15T18:10:42.936Z] 
[2021-11-15T18:10:42.936Z] PlaintextConsumerTest > testInterceptors() STARTED
[2021-11-15T18:10:46.461Z] 
[2021-11-15T18:10:46.461Z] PlaintextConsumerTest > testCommitMetadata() PASSED
[2021-11-15T18:10:46.461Z] 
[2021-11-15T18:10:46.461Z] PlaintextConsumerTest > testRoundRobinAssignment() 
STARTED
[2021-11-15T18:10:48.355Z] 
[2021-11-15T18:10:48.355Z] PlaintextConsumerTest > testInterceptors() PASSED
[2021-11-15T18:10:48.355Z] 
[2021-11-15T18:10:48.355Z] PlaintextConsumerTest > 
testConsumingWithEmptyGroupId() STARTED
[2021-11-15T18:10:51.362Z] 
[2021-11-15T18:10:51.362Z] PlaintextConsumerTest > 
testConsumingWithEmptyGroupId() PASSED
[2021-11-15T18:10:51.362Z] 
[2021-11-15T18:10:51.362Z] PlaintextConsumerTest > testPatternUnsubscription() 
STARTED
[2021-11-15T18:10:55.101Z] 
[2021-11-15T18:10:55.101Z] PlaintextConsumerTest > testRoundRobinAssignment() 
PASSED
[2021-11-15T18:10:55.101Z] 
[2021-11-15T18:10:55.101Z] PlaintextConsumerTest > testPatternSubscription() 
STARTED
[2021-11-15T18:10:58.561Z] 
[2021-11-15T18:10:58.561Z] PlaintextConsumerTest > testPatternUnsubscription() 
PASSED
[2021-11-15T18:10:58.561Z] 
[2021-11-15T18:10:58.561Z] PlaintextConsumerTest > testGroupConsumption() 
STARTED
[2021-11-15T18:11:04.005Z] 
[2021-11-15T18:11:04.005Z] PlaintextConsumerTest > testGroupConsumption() PASSED
[2021-11-15T18:11:04.005Z] 
[2021-11-15T18:11:04.005Z] PlaintextConsumerTest > testPartitionsFor() STARTED
[2021-11-15T18:11:05.464Z] 
[2021-11-15T18:11:05.465Z] PlaintextConsumerTest > testPatternSubscription() 
PASSED
[2021-11-15T18:11:07.189Z] 
[2021-11-15T18:11:07.189Z] PlaintextConsumerTest > testPartitionsFor() PASSED
[2021-11-15T18:11:07.189Z] 
[2021-11-15T18:11:07.189Z] PlaintextConsumerTest > 
testMultiConsumerDefaultAssignorAndVerifyAssignment() STARTED
[2021-11-15T18:11:07.706Z] 
[2021-11-15T18:11:07.706Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-11-15T18:11:07.706Z] 
[2021-11-15T18:11:07.706Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2021-11-15T18:11:07.706Z] 
[2021-11-15T18:11:07.706Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2021-11-15T18:11:07.706Z] 
[2021-11-15T18:11:07.706Z] BUILD SUCCESSFUL in 2h 1m
[2021-11-15T18:11:07.706Z] 202 actionable tasks: 109 executed, 93 up-to-date
[2021-11-15T18:11:07.706Z] 
[2021-11-15T18:11:07.706Z] See the profiling report 

Re: [DISCUSS] KIP-778 KRaft Upgrades

[Guozhang Wang]

Thanks David,

1. Got it. One thing I'm still not very clear is why it's sufficient to
select a metadata.version which is supported by majority of the quorum, but
not the whole quorum (i.e. choosing the lowest version among all the quorum
members)? Since the leader election today does not take this value into
consideration, we are not guaranteed that newly selected leaders would
always be able to recognize and support the initialized metadata.version
right?

2. Yeah I think I agree the behavior-but-not-RPC-change scenario is beyond
the scope of this KIP, we can defer it to later discussions.

On Mon, Nov 15, 2021 at 8:13 AM David Arthur
 wrote:

> Guozhang, thanks for the review!
>
> 1, As we've defined it so far, the initial metadata.version is set by an
> operator via the "kafka-storage.sh" tool. It would be possible for
> different values to be selected, but only the quorum leader would commit a
> FeatureLevelRecord with the version they read locally. See the above reply
> to Jun's question for a little more detail.
>
> We need to enable the KRaft RPCs regardless of metadata.version (vote,
> heartbeat, fetch, etc) so that the quorum can be formed, a leader can be
> elected, and followers can learn about the selected metadata.version. I
> believe the quorum startup procedure would go something like:
>
> * Controller nodes start, read their logs, begin leader election
> * While the elected node is becoming leader (in
> QuorumMetaLogListener#handleLeaderChange), initialize metadata.version if
> necessary
> * Followers replicate the FeatureLevelRecord
>
> This (probably) means the quorum peers must continue to rely on ApiVersions
> to negotiate compatible RPC versions and these versions cannot depend on
> metadata.version.
>
> Does this make sense?
>
>
> 2, ApiVersionResponse includes the broker's supported feature flags as well
> as the cluster-wide finalized feature flags. We probably need to add
> something like the feature flag "epoch" to this response payload in order
> to see which broker is most up to date.
>
> If the new feature flag version included RPC changes, we are helped by the
> fact that a client won't attempt to use the new RPC until it has discovered
> a broker that supports it via ApiVersions. The problem is more difficult
> for cases like you described where the feature flag upgrade changes the
> behavior of the broker, but not its RPCs. This is actually the same problem
> as upgrading the IBP. During a rolling restart, clients may hit different
> brokers with different capabilities and not know it.
>
> This probably warrants further investigation, but hopefully you agree it is
> beyond the scope of this KIP :)
>
> -David
>
>
> On Mon, Nov 15, 2021 at 10:26 AM David Arthur 
> wrote:
>
> > Jun, thanks for the comments!
> >
> > 16, When a new cluster is deployed, we don't select the highest available
> > metadata.version, but rather the quorum leader picks a bootstrap version
> > defined in meta.properties. As mentioned earlier, we should add
> validation
> > here to ensure a majority of the followers could support this version
> > before initializing it. This would avoid a situation where a failover
> > results in a new leader who can't support the selected metadata.version.
> >
> > Thinking a bit more on this, we do need to define a state where we're
> > running newer software, but we don't have the feature flag set. This
> could
> > happen if we were running an older IBP that did not support KIP-778.
> > Following on this, it doesn't seem too difficult to consider a case where
> > the IBP has been upgraded, but we still have not finalized a
> > metadata.version. Here are some possible version combinations (assuming
> > KIP-778 is added to Kafka 3.2):
> >
> > Case  SoftwareIBPmetadata.versioneffective version
> > --
> > A 3.1 3.1-   0  software too old for
> > feature flag
> > B 3.2 3.1-   0  feature flag supported,
> > but IBP too old
> > C 3.2 3.2-   0  feature flag supported,
> > but not initialized
> > D 3.2 3.21   1  feature flag initialized
> > to 1 (via operator or bootstrap process)
> > ...
> > E 3.8 3.1-   0  ...IBP too old
> > F 3.8 3.2-   0  ...not initialized
> > G 3.8 3.24   4
> >
> >
> > Here I'm defining version 0 as "no metadata.version set". So back to your
> > comment, I think the KIP omits discussion of case C from the above table
> > which I can amend. Does that cover your concerns, or am I missing
> something
> > else?
> >
> >
> > > it's inconvenient for a user to manually upgrade every feature version
> >
> > For this, we would probably want to extend the capabilities of KIP-584. I
> > don't think anything we've discussed for KIP-778 will preclude us from
> > adding some 

Re: [DISCUSS] KIP-796: Interactive Query v2

[Sagar]

Hi John,

Thanks for the great writeup! Couple of things I wanted to bring up(may or
mayn't be relevant):

1) The sample implementation that you have presented for KeyQuery is very
helpful. One thing which may be added to it is how it connects to the
KeyValue.get(key) method. That's something that atleast I couldn't totally
figure out-not sure about others though. I understand that it is out of
scope of th KIP to explain for every query that IQ supports but one
implementation just to get a sense of how the changes would feel like.
2) The other thing that I wanted to know is that StateStore on it's own has
a lot of implementations and some of them are wrappers, So at what levels
do users need to implement the query methods? Like a MeteredKeyValueStore
wraps RocksDbStore and calls it internally through a wrapped call. As per
the new changes, how would the scheme of things look like for the same
KeyQuery?

Thanks!
Sagar.


On Mon, Nov 15, 2021 at 6:20 PM Patrick Stuedi 
wrote:

> Hi John,
>
> Thanks for submitting the KIP! One question I have is, assuming one
> instantiates InteractiveQueryRequest via withQuery, and then later calls
> getPositionBound, what will the result be? Also I noticed the Position
> returning method is in InteractiveQueryRequest and InteractiveQueryResult
> is named differently, any particular reason?
>
> Best,
>   Patrick
>
>
> On Fri, Nov 12, 2021 at 12:29 AM John Roesler  wrote:
>
> > Thanks for taking a look, Sophie!
> >
> > Ah, that was a revision error. I had initially been planning
> > an Optional> with Optional.empty() meaning to
> > fetch all partitions, but then decided it was needlessly
> > complex and changed it to the current proposal with two
> > methods:
> >
> > boolean isAllPartitions();
> > Set getPartitions(); (which would throw an
> > exception if it's an "all partitions" request).
> >
> > I've corrected the javadoc and also documented the
> > exception.
> >
> > Thanks!
> > -John
> >
> > On Thu, 2021-11-11 at 15:03 -0800, Sophie Blee-Goldman
> > wrote:
> > > Thanks John, I've been looking forward to this for a while now. It
> > > was pretty horrifying to learn
> > > how present-day IQ works  (or rather, doesn't work) with custom state
> > > stores :/
> > >
> > > One minor cosmetic point, In the InteractiveQueryRequest class, the #
> > > getPartitions
> > > method has a return type of Set, but the javadocs refer to
> > Optional.
> > > Not
> > > sure which is intended for this API, but if is supposed to be the
> return
> > > type, do you perhaps
> > > mean for it to be  Optional.ofEmpty() and Optional.of(non-empty set)
> > > rather than Optional.of(empty set) and Optional.of(non-empty set) ?
> > >
> > > On Thu, Nov 11, 2021 at 12:03 PM John Roesler 
> > wrote:
> > >
> > > > Hello again, all,
> > > >
> > > > Just bumping this discussion on a new, more flexible
> > > > Interactive Query API in Kafka Streams.
> > > >
> > > > If there are no concerns, I'll go ahead and call a vote on
> > > > Monday.
> > > >
> > > > Thanks!
> > > > -John
> > > >
> > > > On Tue, 2021-11-09 at 17:37 -0600, John Roesler wrote:
> > > > > Hello all,
> > > > >
> > > > > I'd like to start the discussion for KIP-796, which proposes
> > > > > a revamp of the Interactive Query APIs in Kafka Streams.
> > > > >
> > > > > The proposal is here:
> > > > > https://cwiki.apache.org/confluence/x/34xnCw
> > > > >
> > > > > I look forward to your feedback!
> > > > >
> > > > > Thank you,
> > > > > -John
> > > > >
> > > >
> > > >
> > > >
> >
> >
> >
>

[jira] [Created] (KAFKA-13456) controller.listener.names is required for all KRaft nodes, not just controllers

[Ron Dagostino (Jira)]

Ron Dagostino created KAFKA-13456:
-

 Summary: controller.listener.names is required for all KRaft 
nodes, not just controllers
 Key: KAFKA-13456
 URL: https://issues.apache.org/jira/browse/KAFKA-13456
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.0.0, 2.8.0, 3.1.0
Reporter: Ron Dagostino
Assignee: Ron Dagostino


The controller.listener.names config is currently checked for existence when 
the process.roles contains the controller role (i.e. process.roles=controller 
or process.roles=broker,contrtoller); it is not checked for existence when 
process.roles=broker.  However, KRaft brokers have to talk to KRaft 
controllers, of course, and they do so by taking the first entry in the 
controller.listener.names list.  Therefore, controller.listener.names is 
required in KRaft mode even when process.roles.broker.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13455) The Apache Kafka quickstart guide does not contain any steps for running Kafka Connect

[Kate Stanley (Jira)]

Kate Stanley created KAFKA-13455:


 Summary: The Apache Kafka quickstart guide does not contain any 
steps for running Kafka Connect
 Key: KAFKA-13455
 URL: https://issues.apache.org/jira/browse/KAFKA-13455
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Affects Versions: 3.0.0, 2.8.1, 2.7.2
Reporter: Kate Stanley


The Apache Kafka quickstart guide does not contain any steps for running Kafka 
Connect. Instead it links to the User guide, which just links back to the 
quickstart. The steps are present in version 2.6 of the documentation, but not 
in the latest.

See [https://kafka.apache.org/26/documentation/#quickstart_kafkaconnect] vs 
https://kafka.apache.org/documentation/#quickstart_kafkaconnect

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-792: Add "generation" field into consumer protocol

[David Jacot]

Hi Luke,

Thanks for the KIP. Overall, I think that the motivation makes sense. I
have a couple of comments/questions:

1. In the Public Interfaces section, it would be great if you could put the
end state not the current one.

2. Do we need to update the Subscription class to expose the
generation? If so, it would be great to mention it in the Public
Interfaces section as well.

3. You mention that the broker will set the generation if the subscription
contains a sentinel value (-1). As of today, the broker does not parse
the subscription so I am not sure how/why we would do this. I suppose
that we could add a `generation` field to the JoinGroupRequest instead
to do the fencing that you describe while handling the sentinel in the
assignor directly. If we would add the `generation` to the request itself,
would we need the `generation` in the subscription protocol as well?

Best,
David

On Fri, Nov 12, 2021 at 3:31 AM Luke Chen  wrote:
>
> Hi all,
>
> I'd like to start the discussion for KIP-792: Add "generation" field into
> consumer protocol.
>
> The goal of this KIP is to allow assignor/consumer coordinator/group
> coordinator to have a way to identify the out-of-date members/assignments.
>
> Detailed description can be found here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336614
>
> Any feedback is welcome.
>
> Thank you.
> Luke

Re: [DISCUSS] KIP-778 KRaft Upgrades

[David Arthur]

Guozhang, thanks for the review!

1, As we've defined it so far, the initial metadata.version is set by an
operator via the "kafka-storage.sh" tool. It would be possible for
different values to be selected, but only the quorum leader would commit a
FeatureLevelRecord with the version they read locally. See the above reply
to Jun's question for a little more detail.

We need to enable the KRaft RPCs regardless of metadata.version (vote,
heartbeat, fetch, etc) so that the quorum can be formed, a leader can be
elected, and followers can learn about the selected metadata.version. I
believe the quorum startup procedure would go something like:

* Controller nodes start, read their logs, begin leader election
* While the elected node is becoming leader (in
QuorumMetaLogListener#handleLeaderChange), initialize metadata.version if
necessary
* Followers replicate the FeatureLevelRecord

This (probably) means the quorum peers must continue to rely on ApiVersions
to negotiate compatible RPC versions and these versions cannot depend on
metadata.version.

Does this make sense?


2, ApiVersionResponse includes the broker's supported feature flags as well
as the cluster-wide finalized feature flags. We probably need to add
something like the feature flag "epoch" to this response payload in order
to see which broker is most up to date.

If the new feature flag version included RPC changes, we are helped by the
fact that a client won't attempt to use the new RPC until it has discovered
a broker that supports it via ApiVersions. The problem is more difficult
for cases like you described where the feature flag upgrade changes the
behavior of the broker, but not its RPCs. This is actually the same problem
as upgrading the IBP. During a rolling restart, clients may hit different
brokers with different capabilities and not know it.

This probably warrants further investigation, but hopefully you agree it is
beyond the scope of this KIP :)

-David


On Mon, Nov 15, 2021 at 10:26 AM David Arthur 
wrote:

> Jun, thanks for the comments!
>
> 16, When a new cluster is deployed, we don't select the highest available
> metadata.version, but rather the quorum leader picks a bootstrap version
> defined in meta.properties. As mentioned earlier, we should add validation
> here to ensure a majority of the followers could support this version
> before initializing it. This would avoid a situation where a failover
> results in a new leader who can't support the selected metadata.version.
>
> Thinking a bit more on this, we do need to define a state where we're
> running newer software, but we don't have the feature flag set. This could
> happen if we were running an older IBP that did not support KIP-778.
> Following on this, it doesn't seem too difficult to consider a case where
> the IBP has been upgraded, but we still have not finalized a
> metadata.version. Here are some possible version combinations (assuming
> KIP-778 is added to Kafka 3.2):
>
> Case  SoftwareIBPmetadata.versioneffective version
> --
> A 3.1 3.1-   0  software too old for
> feature flag
> B 3.2 3.1-   0  feature flag supported,
> but IBP too old
> C 3.2 3.2-   0  feature flag supported,
> but not initialized
> D 3.2 3.21   1  feature flag initialized
> to 1 (via operator or bootstrap process)
> ...
> E 3.8 3.1-   0  ...IBP too old
> F 3.8 3.2-   0  ...not initialized
> G 3.8 3.24   4
>
>
> Here I'm defining version 0 as "no metadata.version set". So back to your
> comment, I think the KIP omits discussion of case C from the above table
> which I can amend. Does that cover your concerns, or am I missing something
> else?
>
>
> > it's inconvenient for a user to manually upgrade every feature version
>
> For this, we would probably want to extend the capabilities of KIP-584. I
> don't think anything we've discussed for KIP-778 will preclude us from
> adding some kind of auto-upgrade in the future.
>
> 21, "disable" sounds good to me. I agree "delete feature-x" sounds a bit
> weird.
>
>
>
> On Mon, Nov 8, 2021 at 8:47 PM Guozhang Wang  wrote:
>
>> Hello David,
>>
>> Thanks for the very nice writeup! It helped me a lot to refresh my memory
>> on KIP-630/590/584 :)
>>
>> I just had two clarification questions after reading through the KIP:
>>
>> 1. For the initialization procedure, do we guarantee that all the quorum
>> nodes (inactive candidates and leaders, a.k.a. controllers) would always
>> initialize with the same metadata.version? If yes, how is that guaranteed?
>> More specifically, when a quorum candidate is starting up, would it avoid
>> handling any controller requests (including APIVersionsRequest) from its
>> peers in the quorum until it completes reading the local log? 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #566

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 500086 lines...]
[2021-11-15T15:46:20.983Z] > Task :raft:testClasses UP-TO-DATE
[2021-11-15T15:46:20.983Z] > Task :connect:json:testJar
[2021-11-15T15:46:20.983Z] > Task :connect:json:testSrcJar
[2021-11-15T15:46:20.983Z] > Task :metadata:compileTestJava UP-TO-DATE
[2021-11-15T15:46:20.983Z] > Task :metadata:testClasses UP-TO-DATE
[2021-11-15T15:46:20.983Z] > Task :core:compileScala UP-TO-DATE
[2021-11-15T15:46:20.983Z] > Task :core:classes UP-TO-DATE
[2021-11-15T15:46:20.983Z] > Task :core:compileTestJava NO-SOURCE
[2021-11-15T15:46:20.983Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-11-15T15:46:20.983Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2021-11-15T15:46:22.007Z] 
[2021-11-15T15:46:22.007Z] > Task :streams:processMessages
[2021-11-15T15:46:22.007Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2021-11-15T15:46:22.007Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-11-15T15:46:22.007Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2021-11-15T15:46:22.007Z] 
[2021-11-15T15:46:22.007Z] > Task :streams:compileJava UP-TO-DATE
[2021-11-15T15:46:22.007Z] > Task :streams:classes UP-TO-DATE
[2021-11-15T15:46:22.007Z] > Task :streams:copyDependantLibs UP-TO-DATE
[2021-11-15T15:46:22.007Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-11-15T15:46:22.007Z] > Task :streams:jar UP-TO-DATE
[2021-11-15T15:46:22.007Z] > Task :core:compileTestScala UP-TO-DATE
[2021-11-15T15:46:22.007Z] > Task :core:testClasses UP-TO-DATE
[2021-11-15T15:46:22.007Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-11-15T15:46:25.706Z] > Task :connect:api:javadoc
[2021-11-15T15:46:25.706Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-11-15T15:46:25.706Z] > Task :connect:api:jar UP-TO-DATE
[2021-11-15T15:46:25.706Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-11-15T15:46:25.706Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-11-15T15:46:25.706Z] > Task :connect:json:jar UP-TO-DATE
[2021-11-15T15:46:25.706Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-11-15T15:46:25.706Z] > Task :connect:api:javadocJar
[2021-11-15T15:46:25.706Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-11-15T15:46:25.706Z] > Task :connect:json:publishToMavenLocal
[2021-11-15T15:46:25.706Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-11-15T15:46:25.706Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-11-15T15:46:25.706Z] > Task :connect:api:testJar
[2021-11-15T15:46:25.706Z] > Task :connect:api:testSrcJar
[2021-11-15T15:46:25.706Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-11-15T15:46:25.706Z] > Task :connect:api:publishToMavenLocal
[2021-11-15T15:46:27.752Z] > Task :streams:javadoc
[2021-11-15T15:46:27.752Z] > Task :streams:javadocJar
[2021-11-15T15:46:28.777Z] > Task :streams:compileTestJava UP-TO-DATE
[2021-11-15T15:46:28.777Z] > Task :streams:testClasses UP-TO-DATE
[2021-11-15T15:46:28.777Z] > Task :streams:testJar
[2021-11-15T15:46:28.777Z] > Task :streams:testSrcJar
[2021-11-15T15:46:28.777Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2021-11-15T15:46:28.777Z] > Task :streams:publishToMavenLocal
[2021-11-15T15:46:30.917Z] > Task :clients:javadoc
[2021-11-15T15:46:30.917Z] > Task :clients:javadocJar
[2021-11-15T15:46:32.112Z] 
[2021-11-15T15:46:32.112Z] > Task :clients:srcJar
[2021-11-15T15:46:32.112Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2021-11-15T15:46:32.112Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/generated/java'. Reason: 
Task ':clients:srcJar' uses this output of task ':clients:processMessages' 
without declaring an explicit or implicit dependency. This can lead to 
incorrect results being produced, depending on what order the tasks are 
executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-11-15T15:46:32.112Z] 
[2021-11-15T15:46:32.112Z] > Task :clients:testJar
[2021-11-15T15:46:33.245Z] > Task :clients:testSrcJar

Re: [DISCUSS] KIP-778 KRaft Upgrades

[David Arthur]

Jun, thanks for the comments!

16, When a new cluster is deployed, we don't select the highest available
metadata.version, but rather the quorum leader picks a bootstrap version
defined in meta.properties. As mentioned earlier, we should add validation
here to ensure a majority of the followers could support this version
before initializing it. This would avoid a situation where a failover
results in a new leader who can't support the selected metadata.version.

Thinking a bit more on this, we do need to define a state where we're
running newer software, but we don't have the feature flag set. This could
happen if we were running an older IBP that did not support KIP-778.
Following on this, it doesn't seem too difficult to consider a case where
the IBP has been upgraded, but we still have not finalized a
metadata.version. Here are some possible version combinations (assuming
KIP-778 is added to Kafka 3.2):

Case  SoftwareIBPmetadata.versioneffective version
--
A 3.1 3.1-   0  software too old for
feature flag
B 3.2 3.1-   0  feature flag supported, but
IBP too old
C 3.2 3.2-   0  feature flag supported, but
not initialized
D 3.2 3.21   1  feature flag initialized to
1 (via operator or bootstrap process)
...
E 3.8 3.1-   0  ...IBP too old
F 3.8 3.2-   0  ...not initialized
G 3.8 3.24   4


Here I'm defining version 0 as "no metadata.version set". So back to your
comment, I think the KIP omits discussion of case C from the above table
which I can amend. Does that cover your concerns, or am I missing something
else?


> it's inconvenient for a user to manually upgrade every feature version

For this, we would probably want to extend the capabilities of KIP-584. I
don't think anything we've discussed for KIP-778 will preclude us from
adding some kind of auto-upgrade in the future.

21, "disable" sounds good to me. I agree "delete feature-x" sounds a bit
weird.



On Mon, Nov 8, 2021 at 8:47 PM Guozhang Wang  wrote:

> Hello David,
>
> Thanks for the very nice writeup! It helped me a lot to refresh my memory
> on KIP-630/590/584 :)
>
> I just had two clarification questions after reading through the KIP:
>
> 1. For the initialization procedure, do we guarantee that all the quorum
> nodes (inactive candidates and leaders, a.k.a. controllers) would always
> initialize with the same metadata.version? If yes, how is that guaranteed?
> More specifically, when a quorum candidate is starting up, would it avoid
> handling any controller requests (including APIVersionsRequest) from its
> peers in the quorum until it completes reading the local log? And even if
> yes, what would happen if there's no FeatureLevelRecord found, and
> different nodes read different values from their local meta.properties file
> or initializing from their binary's hard-code values?
>
> 2. This is not only for the metadata.version itself, but for general
> feature.versions: when a version is upgraded / downgraded, since brokers
> would read the FeatureLevelRecord at their own pace, there will always be a
> race window when some brokers has processed the record and completed the
> upgrade while others have not, hence may behave differently --- I'm
> thinking for the future like the specific replica selector to allow
> fetching from follower, and even more advanced selectors --- i.e. should we
> consider letting clients to only talk to brokers with the highest metadata
> log offsets for example?
>
>
> Guozhang
>
>
>
>
> On Fri, Nov 5, 2021 at 3:18 PM Jun Rao  wrote:
>
> > Hi, David,
> >
> > Thanks for the reply.
> >
> > 16. My first concern is that the KIP picks up meta.version inconsistently
> > during the deployment. If a new cluster is started, we pick up the
> highest
> > version. If we upgrade, we leave the feature version unchanged.
> > Intuitively, it seems that independent of how a cluster is deployed, we
> > should always pick the same feature version. I think we need to think
> > this through in this KIP. My second concern is that as a particular
> version
> > matures, it's inconvenient for a user to manually upgrade every feature
> > version. As long as we have a path to achieve that in the future, we
> don't
> > need to address that in this KIP.
> >
> > 21. "./kafka-features.sh delete": Deleting a feature seems a bit weird
> > since the logic is always there. Would it be better to use disable?
> >
> > Jun
> >
> > On Fri, Nov 5, 2021 at 8:11 AM David Arthur
> >  wrote:
> >
> > > Colin and Jun, thanks for the additional comments!
> > >
> > > Colin:
> > >
> > > > We've been talking about having an automated RPC compatibility
> checker
> > >
> > > Do we have a way to mark fields in schemas as deprecated? It can stay
> in
> > > the RPC, it just 

订阅

[天琊]

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #565

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 499913 lines...]
[2021-11-15T13:30:06.080Z] 
[2021-11-15T13:30:06.080Z] PlaintextConsumerTest > 
testConsumeMessagesWithCreateTime() PASSED
[2021-11-15T13:30:06.080Z] 
[2021-11-15T13:30:06.080Z] PlaintextConsumerTest > testAsyncCommit() STARTED
[2021-11-15T13:30:10.706Z] 
[2021-11-15T13:30:10.706Z] PlaintextConsumerTest > testAsyncCommit() PASSED
[2021-11-15T13:30:10.706Z] 
[2021-11-15T13:30:10.706Z] PlaintextConsumerTest > 
testLowMaxFetchSizeForRequestAndPartition() STARTED
[2021-11-15T13:30:52.560Z] 
[2021-11-15T13:30:52.560Z] PlaintextConsumerTest > 
testLowMaxFetchSizeForRequestAndPartition() PASSED
[2021-11-15T13:30:52.560Z] 
[2021-11-15T13:30:52.560Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnStopPolling() STARTED
[2021-11-15T13:31:06.451Z] 
[2021-11-15T13:31:06.451Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnStopPolling() PASSED
[2021-11-15T13:31:06.451Z] 
[2021-11-15T13:31:06.451Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInRevocation() STARTED
[2021-11-15T13:31:12.304Z] 
[2021-11-15T13:31:12.304Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInRevocation() PASSED
[2021-11-15T13:31:12.304Z] 
[2021-11-15T13:31:12.304Z] PlaintextConsumerTest > 
testPerPartitionLagMetricsCleanUpWithAssign() STARTED
[2021-11-15T13:31:20.977Z] 
[2021-11-15T13:31:20.977Z] PlaintextConsumerTest > 
testPerPartitionLagMetricsCleanUpWithAssign() PASSED
[2021-11-15T13:31:20.977Z] 
[2021-11-15T13:31:20.977Z] PlaintextConsumerTest > 
testPartitionsForInvalidTopic() STARTED
[2021-11-15T13:31:23.612Z] 
[2021-11-15T13:31:23.612Z] PlaintextConsumerTest > 
testPartitionsForInvalidTopic() PASSED
[2021-11-15T13:31:23.612Z] 
[2021-11-15T13:31:23.612Z] PlaintextConsumerTest > 
testPauseStateNotPreservedByRebalance() STARTED
[2021-11-15T13:31:28.407Z] 
[2021-11-15T13:31:28.407Z] PlaintextConsumerTest > 
testPauseStateNotPreservedByRebalance() PASSED
[2021-11-15T13:31:28.407Z] 
[2021-11-15T13:31:28.407Z] PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst() STARTED
[2021-11-15T13:31:33.028Z] 
[2021-11-15T13:31:33.028Z] PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst() PASSED
[2021-11-15T13:31:33.028Z] 
[2021-11-15T13:31:33.028Z] PlaintextConsumerTest > testSeek() STARTED
[2021-11-15T13:31:40.054Z] 
[2021-11-15T13:31:40.054Z] PlaintextConsumerTest > testSeek() PASSED
[2021-11-15T13:31:40.054Z] 
[2021-11-15T13:31:40.054Z] PlaintextConsumerTest > 
testConsumingWithNullGroupId() STARTED
[2021-11-15T13:31:48.700Z] 
[2021-11-15T13:31:48.700Z] PlaintextConsumerTest > 
testConsumingWithNullGroupId() PASSED
[2021-11-15T13:31:48.700Z] 
[2021-11-15T13:31:48.700Z] PlaintextConsumerTest > testPositionAndCommit() 
STARTED
[2021-11-15T13:31:53.447Z] 
[2021-11-15T13:31:53.447Z] PlaintextConsumerTest > testPositionAndCommit() 
PASSED
[2021-11-15T13:31:53.447Z] 
[2021-11-15T13:31:53.447Z] PlaintextConsumerTest > 
testFetchRecordLargerThanMaxPartitionFetchBytes() STARTED
[2021-11-15T13:31:58.079Z] 
[2021-11-15T13:31:58.079Z] PlaintextConsumerTest > 
testFetchRecordLargerThanMaxPartitionFetchBytes() PASSED
[2021-11-15T13:31:58.079Z] 
[2021-11-15T13:31:58.079Z] PlaintextConsumerTest > testUnsubscribeTopic() 
STARTED
[2021-11-15T13:32:01.843Z] 
[2021-11-15T13:32:01.843Z] PlaintextConsumerTest > testUnsubscribeTopic() PASSED
[2021-11-15T13:32:01.843Z] 
[2021-11-15T13:32:01.843Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnClose() STARTED
[2021-11-15T13:32:13.690Z] 
[2021-11-15T13:32:13.690Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnClose() PASSED
[2021-11-15T13:32:13.690Z] 
[2021-11-15T13:32:13.690Z] PlaintextConsumerTest > 
testMultiConsumerStickyAssignor() STARTED
[2021-11-15T13:32:30.034Z] 
[2021-11-15T13:32:30.034Z] PlaintextConsumerTest > 
testMultiConsumerStickyAssignor() PASSED
[2021-11-15T13:32:30.034Z] 
[2021-11-15T13:32:30.034Z] PlaintextConsumerTest > 
testFetchRecordLargerThanFetchMaxBytes() STARTED
[2021-11-15T13:32:33.625Z] 
[2021-11-15T13:32:33.625Z] PlaintextConsumerTest > 
testFetchRecordLargerThanFetchMaxBytes() PASSED
[2021-11-15T13:32:33.625Z] 
[2021-11-15T13:32:33.625Z] PlaintextConsumerTest > testAutoCommitOnClose() 
STARTED
[2021-11-15T13:32:39.577Z] 
[2021-11-15T13:32:39.577Z] PlaintextConsumerTest > testAutoCommitOnClose() 
PASSED
[2021-11-15T13:32:39.577Z] 
[2021-11-15T13:32:39.577Z] PlaintextConsumerTest > testListTopics() STARTED
[2021-11-15T13:32:42.223Z] 
[2021-11-15T13:32:42.223Z] PlaintextConsumerTest > testListTopics() PASSED
[2021-11-15T13:32:42.223Z] 
[2021-11-15T13:32:42.223Z] PlaintextConsumerTest > 
testExpandingTopicSubscriptions() STARTED
[2021-11-15T13:32:48.212Z] 
[2021-11-15T13:32:48.212Z] PlaintextConsumerTest > 
testExpandingTopicSubscriptions() PASSED
[2021-11-15T13:32:48.212Z] 
[2021-11-15T13:32:48.212Z] PlaintextConsumerTest > 

[jira] [Resolved] (KAFKA-13441) improve upgrade doc

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-13441.

Fix Version/s: 3.2.0
   Resolution: Fixed

> improve upgrade doc
> ---
>
> Key: KAFKA-13441
> URL: https://issues.apache.org/jira/browse/KAFKA-13441
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 3.0.0
>Reporter: guo
>Priority: Minor
> Fix For: 3.2.0
>
>
> improve upgrade doc



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #9

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 500798 lines...]
[2021-11-15T12:54:22.393Z] > Task :raft:testClasses UP-TO-DATE
[2021-11-15T12:54:22.393Z] > Task :connect:json:testJar
[2021-11-15T12:54:22.393Z] > Task :connect:json:testSrcJar
[2021-11-15T12:54:22.393Z] > Task :metadata:compileTestJava UP-TO-DATE
[2021-11-15T12:54:22.393Z] > Task :metadata:testClasses UP-TO-DATE
[2021-11-15T12:54:23.473Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-11-15T12:54:23.473Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2021-11-15T12:54:23.473Z] 
[2021-11-15T12:54:23.473Z] > Task :streams:processMessages
[2021-11-15T12:54:23.473Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2021-11-15T12:54:23.473Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-11-15T12:54:23.473Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2021-11-15T12:54:23.473Z] 
[2021-11-15T12:54:23.473Z] > Task :streams:compileJava UP-TO-DATE
[2021-11-15T12:54:23.473Z] > Task :streams:classes UP-TO-DATE
[2021-11-15T12:54:23.473Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-11-15T12:54:23.473Z] > Task :streams:copyDependantLibs
[2021-11-15T12:54:23.473Z] > Task :streams:jar UP-TO-DATE
[2021-11-15T12:54:23.473Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-11-15T12:54:27.406Z] > Task :connect:api:javadoc
[2021-11-15T12:54:27.406Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-11-15T12:54:27.406Z] > Task :connect:api:jar UP-TO-DATE
[2021-11-15T12:54:27.406Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-11-15T12:54:27.406Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-11-15T12:54:27.406Z] > Task :connect:json:jar UP-TO-DATE
[2021-11-15T12:54:27.406Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-11-15T12:54:27.406Z] > Task :connect:api:javadocJar
[2021-11-15T12:54:27.406Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-11-15T12:54:27.406Z] > Task :connect:json:publishToMavenLocal
[2021-11-15T12:54:27.406Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-11-15T12:54:27.406Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-11-15T12:54:27.406Z] > Task :connect:api:testJar
[2021-11-15T12:54:27.406Z] > Task :connect:api:testSrcJar
[2021-11-15T12:54:27.406Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-11-15T12:54:27.406Z] > Task :connect:api:publishToMavenLocal
[2021-11-15T12:54:30.252Z] > Task :streams:javadoc
[2021-11-15T12:54:30.252Z] > Task :streams:javadocJar
[2021-11-15T12:54:32.190Z] > Task :clients:javadoc
[2021-11-15T12:54:32.190Z] > Task :clients:javadocJar
[2021-11-15T12:54:33.119Z] 
[2021-11-15T12:54:33.119Z] > Task :clients:srcJar
[2021-11-15T12:54:33.119Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2021-11-15T12:54:33.119Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/clients/src/generated/java'.
 Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-11-15T12:54:34.047Z] 
[2021-11-15T12:54:34.047Z] > Task :clients:testJar
[2021-11-15T12:54:34.047Z] > Task :clients:testSrcJar
[2021-11-15T12:54:34.047Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2021-11-15T12:54:34.047Z] > Task :clients:publishToMavenLocal
[2021-11-15T12:54:52.966Z] > Task :core:compileScala
[2021-11-15T12:56:00.268Z] > Task :core:classes
[2021-11-15T12:56:00.268Z] > Task :core:compileTestJava NO-SOURCE
[2021-11-15T12:56:30.560Z] > Task :core:compileTestScala
[2021-11-15T12:57:27.526Z] > Task :core:testClasses
[2021-11-15T12:57:38.658Z] > Task :streams:compileTestJava
[2021-11-15T12:57:38.658Z] > Task :streams:testClasses
[2021-11-15T12:57:38.658Z] > Task :streams:testJar
[2021-11-15T12:57:39.420Z] > Task :streams:testSrcJar
[2021-11-15T12:57:39.420Z] > Task 

Re: [DISCUSS] KIP-796: Interactive Query v2

[Patrick Stuedi]

Hi John,

Thanks for submitting the KIP! One question I have is, assuming one
instantiates InteractiveQueryRequest via withQuery, and then later calls
getPositionBound, what will the result be? Also I noticed the Position
returning method is in InteractiveQueryRequest and InteractiveQueryResult
is named differently, any particular reason?

Best,
  Patrick


On Fri, Nov 12, 2021 at 12:29 AM John Roesler  wrote:

> Thanks for taking a look, Sophie!
>
> Ah, that was a revision error. I had initially been planning
> an Optional> with Optional.empty() meaning to
> fetch all partitions, but then decided it was needlessly
> complex and changed it to the current proposal with two
> methods:
>
> boolean isAllPartitions();
> Set getPartitions(); (which would throw an
> exception if it's an "all partitions" request).
>
> I've corrected the javadoc and also documented the
> exception.
>
> Thanks!
> -John
>
> On Thu, 2021-11-11 at 15:03 -0800, Sophie Blee-Goldman
> wrote:
> > Thanks John, I've been looking forward to this for a while now. It
> > was pretty horrifying to learn
> > how present-day IQ works  (or rather, doesn't work) with custom state
> > stores :/
> >
> > One minor cosmetic point, In the InteractiveQueryRequest class, the #
> > getPartitions
> > method has a return type of Set, but the javadocs refer to
> Optional.
> > Not
> > sure which is intended for this API, but if is supposed to be the return
> > type, do you perhaps
> > mean for it to be  Optional.ofEmpty() and Optional.of(non-empty set)
> > rather than Optional.of(empty set) and Optional.of(non-empty set) ?
> >
> > On Thu, Nov 11, 2021 at 12:03 PM John Roesler 
> wrote:
> >
> > > Hello again, all,
> > >
> > > Just bumping this discussion on a new, more flexible
> > > Interactive Query API in Kafka Streams.
> > >
> > > If there are no concerns, I'll go ahead and call a vote on
> > > Monday.
> > >
> > > Thanks!
> > > -John
> > >
> > > On Tue, 2021-11-09 at 17:37 -0600, John Roesler wrote:
> > > > Hello all,
> > > >
> > > > I'd like to start the discussion for KIP-796, which proposes
> > > > a revamp of the Interactive Query APIs in Kafka Streams.
> > > >
> > > > The proposal is here:
> > > > https://cwiki.apache.org/confluence/x/34xnCw
> > > >
> > > > I look forward to your feedback!
> > > >
> > > > Thank you,
> > > > -John
> > > >
> > >
> > >
> > >
>
>
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #564

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Mickael Maison]

Hi Dongjin,

Thanks for the clarifications.

I wonder if a simpler course of action could be:
- Deprecate log4j-appender now
- Document how to use logging-log4j2
- Remove log4j-appender and all the log4j dependencies in Kafka 4.0

This delays KIP-653 till Kafka 4.0 but (so far) Kafka is not directly
affected by the log4j CVEs. At least this gives us a clear and simple
roadmap to follow.

What do you think?

On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee  wrote:
>
> Hi Mickael,
>
> I greatly appreciate you for reading the proposal so carefully! I wrote it
> quite a while ago and rechecked it today.
>
> > Is the KIP proposing to replace the existing log4-appender or simply add
> a new one for log4j2? Reading the KIP and with its current title, it's not
> entirely explicit.
>
> Oh, After re-reading it, I realized that this is not clear. Let me clarify;
>
> 1. Provide a lo4j2 equivalent of traditional log4j-appender,
> log4j2-appender.
> 2. Migrate the modules depending on log4j-appender (i.e., tools, trogdor,
> shell) into log4j2-appender, removing log4j-appender from dependencies.
> 3. Entirely remove log4j-appender from the project dependencies, along with
> log4j.
>
> I think log4j-appender may be published for every new release like before,
> but the committee should make a decision on the policy.
>
> > Under Rejected Alternative, the KIP states: "the Kafka appender provided
> by log4j2 community stores log message in the Record key". Looking at the
> code, it looks like the log message is stored in the Record value:
> https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135
> Am I missing something?
>
> It's totally my fault; I confused it with another appender. The
> compatibility problem in the logging-log4j2 Kafka appender is not the
> format but the configuration. logging-log4j2 Kafka appender supports
> `properties` configuration, which will be directly used to instantiate a
> Kafka producer. However, log4j-appender has been using non-producer config
> names like brokerList (=bootstrap.servers), requiredNumAcks (=acks).
> Instead, logging-log4j2 Kafka appender supports retryCount,
> sendEventTimestamp.
>
> On second thought, using logging-log4j2 Kafka appender internally and
> making log4j2-appender to focus on compatibility facade only would be a
> better approach; As I described above, the goal of this module is just
> keeping the backward-compatibility, and (as you pointed out) the current
> implementation has little value. Since org.apache.logging.log4j:log4j-core
> already includes Kafka appender, we can make use of the 'proven wheel'
> without adding more dependencies. I have not tried it yet, but I think it
> is well worth it. (One additional advantage of this approach is providing a
> bridge to the users who hope to move from/into logging-log4j2 Kafka
> appender.)
>
> > As the current log4j-appender is not even deprecated yet, in theory we
> can't remove it till Kafka 4. If we want to speed up the process, I wonder
> if the lack of documentation and a migration guide could help us. What do
> you think?
>
> In fact, this is what I am doing nowadays. While working with
> log4j-appender, I found that despite a lack of documentation, considerable
> users are already using it[^1][^2][^3][^4][^5]. So, I think providing a
> documentation to those who are already using log4j-appender is
> indispensable. It should include:
>
> - What is the difference between log4j-appender vs. log4j2-appender.
> - Which options are supported and deprecated.
> - Exemplar configurations that show how to migrate.
>
> Here is the summary:
>
> 1. The goal of this proposal is to replace the traditional log4j-appender
> for compatibility concerns. But log4j-appender may be published after the
> deprecation.
> 2. As of present, the description about logging-log4j2 Kafka appender is
> entirely wrong. The problem is interface compatibility, not record format.
> Focusing on the compatibility facade is a good approach.
> 3. A documentation focus on migration should be provided.
>
> If you have any questions or suggestions, don't hesitate to tell me. Thanks
> again for your comments!
>
> Best,
> Dongjin
>
> [^1]:
> https://docs.cloudera.com/csa/1.2.0/monitoring/topics/csa-kafka-logging.html
> [^2]:
> https://stackoverflow.com/questions/22034895/how-to-use-kafka-0-8-log4j-appender
> [^3]:
> https://stackoverflow.com/questions/32402405/delay-in-kafka-log4j-appender
> [^4]:
> https://stackoverflow.com/questions/32301129/kafka-log4j-appender-not-sending-messages
> [^5]:
> https://stackoverflow.com/questions/35628706/kafka-log4j-appender-0-9-does-not-work
>
> On Mon, Nov 8, 2021 at 9:04 PM Mickael Maison 
> wrote:
>
> > Hi Dongjin,
> >
> > Thanks for working on the update to log4j2, it's definitively
> > something we should complete.
> > I have a couple of comments:
> >
> > 1) Is the KIP proposing to replace the existing log4-appender or
> > simply add 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #8

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 498870 lines...]
[2021-11-15T10:53:36.513Z] [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ streams-quickstart-java ---
[2021-11-15T10:53:36.513Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-11-15T10:53:36.513Z] [INFO] Copying 6 resources
[2021-11-15T10:53:36.513Z] [INFO] Copying 3 resources
[2021-11-15T10:53:36.513Z] [INFO] 
[2021-11-15T10:53:36.513Z] [INFO] --- maven-resources-plugin:2.7:testResources 
(default-testResources) @ streams-quickstart-java ---
[2021-11-15T10:53:36.513Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-11-15T10:53:36.513Z] [INFO] Copying 2 resources
[2021-11-15T10:53:36.513Z] [INFO] Copying 3 resources
[2021-11-15T10:53:36.513Z] [INFO] 
[2021-11-15T10:53:36.513Z] [INFO] --- maven-archetype-plugin:2.2:jar 
(default-jar) @ streams-quickstart-java ---
[2021-11-15T10:53:36.513Z] [INFO] Building archetype jar: 
/home/jenkins/workspace/Kafka_kafka_3.1/streams/quickstart/java/target/streams-quickstart-java-3.1.0-SNAPSHOT
[2021-11-15T10:53:36.513Z] [INFO] 
[2021-11-15T10:53:36.513Z] [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart-java ---
[2021-11-15T10:53:36.513Z] [INFO] 
[2021-11-15T10:53:36.513Z] [INFO] --- 
maven-archetype-plugin:2.2:integration-test (default-integration-test) @ 
streams-quickstart-java ---
[2021-11-15T10:53:36.513Z] [INFO] 
[2021-11-15T10:53:36.513Z] [INFO] --- maven-gpg-plugin:1.6:sign 
(sign-artifacts) @ streams-quickstart-java ---
[2021-11-15T10:53:36.513Z] [INFO] 
[2021-11-15T10:53:36.513Z] [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ streams-quickstart-java ---
[2021-11-15T10:53:36.513Z] [INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_3.1/streams/quickstart/java/target/streams-quickstart-java-3.1.0-SNAPSHOT.jar
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.1.0-SNAPSHOT/streams-quickstart-java-3.1.0-SNAPSHOT.jar
[2021-11-15T10:53:36.513Z] [INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_3.1/streams/quickstart/java/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.1.0-SNAPSHOT/streams-quickstart-java-3.1.0-SNAPSHOT.pom
[2021-11-15T10:53:36.513Z] [INFO] 
[2021-11-15T10:53:36.513Z] [INFO] --- 
maven-archetype-plugin:2.2:update-local-catalog (default-update-local-catalog) 
@ streams-quickstart-java ---
[2021-11-15T10:53:36.513Z] [INFO] 

[2021-11-15T10:53:36.513Z] [INFO] Reactor Summary for Kafka Streams :: 
Quickstart 3.1.0-SNAPSHOT:
[2021-11-15T10:53:36.513Z] [INFO] 
[2021-11-15T10:53:36.513Z] [INFO] Kafka Streams :: Quickstart 
 SUCCESS [  1.578 s]
[2021-11-15T10:53:36.513Z] [INFO] streams-quickstart-java 
 SUCCESS [  0.679 s]
[2021-11-15T10:53:36.513Z] [INFO] 

[2021-11-15T10:53:36.513Z] [INFO] BUILD SUCCESS
[2021-11-15T10:53:36.513Z] [INFO] 

[2021-11-15T10:53:36.513Z] [INFO] Total time:  2.501 s
[2021-11-15T10:53:36.513Z] [INFO] Finished at: 2021-11-15T10:53:35Z
[2021-11-15T10:53:36.513Z] [INFO] 

[Pipeline] dir
[2021-11-15T10:53:37.039Z] Running in 
/home/jenkins/workspace/Kafka_kafka_3.1/streams/quickstart/test-streams-archetype
[Pipeline] {
[Pipeline] sh
[2021-11-15T10:53:37.746Z] 
[2021-11-15T10:53:37.746Z] ControllerIntegrationTest > 
testTopicIdUpgradeAfterReassigningPartitions() PASSED
[2021-11-15T10:53:37.746Z] 
[2021-11-15T10:53:37.746Z] ControllerIntegrationTest > 
testTopicIdPersistsThroughControllerReelection() STARTED
[2021-11-15T10:53:39.589Z] + echo Y
[2021-11-15T10:53:39.589Z] + mvn archetype:generate -DarchetypeCatalog=local 
-DarchetypeGroupId=org.apache.kafka 
-DarchetypeArtifactId=streams-quickstart-java -DarchetypeVersion=3.1.0-SNAPSHOT 
-DgroupId=streams.examples -DartifactId=streams.examples -Dversion=0.1 
-Dpackage=myapps
[2021-11-15T10:53:40.539Z] [INFO] Scanning for projects...
[2021-11-15T10:53:40.539Z] [INFO] 
[2021-11-15T10:53:40.539Z] [INFO] --< 
org.apache.maven:standalone-pom >---
[2021-11-15T10:53:40.539Z] [INFO] Building Maven Stub Project (No POM) 1
[2021-11-15T10:53:40.539Z] [INFO] [ pom 
]-
[2021-11-15T10:53:40.539Z] [INFO] 
[2021-11-15T10:53:40.539Z] [INFO] >>> maven-archetype-plugin:3.2.0:generate 
(default-cli) > generate-sources @ standalone-pom >>>
[2021-11-15T10:53:40.539Z] [INFO] 
[2021-11-15T10:53:40.539Z] [INFO] <<< maven-archetype-plugin:3.2.0:generate 
(default-cli) < generate-sources @ standalone-pom <<<

[jira] [Resolved] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-13255.

Fix Version/s: 3.2.0
   Resolution: Fixed

> Mirrormaker config property config.properties.exclude is not working as 
> expected 
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Assignee: Ed Berezitsky
>Priority: Major
> Fix For: 3.2.0
>
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration 
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
>  # Started kafka-connect service.
>  # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and 
> heartbeat connector). Curl commands used to create connectors are in the 
> attached file.  To exclude certain config properties while topic replication, 
> we are using the 'config.properties.exclude' property in the MM2 source 
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created 
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination 
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic 
> dev.portlandDc.anamika.helloMsk. 
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic 
> config name: confluent.value.schema.validation{code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Apache Kafka 3.1.0 release

[David Jacot]

Hi folks,

We reached the code freeze for the Apache Kafka 3.1 release on Friday.
Therefore,
we will only accept blockers from now on.

There already are a couple of blockers identified which were not
completed before
the code freeze. Please, raise any new blockers to this thread.

For all the non-blocker issues targeting 3.1.0, I will move them to
the next release.

Cheers,
David

On Fri, Oct 29, 2021 at 12:20 PM Dongjin Lee  wrote:
>
> Hi David,
>
> Please update the components of the following KIPs:
>
> - KIP-390: Support Compression Level - Core, Clients
> - KIP-653: Upgrade log4j to log4j2 - Clients, Connect, Core, Streams (that
> is, Log4j-appender, Tools, and Trogdor are excluded.)
>
> Best,
> Dongjin
>
> On Fri, Oct 29, 2021 at 2:24 AM Chris Egerton 
> wrote:
>
> > Hi David,
> >
> > I've moved KIP-618 to the "postponed" section as it will not be merged in
> > time due to lack of review.
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Oct 28, 2021 at 1:07 PM David Jacot 
> > wrote:
> >
> > > Hi team,
> > >
> > > Just a quick reminder that the Feature freeze is tomorrow (October 29th).
> > > In order to be fair with everyone in all the time zones, I plan to cut
> > the
> > > release branch early next week.
> > >
> > > Cheers,
> > > David
> > >
> > > On Mon, Oct 18, 2021 at 9:56 AM David Jacot  wrote:
> > >
> > > > Hi team,
> > > >
> > > > KIP freeze for the next major release of Apache Kafka was reached
> > > > last week.
> > > >
> > > > I have updated the release plan with all the adopted KIPs which are
> > > > considered
> > > > for AK 3.1.0. Please, verify the plan and let me know if any KIP should
> > > be
> > > > added
> > > > to or removed from the release plan.
> > > >
> > > > For the KIPs which are still in progress, please work closely with your
> > > > reviewers
> > > > to make sure that they land on time for the feature freeze.
> > > >
> > > > The next milestone for the AK 3.1.0 release is the feature freeze on
> > > > October 29th,
> > > > 2021.
> > > >
> > > > Cheers,
> > > > David
> > > >
> > > > On Fri, Oct 15, 2021 at 9:05 AM David Jacot 
> > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> Just a quick reminder that the KIP freeze is today. Don't forget to
> > > close
> > > >> your ongoing votes.
> > > >>
> > > >> Best,
> > > >> David
> > > >>
> > > >> On Thu, Oct 14, 2021 at 5:31 PM David Jacot 
> > > wrote:
> > > >>
> > > >>> Hi Luke,
> > > >>>
> > > >>> Added it to the plan.
> > > >>>
> > > >>> Thanks,
> > > >>> David
> > > >>>
> > > >>> On Thu, Oct 14, 2021 at 10:09 AM Luke Chen 
> > wrote:
> > > >>>
> > >  Hi David,
> > >  KIP-766 is merged into trunk. Please help add it into the release
> > > plan.
> > > 
> > >  Thank you.
> > >  Luke
> > > 
> > >  On Mon, Oct 11, 2021 at 10:50 PM David Jacot
> > >  
> > >  wrote:
> > > 
> > >  > Hi Michael,
> > >  >
> > >  > Sure. I have updated the release plan to include it. Thanks for
> > the
> > >  > heads up.
> > >  >
> > >  > Best,
> > >  > David
> > >  >
> > >  > On Mon, Oct 11, 2021 at 4:39 PM Mickael Maison <
> > >  mickael.mai...@gmail.com>
> > >  > wrote:
> > >  >
> > >  > > Hi David,
> > >  > >
> > >  > > You can add KIP-690 to the release plan. The vote passed months
> > > ago
> > >  > > and I merged the PR today.
> > >  > >
> > >  > > Thanks
> > >  > >
> > >  > > On Fri, Oct 8, 2021 at 8:32 AM David Jacot
> > >  
> > >  > > wrote:
> > >  > > >
> > >  > > > Hi folks,
> > >  > > >
> > >  > > > Just a quick reminder that KIP Freeze is next Friday, October
> > >  15th.
> > >  > > >
> > >  > > > Cheers,
> > >  > > > David
> > >  > > >
> > >  > > > On Wed, Sep 29, 2021 at 3:52 PM Chris Egerton
> > >  > > 
> > >  > > > wrote:
> > >  > > >
> > >  > > > > Thanks David!
> > >  > > > >
> > >  > > > > On Wed, Sep 29, 2021 at 2:56 AM David Jacot
> > >  > > 
> > >  > > > > wrote:
> > >  > > > >
> > >  > > > > > Hi Chris,
> > >  > > > > >
> > >  > > > > > Sure thing. I have added KIP-618 to the release plan.
> > Thanks
> > >  for
> > >  > the
> > >  > > > > heads
> > >  > > > > > up.
> > >  > > > > >
> > >  > > > > > Best,
> > >  > > > > > David
> > >  > > > > >
> > >  > > > > > On Wed, Sep 29, 2021 at 8:53 AM David Jacot <
> > >  dja...@confluent.io>
> > >  > > wrote:
> > >  > > > > >
> > >  > > > > > > Hi Kirk,
> > >  > > > > > >
> > >  > > > > > > Yes, it is definitely possible if you can get the KIP
> > > voted
> > >  > before
> > >  > > the
> > >  > > > > > KIP
> > >  > > > > > > freeze
> > >  > > > > > > and the code committed before the feature freeze.
> > Please,
> > >  let me
> > >  > > know
> > >  > > > > > when
> > >  > > > > > > the
> > >  > > > > > > KIP is voted and I will add it to the release plan.
> > >  > > > > 

[GitHub] [kafka-site] mimaison commented on pull request #383: MINOR: Updates for 2.7.2

[GitBox]

mimaison commented on pull request #383:
URL: https://github.com/apache/kafka-site/pull/383#issuecomment-968706179


   Waiting for https://github.com/apache/kafka/pull/11497 to be merged first


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mimaison commented on pull request #382: MINOR: Updates for 2.6.3

[GitBox]

mimaison commented on pull request #382:
URL: https://github.com/apache/kafka-site/pull/382#issuecomment-968705989


   Waiting for https://github.com/apache/kafka/pull/11495 to be merged first


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-13111) Re-evaluate Fetch Sessions when using topic IDs

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13111.
-
  Reviewer: David Jacot
Resolution: Fixed

> Re-evaluate Fetch Sessions when using topic IDs
> ---
>
> Key: KAFKA-13111
> URL: https://issues.apache.org/jira/browse/KAFKA-13111
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Justine Olshan
>Assignee: Justine Olshan
>Priority: Blocker
> Fix For: 3.1.0
>
>
> For fetch request version 13 we have the current method of handling unknown 
> topic IDs.
>  * When the receiving broker sees an unknown topic ID in a request or 
> encounters an inconsistent (mismatched) ID in the logs, it sends a top-level 
> error back, delays *all* partitions (in fetcher thread), and closes the 
> session
> Ideally, we want to handle the same way as unknown topic names. We hold the 
> topic partition in the session and try to resolve on a future fetch request. 
> However, there are a few complications with this approach and this is why we 
> opted to simply close the session. One is that many objects in the fetch 
> session are keyed using topic name+partition. We also still need the topic 
> name for a few other checks in the fetch path.
> Still, upon further inspection, closing the session on any new topics (when 
> using topic names, we often see a few unknown topic or partition exceptions) 
> is not ideal.
> One way to see similar behavior in the topic ID case is to store unresolved 
> IDs in the session. For compatibility reasons, we will need to add and/or 
> generify a few classes. The general idea is that for sessions using version 
> 13+ we will use a structure that keys using topic ID and partition with an 
> optional name. That way, we can send partition level errors for unknown topic 
> IDs and handle them later in the same session. We can also remove partitions 
> using topic ID more easily if the unknown topic ID was due to stale metadata.
> Then the new method will be as follows:
>  * When the receiving broker sees an unknown topic ID in a request or 
> encounters an inconsistent (mismatched) ID in the logs, it sends an error on 
> the unknown partition, delay *only those* partitions (in fetcher thread), and 
> keep the session open to try to resolve in the future



--
This message was sent by Atlassian Jira
(v8.20.1#820001)