[jira] [Commented] (KAFKA-8803) Stream will not start due to TimeoutException: Timeout expired after 60000milliseconds while awaiting InitProducerId
[
https://issues.apache.org/jira/browse/KAFKA-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058444#comment-17058444
]
Guozhang Wang commented on KAFKA-8803:
--
[~rocketraman] I took some time on the logs, and I now wonder if you could try
out the just released 2.4.1 which has this ticket resolved:
https://issues.apache.org/jira/browse/KAFKA-9144. From the logs I suspect it
was likely triggered.
> Stream will not start due to TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
>
>
> Key: KAFKA-8803
> URL: https://issues.apache.org/jira/browse/KAFKA-8803
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Raman Gupta
>Assignee: Sophie Blee-Goldman
>Priority: Major
> Attachments: logs-20200311.txt.gz, logs-client-20200311.txt.gz,
> logs.txt.gz, screenshot-1.png
>
>
> One streams app is consistently failing at startup with the following
> exception:
> {code}
> 2019-08-14 17:02:29,568 ERROR --- [2ce1b-StreamThread-2]
> org.apa.kaf.str.pro.int.StreamTask: task [0_36] Timeout
> exception caught when initializing transactions for task 0_36. This might
> happen if the broker is slow to respond, if the network connection to the
> broker was interrupted, or if similar circumstances arise. You can increase
> producer parameter `max.block.ms` to increase this timeout.
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> {code}
> These same brokers are used by many other streams without any issue,
> including some in the very same processes for the stream which consistently
> throws this exception.
> *UPDATE 08/16:*
> The very first instance of this error is August 13th 2019, 17:03:36.754 and
> it happened for 4 different streams. For 3 of these streams, the error only
> happened once, and then the stream recovered. For the 4th stream, the error
> has continued to happen, and continues to happen now.
> I looked up the broker logs for this time, and see that at August 13th 2019,
> 16:47:43, two of four brokers started reporting messages like this, for
> multiple partitions:
> [2019-08-13 20:47:43,658] INFO [ReplicaFetcher replicaId=3, leaderId=1,
> fetcherId=0] Retrying leaderEpoch request for partition xxx-1 as the leader
> reported an error: UNKNOWN_LEADER_EPOCH (kafka.server.ReplicaFetcherThread)
> The UNKNOWN_LEADER_EPOCH messages continued for some time, and then stopped,
> here is a view of the count of these messages over time:
> !screenshot-1.png!
> However, as noted, the stream task timeout error continues to happen.
> I use the static consumer group protocol with Kafka 2.3.0 clients and 2.3.0
> broker. The broker has a patch for KAFKA-8773.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9144) Early expiration of producer state can cause coordinator epoch to regress
[
https://issues.apache.org/jira/browse/KAFKA-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058443#comment-17058443
]
Jason Gustafson commented on KAFKA-9144:
We found that this bug can also result in a hanging transaction. We had one
instance of this and found the following in the log dump:
{code}
baseOffset: 21830 lastOffset: 21830 count: 1 baseSequence: -1 lastSequence: -1
producerId: 15038 producerEpoch: 17 partitionLeaderEpoch: 7 isTransactional:
true isControl: true position: 499861 CreateTime: 1566838946496 size: 78 magic:
2 compresscodec: NONE crc: 4211809197 isvalid: true
| offset: 21830 CreateTime: 1566838946496 keysize: 4 valuesize: 6 sequence: -1
headerKeys: [] endTxnMarker: COMMIT coordinatorEpoch: 7
baseOffset: 22401 lastOffset: 22401 count: 1 baseSequence: -1 lastSequence: -1
producerId: 15038 producerEpoch: 19 partitionLeaderEpoch: 7 isTransactional:
true isControl: true position: 600640 CreateTime: 1566857918542 size: 78 magic:
2 compresscodec: NONE crc: 1432605016 isvalid: true
| offset: 22401 CreateTime: 1566857918542 keysize: 4 valuesize: 6 sequence: -1
headerKeys: [] endTxnMarker: ABORT coordinatorEpoch: 7
baseOffset: 22422 lastOffset: 22422 count: 1 baseSequence: 0 lastSequence: 0
producerId: 15038 producerEpoch: 18 partitionLeaderEpoch: 7 isTransactional:
true isControl: false position: 606629 CreateTime: 1566858389995 size: 187
magic: 2 compresscodec: LZ4 crc: 286798916 isvalid: true
| offset: 22422 CreateTime: 1566858389995 keysize: 83 valuesize: 24 sequence: 0
headerKeys: []
{code}
The interesting thing to note is that the producer epoch went backwards. What
we believe happened is the following:
1. producer opens a transaction with epoch 18 but loses communication with the
cluster
2. coordinator decides to abort the transaction, so bumps the epoch to 19 and
writes markers
3. due to the bug in this JIRA, producer state is cleaned up before proper
expiration
4. the producer who is now a zombie tries to write with epoch 18.
5. the broker accepts the write because the sequence is 0 and previous state
has been expired
Following this sequence, the transaction would be left open because it was
appended to the log after it had been aborted.
> Early expiration of producer state can cause coordinator epoch to regress
> -
>
> Key: KAFKA-9144
> URL: https://issues.apache.org/jira/browse/KAFKA-9144
> Project: Kafka
> Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.4.1
>
>
> Transaction markers are written by the transaction coordinator. In order to
> fence zombie coordinators, we use the leader epoch associated with the
> coordinator partition. Partition leaders verify the epoch in the
> WriteTxnMarker request and ensure that it can only increase. However, when
> producer state expires, we stop tracking the epoch and it is possible for
> monotonicity to be violated. Generally we expect expiration to be on the
> order of days, so it should be unlikely for this to be a problem.
> At least that is the theory. We observed a case where a coordinator epoch
> decreased between nearly consecutive writes within a couple minutes of each
> other. Upon investigation, we found that producer state had been incorrectly
> expired. We believe the sequence of events is the following:
> # Producer writes transactional data and fails before committing
> # Coordinator times out the transaction and writes ABORT markers
> # Upon seeing the ABORT and the bumped epoch, the partition leader deletes
> state from the last epoch, which effectively resets the last timestamp for
> the producer to -1.
> # The coordinator becomes a zombie before getting a successful response and
> continues trying to send
> # The new coordinator notices the incomplete transaction and also sends
> markers
> # The partition leader accepts the write from the new coordinator
> # The producer state is expired because the last timestamp was -1
> # The partition leader accepts the write from the old coordinator
> Basically it takes an alignment of planets to hit this bug, but it is
> possible. If you hit it, then the broker may be unable to start because we
> validate epoch monotonicity during log recovery. The problem is in 3 when the
> timestamp gets reset. We should use the timestamp from the marker instead.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9657) Add configurable throw on unsupported protocol
[ https://issues.apache.org/jira/browse/KAFKA-9657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058435#comment-17058435 ] ASF GitHub Bot commented on KAFKA-9657: --- guozhangwang commented on pull request #8265: KAFKA-9657: Throw upon offset fetch unsupported stable flag protocol URL: https://github.com/apache/kafka/pull/8265 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add configurable throw on unsupported protocol > -- > > Key: KAFKA-9657 > URL: https://issues.apache.org/jira/browse/KAFKA-9657 > Project: Kafka > Issue Type: Sub-task >Reporter: Boyang Chen >Priority: Major > > Right now Stream could not handle the case when the brokers are downgraded, > thus potentially could violate EOS requirement. We could add an (internal) > config to either consumer or producer to actually crash on unsupported > version when the broker connecting to is on an older version unexpectedly, to > prevent this case from causing correctness concern. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9715) TransactionStateManager: Eliminate unused reference to interBrokerProtocolVersion
[ https://issues.apache.org/jira/browse/KAFKA-9715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kowshik Prakasam updated KAFKA-9715: Description: In TransactionStateManager, the attribute interBrokerProtocolVersion is unused within the class. It can therefore be eliminated from the code. [https://github.com/apache/kafka/blob/07db26c20fcbccbf758591607864f7fd4bd8975f/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L78] was: In TransactionStateManager, the attribute interBrokerProtocolVersion is unused. It can therefore be eliminated from the code. [https://github.com/apache/kafka/blob/07db26c20fcbccbf758591607864f7fd4bd8975f/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L78] > TransactionStateManager: Eliminate unused reference to > interBrokerProtocolVersion > - > > Key: KAFKA-9715 > URL: https://issues.apache.org/jira/browse/KAFKA-9715 > Project: Kafka > Issue Type: Improvement >Reporter: Kowshik Prakasam >Priority: Minor > > In TransactionStateManager, the attribute interBrokerProtocolVersion is > unused within the class. It can therefore be eliminated from the code. > [https://github.com/apache/kafka/blob/07db26c20fcbccbf758591607864f7fd4bd8975f/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L78] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9715) TransactionStateManager: Eliminate unused reference to interBrokerProtocolVersion
[ https://issues.apache.org/jira/browse/KAFKA-9715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kowshik Prakasam updated KAFKA-9715: Description: In TransactionStateManager, the attribute interBrokerProtocolVersion is unused within the class. It can therefore be eliminated from the code. Please refer to this LOC: [https://github.com/apache/kafka/blob/07db26c20fcbccbf758591607864f7fd4bd8975f/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L78] was: In TransactionStateManager, the attribute interBrokerProtocolVersion is unused within the class. It can therefore be eliminated from the code. [https://github.com/apache/kafka/blob/07db26c20fcbccbf758591607864f7fd4bd8975f/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L78] > TransactionStateManager: Eliminate unused reference to > interBrokerProtocolVersion > - > > Key: KAFKA-9715 > URL: https://issues.apache.org/jira/browse/KAFKA-9715 > Project: Kafka > Issue Type: Improvement >Reporter: Kowshik Prakasam >Priority: Minor > > In TransactionStateManager, the attribute interBrokerProtocolVersion is > unused within the class. It can therefore be eliminated from the code. Please > refer to this LOC: > [https://github.com/apache/kafka/blob/07db26c20fcbccbf758591607864f7fd4bd8975f/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L78] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9715) TransactionStateManager: Eliminate unused reference to interBrokerProtocolVersion
Kowshik Prakasam created KAFKA-9715: --- Summary: TransactionStateManager: Eliminate unused reference to interBrokerProtocolVersion Key: KAFKA-9715 URL: https://issues.apache.org/jira/browse/KAFKA-9715 Project: Kafka Issue Type: Improvement Reporter: Kowshik Prakasam In TransactionStateManager, the attribute interBrokerProtocolVersion is unused. It can therefore be eliminated from the code. [https://github.com/apache/kafka/blob/07db26c20fcbccbf758591607864f7fd4bd8975f/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L78] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9689) Automatic broker version detection to initialize stream client
[ https://issues.apache.org/jira/browse/KAFKA-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen reassigned KAFKA-9689: -- Assignee: Boyang Chen > Automatic broker version detection to initialize stream client > -- > > Key: KAFKA-9689 > URL: https://issues.apache.org/jira/browse/KAFKA-9689 > Project: Kafka > Issue Type: Sub-task >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Major > > Eventually we shall deprecate the flag to suppress EOS thread producer > feature, instead we take version detection approach on broker to decide which > semantic to use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9714) Flaky Test SslTransportLayerTest#testTLSDefaults
Matthias J. Sax created KAFKA-9714:
--
Summary: Flaky Test SslTransportLayerTest#testTLSDefaults
Key: KAFKA-9714
URL: https://issues.apache.org/jira/browse/KAFKA-9714
Project: Kafka
Issue Type: Bug
Components: core, security, unit tests
Reporter: Matthias J. Sax
[https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/5145/testReport/junit/org.apache.kafka.common.network/SslTransportLayerTest/testTLSDefaults_tlsProtocol_TLSv1_2_/]
{quote}java.lang.AssertionError: Metric not updated failed-authentication-total
expected:<0.0> but was:<1.0> expected:<0.0> but was:<1.0> at
org.junit.Assert.fail(Assert.java:89) at
org.junit.Assert.failNotEquals(Assert.java:835) at
org.junit.Assert.assertEquals(Assert.java:555) at
org.apache.kafka.common.network.NioEchoServer.waitForMetrics(NioEchoServer.java:194)
at
org.apache.kafka.common.network.NioEchoServer.verifyAuthenticationMetrics(NioEchoServer.java:156)
at
org.apache.kafka.common.network.SslTransportLayerTest.testTLSDefaults(SslTransportLayerTest.java:571){quote}
STDOUT
{quote}[2020-03-12 17:03:44,617] ERROR Modification time of key store could not
be obtained: some.truststore.path
(org.apache.kafka.common.security.ssl.SslEngineBuilder:300)
java.nio.file.NoSuchFileException: some.truststore.path at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
at
java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1763) at
java.base/java.nio.file.Files.getLastModifiedTime(Files.java:2314) at
org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.lastModifiedMs(SslEngineBuilder.java:298)
at
org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.(SslEngineBuilder.java:275)
at
org.apache.kafka.common.security.ssl.SslEngineBuilder.createTruststore(SslEngineBuilder.java:182)
at
org.apache.kafka.common.security.ssl.SslEngineBuilder.(SslEngineBuilder.java:100)
at
org.apache.kafka.common.security.ssl.SslFactory.createNewSslEngineBuilder(SslFactory.java:140)
at
org.apache.kafka.common.security.ssl.SslFactory.validateReconfiguration(SslFactory.java:114)
at
org.apache.kafka.common.network.SslChannelBuilder.validateReconfiguration(SslChannelBuilder.java:85)
at
org.apache.kafka.common.network.SslTransportLayerTest.verifyInvalidReconfigure(SslTransportLayerTest.java:1123)
at
org.apache.kafka.common.network.SslTransportLayerTest.testServerTruststoreDynamicUpdate(SslTransportLayerTest.java:1113){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8939) Flaky test ReassignPartitionsClusterTest#shouldTriggerReassignmentWithZnodePrecedenceOnControllerStartup
[ https://issues.apache.org/jira/browse/KAFKA-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058382#comment-17058382 ] Matthias J. Sax commented on KAFKA-8939: [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/5153/testReport/junit/kafka.admin/ReassignPartitionsClusterTest/shouldTriggerReassignmentWithZnodePrecedenceOnControllerStartup/] > Flaky test > ReassignPartitionsClusterTest#shouldTriggerReassignmentWithZnodePrecedenceOnControllerStartup > > > Key: KAFKA-8939 > URL: https://issues.apache.org/jira/browse/KAFKA-8939 > Project: Kafka > Issue Type: Bug > Components: admin, core >Affects Versions: 2.4.0 >Reporter: Sophie Blee-Goldman >Priority: Major > Labels: flaky-test > > h3. Stacktrace > java.lang.AssertionError: expected: but was: at > org.junit.Assert.fail(Assert.java:89) at > org.junit.Assert.failNotEquals(Assert.java:835) at > org.junit.Assert.assertEquals(Assert.java:120) at > org.junit.Assert.assertEquals(Assert.java:146) at > kafka.admin.ReassignPartitionsClusterTest.shouldTriggerReassignmentWithZnodePrecedenceOnControllerStartup(ReassignPartitionsClusterTest.scala:687) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at > org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at > org.junit.runners.ParentRunner.run(ParentRunner.java:412) at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) > at > org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) > at jdk.internal.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) > at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at > org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) > at jdk.internal.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at >
[jira] [Commented] (KAFKA-8939) Flaky test ReassignPartitionsClusterTest#shouldTriggerReassignmentWithZnodePrecedenceOnControllerStartup
[ https://issues.apache.org/jira/browse/KAFKA-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058381#comment-17058381 ] Matthias J. Sax commented on KAFKA-8939: [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/5145/testReport/junit/kafka.admin/ReassignPartitionsClusterTest/shouldTriggerReassignmentWithZnodePrecedenceOnControllerStartup/] > Flaky test > ReassignPartitionsClusterTest#shouldTriggerReassignmentWithZnodePrecedenceOnControllerStartup > > > Key: KAFKA-8939 > URL: https://issues.apache.org/jira/browse/KAFKA-8939 > Project: Kafka > Issue Type: Bug > Components: admin, core >Affects Versions: 2.4.0 >Reporter: Sophie Blee-Goldman >Priority: Major > Labels: flaky-test > > h3. Stacktrace > java.lang.AssertionError: expected: but was: at > org.junit.Assert.fail(Assert.java:89) at > org.junit.Assert.failNotEquals(Assert.java:835) at > org.junit.Assert.assertEquals(Assert.java:120) at > org.junit.Assert.assertEquals(Assert.java:146) at > kafka.admin.ReassignPartitionsClusterTest.shouldTriggerReassignmentWithZnodePrecedenceOnControllerStartup(ReassignPartitionsClusterTest.scala:687) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at > org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at > org.junit.runners.ParentRunner.run(ParentRunner.java:412) at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) > at > org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) > at jdk.internal.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) > at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at > org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) > at jdk.internal.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at >
[jira] [Assigned] (KAFKA-9644) incrementalAlterConfigs OpType.APPEND on unset property fails with NullPointerException
[ https://issues.apache.org/jira/browse/KAFKA-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar reassigned KAFKA-9644: Assignee: Steve Rodrigues > incrementalAlterConfigs OpType.APPEND on unset property fails with > NullPointerException > --- > > Key: KAFKA-9644 > URL: https://issues.apache.org/jira/browse/KAFKA-9644 > Project: Kafka > Issue Type: Bug > Components: admin >Affects Versions: 2.3.0 >Reporter: Steve Rodrigues >Assignee: Steve Rodrigues >Priority: Minor > Fix For: 2.6.0 > > Attachments: incrementalAlterTest.patch, > kafka.api.PlaintextAdminIntegrationTest.testValidIncrementalAlterConfigs.test.stdout > > > Running incrementalAlterConfigs with an OpType.APPEND when the config > property doesn't already exist fails with a NullPointerException on the > broker. > Attached is a patch to the PlaintextAdminIntegrationTest demonstrating this > failure and the test output showing the NPE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9644) incrementalAlterConfigs OpType.APPEND on unset property fails with NullPointerException
[ https://issues.apache.org/jira/browse/KAFKA-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9644. -- Fix Version/s: 2.6.0 Resolution: Fixed Issue resolved by pull request 8216 [https://github.com/apache/kafka/pull/8216] > incrementalAlterConfigs OpType.APPEND on unset property fails with > NullPointerException > --- > > Key: KAFKA-9644 > URL: https://issues.apache.org/jira/browse/KAFKA-9644 > Project: Kafka > Issue Type: Bug > Components: admin >Affects Versions: 2.3.0 >Reporter: Steve Rodrigues >Priority: Minor > Fix For: 2.6.0 > > Attachments: incrementalAlterTest.patch, > kafka.api.PlaintextAdminIntegrationTest.testValidIncrementalAlterConfigs.test.stdout > > > Running incrementalAlterConfigs with an OpType.APPEND when the config > property doesn't already exist fails with a NullPointerException on the > broker. > Attached is a patch to the PlaintextAdminIntegrationTest demonstrating this > failure and the test output showing the NPE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9644) incrementalAlterConfigs OpType.APPEND on unset property fails with NullPointerException
[ https://issues.apache.org/jira/browse/KAFKA-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058330#comment-17058330 ] ASF GitHub Bot commented on KAFKA-9644: --- omkreddy commented on pull request #8216: [KAFKA-9644] Non-existent configs in incrementalAlterConfigs APPEND/SUBTRACT URL: https://github.com/apache/kafka/pull/8216 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > incrementalAlterConfigs OpType.APPEND on unset property fails with > NullPointerException > --- > > Key: KAFKA-9644 > URL: https://issues.apache.org/jira/browse/KAFKA-9644 > Project: Kafka > Issue Type: Bug > Components: admin >Affects Versions: 2.3.0 >Reporter: Steve Rodrigues >Priority: Minor > Attachments: incrementalAlterTest.patch, > kafka.api.PlaintextAdminIntegrationTest.testValidIncrementalAlterConfigs.test.stdout > > > Running incrementalAlterConfigs with an OpType.APPEND when the config > property doesn't already exist fails with a NullPointerException on the > broker. > Attached is a patch to the PlaintextAdminIntegrationTest demonstrating this > failure and the test output showing the NPE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-5604) All producer methods should raise `ProducerFencedException` after the first time.
[
https://issues.apache.org/jira/browse/KAFKA-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058327#comment-17058327
]
Xiang Zhang commented on KAFKA-5604:
[~mjsax] I notice that beginTransaction(), sendOffsesToTransaction(),
commitTransaction(), abortTransaction() call maybeFailWithError() before
anything. myabeFailWithError() is defined as :
{code:java}
private void maybeFailWithError() {
if (hasError()) {
// for ProducerFencedException, do not wrap it as a KafkaException
// but create a new instance without the call trace since it was not
thrown because of the current call
if (lastError instanceof ProducerFencedException) {
throw new ProducerFencedException("The producer has been rejected
from the broker because " + "it tried to use an old epoch with the
transactionalId");
} else {
throw new KafkaException("Cannot execute transactional method
because we are in an error state", lastError);
}
}
}
{code}
[https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L1084]
If lastError indeed tracks last error, it seems it will throw
ProducerFencedException for the second time as expected.
> All producer methods should raise `ProducerFencedException` after the first
> time.
> -
>
> Key: KAFKA-5604
> URL: https://issues.apache.org/jira/browse/KAFKA-5604
> Project: Kafka
> Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>Priority: Major
>
> Currently, when a `ProducerFencedException` is raised from a transactional
> producer, the expectation is that the application should call `close`
> immediately. However, if the application calls other producer methods, they
> would get a `KafkaException`. This is a bit confusing, and results in tickets
> like : https://issues.apache.org/jira/browse/KAFKA-5603.
> We should update the producer so that calls to any method other than `close`
> should raise a `ProducerFencedException` after the first time it is raised.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058296#comment-17058296
]
Konstantine Karantasis commented on KAFKA-9712:
---
[~nigel.liang] I agree with the direction of the proposed fix, because this
seems to correspond to the behavior that was exhibited until recently,
according to which any exception due to a class mismatch or a match that was
not found during class scanning would result in a {{WARN}} log message from the
{{reflections}} library, that was ignored by Kafka Connect. For example, in
this logging template:
[https://github.com/apache/kafka/blob/trunk/config/connect-log4j.properties]
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Nigel Liang
>Assignee: Nigel Liang
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0
>
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9659) Kafka Streams / Consumer configured for static membership fails on "fatal exception: group.instance.id gets fenced"
[
https://issues.apache.org/jira/browse/KAFKA-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058298#comment-17058298
]
ASF GitHub Bot commented on KAFKA-9659:
---
guozhangwang commented on pull request #8269: KAFKA-9659: Add more log4j when
updating static member mappings
URL: https://github.com/apache/kafka/pull/8269
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Kafka Streams / Consumer configured for static membership fails on "fatal
> exception: group.instance.id gets fenced"
> ---
>
> Key: KAFKA-9659
> URL: https://issues.apache.org/jira/browse/KAFKA-9659
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.5.0
>Reporter: Rohan Desai
>Priority: Major
> Attachments: ksql-1.logs
>
>
> I'm running a KSQL query, which underneath is built into a Kafka Streams
> application. The application has been running without issue for a few days,
> until today, when all the streams threads exited with:
>
>
> {{[ERROR] 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator handle -
> [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2-consumer,
> groupId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5]
> Received fatal exception: group.instance.id gets fenced}}
> {{[ERROR] 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator onFailure -
> [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2-consumer,
> groupId=_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5]
> Caught fenced group.instance.id Optional[ksql-1-2] error in heartbeat thread}}
> {{[ERROR] 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.streams.processor.internals.StreamThread run -
> stream-thread
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> Encountered the following unexpected Kafka exception during processing, this
> usually indicate Streams internal errors:}}
> \{{ org.apache.kafka.common.errors.FencedInstanceIdException: The broker
> rejected this static consumer since another consumer with the same
> group.instance.id has registered with a different member.id.}}{{[INFO]
> 2020-03-05 00:57:58,776
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> org.apache.kafka.streams.processor.internals.StreamThread setState -
> stream-thread
> [_confluent-ksql-pksqlc-xm6g1query_CSAS_RATINGS_WITH_USER_AVERAGE_5-39e8046a-b6e6-44fd-8d6d-37cff78649bf-StreamThread-2]
> State transition from RUNNING to PENDING_SHUTDOWN}}
>
> I've attached the KSQL and Kafka Streams logs to this ticket. Here's a
> summary for one of the streams threads (instance id `ksql-1-2`):
>
> Around 00:56:36 the coordinator fails over from b11 to b2:
>
> {{[INFO] 2020-03-05 00:56:36,258
> [_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator handle -
> [Consumer instanceId=ksql-1-2,
> clientId=_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2-consumer,
> groupId=_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0] Attempt to
> heartbeat failed since coordinator
> b11-pkc-lzxjz.us-west-2.aws.devel.cpdev.cloud:9092 (id: 2147483636 rack:
> null) is either not started or not valid.}}
> {{ [INFO] 2020-03-05 00:56:36,258
> [_confluent-ksql-pksqlc-xm6g1query_CTAS_RATINGS_BY_USER_0-c1df9747-f353-47f1-82fd-30b97c20d038-StreamThread-2]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator
> markCoordinatorUnknown - [Consumer instanceId=ksql-1-2,
>
[jira] [Commented] (KAFKA-9677) Low consume bandwidth quota may cause consumer not being able to fetch data
[ https://issues.apache.org/jira/browse/KAFKA-9677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058293#comment-17058293 ] ASF GitHub Bot commented on KAFKA-9677: --- apovzner commented on pull request #8290: KAFKA-9677: Fix consumer fetch with small consume bandwidth quotas URL: https://github.com/apache/kafka/pull/8290 When we changed quota communication with KIP-219, fetch requests get throttled by returning empty response with the delay in `throttle_time_ms` and Kafka consumer retries again after the delay. With default configs, the maximum fetch size could be as big as 50MB (or 10MB per partition). The default broker config (1-second window, 10 full windows of tracked bandwidth/thread utilization usage) means that < 5MB/s consumer quota (per broker) may block consumers from being able to fetch any data. This PR ensures that consumers cannot get blocked by quota by capping `fetchMaxBytes` in KafkaApis.handleFetchRequest() to * . In the example of default configs (10-second quota window) and 1MB/s consumer bandwidth quota, fetchMaxBytes would be capped to 10MB. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Low consume bandwidth quota may cause consumer not being able to fetch data > --- > > Key: KAFKA-9677 > URL: https://issues.apache.org/jira/browse/KAFKA-9677 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.0.1, 2.1.1, 2.2.2, 2.4.0, 2.3.1 >Reporter: Anna Povzner >Assignee: Anna Povzner >Priority: Major > > When we changed quota communication with KIP-219, fetch requests get > throttled by returning empty response with the delay in `throttle_time_ms` > and Kafka consumer retrying again after the delay. > With default configs, the maximum fetch size could be as big as 50MB (or 10MB > per partition). The default broker config (1-second window, 10 full windows > of tracked bandwidth/thread utilization usage) means that < 5MB/s consumer > quota (per broker) may stop fetch request from ever being successful. > Or the other way around: 1 MB/s consumer quota (per broker) means that any > fetch request that gets >= 10MB of data (10 seconds * 1MB/second) in the > response will never get through. From consumer point of view, the behavior > will be: Consumer will get an empty response with throttle_time_ms > 0, Kafka > consumer will wait for throttle time delay and then send fetch request again, > the fetch response is still too big so broker sends another empty response > with throttle time, and so on in never ending loop > h3. Proposed fix > Return less data in fetch response in this case: Cap `fetchMaxBytes` passed > to replicaManager.fetchMessages() from KafkaApis.handleFetchRequest() to > * . In the example of default > configs and 1MB/s consumer bandwidth quota, fetchMaxBytes will be 10MB. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7869) Refactor RocksDBConfigSetter API to separate DBOptions and CFOptions
[
https://issues.apache.org/jira/browse/KAFKA-7869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058269#comment-17058269
]
John Roesler commented on KAFKA-7869:
-
Hey [~dongjin] ,
The last comments on the PR seem sort of existential... Do you think we should
still proceed with this work, or just close the ticket (with documented
reasoning)?
> Refactor RocksDBConfigSetter API to separate DBOptions and CFOptions
>
>
> Key: KAFKA-7869
> URL: https://issues.apache.org/jira/browse/KAFKA-7869
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Guozhang Wang
>Assignee: Dongjin Lee
>Priority: Major
> Labels: needs-kip, newbie
>
> Current RocksDBConfigSetter has the following API:
> {code}
> void setConfig(final String storeName, final Options options, final
> Map configs);
> {code}
> Where `Options` contains configurations for both db-level and cf-level of
> RocksDB.
> As we move on to have multiple CFs following KIP-258, it's better to refactor
> it into
> {code}
> void setConfig(final String storeName, final DBOptions dbOptions, final
> ColumnFamilyOptions cfOptions, final Map configs);
> {code}
> And then inside the internal implementation, if only the default CF is used,
> we can still use the other constructor of `Options` that takes both a
> DBOptions and CFOptions object as parameters.
> This should be started only after KIP-258 is finished.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058261#comment-17058261
]
ASF GitHub Bot commented on KAFKA-9712:
---
ncliang commented on pull request #8289: KAFKA-9712: Catch and handle exception
thrown by reflections scanner
URL: https://github.com/apache/kafka/pull/8289
This commit works around a bug in v0.9.12 in upstream `reflections` library
by catching and handling the exception thrown.
New unittests were introduced to test the behavior.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Nigel Liang
>Assignee: Nigel Liang
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0
>
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[ https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058235#comment-17058235 ] Nigel Liang edited comment on KAFKA-9712 at 3/12/20, 8:49 PM: -- If we were to revert back to v0.9.11, we would bring back dependency on guava. As noted in https://issues.apache.org/jira/browse/KAFKA-3061, this is undesirable since having it in framework leads to potential version conflicts downstream for users of framework that want to use their own Guava. OTOH, this would be the path of lowest risk since no new code would be introduced. reflections8 claims that they are obsoleted by v0.9.12 of reflections - [https://github.com/aschoerk/reflections8] . This library will likely not be maintained in the future, while reflections library will likely be fixed in a future 0.9.13 release. There is already a PR out for the fix - [https://github.com/ronmamo/reflections/pull/278] The try...catch workaround seems like it will be most promising. The proposal is to catch the exception [here|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/DelegatingClassLoader.java#L342] and return empty result on exception. The `Reflections` object we build and use here scans over either a single URL at a time expecting to find an Uber jar or an array of URLs representing contents of the plugin directory. This bug is triggered only when the `Reflections` object could find no classes in the URLs provided. In which case, returning empty results is indeed the correct response. Empty result is also essentially the response that you would get from previous versions of reflections that do not contain the bug. So, making the change should be pretty safe in the sense that existing users upgrading to `2.5` would see the same behavior as before (non-jars or jars containing no classes silently skipped). Exception handling comes with a little bit of a performance penalty, but this is in the worker startup path only and it will be only for exceptional situations. The happy path will be unchanged. was (Author: nigel.liang): If we were to revert back to v0.9.11, we would bring back dependency on guava. As noted in https://issues.apache.org/jira/browse/KAFKA-3061, this is undesirable since having it in framework leads to potential version conflicts downstream for users of framework that want to use their own Guava. OTOH, this would be the path of lowest risk since no new code would be introduced. reflections8 claims that they are obsoleted by v0.9.12 of reflections - [https://github.com/aschoerk/reflections8] . This library will likely not be maintained in the future, while reflections library will likely be fixed in a future 0.9.13 release. There is already a PR out for the fix - [https://github.com/ronmamo/reflections/pull/278] The try...catch workaround seems like it will be most promising. The proposal is to catch the exception [here|[https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/DelegatingClassLoader.java#L342]] and return empty result on exception. The `Reflections` object we build and use here scans over either a single URL at a time expecting to find an Uber jar or an array of URLs representing contents of the plugin directory. This bug is triggered only when the `Reflections` object could find no classes in the URLs provided. In which case, returning empty results is indeed the correct response. Empty result is also essentially the response that you would get from previous versions of reflections that do not contain the bug. So, making the change should be pretty safe in the sense that existing users upgrading to `2.5` would see the same behavior as before (non-jars or jars containing no classes silently skipped). Exception handling comes with a little bit of a performance penalty, but this is in the worker startup path only and it will be only for exceptional situations. The happy path will be unchanged. > Reflections library 0.9.12 introduced in 2.5 causes regression scanning for > plugins on plugin_path > -- > > Key: KAFKA-9712 > URL: https://issues.apache.org/jira/browse/KAFKA-9712 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.5.0, 2.6.0 >Reporter: Nigel Liang >Assignee: Nigel Liang >Priority: Blocker > Fix For: 2.5.0, 2.6.0 > > > Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency - > https://issues.apache.org/jira/browse/KAFKA-3061 > This version, however, contains a > [bug|https://github.com/ronmamo/reflections/issues/273] that will cause > regression when
[jira] [Comment Edited] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[ https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058235#comment-17058235 ] Nigel Liang edited comment on KAFKA-9712 at 3/12/20, 8:48 PM: -- If we were to revert back to v0.9.11, we would bring back dependency on guava. As noted in https://issues.apache.org/jira/browse/KAFKA-3061, this is undesirable since having it in framework leads to potential version conflicts downstream for users of framework that want to use their own Guava. OTOH, this would be the path of lowest risk since no new code would be introduced. reflections8 claims that they are obsoleted by v0.9.12 of reflections - [https://github.com/aschoerk/reflections8] . This library will likely not be maintained in the future, while reflections library will likely be fixed in a future 0.9.13 release. There is already a PR out for the fix - [https://github.com/ronmamo/reflections/pull/278] The try...catch workaround seems like it will be most promising. The proposal is to catch the exception [here|[https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/DelegatingClassLoader.java#L342]] and return empty result on exception. The `Reflections` object we build and use here scans over either a single URL at a time expecting to find an Uber jar or an array of URLs representing contents of the plugin directory. This bug is triggered only when the `Reflections` object could find no classes in the URLs provided. In which case, returning empty results is indeed the correct response. Empty result is also essentially the response that you would get from previous versions of reflections that do not contain the bug. So, making the change should be pretty safe in the sense that existing users upgrading to `2.5` would see the same behavior as before (non-jars or jars containing no classes silently skipped). Exception handling comes with a little bit of a performance penalty, but this is in the worker startup path only and it will be only for exceptional situations. The happy path will be unchanged. was (Author: nigel.liang): If we were to revert back to v0.9.11, we would bring back dependency on guava. As noted in https://issues.apache.org/jira/browse/KAFKA-3061, this is undesirable since having it in framework leads to potential version conflicts downstream for users of framework that want to use their own Guava. OTOH, this would be the path of lowest risk since no new code would be introduced. reflections8 claims that they are obsoleted by v0.9.12 of reflections - [https://github.com/aschoerk/reflections8] . This library will likely not be maintained in the future, while reflections library will likely be fixed in a future 0.9.13 release. There is already a PR out for the fix - [https://github.com/ronmamo/reflections/pull/278] The try...catch workaround seems like it will be most promising. The proposal is to catch the exception [here|#L342]] and return empty result on exception. The `Reflections` object we build and use here scans over either a single URL at a time expecting to find an Uber jar or an array of URLs representing contents of the plugin directory. This bug is triggered only when the `Reflections` object could find no classes in the URLs provided. In which case, returning empty results is indeed the correct response. Empty result is also essentially the response that you would get from previous versions of reflections that do not contain the bug. So, making the change should be pretty safe in the sense that existing users upgrading to `2.5` would see the same behavior as before (non-jars or jars containing no classes silently skipped). Exception handling comes with a little bit of a performance penalty, but this is in the worker startup path only and it will be only for exceptional situations. The happy path will be unchanged. > Reflections library 0.9.12 introduced in 2.5 causes regression scanning for > plugins on plugin_path > -- > > Key: KAFKA-9712 > URL: https://issues.apache.org/jira/browse/KAFKA-9712 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.5.0, 2.6.0 >Reporter: Nigel Liang >Assignee: Nigel Liang >Priority: Blocker > Fix For: 2.5.0, 2.6.0 > > > Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency - > https://issues.apache.org/jira/browse/KAFKA-3061 > This version, however, contains a > [bug|https://github.com/ronmamo/reflections/issues/273] that will cause > regression when scanning for plugins if the `plugins_path` specified does not > contain valid plugins. The case where we were able to repro this, the >
[jira] [Comment Edited] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[ https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058235#comment-17058235 ] Nigel Liang edited comment on KAFKA-9712 at 3/12/20, 8:46 PM: -- If we were to revert back to v0.9.11, we would bring back dependency on guava. As noted in https://issues.apache.org/jira/browse/KAFKA-3061, this is undesirable since having it in framework leads to potential version conflicts downstream for users of framework that want to use their own Guava. OTOH, this would be the path of lowest risk since no new code would be introduced. reflections8 claims that they are obsoleted by v0.9.12 of reflections - [https://github.com/aschoerk/reflections8] . This library will likely not be maintained in the future, while reflections library will likely be fixed in a future 0.9.13 release. There is already a PR out for the fix - [https://github.com/ronmamo/reflections/pull/278] The try...catch workaround seems like it will be most promising. The proposal is to catch the exception [here|#L342]] and return empty result on exception. The `Reflections` object we build and use here scans over either a single URL at a time expecting to find an Uber jar or an array of URLs representing contents of the plugin directory. This bug is triggered only when the `Reflections` object could find no classes in the URLs provided. In which case, returning empty results is indeed the correct response. Empty result is also essentially the response that you would get from previous versions of reflections that do not contain the bug. So, making the change should be pretty safe in the sense that existing users upgrading to `2.5` would see the same behavior as before (non-jars or jars containing no classes silently skipped). Exception handling comes with a little bit of a performance penalty, but this is in the worker startup path only and it will be only for exceptional situations. The happy path will be unchanged. was (Author: nigel.liang): If we were to revert back to v0.9.11, we would bring back dependency on guava. As noted in https://issues.apache.org/jira/browse/KAFKA-3061, this is undesirable since having it in framework leads to potential version conflicts downstream for users of framework that want to use their own Guava. OTOH, this would be the path of lowest risk since no new code would be introduced. reflections8 claims that they are obsoleted by v0.9.12 of reflections - [https://github.com/aschoerk/reflections8] . This library will likely not be maintained in the future, while reflections library will likely be fixed in a future 0.9.13 release. There are already PRs out for the fix - [https://github.com/ronmamo/reflections/pull/278] The try...catch workaround seems like it will be most promising. The proposal is to catch the exception [here|#L342]] and return empty result on exception. The `Reflections` object we build and use here scans over either a single URL at a time expecting to find an Uber jar or an array of URLs representing contents of the plugin directory. This bug is triggered only when the `Reflections` object could find no classes in the URLs provided. In which case, returning empty results is indeed the correct response. Empty result is also essentially the response that you would get from previous versions of reflections that do not contain the bug. So, making the change should be pretty safe in the sense that existing users upgrading to `2.5` would see the same behavior as before (non-jars or jars containing no classes silently skipped). Exception handling comes with a little bit of a performance penalty, but this is in the worker startup path only and it will be only for exceptional situations. The happy path will be unchanged. > Reflections library 0.9.12 introduced in 2.5 causes regression scanning for > plugins on plugin_path > -- > > Key: KAFKA-9712 > URL: https://issues.apache.org/jira/browse/KAFKA-9712 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.5.0, 2.6.0 >Reporter: Nigel Liang >Assignee: Nigel Liang >Priority: Blocker > Fix For: 2.5.0, 2.6.0 > > > Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency - > https://issues.apache.org/jira/browse/KAFKA-3061 > This version, however, contains a > [bug|https://github.com/ronmamo/reflections/issues/273] that will cause > regression when scanning for plugins if the `plugins_path` specified does not > contain valid plugins. The case where we were able to repro this, the > `plugins_path` was misconfigured to point to `~/.ssh` which contained > unrelated files but no plugins. Further testing is needed to figure out if it >
[jira] [Comment Edited] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[ https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058235#comment-17058235 ] Nigel Liang edited comment on KAFKA-9712 at 3/12/20, 8:37 PM: -- If we were to revert back to v0.9.11, we would bring back dependency on guava. As noted in https://issues.apache.org/jira/browse/KAFKA-3061, this is undesirable since having it in framework leads to potential version conflicts downstream for users of framework that want to use their own Guava. OTOH, this would be the path of lowest risk since no new code would be introduced. reflections8 claims that they are obsoleted by v0.9.12 of reflections - [https://github.com/aschoerk/reflections8] . This library will likely not be maintained in the future, while reflections library will likely be fixed in a future 0.9.13 release. There are already PRs out for the fix - [https://github.com/ronmamo/reflections/pull/278] The try...catch workaround seems like it will be most promising. The proposal is to catch the exception [here|#L342]] and return empty result on exception. The `Reflections` object we build and use here scans over either a single URL at a time expecting to find an Uber jar or an array of URLs representing contents of the plugin directory. This bug is triggered only when the `Reflections` object could find no classes in the URLs provided. In which case, returning empty results is indeed the correct response. Empty result is also essentially the response that you would get from previous versions of reflections that do not contain the bug. So, making the change should be pretty safe in the sense that existing users upgrading to `2.5` would see the same behavior as before (non-jars or jars containing no classes silently skipped). Exception handling comes with a little bit of a performance penalty, but this is in the worker startup path only and it will be only for exceptional situations. The happy path will be unchanged. was (Author: nigel.liang): If we were to revert back to v0.9.11, we would bring back dependency on guava. As noted in https://issues.apache.org/jira/browse/KAFKA-3061, this is undesirable since having it in framework leads to potential version conflicts downstream for users of framework that want to use their own Guava. reflections8 claims that they are obsoleted by v0.9.12 of reflections - [https://github.com/aschoerk/reflections8] . This library will likely not be maintained in the future, while reflections library will likely be fixed in a future 0.9.13 release. There are already PRs out for the fix - [https://github.com/ronmamo/reflections/pull/278] The try...catch workaround seems like it will be most promising. The proposal is to catch the exception [here|[https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/DelegatingClassLoader.java#L342]] and return empty result on exception. The `Reflections` object we build and use here scans over either a single URL at a time expecting to find an Uber jar or an array of URLs representing contents of the plugin directory. This bug is triggered only when the `Reflections` object could find no classes in the URLs provided. In which case, returning empty results is indeed the correct response. Empty result is also essentially the response that you would get from previous versions of reflections that do not contain the bug. So, making the change should be pretty safe in the sense that existing users upgrading to `2.5` would see the same behavior as before (non-jars or jars containing no classes silently skipped). Exception handling comes with a little bit of a performance penalty, but this is in the worker startup path only and it will be only for exceptional situations. The happy path will be unchanged. > Reflections library 0.9.12 introduced in 2.5 causes regression scanning for > plugins on plugin_path > -- > > Key: KAFKA-9712 > URL: https://issues.apache.org/jira/browse/KAFKA-9712 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.5.0, 2.6.0 >Reporter: Nigel Liang >Assignee: Nigel Liang >Priority: Blocker > Fix For: 2.5.0, 2.6.0 > > > Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency - > https://issues.apache.org/jira/browse/KAFKA-3061 > This version, however, contains a > [bug|https://github.com/ronmamo/reflections/issues/273] that will cause > regression when scanning for plugins if the `plugins_path` specified does not > contain valid plugins. The case where we were able to repro this, the > `plugins_path` was misconfigured to point to `~/.ssh` which contained > unrelated files but
[jira] [Commented] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058235#comment-17058235
]
Nigel Liang commented on KAFKA-9712:
If we were to revert back to v0.9.11, we would bring back dependency on guava.
As noted in https://issues.apache.org/jira/browse/KAFKA-3061, this is
undesirable since having it in framework leads to potential version conflicts
downstream for users of framework that want to use their own Guava.
reflections8 claims that they are obsoleted by v0.9.12 of reflections -
[https://github.com/aschoerk/reflections8] . This library will likely not be
maintained in the future, while reflections library will likely be fixed in a
future 0.9.13 release. There are already PRs out for the fix -
[https://github.com/ronmamo/reflections/pull/278]
The try...catch workaround seems like it will be most promising. The proposal
is to catch the exception
[here|[https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/DelegatingClassLoader.java#L342]]
and return empty result on exception. The `Reflections` object we build and
use here scans over either a single URL at a time expecting to find an Uber jar
or an array of URLs representing contents of the plugin directory. This bug is
triggered only when the `Reflections` object could find no classes in the URLs
provided. In which case, returning empty results is indeed the correct
response. Empty result is also essentially the response that you would get from
previous versions of reflections that do not contain the bug. So, making the
change should be pretty safe in the sense that existing users upgrading to
`2.5` would see the same behavior as before (non-jars or jars containing no
classes silently skipped). Exception handling comes with a little bit of a
performance penalty, but this is in the worker startup path only and it will be
only for exceptional situations. The happy path will be unchanged.
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Nigel Liang
>Assignee: Nigel Liang
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0
>
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
>
[jira] [Commented] (KAFKA-9713) Remove BufferExhausedException
[
https://issues.apache.org/jira/browse/KAFKA-9713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058228#comment-17058228
]
ASF GitHub Bot commented on KAFKA-9713:
---
bdbyrne commented on pull request #8288: KAFKA-9713: Remove dead
BufferExhaustedException, which was deprecate…
URL: https://github.com/apache/kafka/pull/8288
…d in 0.9.0.0.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remove BufferExhausedException
> --
>
> Key: KAFKA-9713
> URL: https://issues.apache.org/jira/browse/KAFKA-9713
> Project: Kafka
> Issue Type: Task
> Components: producer
>Reporter: Brian Byrne
>Priority: Trivial
>
> BufferExhaustedException was deprecated in 0.9.0.0, and the corresponding
> block.on.buffer.full property has since been removed. The exception should
> follow.
> {quote}Deprecations in 0.9.0.0
> The producer config block.on.buffer.full has been deprecated and will be
> removed in future release. Currently its default value has been changed to
> false. The KafkaProducer will no longer throw BufferExhaustedException but
> instead will use max.block.ms value to block, after which it will throw a
> TimeoutException. If block.on.buffer.full property is set to true explicitly,
> it will set the max.block.ms to Long.MAX_VALUE and metadata.fetch.timeout.ms
> will not be honoured{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9713) Remove BufferExhausedException
Brian Byrne created KAFKA-9713:
--
Summary: Remove BufferExhausedException
Key: KAFKA-9713
URL: https://issues.apache.org/jira/browse/KAFKA-9713
Project: Kafka
Issue Type: Task
Components: producer
Reporter: Brian Byrne
BufferExhaustedException was deprecated in 0.9.0.0, and the corresponding
block.on.buffer.full property has since been removed. The exception should
follow.
{quote}Deprecations in 0.9.0.0
The producer config block.on.buffer.full has been deprecated and will be
removed in future release. Currently its default value has been changed to
false. The KafkaProducer will no longer throw BufferExhaustedException but
instead will use max.block.ms value to block, after which it will throw a
TimeoutException. If block.on.buffer.full property is set to true explicitly,
it will set the max.block.ms to Long.MAX_VALUE and metadata.fetch.timeout.ms
will not be honoured{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch updated KAFKA-9712:
-
Fix Version/s: 2.6.0
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Nigel Liang
>Assignee: Nigel Liang
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0
>
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch updated KAFKA-9712:
-
Affects Version/s: 2.6.0
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Nigel Liang
>Assignee: Nigel Liang
>Priority: Blocker
> Fix For: 2.5.0
>
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch updated KAFKA-9712:
-
Priority: Blocker (was: Major)
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.0
>Reporter: Nigel Liang
>Priority: Blocker
> Fix For: 2.5.0
>
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch updated KAFKA-9712:
-
Component/s: KafkaConnect
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.0
>Reporter: Nigel Liang
>Priority: Major
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch updated KAFKA-9712:
-
Fix Version/s: 2.5.0
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.0
>Reporter: Nigel Liang
>Priority: Major
> Fix For: 2.5.0
>
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9588) Add rocksdb event listeners in KS
[ https://issues.apache.org/jira/browse/KAFKA-9588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9588: --- Fix Version/s: 3.0.0 > Add rocksdb event listeners in KS > - > > Key: KAFKA-9588 > URL: https://issues.apache.org/jira/browse/KAFKA-9588 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Navinder Brar >Priority: Major > Fix For: 3.0.0 > > > Rocsdb is coming up with the support of event listeners(like > onCompactionCompleted) in jni > ([https://github.com/facebook/rocksdb/issues/6343]) which would be really > helpful in KS to trigger checkpointing on flush completed due to filling up > of memtables, rather than doing it periodically etc. This task is currently > blocked on https://issues.apache.org/jira/browse/KAFKA-8897. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore
[ https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058193#comment-17058193 ] Sophie Blee-Goldman commented on KAFKA-9168: Thanks [~sagarrao], marking this as a blocker for 3.0 since we unfortunately need to wait for the next major release before bumping the major version of rocksdb (they made a number of breaking changes that Streams users are unfortunately exposed to). That said, this patch has the potential for some pretty significant performance improvements in addition to opening the door for other desirable features that aren't available to RocksJava users. Definitely one of the stronger motivations to consider a workaround that allows upgrading without introducing breaking changes to user code... > Integrate JNI direct buffer support to RocksDBStore > --- > > Key: KAFKA-9168 > URL: https://issues.apache.org/jira/browse/KAFKA-9168 > Project: Kafka > Issue Type: Task > Components: streams >Reporter: Sagar Rao >Priority: Blocker > Labels: perfomance > Fix For: 3.0.0 > > > There has been a PR created on rocksdb Java client to support direct > ByteBuffers in Java. We can look at integrating it whenever it gets merged. > Link to PR: [https://github.com/facebook/rocksdb/pull/2283] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058191#comment-17058191
]
Nigel Liang edited comment on KAFKA-9712 at 3/12/20, 7:17 PM:
--
I have concluded more testing. My conclusion is that any file on the plugin
path that is either:
# not a jar, or
# a jar containing no classes
will trigger the bug. Consequence is that connect worker will not be able to
start up.
Possible workarounds include but are not limited to:
* Revert back to v0.9.11
* Switch to reflections8 fork similar to what is done here -
[https://github.com/JanusGraph/janusgraph/pull/2029]
* Try to catch and handle the exception by skipping the problematic file and
continuing the scan
was (Author: nigel.liang):
I have concluded more testing. My conclusion is that any file on the plugin
path that is either:
# not a jar, or
# a jar containing no classes
will trigger the bug. Consequence is that connect worker will not be able to
start up.
Possible workarounds include but are not limited to: * Revert back to v0.9.11
* Switch to reflections8 fork similar to what is done here -
[https://github.com/JanusGraph/janusgraph/pull/2029]
* Try to catch and handle the exception by skipping the problematic file and
continuing the scan
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Nigel Liang
>Priority: Major
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore
[ https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9168: --- Priority: Blocker (was: Minor) > Integrate JNI direct buffer support to RocksDBStore > --- > > Key: KAFKA-9168 > URL: https://issues.apache.org/jira/browse/KAFKA-9168 > Project: Kafka > Issue Type: Task > Components: streams >Reporter: Sagar Rao >Priority: Blocker > Labels: perfomance > Fix For: 3.0.0 > > > There has been a PR created on rocksdb Java client to support direct > ByteBuffers in Java. We can look at integrating it whenever it gets merged. > Link to PR: [https://github.com/facebook/rocksdb/pull/2283] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore
[ https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9168: --- Fix Version/s: 3.0.0 > Integrate JNI direct buffer support to RocksDBStore > --- > > Key: KAFKA-9168 > URL: https://issues.apache.org/jira/browse/KAFKA-9168 > Project: Kafka > Issue Type: Task > Components: streams >Reporter: Sagar Rao >Priority: Minor > Labels: perfomance > Fix For: 3.0.0 > > > There has been a PR created on rocksdb Java client to support direct > ByteBuffers in Java. We can look at integrating it whenever it gets merged. > Link to PR: [https://github.com/facebook/rocksdb/pull/2283] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058191#comment-17058191
]
Nigel Liang edited comment on KAFKA-9712 at 3/12/20, 7:16 PM:
--
I have concluded more testing. My conclusion is that any file on the plugin
path that is either:
# not a jar, or
# a jar containing no classes
will trigger the bug. Consequence is that connect worker will not be able to
start up.
Possible workarounds include but are not limited to: * Revert back to v0.9.11
* Switch to reflections8 fork similar to what is done here -
[https://github.com/JanusGraph/janusgraph/pull/2029]
* Try to catch and handle the exception by skipping the problematic file and
continuing the scan
was (Author: nigel.liang):
I have concluded more testing. My conclusion is that any file on the plugin
path that is either: # not a jar, or
# a jar containing no classes
will trigger the bug. Consequence is that connect worker will not be able to
start up.
Possible workarounds include but are not limited to: * Revert back to v0.9.11
* Switch to reflections8 fork similar to what is done here -
[https://github.com/JanusGraph/janusgraph/pull/2029]
* Try to catch and handle the exception by skipping the problematic file and
continuing the scan
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Nigel Liang
>Priority: Major
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058191#comment-17058191
]
Nigel Liang commented on KAFKA-9712:
I have concluded more testing. My conclusion is that any file on the plugin
path that is either: # not a jar, or
# a jar containing no classes
will trigger the bug. Consequence is that connect worker will not be able to
start up.
Possible workarounds include but are not limited to: * Revert back to v0.9.11
* Switch to reflections8 fork similar to what is done here -
[https://github.com/JanusGraph/janusgraph/pull/2029]
* Try to catch and handle the exception by skipping the problematic file and
continuing the scan
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Nigel Liang
>Priority: Major
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
> org.reflections.ReflectionsException: Scanner SubTypesScanner was not
> configured
> at org.reflections.Store.get(Store.java:39)
> at org.reflections.Store.get(Store.java:61)
> at org.reflections.Store.get(Store.java:46)
> at org.reflections.Store.getAll(Store.java:93)
> at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
> at
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9675) RocksDB metrics reported always at zero
[
https://issues.apache.org/jira/browse/KAFKA-9675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058178#comment-17058178
]
ASF GitHub Bot commented on KAFKA-9675:
---
vvcephei commented on pull request #8256: KAFKA-9675: Fix bug that prevents
RocksDB metrics to be updated
URL: https://github.com/apache/kafka/pull/8256
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> RocksDB metrics reported always at zero
> ---
>
> Key: KAFKA-9675
> URL: https://issues.apache.org/jira/browse/KAFKA-9675
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Michael Viamari
>Assignee: Bruno Cadonna
>Priority: Blocker
> Fix For: 2.6.0, 2.4.2
>
>
> The rocksdb metrics listed under {{stream-state-metrics}} are reported as
> zero for all metrics and all rocksdb instances. The metrics are present in
> JMX, but are always zero.
> The streams state is configured with {{MetricsRecordingLevel}} at {{debug}}.
> I am able to see metrics with appropriate values in the
> {{stream-rocksdb-window-state-metrics}}, {{stream-record-cache-metrics}},
> {{stream-task-metrics}}, and {{stream-processor-node-metrics}}.
> Additionally, my DEBUG logs show the appropriate messages for recording
> events, i.e.
> {{org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder
> [RocksDB Metrics Recorder for agg-store] Recording metrics for store
> agg-store}}
> It happens that all of my rocksdb instances are windowed stores, not key
> value stores, so I haven't been able to check if this issue is unique to
> windowed stores.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9364) Fix misleading consumer logs on throttling
[ https://issues.apache.org/jira/browse/KAFKA-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9364. - Fix Version/s: 2.5.0 Resolution: Fixed > Fix misleading consumer logs on throttling > -- > > Key: KAFKA-9364 > URL: https://issues.apache.org/jira/browse/KAFKA-9364 > Project: Kafka > Issue Type: Bug >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Minor > Fix For: 2.5.0 > > > Fix misleading consumer logs on throttling -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
[
https://issues.apache.org/jira/browse/KAFKA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nigel Liang updated KAFKA-9712:
---
Description:
Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
https://issues.apache.org/jira/browse/KAFKA-3061
This version, however, contains a
[bug|https://github.com/ronmamo/reflections/issues/273] that will cause
regression when scanning for plugins if the `plugins_path` specified does not
contain valid plugins. The case where we were able to repro this, the
`plugins_path` was misconfigured to point to `~/.ssh` which contained unrelated
files but no plugins. Further testing is needed to figure out if it will repro
for other cases such as empty directory, combination of valid and invalid
plugins in directory, just some types of plugins and not others in directory,
etc.
{code}
[2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2020-03-12 18:07:01,062] ERROR Stopping due to error
(org.apache.kafka.connect.cli.ConnectDistributed)
org.reflections.ReflectionsException: Scanner SubTypesScanner was not configured
at org.reflections.Store.get(Store.java:39)
at org.reflections.Store.get(Store.java:61)
at org.reflections.Store.get(Store.java:46)
at org.reflections.Store.getAll(Store.java:93)
at org.reflections.Reflections.getSubTypesOf(Reflections.java:404)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:342)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:327)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:260)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:252)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:221)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:198)
at
org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:60)
at
org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
at
org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
{code}
was:
Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
https://issues.apache.org/jira/browse/KAFKA-3061
This version, however, contains a
[bug|https://github.com/ronmamo/reflections/issues/273] that will cause
regression when scanning for plugins if the `plugins_path` specified does not
contain valid plugins. The case where we were able to repro this, the
`plugins_path` was misconfigured to point to `~/.ssh` which contained unrelated
files but no plugins. Further testing is needed to figure out if it will repro
for other cases such as empty directory, combination of valid and invalid
plugins in directory, just some types of plugins and not others in directory,
etc.
> Reflections library 0.9.12 introduced in 2.5 causes regression scanning for
> plugins on plugin_path
> --
>
> Key: KAFKA-9712
> URL: https://issues.apache.org/jira/browse/KAFKA-9712
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Nigel Liang
>Priority: Major
>
> Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency -
> https://issues.apache.org/jira/browse/KAFKA-3061
> This version, however, contains a
> [bug|https://github.com/ronmamo/reflections/issues/273] that will cause
> regression when scanning for plugins if the `plugins_path` specified does not
> contain valid plugins. The case where we were able to repro this, the
> `plugins_path` was misconfigured to point to `~/.ssh` which contained
> unrelated files but no plugins. Further testing is needed to figure out if it
> will repro for other cases such as empty directory, combination of valid and
> invalid plugins in directory, just some types of plugins and not others in
> directory, etc.
> {code}
> [2020-03-12 18:07:01,045] INFO Loading plugin from: /home/ducker/.ssh
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,047] DEBUG Loading plugin urls: []
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2020-03-12 18:07:01,062] ERROR Stopping due to error
> (org.apache.kafka.connect.cli.ConnectDistributed)
>
[jira] [Commented] (KAFKA-9603) Number of open files keeps increasing in Streams application
[
https://issues.apache.org/jira/browse/KAFKA-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058145#comment-17058145
]
Sophie Blee-Goldman commented on KAFKA-9603:
Hey [~biljazovic] any new info or results? It does seem like there may be
something odd going on here. Also, just to clarify this occurred when upgrading
Streams from 2.0 to 2.3 right? Is the "2.4" listed in the affected versions the
broker version?
> Number of open files keeps increasing in Streams application
>
>
> Key: KAFKA-9603
> URL: https://issues.apache.org/jira/browse/KAFKA-9603
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0, 2.3.1
> Environment: Spring Boot 2.2.4, OpenJDK 13, Centos image
>Reporter: Bruno Iljazovic
>Priority: Major
>
> Problem appeared when upgrading from *2.0.1* to *2.3.1*.
> Relevant Kafka Streams code:
> {code:java}
> KStream events1 =
> builder.stream(FIRST_TOPIC_NAME, Consumed.with(stringSerde, event1Serde,
> event1TimestampExtractor(), null))
>.mapValues(...);
> KStream events2 =
> builder.stream(SECOND_TOPIC_NAME, Consumed.with(stringSerde, event2Serde,
> event2TimestampExtractor(), null))
>.mapValues(...);
> var joinWindows = JoinWindows.of(Duration.of(1, MINUTES).toMillis())
> .until(Duration.of(1, HOURS).toMillis());
> events2.join(events1, this::join, joinWindows, Joined.with(stringSerde,
> event2Serde, event1Serde))
>.foreach(...);
> {code}
> Number of open *.sst files keeps increasing until eventually it hits the os
> limit (65536) and causes this exception:
> {code:java}
> Caused by: org.rocksdb.RocksDBException: While open a file for appending:
> /.../0_8/KSTREAM-JOINOTHER-10-store/KSTREAM-JOINOTHER-10-store.157943520/001354.sst:
> Too many open files
> at org.rocksdb.RocksDB.flush(Native Method)
> at org.rocksdb.RocksDB.flush(RocksDB.java:2394)
> {code}
> Here are example files that are opened and never closed:
> {code:java}
> /.../0_27/KSTREAM-JOINTHIS-09-store/KSTREAM-JOINTHIS-09-store.158245920/000114.sst
> /.../0_27/KSTREAM-JOINOTHER-10-store/KSTREAM-JOINOTHER-10-store.158245920/65.sst
> /.../0_29/KSTREAM-JOINTHIS-09-store/KSTREAM-JOINTHIS-09-store.158215680/000115.sst
> /.../0_29/KSTREAM-JOINTHIS-09-store/KSTREAM-JOINTHIS-09-store.158245920/000112.sst
> /.../0_31/KSTREAM-JOINTHIS-09-store/KSTREAM-JOINTHIS-09-store.158185440/51.sst
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9712) Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path
Nigel Liang created KAFKA-9712: -- Summary: Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path Key: KAFKA-9712 URL: https://issues.apache.org/jira/browse/KAFKA-9712 Project: Kafka Issue Type: Bug Affects Versions: 2.5.0 Reporter: Nigel Liang Reflections v0.9.12 was introduced in 2.5 branch to remove Guava dependency - https://issues.apache.org/jira/browse/KAFKA-3061 This version, however, contains a [bug|https://github.com/ronmamo/reflections/issues/273] that will cause regression when scanning for plugins if the `plugins_path` specified does not contain valid plugins. The case where we were able to repro this, the `plugins_path` was misconfigured to point to `~/.ssh` which contained unrelated files but no plugins. Further testing is needed to figure out if it will repro for other cases such as empty directory, combination of valid and invalid plugins in directory, just some types of plugins and not others in directory, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9225) kafka fail to run on linux-aarch64
[
https://issues.apache.org/jira/browse/KAFKA-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058127#comment-17058127
]
Sophie Blee-Goldman commented on KAFKA-9225:
Nice! Big thanks to the RocksDB folks for backporting this and doing another
release. Now we don't have to wait for the next major release
> kafka fail to run on linux-aarch64
> --
>
> Key: KAFKA-9225
> URL: https://issues.apache.org/jira/browse/KAFKA-9225
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: jiamei xie
>Priority: Blocker
> Labels: incompatible
> Attachments: compat_report.html
>
>
> *Steps to reproduce:*
> 1. Download Kafka latest source code
> 2. Build it with gradle
> 3. Run
> [streamDemo|[https://kafka.apache.org/23/documentation/streams/quickstart]]
> when running bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo, it crashed with
> the following error message
> {code:java}
> xjm@ubuntu-arm01:~/kafka$bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/core/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/tools/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/api/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/transforms/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/file/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror-client/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/json/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/basic-auth-extension/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [2019-11-19 15:42:23,277] WARN The configuration 'admin.retries' was supplied
> but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:23,278] WARN The configuration 'admin.retry.backoff.ms' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:24,278] ERROR stream-client
> [streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apach e.kafka.streams.KafkaStreams)
> Exception in thread
> "streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48-StreamThread-1"
> java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni1377754636857652484.so:
> /tmp/librocksdbjni13777546368576524 84.so:
> cannot open shared object file: No such file or directory (Possible cause:
> can't load AMD 64-bit .so on a AARCH64-bit platform)
> {code}
> *Analyze:*
> This issue is caused by rocksdbjni-5.18.3.jar which doesn't come with aarch64
> native support. Replace rocksdbjni-5.18.3.jar with rocksdbjni-6.3.6.jar from
> [https://mvnrepository.com/artifact/org.rocksdb/rocksdbjni/6.3.6] can fix
> this problem.
> Attached is the binary compatibility report of rocksdbjni.jar between 5.18.3
> and 6.3.6. The result is 81.8%. So is it possible to upgrade rocksdbjni to
> 6.3.6 in upstream? Should there be any kind of tests to execute, please
> kindly point me. Thanks a lot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9225) kafka fail to run on linux-aarch64
[
https://issues.apache.org/jira/browse/KAFKA-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sophie Blee-Goldman updated KAFKA-9225:
---
Fix Version/s: (was: 3.0.0)
> kafka fail to run on linux-aarch64
> --
>
> Key: KAFKA-9225
> URL: https://issues.apache.org/jira/browse/KAFKA-9225
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: jiamei xie
>Priority: Blocker
> Labels: incompatible
> Attachments: compat_report.html
>
>
> *Steps to reproduce:*
> 1. Download Kafka latest source code
> 2. Build it with gradle
> 3. Run
> [streamDemo|[https://kafka.apache.org/23/documentation/streams/quickstart]]
> when running bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo, it crashed with
> the following error message
> {code:java}
> xjm@ubuntu-arm01:~/kafka$bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/core/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/tools/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/api/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/transforms/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/file/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror-client/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/json/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/basic-auth-extension/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [2019-11-19 15:42:23,277] WARN The configuration 'admin.retries' was supplied
> but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:23,278] WARN The configuration 'admin.retry.backoff.ms' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:24,278] ERROR stream-client
> [streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apach e.kafka.streams.KafkaStreams)
> Exception in thread
> "streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48-StreamThread-1"
> java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni1377754636857652484.so:
> /tmp/librocksdbjni13777546368576524 84.so:
> cannot open shared object file: No such file or directory (Possible cause:
> can't load AMD 64-bit .so on a AARCH64-bit platform)
> {code}
> *Analyze:*
> This issue is caused by rocksdbjni-5.18.3.jar which doesn't come with aarch64
> native support. Replace rocksdbjni-5.18.3.jar with rocksdbjni-6.3.6.jar from
> [https://mvnrepository.com/artifact/org.rocksdb/rocksdbjni/6.3.6] can fix
> this problem.
> Attached is the binary compatibility report of rocksdbjni.jar between 5.18.3
> and 6.3.6. The result is 81.8%. So is it possible to upgrade rocksdbjni to
> 6.3.6 in upstream? Should there be any kind of tests to execute, please
> kindly point me. Thanks a lot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-5603) Streams should not abort transaction when closing zombie task
[
https://issues.apache.org/jira/browse/KAFKA-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058039#comment-17058039
]
Matthias J. Sax commented on KAFKA-5603:
Well, nobody worked on KAFKA-5604 and hence it would be nice to get it fixed.
The point being is that after the producer got fenced, `abortTransaction()`
should not be called what Streams did, but we fixed it. I can't remember if the
call to `abortTransaction` resulted in another `ProducerFencedException` or a
`KafkaException` (due to KAFKA-5604). But for this ticket it does not really
matter as an exception would be thrown anyway if the call is made and thus we
now don't make the call any longer.
> Streams should not abort transaction when closing zombie task
> -
>
> Key: KAFKA-5603
> URL: https://issues.apache.org/jira/browse/KAFKA-5603
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Critical
> Fix For: 0.11.0.1, 1.0.0
>
>
> The contract of the transactional producer API is to not call any
> transactional method after a {{ProducerFenced}} exception was thrown.
> Streams however, does an unconditional call within {{StreamTask#close()}} to
> {{abortTransaction()}} in case of unclean shutdown. We need to distinguish
> between a {{ProducerFenced}} and other unclean shutdown cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9680) Negative consumer lag
[
https://issues.apache.org/jira/browse/KAFKA-9680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057970#comment-17057970
]
Sayed Mohammad Hossein Torabi commented on KAFKA-9680:
--
So, how can i fix this issue?
> Negative consumer lag
> -
>
> Key: KAFKA-9680
> URL: https://issues.apache.org/jira/browse/KAFKA-9680
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 2.3.1
>Reporter: Sayed Mohammad Hossein Torabi
>Assignee: Evelyn Bayes
>Priority: Major
>
> I'm using Kafka 2.3.1 and got negative consumer lag on it. Here is the result
> of my consumergroup
> {code:java}
> // 1 18985 8576-10409
> connector-consumer-hdfs-sink-1-20e86f06-cf92-4b68-a67a-32e1876118f7
> /172.16.2.220 connector-consumer-hdfs-sink-1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9711) The authentication failure caused by SSLEngine#beginHandshake is not properly caught and handled
[
https://issues.apache.org/jira/browse/KAFKA-9711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057962#comment-17057962
]
ASF GitHub Bot commented on KAFKA-9711:
---
chia7712 commented on pull request #8287: KAFKA-9711 The authentication failure
caused by SSLEngine#beginHandsh…
URL: https://github.com/apache/kafka/pull/8287
https://issues.apache.org/jira/browse/KAFKA-9711
```java
@Override
public void handshake() throws IOException {
if (state == State.NOT_INITALIZED)
startHandshake(); // this line
if (ready())
throw renegotiationException();
if (state == State.CLOSING)
throw closingException();
```
SSLEngine#beginHandshake is possible to throw authentication failures (for
example, no suitable cipher suites) so we ought to catch SSLException and then
convert it to SslAuthenticationException so as to process authentication
failures correctly.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> The authentication failure caused by SSLEngine#beginHandshake is not properly
> caught and handled
>
>
> Key: KAFKA-9711
> URL: https://issues.apache.org/jira/browse/KAFKA-9711
> Project: Kafka
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
>
> {code:java}
> @Override
> public void handshake() throws IOException {
> if (state == State.NOT_INITALIZED)
> startHandshake(); // this line
> if (ready())
> throw renegotiationException();
> if (state == State.CLOSING)
> throw closingException();
> {code}
> SSLEngine#beginHandshake is possible to throw authentication failures (for
> example, no suitable cipher suites) so we ought to catch SSLException and
> then convert it to SslAuthenticationException so as to process authentication
> failures correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9711) The authentication failure caused by SSLEngine#beginHandshake is not properly caught and handled
Chia-Ping Tsai created KAFKA-9711:
-
Summary: The authentication failure caused by
SSLEngine#beginHandshake is not properly caught and handled
Key: KAFKA-9711
URL: https://issues.apache.org/jira/browse/KAFKA-9711
Project: Kafka
Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
{code:java}
@Override
public void handshake() throws IOException {
if (state == State.NOT_INITALIZED)
startHandshake(); // this line
if (ready())
throw renegotiationException();
if (state == State.CLOSING)
throw closingException();
{code}
SSLEngine#beginHandshake is possible to throw authentication failures (for
example, no suitable cipher suites) so we ought to catch SSLException and then
convert it to SslAuthenticationException so as to process authentication
failures correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-5603) Streams should not abort transaction when closing zombie task
[
https://issues.apache.org/jira/browse/KAFKA-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057942#comment-17057942
]
Xiang Zhang commented on KAFKA-5603:
[~mjsax] How is this ticket related to KAFKA-5604? Does KAFKA-5604 need more
work ?
> Streams should not abort transaction when closing zombie task
> -
>
> Key: KAFKA-5603
> URL: https://issues.apache.org/jira/browse/KAFKA-5603
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Critical
> Fix For: 0.11.0.1, 1.0.0
>
>
> The contract of the transactional producer API is to not call any
> transactional method after a {{ProducerFenced}} exception was thrown.
> Streams however, does an unconditional call within {{StreamTask#close()}} to
> {{abortTransaction()}} in case of unclean shutdown. We need to distinguish
> between a {{ProducerFenced}} and other unclean shutdown cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-9590) Add read-only configuration to limit number of topic partitions
[ https://issues.apache.org/jira/browse/KAFKA-9590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gokul Ramanan Subramanian reassigned KAFKA-9590: Assignee: Gokul Ramanan Subramanian > Add read-only configuration to limit number of topic partitions > --- > > Key: KAFKA-9590 > URL: https://issues.apache.org/jira/browse/KAFKA-9590 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Gokul Ramanan Subramanian >Assignee: Gokul Ramanan Subramanian >Priority: Minor > > Currently, a Kafka cluster does not limit the number of topic partitions > despite the fact that on a given system, having more than a certain number of > topic partitions (without even any messages written / read) renders the > cluster unusable. > > The ask here is to have a read-only Kafka configuration named something like > *max.broker.partitions* (an integer, which applies at the broker level and > defaults to INT.MAX) that the administrator the cluster can set to ensure > smooth operation. > > Things to keep in mind: > # If a broker already has more than *max.broker.partitions* number of > partitions at the time when the *max.broker.partitions* configuration is > first applied on the broker, the broker should disallow any more topic > partitions. This allows for graceful reassignments and in-place upgrade to a > Kafka version that supports the *max.broker.partitions* configuration. > # Different brokers can technically have different *max.broker.partitions* > in case of a heterogenous cluster, where not all broker nodes operate with > the same system limits (such as CPU, memory etc.) > # When creating a topic partition, the controller has to identify brokers > which will replicate the topic partition. Currently, the controller > distributes the leadership of the topic partitions to ensure uniform load. > This algorithm will need to account for the *max.broker.partitions* > configuration. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9590) Add read-only configuration to limit number of topic partitions
[ https://issues.apache.org/jira/browse/KAFKA-9590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gokul Ramanan Subramanian updated KAFKA-9590: - Description: Currently, a Kafka cluster does not limit the number of topic partitions despite the fact that on a given system, having more than a certain number of topic partitions (without even any messages written / read) renders the cluster unusable. The ask here is to have a read-only Kafka configuration named something like *max.broker.partitions* (an integer, which applies at the broker level and defaults to INT.MAX) that the administrator the cluster can set to ensure smooth operation. Things to keep in mind: # If a broker already has more than *max.broker.partitions* number of partitions at the time when the *max.broker.partitions* configuration is first applied on the broker, the broker should disallow any more topic partitions. This allows for graceful reassignments and in-place upgrade to a Kafka version that supports the *max.broker.partitions* configuration. # Different brokers can technically have different *max.broker.partitions* in case of a heterogenous cluster, where not all broker nodes operate with the same system limits (such as CPU, memory etc.) # When creating a topic partition, the controller has to identify brokers which will replicate the topic partition. Currently, the controller distributes the leadership of the topic partitions to ensure uniform load. This algorithm will need to account for the *max.broker.partitions* configuration. was: Currently, a Kafka cluster does not limit the number of topic partitions despite the fact that on a given system, having more than a certain number of topic partitions (without even any messages written / read) renders the cluster unusable. The ask here is to have a read-only Kafka configuration named something like *max.topic.partitions* (an integer, which applies at the broker level and defaults to INT.MAX) that the administrator the cluster can set to ensure smooth operation. Things to keep in mind: # If a broker already has more than *max.topic.partitions* number of partitions at the time when the *max.topic.partitions* configuration is first applied on the broker, the broker should disallow any more topic partitions. This allows for graceful reassignments and in-place upgrade to a Kafka version that supports the *max.topic.partitions* configuration. # Different brokers can technically have different *max.topic.partitions* in case of a heterogenous cluster, where not all broker nodes operate with the same system limits (such as CPU, memory etc.) # When creating a topic partition, the controller has to identify brokers which will replicate the topic partition. Currently, the controller distributes the leadership of the topic partitions to ensure uniform load. This algorithm will need to account for the *max.topic.partitions* configuration. > Add read-only configuration to limit number of topic partitions > --- > > Key: KAFKA-9590 > URL: https://issues.apache.org/jira/browse/KAFKA-9590 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Gokul Ramanan Subramanian >Priority: Minor > > Currently, a Kafka cluster does not limit the number of topic partitions > despite the fact that on a given system, having more than a certain number of > topic partitions (without even any messages written / read) renders the > cluster unusable. > > The ask here is to have a read-only Kafka configuration named something like > *max.broker.partitions* (an integer, which applies at the broker level and > defaults to INT.MAX) that the administrator the cluster can set to ensure > smooth operation. > > Things to keep in mind: > # If a broker already has more than *max.broker.partitions* number of > partitions at the time when the *max.broker.partitions* configuration is > first applied on the broker, the broker should disallow any more topic > partitions. This allows for graceful reassignments and in-place upgrade to a > Kafka version that supports the *max.broker.partitions* configuration. > # Different brokers can technically have different *max.broker.partitions* > in case of a heterogenous cluster, where not all broker nodes operate with > the same system limits (such as CPU, memory etc.) > # When creating a topic partition, the controller has to identify brokers > which will replicate the topic partition. Currently, the controller > distributes the leadership of the topic partitions to ensure uniform load. > This algorithm will need to account for the *max.broker.partitions* > configuration. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9695) AdminClient allows null topic configs, but broker throws NPE
[
https://issues.apache.org/jira/browse/KAFKA-9695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajini Sivaram resolved KAFKA-9695.
---
Reviewer: Manikumar
Resolution: Fixed
Updated broker to throw InvalidRequestException for null config values in
CreateTopics, AlterConfigs and IncrementalAlterConfigs.
> AdminClient allows null topic configs, but broker throws NPE
>
>
> Key: KAFKA-9695
> URL: https://issues.apache.org/jira/browse/KAFKA-9695
> Project: Kafka
> Issue Type: Bug
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.6.0
>
>
> Config entries may contain null values, but broker's AdminManager throws NPE
> resulting in UnknownServerException. We should handle null values in configs.
> {code:java}
> [2020-03-10 21:56:07,904] ERROR [Admin Manager on Broker 0]: Error processing
> create topic request CreatableTopic(name='topic', numPartitions=2,
> replicationFactor=3, assignments=[],
> configs=[CreateableTopicConfig(name='message.format.version', value=null),
> CreateableTopicConfig(name='compression.type', value='producer')])
> (kafka.server.AdminManager:76)
> java.lang.NullPointerException
> at java.util.Hashtable.put(Hashtable.java:460)
> at java.util.Properties.setProperty(Properties.java:166)
> at
> kafka.server.AdminManager.$anonfun$createTopics$3(AdminManager.scala:99)
> at scala.collection.Iterator.foreach(Iterator.scala:941)
> at scala.collection.Iterator.foreach$(Iterator.scala:941)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at
> kafka.server.AdminManager.$anonfun$createTopics$2(AdminManager.scala:98)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at
> scala.collection.mutable.HashMap$$anon$2.$anonfun$foreach$3(HashMap.scala:158)
> at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
> at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
> at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:158)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.AbstractTraversable.map(Traversable.scala:108)
> at kafka.server.AdminManager.createTopics(AdminManager.scala:91)
> at
> kafka.server.KafkaApis.handleCreateTopicsRequest(KafkaApis.scala:1701)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:147)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:70)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9695) AdminClient allows null topic configs, but broker throws NPE
[
https://issues.apache.org/jira/browse/KAFKA-9695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057799#comment-17057799
]
ASF GitHub Bot commented on KAFKA-9695:
---
rajinisivaram commented on pull request #8266: KAFKA-9695; Handle null config
values for createTopics, alterConfigs
URL: https://github.com/apache/kafka/pull/8266
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> AdminClient allows null topic configs, but broker throws NPE
>
>
> Key: KAFKA-9695
> URL: https://issues.apache.org/jira/browse/KAFKA-9695
> Project: Kafka
> Issue Type: Bug
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.6.0
>
>
> Config entries may contain null values, but broker's AdminManager throws NPE
> resulting in UnknownServerException. We should handle null values in configs.
> {code:java}
> [2020-03-10 21:56:07,904] ERROR [Admin Manager on Broker 0]: Error processing
> create topic request CreatableTopic(name='topic', numPartitions=2,
> replicationFactor=3, assignments=[],
> configs=[CreateableTopicConfig(name='message.format.version', value=null),
> CreateableTopicConfig(name='compression.type', value='producer')])
> (kafka.server.AdminManager:76)
> java.lang.NullPointerException
> at java.util.Hashtable.put(Hashtable.java:460)
> at java.util.Properties.setProperty(Properties.java:166)
> at
> kafka.server.AdminManager.$anonfun$createTopics$3(AdminManager.scala:99)
> at scala.collection.Iterator.foreach(Iterator.scala:941)
> at scala.collection.Iterator.foreach$(Iterator.scala:941)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at
> kafka.server.AdminManager.$anonfun$createTopics$2(AdminManager.scala:98)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at
> scala.collection.mutable.HashMap$$anon$2.$anonfun$foreach$3(HashMap.scala:158)
> at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
> at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
> at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:158)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.AbstractTraversable.map(Traversable.scala:108)
> at kafka.server.AdminManager.createTopics(AdminManager.scala:91)
> at
> kafka.server.KafkaApis.handleCreateTopicsRequest(KafkaApis.scala:1701)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:147)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:70)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9458) Kafka crashed in windows environment
[ https://issues.apache.org/jira/browse/KAFKA-9458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057699#comment-17057699 ] ASF GitHub Bot commented on KAFKA-9458: --- hirik commented on pull request #8276: KAFKA-9458: Kafka broker crashes in windows platform URL: https://github.com/apache/kafka/pull/8276 Reason: I found that in LogSegment, files are being renamed as per the status flag(deleted, cleaned, swap). In windows renaming or deleting an opened file is not allowed. Existing behavior (LogSegment Management): While deleting a segment or creating a clean segment, we are trying to change the status of the segment. Segment status is maintained by the file suffix. Possible solutions: 1. Close the existing files before renaming or deleting 2. Maintain a separate status file to store the status of the segment. I've implemented the second solution and it's successfully running on my windows environment. Please review the changes and let me know your comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kafka crashed in windows environment > > > Key: KAFKA-9458 > URL: https://issues.apache.org/jira/browse/KAFKA-9458 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 2.4.0 > Environment: Windows Server 2019 >Reporter: hirik >Priority: Critical > Labels: windows > Fix For: 2.6.0 > > Attachments: Windows_crash_fix.patch, logs.zip > > > Hi, > while I was trying to validate Kafka retention policy, Kafka Server crashed > with below exception trace. > [2020-01-21 17:10:40,475] INFO [Log partition=test1-3, > dir=C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka] > Rolled new log segment at offset 1 in 52 ms. (kafka.log.Log) > [2020-01-21 17:10:40,484] ERROR Error while deleting segments for test1-3 in > dir C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka > (kafka.server.LogDirFailureChannel) > java.nio.file.FileSystemException: > C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka\test1-3\.timeindex > -> > C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka\test1-3\.timeindex.deleted: > The process cannot access the file because it is being used by another > process. > at > java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) > at > java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) > at java.base/sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:395) > at > java.base/sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:292) > at java.base/java.nio.file.Files.move(Files.java:1425) > at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:795) > at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:209) > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:497) > at kafka.log.Log.$anonfun$deleteSegmentFiles$1(Log.scala:2206) > at kafka.log.Log.$anonfun$deleteSegmentFiles$1$adapted(Log.scala:2206) > at scala.collection.immutable.List.foreach(List.scala:305) > at kafka.log.Log.deleteSegmentFiles(Log.scala:2206) > at kafka.log.Log.removeAndDeleteSegments(Log.scala:2191) > at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1700) > at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.scala:17) > at kafka.log.Log.maybeHandleIOException(Log.scala:2316) > at kafka.log.Log.deleteSegments(Log.scala:1691) > at kafka.log.Log.deleteOldSegments(Log.scala:1686) > at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1763) > at kafka.log.Log.deleteOldSegments(Log.scala:1753) > at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:982) > at kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:979) > at scala.collection.immutable.List.foreach(List.scala:305) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:979) > at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:403) > at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:116) > at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:65) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) > at >
[jira] [Commented] (KAFKA-9458) Kafka crashed in windows environment
[ https://issues.apache.org/jira/browse/KAFKA-9458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057698#comment-17057698 ] ASF GitHub Bot commented on KAFKA-9458: --- hirik commented on pull request #8276: KAFKA-9458: Kafka broker crashes in windows platform URL: https://github.com/apache/kafka/pull/8276 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kafka crashed in windows environment > > > Key: KAFKA-9458 > URL: https://issues.apache.org/jira/browse/KAFKA-9458 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 2.4.0 > Environment: Windows Server 2019 >Reporter: hirik >Priority: Critical > Labels: windows > Fix For: 2.6.0 > > Attachments: Windows_crash_fix.patch, logs.zip > > > Hi, > while I was trying to validate Kafka retention policy, Kafka Server crashed > with below exception trace. > [2020-01-21 17:10:40,475] INFO [Log partition=test1-3, > dir=C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka] > Rolled new log segment at offset 1 in 52 ms. (kafka.log.Log) > [2020-01-21 17:10:40,484] ERROR Error while deleting segments for test1-3 in > dir C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka > (kafka.server.LogDirFailureChannel) > java.nio.file.FileSystemException: > C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka\test1-3\.timeindex > -> > C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka\test1-3\.timeindex.deleted: > The process cannot access the file because it is being used by another > process. > at > java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) > at > java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) > at java.base/sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:395) > at > java.base/sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:292) > at java.base/java.nio.file.Files.move(Files.java:1425) > at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:795) > at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:209) > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:497) > at kafka.log.Log.$anonfun$deleteSegmentFiles$1(Log.scala:2206) > at kafka.log.Log.$anonfun$deleteSegmentFiles$1$adapted(Log.scala:2206) > at scala.collection.immutable.List.foreach(List.scala:305) > at kafka.log.Log.deleteSegmentFiles(Log.scala:2206) > at kafka.log.Log.removeAndDeleteSegments(Log.scala:2191) > at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1700) > at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.scala:17) > at kafka.log.Log.maybeHandleIOException(Log.scala:2316) > at kafka.log.Log.deleteSegments(Log.scala:1691) > at kafka.log.Log.deleteOldSegments(Log.scala:1686) > at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1763) > at kafka.log.Log.deleteOldSegments(Log.scala:1753) > at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:982) > at kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:979) > at scala.collection.immutable.List.foreach(List.scala:305) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:979) > at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:403) > at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:116) > at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:65) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:830) > Suppressed: java.nio.file.FileSystemException: > C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka\test1-3\.timeindex > -> > C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka\test1-3\.timeindex.deleted: > The process cannot access the file because it is being used by another > process. > at > java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) > at >
[jira] [Created] (KAFKA-9710) Windows kafka-server-start.bat script fails when Kafka is located in path with whitespace
Tomas Kovan created KAFKA-9710:
--
Summary: Windows kafka-server-start.bat script fails when Kafka is
located in path with whitespace
Key: KAFKA-9710
URL: https://issues.apache.org/jira/browse/KAFKA-9710
Project: Kafka
Issue Type: Bug
Components: tools
Affects Versions: 2.4.0
Reporter: Tomas Kovan
Fix For: 2.4.0
Attachments: kafka_image.png
*Steps to reproduce:*
On Windows when you place Kafka to directory path with space, for example
"C:\kafka path with whitespace" in it and run
".\bin\windows\kafka-server-start.bat", script will fail with error message
"Error: Could not find or load main class path" (example in image).
*Possible solution:*
Working solution is to remove double quotes around %CLASSPATH% variable in
"kafka-run-class.bat" since all paths included in %CLASSPATH% variable already
includes double quotes around them. So changing line 179 in file
"kafka-run-class.bat"
FROM:
{code:java}
set COMMAND=%JAVA% %KAFKA_HEAP_OPTS% %KAFKA_JVM_PERFORMANCE_OPTS%
%KAFKA_JMX_OPTS% %KAFKA_LOG4J_OPTS% -cp "%CLASSPATH%" %KAFKA_OPTS% %*
{code}
TO:
{code:java}
set COMMAND=%JAVA% %KAFKA_HEAP_OPTS% %KAFKA_JVM_PERFORMANCE_OPTS%
%KAFKA_JMX_OPTS% %KAFKA_LOG4J_OPTS% -cp %CLASSPATH% %KAFKA_OPTS% %*
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9225) kafka fail to run on linux-aarch64
[
https://issues.apache.org/jira/browse/KAFKA-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057668#comment-17057668
]
jiamei xie commented on KAFKA-9225:
---
Rocksdb 5.18.4 has been released. It supports all platforms. About this
version, you can refer to
https://github.com/facebook/rocksdb/pull/6497
https://github.com/facebook/rocksdb/issues/6188
I made a PR https://github.com/apache/kafka/pull/8284 to bump to 5.18.4.
> kafka fail to run on linux-aarch64
> --
>
> Key: KAFKA-9225
> URL: https://issues.apache.org/jira/browse/KAFKA-9225
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: jiamei xie
>Priority: Blocker
> Labels: incompatible
> Fix For: 3.0.0
>
> Attachments: compat_report.html
>
>
> *Steps to reproduce:*
> 1. Download Kafka latest source code
> 2. Build it with gradle
> 3. Run
> [streamDemo|[https://kafka.apache.org/23/documentation/streams/quickstart]]
> when running bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo, it crashed with
> the following error message
> {code:java}
> xjm@ubuntu-arm01:~/kafka$bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/core/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/tools/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/api/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/transforms/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/file/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror-client/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/json/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/basic-auth-extension/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [2019-11-19 15:42:23,277] WARN The configuration 'admin.retries' was supplied
> but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:23,278] WARN The configuration 'admin.retry.backoff.ms' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:24,278] ERROR stream-client
> [streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apach e.kafka.streams.KafkaStreams)
> Exception in thread
> "streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48-StreamThread-1"
> java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni1377754636857652484.so:
> /tmp/librocksdbjni13777546368576524 84.so:
> cannot open shared object file: No such file or directory (Possible cause:
> can't load AMD 64-bit .so on a AARCH64-bit platform)
> {code}
> *Analyze:*
> This issue is caused by rocksdbjni-5.18.3.jar which doesn't come with aarch64
> native support. Replace rocksdbjni-5.18.3.jar with rocksdbjni-6.3.6.jar from
> [https://mvnrepository.com/artifact/org.rocksdb/rocksdbjni/6.3.6] can fix
> this problem.
> Attached is the binary compatibility report of rocksdbjni.jar between 5.18.3
> and 6.3.6. The result is 81.8%. So is it possible to upgrade rocksdbjni to
> 6.3.6 in upstream? Should there be any kind of tests to execute, please
> kindly point me. Thanks a lot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (KAFKA-9703) ProducerBatch.split takes up too many resources if the bigBatch is huge
[
https://issues.apache.org/jira/browse/KAFKA-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiamei xie updated KAFKA-9703:
--
Comment: was deleted
(was: https://issues.apache.org/jira/browse/KAFKA-9703)
> ProducerBatch.split takes up too many resources if the bigBatch is huge
> ---
>
> Key: KAFKA-9703
> URL: https://issues.apache.org/jira/browse/KAFKA-9703
> Project: Kafka
> Issue Type: Bug
>Reporter: jiamei xie
>Priority: Major
>
> ProducerBatch.split takes up too many resources and might cause outOfMemory
> error if the bigBatch is huge. About how I found this issue is in
> https://lists.apache.org/list.html?us...@kafka.apache.org:lte=1M:MESSAGE_TOO_LARGE
> Following is the code which takes a lot of resources.
> {code:java}
> for (Record record : recordBatch) {
> assert thunkIter.hasNext();
> Thunk thunk = thunkIter.next();
> if (batch == null)
> batch = createBatchOffAccumulatorForRecord(record,
> splitBatchSize);
> // A newly created batch can always host the first message.
> if (!batch.tryAppendForSplit(record.timestamp(), record.key(),
> record.value(), record.headers(), thunk)) {
> batches.add(batch);
> batch = createBatchOffAccumulatorForRecord(record,
> splitBatchSize);
> batch.tryAppendForSplit(record.timestamp(), record.key(),
> record.value(), record.headers(), thunk);
> }
> {code}
> Refer to RecordAccumulator#tryAppend, we can call closeForRecordAppends()
> after a batch is full.
> {code:java}
> private RecordAppendResult tryAppend(long timestamp, byte[] key, byte[]
> value, Header[] headers,
> Callback callback,
> Deque deque, long nowMs) {
> ProducerBatch last = deque.peekLast();
> if (last != null) {
> FutureRecordMetadata future = last.tryAppend(timestamp, key,
> value, headers, callback, nowMs);
> if (future == null)
> last.closeForRecordAppends();
> else
> return new RecordAppendResult(future, deque.size() > 1 ||
> last.isFull(), false, false);
> }
> return null;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (KAFKA-9703) ProducerBatch.split takes up too many resources if the bigBatch is huge
[
https://issues.apache.org/jira/browse/KAFKA-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiamei xie updated KAFKA-9703:
--
Comment: was deleted
(was: PR has been created in github.com/apache/kafka/pull/8286)
> ProducerBatch.split takes up too many resources if the bigBatch is huge
> ---
>
> Key: KAFKA-9703
> URL: https://issues.apache.org/jira/browse/KAFKA-9703
> Project: Kafka
> Issue Type: Bug
>Reporter: jiamei xie
>Priority: Major
>
> ProducerBatch.split takes up too many resources and might cause outOfMemory
> error if the bigBatch is huge. About how I found this issue is in
> https://lists.apache.org/list.html?us...@kafka.apache.org:lte=1M:MESSAGE_TOO_LARGE
> Following is the code which takes a lot of resources.
> {code:java}
> for (Record record : recordBatch) {
> assert thunkIter.hasNext();
> Thunk thunk = thunkIter.next();
> if (batch == null)
> batch = createBatchOffAccumulatorForRecord(record,
> splitBatchSize);
> // A newly created batch can always host the first message.
> if (!batch.tryAppendForSplit(record.timestamp(), record.key(),
> record.value(), record.headers(), thunk)) {
> batches.add(batch);
> batch = createBatchOffAccumulatorForRecord(record,
> splitBatchSize);
> batch.tryAppendForSplit(record.timestamp(), record.key(),
> record.value(), record.headers(), thunk);
> }
> {code}
> Refer to RecordAccumulator#tryAppend, we can call closeForRecordAppends()
> after a batch is full.
> {code:java}
> private RecordAppendResult tryAppend(long timestamp, byte[] key, byte[]
> value, Header[] headers,
> Callback callback,
> Deque deque, long nowMs) {
> ProducerBatch last = deque.peekLast();
> if (last != null) {
> FutureRecordMetadata future = last.tryAppend(timestamp, key,
> value, headers, callback, nowMs);
> if (future == null)
> last.closeForRecordAppends();
> else
> return new RecordAppendResult(future, deque.size() > 1 ||
> last.isFull(), false, false);
> }
> return null;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9703) ProducerBatch.split takes up too many resources if the bigBatch is huge
[
https://issues.apache.org/jira/browse/KAFKA-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057655#comment-17057655
]
jiamei xie commented on KAFKA-9703:
---
PR has been created in ithub.com/apache/kafka/pull/8286
> ProducerBatch.split takes up too many resources if the bigBatch is huge
> ---
>
> Key: KAFKA-9703
> URL: https://issues.apache.org/jira/browse/KAFKA-9703
> Project: Kafka
> Issue Type: Bug
>Reporter: jiamei xie
>Priority: Major
>
> ProducerBatch.split takes up too many resources and might cause outOfMemory
> error if the bigBatch is huge. About how I found this issue is in
> https://lists.apache.org/list.html?us...@kafka.apache.org:lte=1M:MESSAGE_TOO_LARGE
> Following is the code which takes a lot of resources.
> {code:java}
> for (Record record : recordBatch) {
> assert thunkIter.hasNext();
> Thunk thunk = thunkIter.next();
> if (batch == null)
> batch = createBatchOffAccumulatorForRecord(record,
> splitBatchSize);
> // A newly created batch can always host the first message.
> if (!batch.tryAppendForSplit(record.timestamp(), record.key(),
> record.value(), record.headers(), thunk)) {
> batches.add(batch);
> batch = createBatchOffAccumulatorForRecord(record,
> splitBatchSize);
> batch.tryAppendForSplit(record.timestamp(), record.key(),
> record.value(), record.headers(), thunk);
> }
> {code}
> Refer to RecordAccumulator#tryAppend, we can call closeForRecordAppends()
> after a batch is full.
> {code:java}
> private RecordAppendResult tryAppend(long timestamp, byte[] key, byte[]
> value, Header[] headers,
> Callback callback,
> Deque deque, long nowMs) {
> ProducerBatch last = deque.peekLast();
> if (last != null) {
> FutureRecordMetadata future = last.tryAppend(timestamp, key,
> value, headers, callback, nowMs);
> if (future == null)
> last.closeForRecordAppends();
> else
> return new RecordAppendResult(future, deque.size() > 1 ||
> last.isFull(), false, false);
> }
> return null;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9703) ProducerBatch.split takes up too many resources if the bigBatch is huge
[
https://issues.apache.org/jira/browse/KAFKA-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057655#comment-17057655
]
jiamei xie edited comment on KAFKA-9703 at 3/12/20, 7:17 AM:
-
PR has been created in github.com/apache/kafka/pull/8286
was (Author: adally):
PR has been created in ithub.com/apache/kafka/pull/8286
> ProducerBatch.split takes up too many resources if the bigBatch is huge
> ---
>
> Key: KAFKA-9703
> URL: https://issues.apache.org/jira/browse/KAFKA-9703
> Project: Kafka
> Issue Type: Bug
>Reporter: jiamei xie
>Priority: Major
>
> ProducerBatch.split takes up too many resources and might cause outOfMemory
> error if the bigBatch is huge. About how I found this issue is in
> https://lists.apache.org/list.html?us...@kafka.apache.org:lte=1M:MESSAGE_TOO_LARGE
> Following is the code which takes a lot of resources.
> {code:java}
> for (Record record : recordBatch) {
> assert thunkIter.hasNext();
> Thunk thunk = thunkIter.next();
> if (batch == null)
> batch = createBatchOffAccumulatorForRecord(record,
> splitBatchSize);
> // A newly created batch can always host the first message.
> if (!batch.tryAppendForSplit(record.timestamp(), record.key(),
> record.value(), record.headers(), thunk)) {
> batches.add(batch);
> batch = createBatchOffAccumulatorForRecord(record,
> splitBatchSize);
> batch.tryAppendForSplit(record.timestamp(), record.key(),
> record.value(), record.headers(), thunk);
> }
> {code}
> Refer to RecordAccumulator#tryAppend, we can call closeForRecordAppends()
> after a batch is full.
> {code:java}
> private RecordAppendResult tryAppend(long timestamp, byte[] key, byte[]
> value, Header[] headers,
> Callback callback,
> Deque deque, long nowMs) {
> ProducerBatch last = deque.peekLast();
> if (last != null) {
> FutureRecordMetadata future = last.tryAppend(timestamp, key,
> value, headers, callback, nowMs);
> if (future == null)
> last.closeForRecordAppends();
> else
> return new RecordAppendResult(future, deque.size() > 1 ||
> last.isFull(), false, false);
> }
> return null;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9703) ProducerBatch.split takes up too many resources if the bigBatch is huge
[
https://issues.apache.org/jira/browse/KAFKA-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057652#comment-17057652
]
ASF GitHub Bot commented on KAFKA-9703:
---
jiameixie commented on pull request #8286: KAFKA-9703:Free up resources when
splitting huge batches
URL: https://github.com/apache/kafka/pull/8286
Method split takes up too many resources and might
cause outOfMemory error when the bigBatch is huge.
Call closeForRecordAppends() to free up resources
like compression buffers.
Change-Id: Iac6519fcc2e432330b8af2d9f68a8d4d4a07646b
Signed-off-by: Jiamei Xie
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> ProducerBatch.split takes up too many resources if the bigBatch is huge
> ---
>
> Key: KAFKA-9703
> URL: https://issues.apache.org/jira/browse/KAFKA-9703
> Project: Kafka
> Issue Type: Bug
>Reporter: jiamei xie
>Priority: Major
>
> ProducerBatch.split takes up too many resources and might cause outOfMemory
> error if the bigBatch is huge. About how I found this issue is in
> https://lists.apache.org/list.html?us...@kafka.apache.org:lte=1M:MESSAGE_TOO_LARGE
> Following is the code which takes a lot of resources.
> {code:java}
> for (Record record : recordBatch) {
> assert thunkIter.hasNext();
> Thunk thunk = thunkIter.next();
> if (batch == null)
> batch = createBatchOffAccumulatorForRecord(record,
> splitBatchSize);
> // A newly created batch can always host the first message.
> if (!batch.tryAppendForSplit(record.timestamp(), record.key(),
> record.value(), record.headers(), thunk)) {
> batches.add(batch);
> batch = createBatchOffAccumulatorForRecord(record,
> splitBatchSize);
> batch.tryAppendForSplit(record.timestamp(), record.key(),
> record.value(), record.headers(), thunk);
> }
> {code}
> Refer to RecordAccumulator#tryAppend, we can call closeForRecordAppends()
> after a batch is full.
> {code:java}
> private RecordAppendResult tryAppend(long timestamp, byte[] key, byte[]
> value, Header[] headers,
> Callback callback,
> Deque deque, long nowMs) {
> ProducerBatch last = deque.peekLast();
> if (last != null) {
> FutureRecordMetadata future = last.tryAppend(timestamp, key,
> value, headers, callback, nowMs);
> if (future == null)
> last.closeForRecordAppends();
> else
> return new RecordAppendResult(future, deque.size() > 1 ||
> last.isFull(), false, false);
> }
> return null;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9700) Negative estimatedCompressionRatio leads to misjudgment about if there is no room
[
https://issues.apache.org/jira/browse/KAFKA-9700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057651#comment-17057651
]
ASF GitHub Bot commented on KAFKA-9700:
---
jiameixie commented on pull request #8285: KAFKA-9700:Fix negative
estimatedCompressionRatio issue
URL: https://github.com/apache/kafka/pull/8285
There are cases that currentEstimation is smaller than
COMPRESSION_RATIO_IMPROVING_STEP and it will get negative
estimatedCompressionRatio,which leads to misjudgment
about if there is no room and MESSAGE_TOO_LARGE might occur.
Change-Id: I0932a2a6ca669f673ab5d862d3fe7b2bb6d96ff6
Signed-off-by: Jiamei Xie
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Negative estimatedCompressionRatio leads to misjudgment about if there is no
> room
> -
>
> Key: KAFKA-9700
> URL: https://issues.apache.org/jira/browse/KAFKA-9700
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Reporter: jiamei xie
>Priority: Major
>
> * When I run the following command
> bin/kafka-producer-perf-test.sh --topic test --num-records 5000
> --throughput -1 --record-size 5000 --producer-props
> bootstrap.servers=server04:9092 acks=1 buffer.memory=67108864 batch.size
> 65536 compression.type=zstd
> There was a warning:
> [2020-03-06 17:36:50,216] WARN [Producer clientId=producer-1] Got error
> produce response in correlation id 3261 on topic-partition test-1, splitting
> and retrying (2147483647 attempts left). Error: MESSAGE_TOO_LARGE
> (org.apache.kafka.clients.producer.internals.Sender)
> * The batch size(65536) is smaller than max.message.bytes (1048588) . So
> it's not the root cause.
> * I added some logs in CompressionRatioEstimator.updateEstimation and found
> there were negative currentEstimation values. The following were logs I added
> public static float updateEstimation(String topic, CompressionType type,
> float observedRatio) {
> float[] compressionRatioForTopic = getAndCreateEstimationIfAbsent(topic);
> float currentEstimation = compressionRatioForTopic[type.id];
> synchronized (compressionRatioForTopic) {
> if (observedRatio > currentEstimation)
> {
> compressionRatioForTopic[type.id] =
> Math.max(currentEstimation + COMPRESSION_RATIO_DETERIORATE_STEP,
> observedRatio);
> }
> else if (observedRatio < currentEstimation) {
> compressionRatioForTopic[type.id] = currentEstimation -
> COMPRESSION_RATIO_IMPROVING_STEP;
> log.warn("currentEstimation is {} ,
> COMPRESSION_RATIO_IMPROVING_STEP is {} , compressionRatioForTopic[type.id] is
> {}, type.id is {}", currentEstimation,
> COMPRESSION_RATIO_IMPROVING_STEP,compressionRatioForTopic[type.id], type.id);
> }
> }
> return compressionRatioForTopic[type.id];
> }
> The observedRatio is smaller than COMPRESSION_RATIO_IMPROVING_STEP in some
> cases. Some I think the else if block should be changed into
> else if (observedRatio < currentEstimation) {
> compressionRatioForTopic[type.id] =
> Math.max(currentEstimation - COMPRESSION_RATIO_IMPROVING_STEP, observedRatio);
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9225) kafka fail to run on linux-aarch64
[
https://issues.apache.org/jira/browse/KAFKA-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057644#comment-17057644
]
ASF GitHub Bot commented on KAFKA-9225:
---
jiameixie commented on pull request #8284: KAFKA-9225: rocksdb 5.18.3 to 5.18.4
URL: https://github.com/apache/kafka/pull/8284
Bump rocksdb 5.18.3 to 5.18.4 that supports all platforms.
Issues about this version are https://github.com/facebook/rocksdb/pull/6497
and https://github.com/facebook/rocksdb/issues/6188
Change-Id: I3febec8e36550edcb7f88839cc1e2b2a54984564
Signed-off-by: Jiamei Xie
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> kafka fail to run on linux-aarch64
> --
>
> Key: KAFKA-9225
> URL: https://issues.apache.org/jira/browse/KAFKA-9225
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: jiamei xie
>Priority: Blocker
> Labels: incompatible
> Fix For: 3.0.0
>
> Attachments: compat_report.html
>
>
> *Steps to reproduce:*
> 1. Download Kafka latest source code
> 2. Build it with gradle
> 3. Run
> [streamDemo|[https://kafka.apache.org/23/documentation/streams/quickstart]]
> when running bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo, it crashed with
> the following error message
> {code:java}
> xjm@ubuntu-arm01:~/kafka$bin/kafka-run-class.sh
> org.apache.kafka.streams.examples.wordcount.WordCountDemo
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/core/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/tools/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/api/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/transforms/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/file/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/mirror-client/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/json/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/xjm/kafka/connect/basic-auth-extension/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [2019-11-19 15:42:23,277] WARN The configuration 'admin.retries' was supplied
> but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:23,278] WARN The configuration 'admin.retry.backoff.ms' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2019-11-19 15:42:24,278] ERROR stream-client
> [streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apach e.kafka.streams.KafkaStreams)
> Exception in thread
>
[jira] [Commented] (KAFKA-7983) supporting replication.throttled.replicas in dynamic broker configuration
[ https://issues.apache.org/jira/browse/KAFKA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057634#comment-17057634 ] Cheng Tan commented on KAFKA-7983: -- [https://github.com/apache/kafka/pull/8283] > supporting replication.throttled.replicas in dynamic broker configuration > - > > Key: KAFKA-7983 > URL: https://issues.apache.org/jira/browse/KAFKA-7983 > Project: Kafka > Issue Type: New Feature > Components: core >Reporter: Jun Rao >Assignee: Cheng Tan >Priority: Major > > In > [KIP-226|https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration#KIP-226-DynamicBrokerConfiguration-DefaultTopicconfigs], > we added the support to change broker defaults dynamically. However, it > didn't support changing leader.replication.throttled.replicas and > follower.replication.throttled.replicas. These 2 configs were introduced in > [KIP-73|https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas] > and controls the set of topic partitions on which replication throttling > will be engaged. One useful case is to be able to set a default value for > both configs to * to allow throttling to be engaged for all topic partitions. > Currently, the static default value for both configs are ignored for > replication throttling, it would be useful to fix that as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7983) supporting replication.throttled.replicas in dynamic broker configuration
[ https://issues.apache.org/jira/browse/KAFKA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057633#comment-17057633 ] ASF GitHub Bot commented on KAFKA-7983: --- d8tltanc commented on pull request #8283: [WIP] KAFKA-7983: supporting replication.throttled.replicas in dynamic broker configuration URL: https://github.com/apache/kafka/pull/8283 **More detailed description of your change** > In KIP-226, we added the support to change broker defaults dynamically. However, it didn't support changing leader.replication.throttled.replicas and follower.replication.throttled.replicas. These 2 configs were introduced in KIP-73 and controls the set of topic partitions on which replication throttling will be engaged. One useful case is to be able to set a default value for both configs to * to allow throttling to be engaged for all topic partitions. Currently, the static default value for both configs are ignored for replication throttling, it would be useful to fix that as well. > leader.replication.throttled.replicas and follower.replication.throttled.replicas are dynamically set through ReplicationQuotaManager.markThrottled() at the topic level. However, these two properties don't exist at the broker level config and BrokerConfigHandler doesn't call ReplicationQuotaManager.markThrottled(). So, currently, we can't set leader.replication.throttled.replicas and follower.replication.throttled.replicas at the broker level either statically or dynamically. In this patch, we introduced two new dynamic broker configs, both of them are type of boolean: "leader.replication.throttled" (default: false) "follower.replication.throttled" (default: false) If "leader.replication.throttled" is set to "true", all leader brokers will be throttled. Similarly, if "follower.replication.throttled" is set to "true", all follower brokers will be throttled. The throttle mechanism is introduced in KIP-73. To implement the broker level throttle, I added a new class variable to ReplicationQuotaManager. The BrokerConfigHandler will call updateBrokerThrottle() and update this class variable upon receiving the config change notification from ZooKeeper. ReplicationQuotaManager::isThrottled() *Summary of testing strategy (including rationale) Added ReplicationQuotaManagerTest::shouldBrokerLevelThrottleAffectAllTopicPartition() to test if all topic partitions will be throttled when the broker is throttled. I'm currently working on adding more tests. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > supporting replication.throttled.replicas in dynamic broker configuration > - > > Key: KAFKA-7983 > URL: https://issues.apache.org/jira/browse/KAFKA-7983 > Project: Kafka > Issue Type: New Feature > Components: core >Reporter: Jun Rao >Assignee: Cheng Tan >Priority: Major > > In > [KIP-226|https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration#KIP-226-DynamicBrokerConfiguration-DefaultTopicconfigs], > we added the support to change broker defaults dynamically. However, it > didn't support changing leader.replication.throttled.replicas and > follower.replication.throttled.replicas. These 2 configs were introduced in > [KIP-73|https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas] > and controls the set of topic partitions on which replication throttling > will be engaged. One useful case is to be able to set a default value for > both configs to * to allow throttling to be engaged for all topic partitions. > Currently, the static default value for both configs are ignored for > replication throttling, it would be useful to fix that as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
