[jira] [Updated] (KAFKA-8035) Add tests for generics in KStream API
[
https://issues.apache.org/jira/browse/KAFKA-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-8035:
-
Description:
During the work on KAFKA-4217, it was discovered that some combinations of Java
generics block the usage of lambda expressions as parameters of the KStream API
([see this
discussion|https://github.com/apache/kafka/pull/5273#discussion_r216810275]).
To avoid using those blocking combinations of generics, tests shall be
implemented that verify that both lambda expressions as well as classes
(anonymous and declared) can be used as parameters with the KStream API. Those
tests may also serve as regression tests to ensure that future changes to the
generics in the KStream API may not make the API incompatible with previous
versions.
Unlike other tests, the tests required here pass if they compile. For example,
to verify that the parameter {{mapper}} in
{code:java}
KStream flatMap(final KeyValueMapper>> mapper);
{code}
accepts a {{KeyValueMapper}} specified as a lambda expression that returns an
implementation of the {{Iterable}} interface, the following stream could be
specified in the test:
{code:java}
stream
.flatMap((Integer key, Integer value) -> Arrays.asList(
KeyValue.pair(key, value),
KeyValue.pair(key, value),
KeyValue.pair(key, value)))
.foreach(action);
{code}
If the test compiles, the test passes.
Other tests for {{flatMap}} need to check the bounds of the generics, e.g., if
the {{mapper}} accepts a {{KeyValueMapper}} specified as a lambda expression
that takes a super class of K and V as inputs.
was:
During the work on KAFKA-4217, it was discovered that some combinations of Java
generics block the usage of lambda functions ([see this
discussion|https://github.com/apache/kafka/pull/5273#discussion_r216810275]).
To avoid using those blocking combinations of generics, tests shall be
implemented that verify that lambda functions can be used with the KStream API.
Those tests may also serve as regression tests to ensure that future changes to
the generics in the KStream API may not block lambda functions and make the API
incompatible with previous versions.
Unlike other tests, the tests required here pass if they compile. For example,
to verify that the parameter {{mapper}} in
{code:java}
KStream flatMap(final KeyValueMapper>> mapper);
{code}
accepts a {{KeyValueMapper}} specified as a lambda function that returns an
implementation of the {{Iterable}} interface, the following stream could be
specified in the test:
{code:java}
stream
.flatMap((Integer key, Integer value) -> Arrays.asList(
KeyValue.pair(key, value),
KeyValue.pair(key, value),
KeyValue.pair(key, value)))
.foreach(action);
{code}
If the test compiles, the test passes.
Other tests for {{flatMap}} need to check the bounds of the generics, e.g., if
the {{mapper}} accepts a {{KeyValueMapper}} specified as a lambda function that
takes a super class of K and V as inputs.
> Add tests for generics in KStream API
> --
>
> Key: KAFKA-8035
> URL: https://issues.apache.org/jira/browse/KAFKA-8035
> Project: Kafka
> Issue Type: Test
> Components: streams
>Reporter: Bruno Cadonna
>Priority: Major
> Labels: unit-test
>
> During the work on KAFKA-4217, it was discovered that some combinations of
> Java generics block the usage of lambda expressions as parameters of the
> KStream API ([see this
> discussion|https://github.com/apache/kafka/pull/5273#discussion_r216810275]).
> To avoid using those blocking combinations of generics, tests shall be
> implemented that verify that both lambda expressions as well as classes
> (anonymous and declared) can be used as parameters with the KStream API.
> Those tests may also serve as regression tests to ensure that future changes
> to the generics in the KStream API may not make the API incompatible with
> previous versions.
> Unlike other tests, the tests required here pass if they compile. For
> example, to verify that the parameter {{mapper}} in
> {code:java}
> KStream flatMap(final KeyValueMapper extends Iterable>> mapper);
> {code}
> accepts a {{KeyValueMapper}} specified as a lambda expression that returns an
> implementation of the {{Iterable}} interface, the following stream could be
> specified in the test:
> {code:java}
> stream
> .flatMap((Integer key, Integer value) -> Arrays.asList(
> KeyValue.pair(key, value),
> KeyValue.pair(key, value),
> KeyValue.pair(key, value)))
> .foreach(action);
> {code}
> If the test compiles, the test passes.
> Other tests for {{flatMap}} need to check the bounds of the generics, e.g.,
> if the {{mapper}} accepts a {{KeyValueMapper}} specified as a lambda
> expression that takes a super class
[jira] [Commented] (KAFKA-7965) Flaky Test ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
[
https://issues.apache.org/jira/browse/KAFKA-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818336#comment-16818336
]
Bruno Cadonna commented on KAFKA-7965:
--
Two more failure with Java 8 and Java 11 with the same error messages posted
above.
> Flaky Test
> ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
>
>
> Key: KAFKA-7965
> URL: https://issues.apache.org/jira/browse/KAFKA-7965
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: huxihx
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/21/]
> {quote}java.lang.AssertionError: Received 0, expected at least 68 at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.ConsumerBounceTest.receiveAndCommit(ConsumerBounceTest.scala:557)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1(ConsumerBounceTest.scala:320)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1$adapted(ConsumerBounceTest.scala:319)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup(ConsumerBounceTest.scala:319){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-6819) Refactor build-in StreamsMetrics internal implementations
[
https://issues.apache.org/jira/browse/KAFKA-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna reassigned KAFKA-6819:
Assignee: Bruno Cadonna
> Refactor build-in StreamsMetrics internal implementations
> -
>
> Key: KAFKA-6819
> URL: https://issues.apache.org/jira/browse/KAFKA-6819
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Guozhang Wang
>Assignee: Bruno Cadonna
>Priority: Major
>
> Our current internal implementations of StreamsMetrics and different layered
> metrics like StreamMetricsThreadImpl, TaskMetrics, NodeMetrics etc are a bit
> messy nowadays. We could improve on the current situation by doing the
> following:
> 0. For thread-level metrics, refactor the {{StreamsMetricsThreadImpl}} class
> to {{ThreadMetrics}} such that a) it does not extend from
> {{StreamsMetricsImpl}} but just include the {{StreamsMetricsThreadImpl}} as
> its constructor parameters. And make its constructor, replacing with a static
> {{addAllSensors(threadName)}} that tries to register all the thread-level
> sensors for the given thread name.
> 1. Add a static function for each of the built-in sensors of the thread-level
> metrics in {{ThreadMetrics}} that relies on the internal
> {{StreamsMetricsConventions}} to get thread level sensor names. If the sensor
> cannot be found from the internal {{Metrics}} registry, create the sensor
> on-the-fly.
> 2.a Add a static {{removeAllSensors(threadName)}} function in
> {{ThreadMetrics}} that tries to de-register all the thread-level metrics for
> this thread, if there is no sensors then it will be a no-op. In
> {{StreamThread#close()}} we will trigger this function; and similarly in
> `TopologyTestDriver` when we close the driver we will also call this function
> as well. As a result, the {{ThreadMetrics}} class itself would only contain
> static functions with no member fields at all.
> 2.b We can consider doing the same for {{TaskMetrics}}, {{NodeMetrics}} and
> {{NamedCacheMetrics}} as well, and add a {{StoreMetrics}} following the
> similar pattern: although these metrics are not accessed externally to their
> enclosing class in the future this may be changed as well.
> 3. Then, we only pass {{StreamsMetricsImpl}} around between the internal
> classes, to access the specific sensor whenever trying to record it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna reassigned KAFKA-6498:
Assignee: Bruno Cadonna (was: james chien)
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Assignee: Bruno Cadonna
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8412) Still a nullpointer exception thrown on shutdown while flushing before closing producers
[
https://issues.apache.org/jira/browse/KAFKA-8412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-8412:
-
Labels: triaged (was: )
> Still a nullpointer exception thrown on shutdown while flushing before
> closing producers
>
>
> Key: KAFKA-8412
> URL: https://issues.apache.org/jira/browse/KAFKA-8412
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.1
>Reporter: Sebastiaan
>Priority: Minor
> Labels: triaged
>
> I found a closed issue and replied there but decided to open one myself
> because although they're related they're slightly different. The original
> issue is at https://issues.apache.org/jira/browse/KAFKA-7678
> The fix there has been to implement a null check around closing a producer
> because in some cases the producer is already null there (has been closed
> already)
> In version 2.1.1 we are getting a very similar exception, but in the 'flush'
> method that is called pre-close. This is in the log:
> {code:java}
> message: stream-thread
> [webhook-poster-7034dbb0-7423-476b-98f3-d18db675d6d6-StreamThread-1] Failed
> while closing StreamTask 1_26 due to the following error:
> logger_name: org.apache.kafka.streams.processor.internals.AssignedStreamsTasks
> java.lang.NullPointerException: null
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.flush(RecordCollectorImpl.java:245)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.flushState(StreamTask.java:493)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:443)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.suspend(StreamTask.java:568)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:691)
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks.close(AssignedTasks.java:397)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.shutdown(TaskManager.java:260)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.completeShutdown(StreamThread.java:1181)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:758){code}
> Followed by:
>
> {code:java}
> message: task [1_26] Could not close task due to the following error:
> logger_name: org.apache.kafka.streams.processor.internals.StreamTask
> java.lang.NullPointerException: null
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.flush(RecordCollectorImpl.java:245)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.flushState(StreamTask.java:493)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:443)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.suspend(StreamTask.java:568)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:691)
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks.close(AssignedTasks.java:397)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.shutdown(TaskManager.java:260)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.completeShutdown(StreamThread.java:1181)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:758){code}
> If I look at the source code at this point, I see a nice null check in the
> close method, but not in the flush method that is called just before that:
> {code:java}
> public void flush() {
> this.log.debug("Flushing producer");
> this.producer.flush();
> this.checkForException();
> }
> public void close() {
> this.log.debug("Closing producer");
> if (this.producer != null) {
> this.producer.close();
> this.producer = null;
> }
> this.checkForException();
> }{code}
> Seems to my (ignorant) eye that the flush method should also be wrapped in a
> null check in the same way as has been done for close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8412) Still a nullpointer exception thrown on shutdown while flushing before closing producers
[
https://issues.apache.org/jira/browse/KAFKA-8412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-8412:
-
Labels: (was: triaged)
> Still a nullpointer exception thrown on shutdown while flushing before
> closing producers
>
>
> Key: KAFKA-8412
> URL: https://issues.apache.org/jira/browse/KAFKA-8412
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.1
>Reporter: Sebastiaan
>Priority: Minor
>
> I found a closed issue and replied there but decided to open one myself
> because although they're related they're slightly different. The original
> issue is at https://issues.apache.org/jira/browse/KAFKA-7678
> The fix there has been to implement a null check around closing a producer
> because in some cases the producer is already null there (has been closed
> already)
> In version 2.1.1 we are getting a very similar exception, but in the 'flush'
> method that is called pre-close. This is in the log:
> {code:java}
> message: stream-thread
> [webhook-poster-7034dbb0-7423-476b-98f3-d18db675d6d6-StreamThread-1] Failed
> while closing StreamTask 1_26 due to the following error:
> logger_name: org.apache.kafka.streams.processor.internals.AssignedStreamsTasks
> java.lang.NullPointerException: null
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.flush(RecordCollectorImpl.java:245)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.flushState(StreamTask.java:493)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:443)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.suspend(StreamTask.java:568)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:691)
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks.close(AssignedTasks.java:397)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.shutdown(TaskManager.java:260)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.completeShutdown(StreamThread.java:1181)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:758){code}
> Followed by:
>
> {code:java}
> message: task [1_26] Could not close task due to the following error:
> logger_name: org.apache.kafka.streams.processor.internals.StreamTask
> java.lang.NullPointerException: null
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.flush(RecordCollectorImpl.java:245)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.flushState(StreamTask.java:493)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:443)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.suspend(StreamTask.java:568)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:691)
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks.close(AssignedTasks.java:397)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.shutdown(TaskManager.java:260)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.completeShutdown(StreamThread.java:1181)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:758){code}
> If I look at the source code at this point, I see a nice null check in the
> close method, but not in the flush method that is called just before that:
> {code:java}
> public void flush() {
> this.log.debug("Flushing producer");
> this.producer.flush();
> this.checkForException();
> }
> public void close() {
> this.log.debug("Closing producer");
> if (this.producer != null) {
> this.producer.close();
> this.producer = null;
> }
> this.checkForException();
> }{code}
> Seems to my (ignorant) eye that the flush method should also be wrapped in a
> null check in the same way as has been done for close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7994) Improve Stream-Time for rebalances and restarts
[
https://issues.apache.org/jira/browse/KAFKA-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848791#comment-16848791
]
Bruno Cadonna commented on KAFKA-7994:
--
I run into a case where the reset of the stream time to -1 after a restart
resulted in incorrect results. More specifically, an event that should have
been dropped because the grace period was exceeded opened a window.
Suppose the following application:
{code:java}
builder
.stream(INPUT_TOPIC, Consumed.with(Serdes.String(), Serdes.String()))
.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofMinutes(60)).grace(Duration.ofMinutes(30)))
...
{code}
with the following input (numbers are timestamps)
m1, m2, m3, m61, m4, m90, m5
application restart
m6, m150
With the reset of the stream time to -1 after restart, the application will
create the following windows:
[0, 60), {m1, m2, m3, m4}
[0, 60), {m6}
[60, 120), {m61, m90}
Window [0, 60), {m6} is wrong because if there had not be a restart, stream
time would be at 90 when m6 reaches the processor. Stream time 90 is outside
the grace period and m6 should be dropped. However, with the restart stream
time is reset to -1 and m6 is not dropped because the condition to drop events
timestamp < stream-time - grace-period is not satisfied, i.e., 6 < -1 - 30.
Hence, a second window [0, 60) is incorrectly opened.
> Improve Stream-Time for rebalances and restarts
> ---
>
> Key: KAFKA-7994
> URL: https://issues.apache.org/jira/browse/KAFKA-7994
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Richard Yu
>Priority: Major
> Attachments: possible-patch.diff
>
>
> We compute a per-partition partition-time as the maximum timestamp over all
> records processed so far. Furthermore, we use partition-time to compute
> stream-time for each task as maximum over all partition-times (for all
> corresponding task partitions). This stream-time is used to make decisions
> about processing out-of-order records or drop them if they are late (ie,
> timestamp < stream-time - grace-period).
> During rebalances and restarts, stream-time is initialized as UNKNOWN (ie,
> -1) for tasks that are newly created (or migrated). In net effect, we forget
> current stream-time for this case what may lead to non-deterministic behavior
> if we stop processing right before a late record, that would be dropped if we
> continue processing, but is not dropped after rebalance/restart. Let's look
> at an examples with a grade period of 5ms for a tumbling windowed of 5ms, and
> the following records (timestamps in parenthesis):
>
> {code:java}
> r1(0) r2(5) r3(11) r4(2){code}
> In the example, stream-time advances as 0, 5, 11, 11 and thus record `r4` is
> dropped as late because 2 < 6 = 11 - 5. However, if we stop processing or
> rebalance after processing `r3` but before processing `r4`, we would
> reinitialize stream-time as -1, and thus would process `r4` on restart/after
> rebalance. The problem is, that stream-time does advance differently from a
> global point of view: 0, 5, 11, 2.
> Note, this is a corner case, because if we would stop processing one record
> earlier, ie, after processing `r2` but before processing `r3`, stream-time
> would be advance correctly from a global point of view.
> A potential fix would be, to store latest observed partition-time in the
> metadata of committed offsets. Thus way, on restart/rebalance we can
> re-initialize time correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-8440) Flaky Test KStreamAggregationIntegrationTest#shouldReduceSessionWindows
Bruno Cadonna created KAFKA-8440: Summary: Flaky Test KStreamAggregationIntegrationTest#shouldReduceSessionWindows Key: KAFKA-8440 URL: https://issues.apache.org/jira/browse/KAFKA-8440 Project: Kafka Issue Type: Bug Components: streams, unit tests Affects Versions: 2.3.0 Reporter: Bruno Cadonna -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8441) CLONE - Flaky Test KStreamAggregationIntegrationTest#shouldReduceSessionWindows
Bruno Cadonna created KAFKA-8441:
Summary: CLONE - Flaky Test
KStreamAggregationIntegrationTest#shouldReduceSessionWindows
Key: KAFKA-8441
URL: https://issues.apache.org/jira/browse/KAFKA-8441
Project: Kafka
Issue Type: Bug
Components: streams, unit tests
Affects Versions: 2.3.0
Reporter: Bruno Cadonna
h1. Stacktrace:
{noformat}
java.lang.AssertionError:
Expected:
but: was null
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
at
org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest.shouldReduceSessionWindows(KStreamAggregationIntegrationTest.java:663)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
[jira] [Updated] (KAFKA-8441) Flaky Test KStreamAggregationIntegrationTest#shouldReduceSessionWindows
[
https://issues.apache.org/jira/browse/KAFKA-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-8441:
-
Summary: Flaky Test
KStreamAggregationIntegrationTest#shouldReduceSessionWindows (was: CLONE -
Flaky Test KStreamAggregationIntegrationTest#shouldReduceSessionWindows)
> Flaky Test KStreamAggregationIntegrationTest#shouldReduceSessionWindows
> ---
>
> Key: KAFKA-8441
> URL: https://issues.apache.org/jira/browse/KAFKA-8441
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.3.0
>Reporter: Bruno Cadonna
>Priority: Critical
> Labels: flaky-test
>
> h1. Stacktrace:
> {noformat}
> java.lang.AssertionError:
> Expected:
> but: was null
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
> at
> org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest.shouldReduceSessionWindows(KStreamAggregationIntegrationTest.java:663)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gra
[jira] [Updated] (KAFKA-8441) Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
[
https://issues.apache.org/jira/browse/KAFKA-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-8441:
-
Summary: Flaky Test
RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated (was: Flaky Test
KStreamAggregationIntegrationTest#shouldReduceSessionWindows)
> Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
>
>
> Key: KAFKA-8441
> URL: https://issues.apache.org/jira/browse/KAFKA-8441
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.3.0
>Reporter: Bruno Cadonna
>Priority: Critical
> Labels: flaky-test
>
> h1. Stacktrace:
> {noformat}
> java.lang.AssertionError: Condition not met within timeout 3. Topics not
> deleted after 3 milli seconds.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:375)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:352)
> at
> org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster.deleteTopicsAndWait(EmbeddedKafkaCluster.java:265)
> at
> org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster.deleteAndRecreateTopics(EmbeddedKafkaCluster.java:288)
> at
> org.apache.kafka.streams.integration.RegexSourceIntegrationTest.setUp(RegexSourceIntegrationTest.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTe
[jira] [Resolved] (KAFKA-8440) Flaky Test KStreamAggregationIntegrationTest#shouldReduceSessionWindows
[
https://issues.apache.org/jira/browse/KAFKA-8440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna resolved KAFKA-8440.
--
Resolution: Invalid
> Flaky Test KStreamAggregationIntegrationTest#shouldReduceSessionWindows
> ---
>
> Key: KAFKA-8440
> URL: https://issues.apache.org/jira/browse/KAFKA-8440
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.3.0
>Reporter: Bruno Cadonna
>Priority: Major
> Labels: flaky-test
>
> h1. Stacktrace:
> {noformat}
> java.lang.AssertionError:
> Expected:
> but: was null
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
> at
> org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest.shouldReduceSessionWindows(KStreamAggregationIntegrationTest.java:663)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(Ref
[jira] [Resolved] (KAFKA-8441) Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
[
https://issues.apache.org/jira/browse/KAFKA-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna resolved KAFKA-8441.
--
Resolution: Invalid
> Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
>
>
> Key: KAFKA-8441
> URL: https://issues.apache.org/jira/browse/KAFKA-8441
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.3.0
>Reporter: Bruno Cadonna
>Priority: Major
> Labels: flaky-test
>
> h1. Stacktrace:
> {noformat}
> java.lang.AssertionError: Condition not met within timeout 3. Topics not
> deleted after 3 milli seconds.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:375)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:352)
> at
> org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster.deleteTopicsAndWait(EmbeddedKafkaCluster.java:265)
> at
> org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster.deleteAndRecreateTopics(EmbeddedKafkaCluster.java:288)
> at
> org.apache.kafka.streams.integration.RegexSourceIntegrationTest.setUp(RegexSourceIntegrationTest.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
> at su
[jira] [Commented] (KAFKA-8497) kafka streams application占用内存很高
[
https://issues.apache.org/jira/browse/KAFKA-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16857357#comment-16857357
]
Bruno Cadonna commented on KAFKA-8497:
--
Hi [~dongni98]
Could you please use English for Jira issues (and in all other communication
channels)?
Using English makes it much easier for us to react on your messages.
> kafka streams application占用内存很高
> ---
>
> Key: KAFKA-8497
> URL: https://issues.apache.org/jira/browse/KAFKA-8497
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
>Reporter: sunqing
>Priority: Major
>
> 一个简单的kafka streams测试应用，使用KStream来消费数据，当所消费的kafka
> Topic中的数据暴涨时，或者要消费的Topic中待消费数据量很大时，消费程序占用的内存会非常高，能达到20多G，
> 疑问：kafka streams不是逐条消费吗，为啥topic中的数据量很大时会导致程序内存飙升
>
> 测试程序代码如下：
>
> 代码如下：
> public class RunMain {
> public static StreamsBuilder builder = new StreamsBuilder();
> public static void kafkaStreamStart() {
> KStream stream =
> builder.stream(Arrays.asList("wk_wangxin_po"));
> Properties props = new Properties();
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, "testwang_xin");
>
> props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
>
> "zktj-kafka-broker-out-1:29092,zktj-kafka-broker-out-2:29092,zktj-kafka-broker-out-3:29092");
> props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
> Serdes.String().getClass());
> props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
> Serdes.String().getClass());
> props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 3000);
> props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);
> props.setProperty("security.protocol", "SASL_PLAINTEXT");
> props.setProperty("sasl.mechanism", "PLAIN");
> props.setProperty("sasl.kerberos.service.name", "kafka");
> System.setProperty("java.security.auth.login.config",
> "./conf/kafka_client_jaas.conf");
>
> stream.foreach(new ForeachAction() {
> @Override
> public void apply(String key, String value) {
> System.out.println("");
> System.out.println(key);
> }
> });
> Topology topo = builder.build();
> KafkaStreams streams = new KafkaStreams(topo, props);
> streams.start();
> }
> public static void main(String[] args) {
> kafkaStreamStart();
> }
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8497) kafka streams application takes up a lot of memory
[
https://issues.apache.org/jira/browse/KAFKA-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16857483#comment-16857483
]
Bruno Cadonna commented on KAFKA-8497:
--
Hi [~dongni98],
Thank you for the translation to English.
Could you ask this question on the user mailing list under
us...@kafka.apache.org?
I think the user mailing list is the more appropriate place to post your issue,
because we do not know yet if this is a bug.
If you have not already subscribed to the mailing list, you have first to
subscribe to it by sending an e-mail to users-subscr...@kafka.apache.org.
There is a good chance, that somebody on the mailing list can help you with
your issue. If it turns out that this is a bug you should open a bug here in
Jira.
I will now close this bug report, because we actually do not know if this is
really a bug.
> kafka streams application takes up a lot of memory
> --
>
> Key: KAFKA-8497
> URL: https://issues.apache.org/jira/browse/KAFKA-8497
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
>Reporter: sunqing
>Priority: Major
>
>
> A simple kafka streams application, use KStream to consume data, as below。
> Memory usage is very high when there is a large amount of data under the
> consuming topic.
> Sometimes it goes up to 20G.
> This is very strange. The program doesn't do anything. It just reads the data
> and prints it to the screen. Why is the memory usage so high when there is a
> lot of data in the topic?
>
>
>
> The program code：
>
> public class TestMain {
> public static StreamsBuilder builder = new StreamsBuilder();
> public static void kafkaStreamStart() {
> KStream stream =
> builder.stream(Arrays.asList("wk_wangxin_po"));
> Properties props = new Properties();
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, "testwang_xin");
> props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
>
> "zktj-kafka-broker-out-1:29092,zktj-kafka-broker-out-2:29092,zktj-kafka-broker-out-3:29092");
> props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
> Serdes.String().getClass());
> props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
> Serdes.String().getClass());
> props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 3000);
> props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);
> props.setProperty("security.protocol", "SASL_PLAINTEXT");
> props.setProperty("sasl.mechanism", "PLAIN");
> props.setProperty("sasl.kerberos.service.name", "kafka");
> System.setProperty("java.security.auth.login.config",
> "./conf/kafka_client_jaas.conf");
> stream.foreach(new ForeachAction() {
> @Override
> public void apply(String key, String value)
> { System.out.println(""); System.out.println(key); }
> });
> Topology topo = builder.build();
> KafkaStreams streams = new KafkaStreams(topo, props);
> streams.start();
> }
> public static void main(String[] args)
> { kafkaStreamStart(); }
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-8497) kafka streams application takes up a lot of memory
[
https://issues.apache.org/jira/browse/KAFKA-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna resolved KAFKA-8497.
--
Resolution: Invalid
> kafka streams application takes up a lot of memory
> --
>
> Key: KAFKA-8497
> URL: https://issues.apache.org/jira/browse/KAFKA-8497
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
>Reporter: sunqing
>Priority: Major
>
>
> A simple kafka streams application, use KStream to consume data, as below。
> Memory usage is very high when there is a large amount of data under the
> consuming topic.
> Sometimes it goes up to 20G.
> This is very strange. The program doesn't do anything. It just reads the data
> and prints it to the screen. Why is the memory usage so high when there is a
> lot of data in the topic?
>
>
>
> The program code：
>
> public class TestMain {
> public static StreamsBuilder builder = new StreamsBuilder();
> public static void kafkaStreamStart() {
> KStream stream =
> builder.stream(Arrays.asList("wk_wangxin_po"));
> Properties props = new Properties();
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, "testwang_xin");
> props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
>
> "zktj-kafka-broker-out-1:29092,zktj-kafka-broker-out-2:29092,zktj-kafka-broker-out-3:29092");
> props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
> Serdes.String().getClass());
> props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
> Serdes.String().getClass());
> props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 3000);
> props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);
> props.setProperty("security.protocol", "SASL_PLAINTEXT");
> props.setProperty("sasl.mechanism", "PLAIN");
> props.setProperty("sasl.kerberos.service.name", "kafka");
> System.setProperty("java.security.auth.login.config",
> "./conf/kafka_client_jaas.conf");
> stream.foreach(new ForeachAction() {
> @Override
> public void apply(String key, String value)
> { System.out.println(""); System.out.println(key); }
> });
> Topology topo = builder.build();
> KafkaStreams streams = new KafkaStreams(topo, props);
> streams.start();
> }
> public static void main(String[] args)
> { kafkaStreamStart(); }
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8263) Flaky Test MetricsIntegrationTest#testStreamMetricOfWindowStore
[
https://issues.apache.org/jira/browse/KAFKA-8263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16862037#comment-16862037
]
Bruno Cadonna commented on KAFKA-8263:
--
Pull request: [https://github.com/apache/kafka/pull/6922]
> Flaky Test MetricsIntegrationTest#testStreamMetricOfWindowStore
> ---
>
> Key: KAFKA-8263
> URL: https://issues.apache.org/jira/browse/KAFKA-8263
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Bruno Cadonna
>Priority: Major
> Labels: flaky-test
> Fix For: 2.4.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/3900/testReport/junit/org.apache.kafka.streams.integration/MetricsIntegrationTest/testStreamMetricOfWindowStore/]
> {quote}java.lang.AssertionError: Condition not met within timeout 1.
> testStoreMetricWindow -> Size of metrics of type:'put-latency-avg' must be
> equal to:2 but it's equal to 0 expected:<2> but was:<0> at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:361){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-8578) Add Sensors and Metrics to Expose RocksDB Metrics
Bruno Cadonna created KAFKA-8578: Summary: Add Sensors and Metrics to Expose RocksDB Metrics Key: KAFKA-8578 URL: https://issues.apache.org/jira/browse/KAFKA-8578 Project: Kafka Issue Type: Sub-task Reporter: Bruno Cadonna To expose RocksDB metrics as specified in KIP-471, a bunch of sensors and metrics in the Kafka metrics registry are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8579) Expose RocksDB Metrics to JMX
Bruno Cadonna created KAFKA-8579: Summary: Expose RocksDB Metrics to JMX Key: KAFKA-8579 URL: https://issues.apache.org/jira/browse/KAFKA-8579 Project: Kafka Issue Type: Sub-task Components: streams Reporter: Bruno Cadonna Sensors and the corresponding metrics specified to record RocksDB metrics need to be created in the RocksDB state stores. Once the metrics are created and registered in the Kafka metrics registry, they are also exposed in JMX. This ticker does not include the computation of the RocksDB metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8580) Compute RocksDB Metrics
Bruno Cadonna created KAFKA-8580: Summary: Compute RocksDB Metrics Key: KAFKA-8580 URL: https://issues.apache.org/jira/browse/KAFKA-8580 Project: Kafka Issue Type: Sub-task Components: streams Reporter: Bruno Cadonna Once the metrics for RocksDB are exposed, the metrics need to be recorded in the RocksDB state stores. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8578) Add Sensors and Metrics to Expose RocksDB Metrics
[ https://issues.apache.org/jira/browse/KAFKA-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8578: - Fix Version/s: 2.4.0 > Add Sensors and Metrics to Expose RocksDB Metrics > - > > Key: KAFKA-8578 > URL: https://issues.apache.org/jira/browse/KAFKA-8578 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Bruno Cadonna >Priority: Major > Fix For: 2.4.0 > > > To expose RocksDB metrics as specified in KIP-471, a bunch of sensors and > metrics in the Kafka metrics registry are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8578) Add Sensors and Metrics to Expose RocksDB Metrics
[ https://issues.apache.org/jira/browse/KAFKA-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8578: - Component/s: streams > Add Sensors and Metrics to Expose RocksDB Metrics > - > > Key: KAFKA-8578 > URL: https://issues.apache.org/jira/browse/KAFKA-8578 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Bruno Cadonna >Priority: Major > > To expose RocksDB metrics as specified in KIP-471, a bunch of sensors and > metrics in the Kafka metrics registry are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8579) Expose RocksDB Metrics to JMX
[ https://issues.apache.org/jira/browse/KAFKA-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8579: - Fix Version/s: 2.4.0 > Expose RocksDB Metrics to JMX > - > > Key: KAFKA-8579 > URL: https://issues.apache.org/jira/browse/KAFKA-8579 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Bruno Cadonna >Priority: Major > Fix For: 2.4.0 > > > Sensors and the corresponding metrics specified to record RocksDB metrics > need to be created in the RocksDB state stores. Once the metrics are created > and registered in the Kafka metrics registry, they are also exposed in JMX. > This ticker does not include the computation of the RocksDB metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8579) Expose RocksDB Metrics to JMX
[ https://issues.apache.org/jira/browse/KAFKA-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8579: - Description: Sensors and the corresponding metrics specified to record RocksDB metrics need to be created in the RocksDB state stores. Once the metrics are created and registered in the Kafka metrics registry, they are also exposed in JMX. This ticket does not include the computation of the RocksDB metrics. (was: Sensors and the corresponding metrics specified to record RocksDB metrics need to be created in the RocksDB state stores. Once the metrics are created and registered in the Kafka metrics registry, they are also exposed in JMX. This ticker does not include the computation of the RocksDB metrics. ) > Expose RocksDB Metrics to JMX > - > > Key: KAFKA-8579 > URL: https://issues.apache.org/jira/browse/KAFKA-8579 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Bruno Cadonna >Priority: Major > Fix For: 2.4.0 > > > Sensors and the corresponding metrics specified to record RocksDB metrics > need to be created in the RocksDB state stores. Once the metrics are created > and registered in the Kafka metrics registry, they are also exposed in JMX. > This ticket does not include the computation of the RocksDB metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-8578) Add Sensors and Metrics to Expose RocksDB Metrics
[ https://issues.apache.org/jira/browse/KAFKA-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna reassigned KAFKA-8578: Assignee: Bruno Cadonna > Add Sensors and Metrics to Expose RocksDB Metrics > - > > Key: KAFKA-8578 > URL: https://issues.apache.org/jira/browse/KAFKA-8578 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > Fix For: 2.4.0 > > > To expose RocksDB metrics as specified in KIP-471, a bunch of sensors and > metrics in the Kafka metrics registry are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-8579) Expose RocksDB Metrics to JMX
[ https://issues.apache.org/jira/browse/KAFKA-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna reassigned KAFKA-8579: Assignee: Bruno Cadonna > Expose RocksDB Metrics to JMX > - > > Key: KAFKA-8579 > URL: https://issues.apache.org/jira/browse/KAFKA-8579 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > Fix For: 2.4.0 > > > Sensors and the corresponding metrics specified to record RocksDB metrics > need to be created in the RocksDB state stores. Once the metrics are created > and registered in the Kafka metrics registry, they are also exposed in JMX. > This ticket does not include the computation of the RocksDB metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-8580) Compute RocksDB Metrics
[ https://issues.apache.org/jira/browse/KAFKA-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna reassigned KAFKA-8580: Assignee: Bruno Cadonna > Compute RocksDB Metrics > --- > > Key: KAFKA-8580 > URL: https://issues.apache.org/jira/browse/KAFKA-8580 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > > Once the metrics for RocksDB are exposed, the metrics need to be recorded in > the RocksDB state stores. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8578) Add Sensors and Metrics to Expose RocksDB Metrics
[ https://issues.apache.org/jira/browse/KAFKA-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8578: - Description: To expose RocksDB metrics as specified in KIP-471, functionality to create and record metrics in the Kafka metrics registry is required. (was: To expose RocksDB metrics as specified in KIP-471, a bunch of sensors and metrics in the Kafka metrics registry are required.) > Add Sensors and Metrics to Expose RocksDB Metrics > - > > Key: KAFKA-8578 > URL: https://issues.apache.org/jira/browse/KAFKA-8578 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > Fix For: 2.4.0 > > > To expose RocksDB metrics as specified in KIP-471, functionality to create > and record metrics in the Kafka metrics registry is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8578) Add Functionality to Expose RocksDB Metrics
[ https://issues.apache.org/jira/browse/KAFKA-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8578: - Summary: Add Functionality to Expose RocksDB Metrics (was: Add Sensors and Metrics to Expose RocksDB Metrics) > Add Functionality to Expose RocksDB Metrics > --- > > Key: KAFKA-8578 > URL: https://issues.apache.org/jira/browse/KAFKA-8578 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > Fix For: 2.4.0 > > > To expose RocksDB metrics as specified in KIP-471, functionality to create > and record metrics in the Kafka metrics registry is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8602) StreamThread Dies Because Restore Consumer is not Subscribed to Any Topic
Bruno Cadonna created KAFKA-8602:
Summary: StreamThread Dies Because Restore Consumer is not
Subscribed to Any Topic
Key: KAFKA-8602
URL: https://issues.apache.org/jira/browse/KAFKA-8602
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.1.1
Reporter: Bruno Cadonna
StreamThread dies with the following exception:
{code:java}
java.lang.IllegalStateException: Consumer is not subscribed to any topics or
assigned any partitions
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1199)
at
org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1126)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:923)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)
{code}
The reason is that the restore consumer is not subscribed to any topic. This
happens when a StreamThread gets assigned standby tasks for sub-topologies with
just state stores with disabled logging.
To reproduce the bug start two applications with one StreamThread and one
standby replica each and the following topology. The input topic should have
two partitions:
{code:java}
final StreamsBuilder builder = new StreamsBuilder();
final String stateStoreName = "myTransformState";
final StoreBuilder> keyValueStoreBuilder =
Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore(stateStoreName),
Serdes.Integer(),
Serdes.Integer())
.withLoggingDisabled();
builder.addStateStore(keyValueStoreBuilder);
builder.stream(INPUT_TOPIC, Consumed.with(Serdes.Integer(), Serdes.Integer()))
.transform(() -> new Transformer>() {
private KeyValueStore state;
@SuppressWarnings("unchecked")
@Override
public void init(final ProcessorContext context) {
state = (KeyValueStore)
context.getStateStore(stateStoreName);
}
@Override
public KeyValue transform(final Integer key,
final Integer value) {
final KeyValue result = new KeyValue<>(key,
value);
return result;
}
@Override
public void close() {}
}, stateStoreName)
.to(OUTPUT_TOPIC);
{code}
Both StreamThreads should die with the above exception.
The root cause is that standby tasks are created although all state stores of
the sub-topology have a logging disabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-8602) StreamThread Dies Because Restore Consumer is not Subscribed to Any Topic
[
https://issues.apache.org/jira/browse/KAFKA-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna reassigned KAFKA-8602:
Assignee: Bruno Cadonna
> StreamThread Dies Because Restore Consumer is not Subscribed to Any Topic
> -
>
> Key: KAFKA-8602
> URL: https://issues.apache.org/jira/browse/KAFKA-8602
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.1
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Critical
>
> StreamThread dies with the following exception:
> {code:java}
> java.lang.IllegalStateException: Consumer is not subscribed to any topics or
> assigned any partitions
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1199)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1126)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:923)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)
> {code}
> The reason is that the restore consumer is not subscribed to any topic. This
> happens when a StreamThread gets assigned standby tasks for sub-topologies
> with just state stores with disabled logging.
> To reproduce the bug start two applications with one StreamThread and one
> standby replica each and the following topology. The input topic should have
> two partitions:
> {code:java}
> final StreamsBuilder builder = new StreamsBuilder();
> final String stateStoreName = "myTransformState";
> final StoreBuilder> keyValueStoreBuilder =
>
> Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore(stateStoreName),
> Serdes.Integer(),
> Serdes.Integer())
> .withLoggingDisabled();
> builder.addStateStore(keyValueStoreBuilder);
> builder.stream(INPUT_TOPIC, Consumed.with(Serdes.Integer(), Serdes.Integer()))
> .transform(() -> new Transformer Integer>>() {
> private KeyValueStore state;
> @SuppressWarnings("unchecked")
> @Override
> public void init(final ProcessorContext context) {
> state = (KeyValueStore)
> context.getStateStore(stateStoreName);
> }
> @Override
> public KeyValue transform(final Integer key,
> final Integer value) {
> final KeyValue result = new KeyValue<>(key,
> value);
> return result;
> }
> @Override
> public void close() {}
> }, stateStoreName)
> .to(OUTPUT_TOPIC);
> {code}
> Both StreamThreads should die with the above exception.
> The root cause is that standby tasks are created although all state stores of
> the sub-topology have a logging disabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8602) StreamThread Dies Because Restore Consumer is not Subscribed to Any Topic
[
https://issues.apache.org/jira/browse/KAFKA-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-8602:
-
Description:
StreamThread dies with the following exception:
{code:java}
java.lang.IllegalStateException: Consumer is not subscribed to any topics or
assigned any partitions
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1199)
at
org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1126)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:923)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)
{code}
The reason is that the restore consumer is not subscribed to any topic. This
happens when a StreamThread gets assigned standby tasks for sub-topologies with
just state stores with disabled logging.
To reproduce the bug start two applications with one StreamThread and one
standby replica each and the following topology. The input topic should have
two partitions:
{code:java}
final StreamsBuilder builder = new StreamsBuilder();
final String stateStoreName = "myTransformState";
final StoreBuilder> keyValueStoreBuilder =
Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore(stateStoreName),
Serdes.Integer(),
Serdes.Integer())
.withLoggingDisabled();
builder.addStateStore(keyValueStoreBuilder);
builder.stream(INPUT_TOPIC, Consumed.with(Serdes.Integer(), Serdes.Integer()))
.transform(() -> new Transformer>() {
private KeyValueStore state;
@SuppressWarnings("unchecked")
@Override
public void init(final ProcessorContext context) {
state = (KeyValueStore)
context.getStateStore(stateStoreName);
}
@Override
public KeyValue transform(final Integer key,
final Integer value) {
final KeyValue result = new KeyValue<>(key,
value);
return result;
}
@Override
public void close() {}
}, stateStoreName)
.to(OUTPUT_TOPIC);
{code}
Both StreamThreads should die with the above exception.
The root cause is that standby tasks are created although all state stores of
the sub-topology have logging disabled.
was:
StreamThread dies with the following exception:
{code:java}
java.lang.IllegalStateException: Consumer is not subscribed to any topics or
assigned any partitions
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1199)
at
org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1126)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:923)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)
{code}
The reason is that the restore consumer is not subscribed to any topic. This
happens when a StreamThread gets assigned standby tasks for sub-topologies with
just state stores with disabled logging.
To reproduce the bug start two applications with one StreamThread and one
standby replica each and the following topology. The input topic should have
two partitions:
{code:java}
final StreamsBuilder builder = new StreamsBuilder();
final String stateStoreName = "myTransformState";
final StoreBuilder> keyValueStoreBuilder =
Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore(stateStoreName),
Serdes.Integer(),
Serdes.Integer())
.withLoggingDisabled();
builder.addStateStore(keyValueStoreBuilder);
builder.stream(INPUT_TOPIC, Consumed.with(Serdes.Integer(), Serdes.Integer()))
.transform(() -> new Transformer>() {
private KeyValueStore state;
@SuppressWarnings("unchecked")
@Override
public void init(final ProcessorContext context) {
state = (KeyValueStore)
context.getStateStore(stateStoreName);
}
@Override
public KeyValue transform(final Integer key,
final Integer value) {
final KeyValue result = new KeyValue<>(key,
value);
return result;
}
@Override
public void close() {}
}, stateStoreName)
.to(OUTPUT_TOPIC);
{code}
Both StreamThreads should die with the above exception.
The root cause is that standby tasks are created although all state stores of
the sub-topology hav
[jira] [Created] (KAFKA-8613) Set default grace period to 0
Bruno Cadonna created KAFKA-8613:
Summary: Set default grace period to 0
Key: KAFKA-8613
URL: https://issues.apache.org/jira/browse/KAFKA-8613
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 3.0.0
Reporter: Bruno Cadonna
Currently, the grace period is set to retention time if the grace period is not
specified explicitly. The reason for setting the default grace period to
retention time was backward compatibility. Topologies that were implemented
before the introduction of the grace period, added late arriving records to a
window as long as the window existed, i.e., as long as its retention time was
not elapsed.
This unintuitive default grace period has already caused confusion among users.
For the next major release, we should set the default grace period to
{{Duration.ZERO}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8617) Replace EndTxn request/response with automated protocol
[
https://issues.apache.org/jira/browse/KAFKA-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875999#comment-16875999
]
Bruno Cadonna commented on KAFKA-8617:
--
Hi [~manish.c.ghildi...@gmail.com],
Thank you for your interest. Newbie tickets are tagged with label {{newbie}}.
You can search for them by yourself in JIRA or you can click on the following
pre-defined search link that you can also find on the ["How to
contribute"-page|https://kafka.apache.org/contributing].
https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20labels%20%3D%20newbie%20AND%20status%20%3D%20Open
> Replace EndTxn request/response with automated protocol
> ---
>
> Key: KAFKA-8617
> URL: https://issues.apache.org/jira/browse/KAFKA-8617
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8555: - Attachment: log-job23145.txt > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job23145.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[
https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16876795#comment-16876795
]
Bruno Cadonna commented on KAFKA-8555:
--
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/23145/
{code:java}
org.apache.kafka.connect.errors.DataException: Insufficient records committed
by connector simple-conn in 15000 millis. Records expected=2000, actual=1517
at
org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)
at
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
...
{code}
log attached: log-job23145.txt
> Flaky test ExampleConnectIntegrationTest#testSourceConnector
>
>
> Key: KAFKA-8555
> URL: https://issues.apache.org/jira/browse/KAFKA-8555
> Project: Kafka
> Issue Type: Bug
>Reporter: Boyang Chen
>Priority: Major
> Attachments: log-job23145.txt
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console]
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21*
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest >
> testSourceConnector FAILED*02:03:21*
> org.apache.kafka.connect.errors.DataException: Insufficient records committed
> by connector simple-conn in 15000 millis. Records expected=2000,
> actual=1013*02:03:21* at
> org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21*
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[
https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878601#comment-16878601
]
Bruno Cadonna commented on KAFKA-8555:
--
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/23215
{code:java}
org.apache.kafka.connect.errors.DataException: Insufficient records committed
by connector simple-conn in 15000 millis. Records expected=2000, actual=1185
at
org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)
at
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
{code}
log attached: log-job23215.txt
> Flaky test ExampleConnectIntegrationTest#testSourceConnector
>
>
> Key: KAFKA-8555
> URL: https://issues.apache.org/jira/browse/KAFKA-8555
> Project: Kafka
> Issue Type: Bug
>Reporter: Boyang Chen
>Priority: Major
> Attachments: log-job23145.txt
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console]
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21*
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest >
> testSourceConnector FAILED*02:03:21*
> org.apache.kafka.connect.errors.DataException: Insufficient records committed
> by connector simple-conn in 15000 millis. Records expected=2000,
> actual=1013*02:03:21* at
> org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21*
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8555: - Attachment: log-job23215.txt > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job23145.txt, log-job23215.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8555: - Attachment: log-job6046.txt > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job23145.txt, log-job23215.txt, log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[
https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879196#comment-16879196
]
Bruno Cadonna commented on KAFKA-8555:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6046
{code:java}
org.apache.kafka.connect.errors.DataException: Insufficient records committed
by connector simple-conn in 15000 millis. Records expected=2000, actual=1898
at
org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)
at
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
{code}
log attached: log-job6046.txt
> Flaky test ExampleConnectIntegrationTest#testSourceConnector
>
>
> Key: KAFKA-8555
> URL: https://issues.apache.org/jira/browse/KAFKA-8555
> Project: Kafka
> Issue Type: Bug
>Reporter: Boyang Chen
>Priority: Major
> Attachments: log-job23145.txt, log-job23215.txt, log-job6046.txt
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console]
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21*
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest >
> testSourceConnector FAILED*02:03:21*
> org.apache.kafka.connect.errors.DataException: Insufficient records committed
> by connector simple-conn in 15000 millis. Records expected=2000,
> actual=1013*02:03:21* at
> org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21*
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8502) Flakey test AdminClientIntegrationTest#testElectUncleanLeadersForAllPartitions
[ https://issues.apache.org/jira/browse/KAFKA-8502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8502: - Attachment: log-job6046.txt > Flakey test AdminClientIntegrationTest#testElectUncleanLeadersForAllPartitions > -- > > Key: KAFKA-8502 > URL: https://issues.apache.org/jira/browse/KAFKA-8502 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/5355/consoleFull] > > *18:06:01* *18:06:01* kafka.api.AdminClientIntegrationTest > > testElectUncleanLeadersForAllPartitions FAILED*18:06:01* > java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.TimeoutException: Aborted due to > timeout.*18:06:01* at > org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)*18:06:01* > at > org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)*18:06:01* > at > org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)*18:06:01* > at > org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)*18:06:01* > at > kafka.api.AdminClientIntegrationTest.testElectUncleanLeadersForAllPartitions(AdminClientIntegrationTest.scala:1496)*18:06:01* > *18:06:01* Caused by:*18:06:01* > org.apache.kafka.common.errors.TimeoutException: Aborted due to timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8502) Flakey test AdminClientIntegrationTest#testElectUncleanLeadersForAllPartitions
[
https://issues.apache.org/jira/browse/KAFKA-8502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879202#comment-16879202
]
Bruno Cadonna commented on KAFKA-8502:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6046
{code:java}
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.TimeoutException: Aborted due to timeout.
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at
kafka.api.AdminClientIntegrationTest.testElectUncleanLeadersForAllPartitions(AdminClientIntegrationTest.scala:1496)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.kafka.common.errors.TimeoutException: Aborted due to
timeout.
{code}
attached logs: log-job6046.txt
> Flakey test AdminClientIntegrationTest#testElectUncleanLeadersForAllPartitions
> --
>
> Key: KAFKA-8502
> URL: https://issues.apache.org/jira/browse/KAFKA-8502
> Project: Kafka
> Issue Type: Bug
>Reporter: Boyang Chen
>Priority: Major
> Attachments: log-job6046.txt
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/5355/consoleFull]
>
> *18:06:01* *18:06:01* kafka.api.AdminClientIntegrationTest >
> testElectUncleanLeadersForAllPartitions FAILED*18:06:01*
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TimeoutException: Aborted due to
> timeout.*18:06:01* at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)*18:06:01*
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)*18:06:01*
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)*18:06:01*
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)*18:06:01*
> at
> kafka.api.AdminClientIntegrationTest.testElectUncleanLeadersForAllPartitions(AdminClientIntegrationTest.scala:1496)*18:06:01*
> *18:06:01* Caused by:*18:06:01*
> org.apache.kafka.common.errors.TimeoutException: Aborted due to timeout.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8555: - Attachment: log-job139.txt > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job139.txt, log-job23145.txt, log-job23215.txt, > log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[
https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881907#comment-16881907
]
Bruno Cadonna commented on KAFKA-8555:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/139
{code:java}
org.apache.kafka.connect.errors.DataException: Insufficient records committed
by connector simple-conn in 15000 millis. Records expected=2000, actual=1146
at
org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)
at
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
{code}
log attached: log-job139.txt
> Flaky test ExampleConnectIntegrationTest#testSourceConnector
>
>
> Key: KAFKA-8555
> URL: https://issues.apache.org/jira/browse/KAFKA-8555
> Project: Kafka
> Issue Type: Bug
>Reporter: Boyang Chen
>Priority: Major
> Attachments: log-job139.txt, log-job23145.txt, log-job23215.txt,
> log-job6046.txt
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console]
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21*
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest >
> testSourceConnector FAILED*02:03:21*
> org.apache.kafka.connect.errors.DataException: Insufficient records committed
> by connector simple-conn in 15000 millis. Records expected=2000,
> actual=1013*02:03:21* at
> org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21*
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7937) Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
[
https://issues.apache.org/jira/browse/KAFKA-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-7937:
-
Attachment: log-job6122.txt
> Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
>
>
> Key: KAFKA-7937
> URL: https://issues.apache.org/jira/browse/KAFKA-7937
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0, 2.1.1, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Gwen Shapira
>Priority: Critical
> Fix For: 2.4.0
>
> Attachments: log-job6122.txt
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/19/pipeline
> {quote}kafka.admin.ResetConsumerGroupOffsetTest >
> testResetOffsetsNotExistingGroup FAILED
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
> coordinator is not available. at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:306)
> at
> kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:89)
> Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException:
> The coordinator is not available.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7937) Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
[
https://issues.apache.org/jira/browse/KAFKA-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881924#comment-16881924
]
Bruno Cadonna commented on KAFKA-7937:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6122
{code:java}
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
coordinator is not available.
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$describeConsumerGroups$1(ConsumerGroupCommand.scala:402)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:237)
at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeConsumerGroups(ConsumerGroupCommand.scala:401)
at
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.collectGroupsState(ConsumerGroupCommand.scala:478)
at
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.collectGroupState(ConsumerGroupCommand.scala:474)
at
kafka.admin.ResetConsumerGroupOffsetTest.$anonfun$testResetOffsetsNotExistingGroup$1(ResetConsumerGroupOffsetTest.scala:113)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:817)
at
kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:112)
{code}
logs attached: log-job6122.txt
> Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
>
>
> Key: KAFKA-7937
> URL: https://issues.apache.org/jira/browse/KAFKA-7937
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0, 2.1.1, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Gwen Shapira
>Priority: Critical
> Fix For: 2.4.0
>
> Attachments: log-job6122.txt
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/19/pipeline
> {quote}kafka.admin.ResetConsumerGroupOffsetTest >
> testResetOffsetsNotExistingGroup FAILED
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
> coordinator is not available. at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:306)
> at
> kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:89)
> Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException:
> The coordinator is not available.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[
https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881926#comment-16881926
]
Bruno Cadonna commented on KAFKA-8555:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/141
{code:java}
org.apache.kafka.connect.errors.DataException: Insufficient records committed
by connector simple-conn in 15000 millis. Records expected=2000, actual=924
at
org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)
at
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
{code}
log attached: log-job141.txt
> Flaky test ExampleConnectIntegrationTest#testSourceConnector
>
>
> Key: KAFKA-8555
> URL: https://issues.apache.org/jira/browse/KAFKA-8555
> Project: Kafka
> Issue Type: Bug
>Reporter: Boyang Chen
>Priority: Major
> Attachments: log-job139.txt, log-job141.txt, log-job23145.txt,
> log-job23215.txt, log-job6046.txt
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console]
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21*
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest >
> testSourceConnector FAILED*02:03:21*
> org.apache.kafka.connect.errors.DataException: Insufficient records committed
> by connector simple-conn in 15000 millis. Records expected=2000,
> actual=1013*02:03:21* at
> org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21*
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8555: - Attachment: log-job141.txt > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job139.txt, log-job141.txt, log-job23145.txt, > log-job23215.txt, log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881972#comment-16881972 ] Bruno Cadonna commented on KAFKA-8555: -- Related pull requests: * https://github.com/apache/kafka/pull/7061 * https://github.com/apache/kafka/pull/7019 > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job139.txt, log-job141.txt, log-job23145.txt, > log-job23215.txt, log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-9088) Consolidate InternalMockProcessorContext and MockInternalProcessorContext
[
https://issues.apache.org/jira/browse/KAFKA-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987881#comment-16987881
]
Bruno Cadonna commented on KAFKA-9088:
--
[~vvcephei] I have the following justifications to consolidate the two classes.
1. {{InternalMockProcessorContext}} uses production code. Such a partial mock
leaks code from a unit that is NOT under test into a test. We should avoid such
leakage because it makes the maintenance of unit tests harder.
2. Both mocks grew over time and together they have a myriad of different
constructors. That should be cleaned up (probably by using a builder pattern).
3. During the Streams metrics refactoring it wasted quite some of my time in
adapting two different mocks that basically do the same. That should be avoided
in future.
IMO, we need to clean up those mocks. Since, we have to touch that code we
should think beyond and try to simplify the use and the maintenance of the mock
also by trying out new ways. For that, I thought it would be worthwhile to see
what EasyMock can do for us. My expectation is that it gives us a better
isolation of unit tests from production code that is not under test and less
test code to maintain.
My approach would be to implement a Builder that builds an EasyMock that
already returns default values or the values set during the build. If then some
additional expectations needs to be set, they can be set in the way you would
set them when using EasyMock. But most of the setup should be hidden in the
build process.
This experiment should also kind of assess whether we should use EasyMock in
such cases in future. If we see that EasyMock does not fulfil the expectations
described above, let's do it differently. But let's at least try.
Finally, I agree that the consolidation should result in less code lines than
before.
> Consolidate InternalMockProcessorContext and MockInternalProcessorContext
> -
>
> Key: KAFKA-9088
> URL: https://issues.apache.org/jira/browse/KAFKA-9088
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Bruno Cadonna
>Assignee: Pierangelo Di Pilato
>Priority: Minor
> Labels: newbie
>
> Currently, we have two mocks for the {{InternalProcessorContext}}. The goal
> of this ticket is to merge both into one mock or replace it with an
> {{EasyMock}} mock.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9195) Use Randomized State Directory Names in Streams System Tests
[
https://issues.apache.org/jira/browse/KAFKA-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987922#comment-16987922
]
Bruno Cadonna commented on KAFKA-9195:
--
I guess your proposal would also work. However, I guess your proposal would be
harder to maintain, because using a randomized path is a environment setting
that can be done once for all (or at least a bunch of) system tests whereas
{{KafkaStreams#cleanup()}} needs to be called in each Streams application.
Every person that adds a test need to know/remember to add
{{KafkaStreams#cleanup()}}.
Maybe not a big deal because the same Streams application code is shared by
multiple tests. Just wanted to point out the difference.
> Use Randomized State Directory Names in Streams System Tests
> -
>
> Key: KAFKA-9195
> URL: https://issues.apache.org/jira/browse/KAFKA-9195
> Project: Kafka
> Issue Type: Improvement
> Components: streams, system tests
>Reporter: Bruno Cadonna
>Priority: Major
>
> Currently, the state directory property in streams' system tests is set to
> the {{PERSISTENT_ROOT}} variable. Since Streams applications in different
> tests have the same application ID and the state directory path consists of
> state directory property + application ID + task ID, it might happen that a
> dirty state directory of one test is re-used by another test if the state
> directory is not properly cleaned up. This may lead to unexpected results and
> false positive and/or flaky failures.
> The state directory property shall be set to a randomized path inside
> {{PERSISTENT_ROOT}} to ensure that tests may not interfere with each other in
> the case of missing state clean-ups.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9088) Consolidate InternalMockProcessorContext and MockInternalProcessorContext
[
https://issues.apache.org/jira/browse/KAFKA-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988579#comment-16988579
]
Bruno Cadonna commented on KAFKA-9088:
--
I understand all your concerns and sometimes I also had hard times with
EasyMock. However, I think the issues are only partly caused by EasyMock. The
design of the production code may leak internals, the unit test may have issues
because it tests too much or the wrong aspects, and EasyMock may be used
wrongly. For instance, sometimes it does not make sense to specify how often a
method on the mock is called or if it is called at all. That would be the case
for the {{InternalProcessorContext}} mock and EasyMock offers functionality
that allows to specify that. I would like to explore this part of EasyMock in
this ticket.
> Consolidate InternalMockProcessorContext and MockInternalProcessorContext
> -
>
> Key: KAFKA-9088
> URL: https://issues.apache.org/jira/browse/KAFKA-9088
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Bruno Cadonna
>Assignee: Pierangelo Di Pilato
>Priority: Minor
> Labels: newbie
>
> Currently, we have two mocks for the {{InternalProcessorContext}}. The goal
> of this ticket is to merge both into one mock or replace it with an
> {{EasyMock}} mock.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9355) RocksDB statistics are removed from JMX when EOS enabled and empty local state dir
[
https://issues.apache.org/jira/browse/KAFKA-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009469#comment-17009469
]
Bruno Cadonna commented on KAFKA-9355:
--
[~savulchik] Thank you for reporting this. I will take a look.
> RocksDB statistics are removed from JMX when EOS enabled and empty local
> state dir
> --
>
> Key: KAFKA-9355
> URL: https://issues.apache.org/jira/browse/KAFKA-9355
> Project: Kafka
> Issue Type: Bug
> Components: metrics, streams
>Affects Versions: 2.4.0
>Reporter: Stanislav Savulchik
>Priority: Major
> Attachments: metric-removal.log
>
>
> *Steps to Reproduce*
> Set processing.guarantee = exactly_once and remove local state dir in order
> to force state restoration from changelog topics that have to be non empty.
> *Expected Behavior*
> There are registered MBeans like
> kafka.streams:type=stream-state-metrics,client-id=-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,task-id=0_0,rocksdb-state-id=
> for persistent RocksDB KeyValueStore-s after streams task state restoration.
> *Actual Behavior*
> There are no registered MBeans like above after streams task state
> restoration.
> *Details*
> I managed to inject custom MetricsReporter in order to log metricChange and
> metricRemoval calls. According to the logs at some point the missing metrics
> are removed and never restored later. Here is an excerpt for
> number-open-files metric:
> {noformat}
> 16:33:40.403 DEBUG c.m.r.LoggingMetricsReporter - metricChange MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> 16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
> recording trigger
> 16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding statistics for store buffered-event of
> task 0_0
> 16:33:40.610 INFO o.a.k.s.p.i.StoreChangelogReader - stream-thread
> [morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1] No
> checkpoint found for task 0_0 state store buffered-event changelog
> morpheus.conversion-buffered-event-changelog-0 with EOS turned on.
> Reinitializing the task and restore its state from the beginning.
> 16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Removing statistics for store buffered-event of
> task 0_0
> 16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Removing metrics recorder for store
> buffered-event of task 0_0 from metrics recording trigger
> 16:33
> 16:33:40.611 DEBUG c.m.r.LoggingMetricsReporter - metricRemoval MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> 16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
> recording trigger
> 16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding statistics for store buffered-event of
> task 0_0
> ...
> (no more calls to metricChange for the removed number-open-files
> metric){noformat}
> Also a complete log is attached [^metric-removal.log]
> Metric removal happens along this call stack:
> {noformat}
> 19:27:35.509 DEBUG c.m.r.LoggingMetricsReporter - metricRemoval MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-9b76f302-7149-47de-b17b-362d642e05d5-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> java.lang.Exception: null
>at
> casino.morpheus.reporter.LoggingMetricsReporter.metricRemoval(LoggingMetricsReporter.scala:24)
>at org.apache.kafka.common.metrics.Metrics.removeMetric(Metrics.java:534)
>at org.apache.kafka.common.metrics.Metrics.removeSensor(Metrics.java:448)
>at
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.removeAllStoreLevelSensors(StreamsMetricsImpl.java:440)
>at
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.close(MeteredKeyValueStore.java:345)
>at
> org.apache.kafka.streams.processor.internals.StateManagerUtil.reinitializeStateStoresForPartitions(StateManagerUtil.java:93)
>at
> org.apache.kafka.streams.pro
[jira] [Assigned] (KAFKA-9355) RocksDB statistics are removed from JMX when EOS enabled and empty local state dir
[
https://issues.apache.org/jira/browse/KAFKA-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna reassigned KAFKA-9355:
Assignee: Bruno Cadonna
> RocksDB statistics are removed from JMX when EOS enabled and empty local
> state dir
> --
>
> Key: KAFKA-9355
> URL: https://issues.apache.org/jira/browse/KAFKA-9355
> Project: Kafka
> Issue Type: Bug
> Components: metrics, streams
>Affects Versions: 2.4.0
>Reporter: Stanislav Savulchik
>Assignee: Bruno Cadonna
>Priority: Major
> Attachments: metric-removal.log
>
>
> *Steps to Reproduce*
> Set processing.guarantee = exactly_once and remove local state dir in order
> to force state restoration from changelog topics that have to be non empty.
> *Expected Behavior*
> There are registered MBeans like
> kafka.streams:type=stream-state-metrics,client-id=-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,task-id=0_0,rocksdb-state-id=
> for persistent RocksDB KeyValueStore-s after streams task state restoration.
> *Actual Behavior*
> There are no registered MBeans like above after streams task state
> restoration.
> *Details*
> I managed to inject custom MetricsReporter in order to log metricChange and
> metricRemoval calls. According to the logs at some point the missing metrics
> are removed and never restored later. Here is an excerpt for
> number-open-files metric:
> {noformat}
> 16:33:40.403 DEBUG c.m.r.LoggingMetricsReporter - metricChange MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> 16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
> recording trigger
> 16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding statistics for store buffered-event of
> task 0_0
> 16:33:40.610 INFO o.a.k.s.p.i.StoreChangelogReader - stream-thread
> [morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1] No
> checkpoint found for task 0_0 state store buffered-event changelog
> morpheus.conversion-buffered-event-changelog-0 with EOS turned on.
> Reinitializing the task and restore its state from the beginning.
> 16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Removing statistics for store buffered-event of
> task 0_0
> 16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Removing metrics recorder for store
> buffered-event of task 0_0 from metrics recording trigger
> 16:33
> 16:33:40.611 DEBUG c.m.r.LoggingMetricsReporter - metricRemoval MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> 16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
> recording trigger
> 16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding statistics for store buffered-event of
> task 0_0
> ...
> (no more calls to metricChange for the removed number-open-files
> metric){noformat}
> Also a complete log is attached [^metric-removal.log]
> Metric removal happens along this call stack:
> {noformat}
> 19:27:35.509 DEBUG c.m.r.LoggingMetricsReporter - metricRemoval MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-9b76f302-7149-47de-b17b-362d642e05d5-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> java.lang.Exception: null
>at
> casino.morpheus.reporter.LoggingMetricsReporter.metricRemoval(LoggingMetricsReporter.scala:24)
>at org.apache.kafka.common.metrics.Metrics.removeMetric(Metrics.java:534)
>at org.apache.kafka.common.metrics.Metrics.removeSensor(Metrics.java:448)
>at
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.removeAllStoreLevelSensors(StreamsMetricsImpl.java:440)
>at
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.close(MeteredKeyValueStore.java:345)
>at
> org.apache.kafka.streams.processor.internals.StateManagerUtil.reinitializeStateStoresForPartitions(StateManagerUtil.java:93)
>at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.reinitiali
[jira] [Commented] (KAFKA-9355) RocksDB statistics are removed from JMX when EOS enabled and empty local state dir
[
https://issues.apache.org/jira/browse/KAFKA-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020733#comment-17020733
]
Bruno Cadonna commented on KAFKA-9355:
--
The cause of the bug is the following. In EOS when the a task is not shutdown
gracefully (i.e., no checkpoint found, which happens when the local state
directory is wiped out), the state stores are reinitialized. During
reinitialization all metrics on state store level are removed when the state
store is closed on `Metered*Store` level. Afterwards when the store is
initialised again the RocksDB metrics recorder does not know that the RocksDB
metrics were removed by `Metered*Store` and does not reinitialize them. The
above PR fixes this issue.
> RocksDB statistics are removed from JMX when EOS enabled and empty local
> state dir
> --
>
> Key: KAFKA-9355
> URL: https://issues.apache.org/jira/browse/KAFKA-9355
> Project: Kafka
> Issue Type: Bug
> Components: metrics, streams
>Affects Versions: 2.4.0
>Reporter: Stanislav Savulchik
>Assignee: Bruno Cadonna
>Priority: Major
> Attachments: metric-removal.log
>
>
> *Steps to Reproduce*
> Set processing.guarantee = exactly_once and remove local state dir in order
> to force state restoration from changelog topics that have to be non empty.
> *Expected Behavior*
> There are registered MBeans like
> kafka.streams:type=stream-state-metrics,client-id=-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,task-id=0_0,rocksdb-state-id=
> for persistent RocksDB KeyValueStore-s after streams task state restoration.
> *Actual Behavior*
> There are no registered MBeans like above after streams task state
> restoration.
> *Details*
> I managed to inject custom MetricsReporter in order to log metricChange and
> metricRemoval calls. According to the logs at some point the missing metrics
> are removed and never restored later. Here is an excerpt for
> number-open-files metric:
> {noformat}
> 16:33:40.403 DEBUG c.m.r.LoggingMetricsReporter - metricChange MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> 16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
> recording trigger
> 16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding statistics for store buffered-event of
> task 0_0
> 16:33:40.610 INFO o.a.k.s.p.i.StoreChangelogReader - stream-thread
> [morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1] No
> checkpoint found for task 0_0 state store buffered-event changelog
> morpheus.conversion-buffered-event-changelog-0 with EOS turned on.
> Reinitializing the task and restore its state from the beginning.
> 16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Removing statistics for store buffered-event of
> task 0_0
> 16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Removing metrics recorder for store
> buffered-event of task 0_0 from metrics recording trigger
> 16:33
> 16:33:40.611 DEBUG c.m.r.LoggingMetricsReporter - metricRemoval MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> 16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
> recording trigger
> 16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding statistics for store buffered-event of
> task 0_0
> ...
> (no more calls to metricChange for the removed number-open-files
> metric){noformat}
> Also a complete log is attached [^metric-removal.log]
> Metric removal happens along this call stack:
> {noformat}
> 19:27:35.509 DEBUG c.m.r.LoggingMetricsReporter - metricRemoval MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-9b76f302-7149-47de-b17b-362d642e05d5-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> java.lang.Exception: null
>at
> casino.morpheus.reporter.LoggingMetricsReporter.metricRemoval(LoggingMetricsReporter.scala:24)
>at org.apache.kafka.common.metrics.Metrics.rem
[jira] [Created] (KAFKA-9480) Value for Task-level Metric process-rate is Constant Zero
Bruno Cadonna created KAFKA-9480:
Summary: Value for Task-level Metric process-rate is Constant Zero
Key: KAFKA-9480
URL: https://issues.apache.org/jira/browse/KAFKA-9480
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.4.0
Reporter: Bruno Cadonna
The value for task-level metric process-rate is constant zero. The value should
reflect the number of calls to {{process()}} on source processors which
clearly cannot be constant zero.
This behavior applies to built-in metrics version {{latest}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-9480) Value for Task-level Metric process-rate is Constant Zero
[
https://issues.apache.org/jira/browse/KAFKA-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna reassigned KAFKA-9480:
Assignee: Bruno Cadonna
> Value for Task-level Metric process-rate is Constant Zero
> --
>
> Key: KAFKA-9480
> URL: https://issues.apache.org/jira/browse/KAFKA-9480
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
>
> The value for task-level metric process-rate is constant zero. The value
> should reflect the number of calls to {{process()}} on source processors
> which clearly cannot be constant zero.
> This behavior applies to built-in metrics version {{latest}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9230) Change User Customizable Metrics API in StreamsMetrics interface
[ https://issues.apache.org/jira/browse/KAFKA-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna resolved KAFKA-9230. -- Resolution: Fixed > Change User Customizable Metrics API in StreamsMetrics interface > > > Key: KAFKA-9230 > URL: https://issues.apache.org/jira/browse/KAFKA-9230 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > > As proposed in KIP-444, the user-customizable metrics API in the > StreamsMetrics interface shall be improved. For more details, see > https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9115) Add Roll-up Throughput Sensor for Processing on Source Nodes
[
https://issues.apache.org/jira/browse/KAFKA-9115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna resolved KAFKA-9115.
--
Resolution: Fixed
> Add Roll-up Throughput Sensor for Processing on Source Nodes
> -
>
> Key: KAFKA-9115
> URL: https://issues.apache.org/jira/browse/KAFKA-9115
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
> Fix For: 2.5.0
>
>
> A sensor should be added with metrics {{process-rate}} and {{process-total}}
> that measure the throughput and latency of calls to process on all source
> nodes of a topology. This sensor is described in KIP-444.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9533) ValueTransform forwards `null` values
[
https://issues.apache.org/jira/browse/KAFKA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034298#comment-17034298
]
Bruno Cadonna commented on KAFKA-9533:
--
[~mviamari] Thank you for opening this ticket. This seems to be a bug indeed.
> ValueTransform forwards `null` values
> -
>
> Key: KAFKA-9533
> URL: https://issues.apache.org/jira/browse/KAFKA-9533
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Michael Viamari
>Priority: Minor
>
> According to the documentation for `KStream#transformValues`, nulls returned
> from `ValueTransformer#transform` are not forwarded. (see
> [KStream#transformValues|https://kafka.apache.org/24/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-]
> However, this does not appear to be the case. In
> `KStreamTransformValuesProcessor#process` the result of the transform is
> forwarded directly.
> {code:java}
> @Override
> public void process(final K key, final V value) {
> context.forward(key, valueTransformer.transform(key, value));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9533) ValueTransform forwards `null` values
[
https://issues.apache.org/jira/browse/KAFKA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035407#comment-17035407
]
Bruno Cadonna commented on KAFKA-9533:
--
[~mviamari] Would you be interested in fixing this bug?
> ValueTransform forwards `null` values
> -
>
> Key: KAFKA-9533
> URL: https://issues.apache.org/jira/browse/KAFKA-9533
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Michael Viamari
>Priority: Minor
>
> According to the documentation for `KStream#transformValues`, nulls returned
> from `ValueTransformer#transform` are not forwarded. (see
> [KStream#transformValues|https://kafka.apache.org/24/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-]
> However, this does not appear to be the case. In
> `KStreamTransformValuesProcessor#process` the result of the transform is
> forwarded directly.
> {code:java}
> @Override
> public void process(final K key, final V value) {
> context.forward(key, valueTransformer.transform(key, value));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9533) ValueTransform forwards `null` values
[
https://issues.apache.org/jira/browse/KAFKA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036099#comment-17036099
]
Bruno Cadonna commented on KAFKA-9533:
--
Thank you for picking this up!
Ad 1) We do not support backwards compatibility of bugs. The documentation is
clear, we just failed to implement it correctly.
Ad 2) If you want to try to simplify the code with the adapter as in
{{transform()}} and {{flatTransform()}} you are very welcome to do so. For a
smoother review process, I would like to ask you to create two PRs, one for the
refactoring and one for the bug fix.
Are you able to assign the ticket to yourself?
> ValueTransform forwards `null` values
> -
>
> Key: KAFKA-9533
> URL: https://issues.apache.org/jira/browse/KAFKA-9533
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Michael Viamari
>Priority: Minor
>
> According to the documentation for `KStream#transformValues`, nulls returned
> from `ValueTransformer#transform` are not forwarded. (see
> [KStream#transformValues|https://kafka.apache.org/24/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-]
> However, this does not appear to be the case. In
> `KStreamTransformValuesProcessor#process` the result of the transform is
> forwarded directly.
> {code:java}
> @Override
> public void process(final K key, final V value) {
> context.forward(key, valueTransformer.transform(key, value));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9540) Application getting "Could not find the standby task 0_4 while closing it" error
[
https://issues.apache.org/jira/browse/KAFKA-9540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038868#comment-17038868
]
Bruno Cadonna commented on KAFKA-9540:
--
[~ableegoldman] [~guozhang] Can we close this ticket?
> Application getting "Could not find the standby task 0_4 while closing it"
> error
>
>
> Key: KAFKA-9540
> URL: https://issues.apache.org/jira/browse/KAFKA-9540
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Badai Aqrandista
>Priority: Minor
> Fix For: 2.4.1
>
>
> Because of this the following line, there is a possibility that some standby
> tasks might not be created:
> https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L436
> Then causing this line to not adding the task to standby task list:
> https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L299
> But this line assumes that all standby tasks are to be created and add it to
> the standby list:
> https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java#L168
> This results in user getting this error message on the next
> PARTITION_ASSIGNMENT state:
> {noformat}
> Could not find the standby task 0_4 while closing it
> (org.apache.kafka.streams.processor.internals.AssignedStandbyTasks:74)
> {noformat}
> But the harm caused by this issue is minimal: No standby task for some
> partitions. And it is recreated on the next rebalance anyway. So, I suggest
> lowering this message to WARN. Or probably check to WARN when standby task
> could not be created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9512) Flaky Test LagFetchIntegrationTest.shouldFetchLagsDuringRestoration
[
https://issues.apache.org/jira/browse/KAFKA-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038870#comment-17038870
]
Bruno Cadonna commented on KAFKA-9512:
--
[~vinoth][~guozhang] Can we close this ticket?
> Flaky Test LagFetchIntegrationTest.shouldFetchLagsDuringRestoration
> ---
>
> Key: KAFKA-9512
> URL: https://issues.apache.org/jira/browse/KAFKA-9512
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.5.0
>Reporter: Matthias J. Sax
>Assignee: Vinoth Chandar
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.5.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/497/testReport/junit/org.apache.kafka.streams.integration/LagFetchIntegrationTest/shouldFetchLagsDuringRestoration/]
> {quote}java.lang.NullPointerException at
> org.apache.kafka.streams.integration.LagFetchIntegrationTest.shouldFetchLagsDuringRestoration(LagFetchIntegrationTest.java:306){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9572) Sum Computation with Exactly-Once Enabled and Injected Failures Misses Some Records
Bruno Cadonna created KAFKA-9572:
Summary: Sum Computation with Exactly-Once Enabled and Injected
Failures Misses Some Records
Key: KAFKA-9572
URL: https://issues.apache.org/jira/browse/KAFKA-9572
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.4.0
Reporter: Bruno Cadonna
Attachments: 7-changelog-1.txt, data-1.txt, streams22.log,
streams23.log, streams30.log, sum-1.txt
System test {{StreamsEosTest.test_failure_and_recovery}} failed due to a
wrongly computed aggregation under exactly-once (EOS). The specific error is:
{code:java}
Exception in thread "main" java.lang.RuntimeException: Result verification
failed for ConsumerRecord(topic = sum, partition = 1, leaderEpoch = 0, offset =
2805, CreateTime = 1580719595164, serialized key size = 4, serialized value
size = 8, headers = RecordHeaders(headers = [], isReadOnly = false), key =
[B@6c779568, value = [B@f381794) expected <6069,17269> but was <6069,10698>
at
org.apache.kafka.streams.tests.EosTestDriver.verifySum(EosTestDriver.java:444)
at
org.apache.kafka.streams.tests.EosTestDriver.verify(EosTestDriver.java:196)
at
org.apache.kafka.streams.tests.StreamsEosTest.main(StreamsEosTest.java:69)
{code}
That means, the sum computed by the Streams app seems to be wrong for key 6069.
I checked the dumps of the log segments of the input topic partition (attached:
data-1.txt) and indeed two input records are not considered in the sum. With
those two missed records the sum would be correct. More concretely, the input
values for key 6069 are:
# 147
# 9250
# 5340
# 1231
# 1301
The sum of this values is 17269 as stated in the exception above as expected
sum. If you subtract values 3 and 4, i.e., 5340 and 1231 from 17269, you get
10698 , which is the actual sum in the exception above. Somehow those two
values are missing.
In the log dump of the output topic partition (attached: sum-1.txt), the sum is
correct until the 4th value 1231 , i.e. 15968, then it is overwritten with
10698.
In the log dump of the changelog topic of the state store that stores the sum
(attached: 7-changelog-1.txt), the sum is also overwritten as in the output
topic.
I attached the logs of the three Streams instances involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9572) Sum Computation with Exactly-Once Enabled and Injected Failures Misses Some Records
[
https://issues.apache.org/jira/browse/KAFKA-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041635#comment-17041635
]
Bruno Cadonna commented on KAFKA-9572:
--
No, it was on 2.4.
> Sum Computation with Exactly-Once Enabled and Injected Failures Misses Some
> Records
> ---
>
> Key: KAFKA-9572
> URL: https://issues.apache.org/jira/browse/KAFKA-9572
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Bruno Cadonna
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 2.5.0
>
> Attachments: 7-changelog-1.txt, data-1.txt, streams22.log,
> streams23.log, streams30.log, sum-1.txt
>
>
> System test {{StreamsEosTest.test_failure_and_recovery}} failed due to a
> wrongly computed aggregation under exactly-once (EOS). The specific error is:
> {code:java}
> Exception in thread "main" java.lang.RuntimeException: Result verification
> failed for ConsumerRecord(topic = sum, partition = 1, leaderEpoch = 0, offset
> = 2805, CreateTime = 1580719595164, serialized key size = 4, serialized value
> size = 8, headers = RecordHeaders(headers = [], isReadOnly = false), key =
> [B@6c779568, value = [B@f381794) expected <6069,17269> but was <6069,10698>
> at
> org.apache.kafka.streams.tests.EosTestDriver.verifySum(EosTestDriver.java:444)
> at
> org.apache.kafka.streams.tests.EosTestDriver.verify(EosTestDriver.java:196)
> at
> org.apache.kafka.streams.tests.StreamsEosTest.main(StreamsEosTest.java:69)
> {code}
> That means, the sum computed by the Streams app seems to be wrong for key
> 6069. I checked the dumps of the log segments of the input topic partition
> (attached: data-1.txt) and indeed two input records are not considered in the
> sum. With those two missed records the sum would be correct. More concretely,
> the input values for key 6069 are:
> # 147
> # 9250
> # 5340
> # 1231
> # 1301
> The sum of this values is 17269 as stated in the exception above as expected
> sum. If you subtract values 3 and 4, i.e., 5340 and 1231 from 17269, you get
> 10698 , which is the actual sum in the exception above. Somehow those two
> values are missing.
> In the log dump of the output topic partition (attached: sum-1.txt), the sum
> is correct until the 4th value 1231 , i.e. 15968, then it is overwritten with
> 10698.
> In the log dump of the changelog topic of the state store that stores the sum
> (attached: 7-changelog-1.txt), the sum is also overwritten as in the output
> topic.
> I attached the logs of the three Streams instances involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9572) Sum Computation with Exactly-Once Enabled and Injected Failures Misses Some Records
[
https://issues.apache.org/jira/browse/KAFKA-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044308#comment-17044308
]
Bruno Cadonna commented on KAFKA-9572:
--
Sounds good to me.
It is a pity that we cannot backport the potential fix, but I see that it would
be really hard.
> Sum Computation with Exactly-Once Enabled and Injected Failures Misses Some
> Records
> ---
>
> Key: KAFKA-9572
> URL: https://issues.apache.org/jira/browse/KAFKA-9572
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Bruno Cadonna
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 2.5.0
>
> Attachments: 7-changelog-1.txt, data-1.txt, streams22.log,
> streams23.log, streams30.log, sum-1.txt
>
>
> System test {{StreamsEosTest.test_failure_and_recovery}} failed due to a
> wrongly computed aggregation under exactly-once (EOS). The specific error is:
> {code:java}
> Exception in thread "main" java.lang.RuntimeException: Result verification
> failed for ConsumerRecord(topic = sum, partition = 1, leaderEpoch = 0, offset
> = 2805, CreateTime = 1580719595164, serialized key size = 4, serialized value
> size = 8, headers = RecordHeaders(headers = [], isReadOnly = false), key =
> [B@6c779568, value = [B@f381794) expected <6069,17269> but was <6069,10698>
> at
> org.apache.kafka.streams.tests.EosTestDriver.verifySum(EosTestDriver.java:444)
> at
> org.apache.kafka.streams.tests.EosTestDriver.verify(EosTestDriver.java:196)
> at
> org.apache.kafka.streams.tests.StreamsEosTest.main(StreamsEosTest.java:69)
> {code}
> That means, the sum computed by the Streams app seems to be wrong for key
> 6069. I checked the dumps of the log segments of the input topic partition
> (attached: data-1.txt) and indeed two input records are not considered in the
> sum. With those two missed records the sum would be correct. More concretely,
> the input values for key 6069 are:
> # 147
> # 9250
> # 5340
> # 1231
> # 1301
> The sum of this values is 17269 as stated in the exception above as expected
> sum. If you subtract values 3 and 4, i.e., 5340 and 1231 from 17269, you get
> 10698 , which is the actual sum in the exception above. Somehow those two
> values are missing.
> In the log dump of the output topic partition (attached: sum-1.txt), the sum
> is correct until the 4th value 1231 , i.e. 15968, then it is overwritten with
> 10698.
> In the log dump of the changelog topic of the state store that stores the sum
> (attached: 7-changelog-1.txt), the sum is also overwritten as in the output
> topic.
> I attached the logs of the three Streams instances involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-9606) Document Metrics Changes from KIP-444
[ https://issues.apache.org/jira/browse/KAFKA-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna reassigned KAFKA-9606: Assignee: Bruno Cadonna > Document Metrics Changes from KIP-444 > - > > Key: KAFKA-9606 > URL: https://issues.apache.org/jira/browse/KAFKA-9606 > Project: Kafka > Issue Type: Task > Components: docs, streams >Affects Versions: 2.5.0 >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > Fix For: 2.5.0 > > > Changes introduced in KIP-444 shall be documented. See > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams|https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%253A+Augment+metrics+for+Kafka+Streams] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9129) Add Thread ID to the InternalProcessorContext
[
https://issues.apache.org/jira/browse/KAFKA-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna reassigned KAFKA-9129:
Assignee: Bruno Cadonna
> Add Thread ID to the InternalProcessorContext
> -
>
> Key: KAFKA-9129
> URL: https://issues.apache.org/jira/browse/KAFKA-9129
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
>
> When we added client metrics we had to move the {{StreamsMetricsImpl}} object
> to the client level. That means that now instead of having one
> {{StreamsMetricsImpl}} object per thread, we have now one per client. That
> also means that we cannot store the thread ID in the {{StreamsMetricsImpl}}
> anymore. Currently, we get the thread ID from
> {{Thread.currentThread().getName()}} when we need to create a sensor.
> However, that is not robust against code refactoring because we need to
> ensure that the thread that creates the sensor is also the one that records
> the metrics. To be more flexible, we should expose the ID of the thread that
> executes a processor in the {{InternalProcessorContext}} like it already
> exposes the task ID.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9606) Document Metrics Changes from KIP-444
Bruno Cadonna created KAFKA-9606: Summary: Document Metrics Changes from KIP-444 Key: KAFKA-9606 URL: https://issues.apache.org/jira/browse/KAFKA-9606 Project: Kafka Issue Type: Task Components: docs, streams Affects Versions: 2.5.0 Reporter: Bruno Cadonna Fix For: 2.5.0 Changes introduced in KIP-444 shall be documented. See [https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams|https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%253A+Augment+metrics+for+Kafka+Streams] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9611) KGroupedTable.aggregate(...) emits incorrect values
[
https://issues.apache.org/jira/browse/KAFKA-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045775#comment-17045775
]
Bruno Cadonna commented on KAFKA-9611:
--
[~neilgreen] Thank you for the bug report!
When you say "Then the output should contain", is this the output that you see?
Actually, it is not true that the sum of the signal is never 4. If the update
{{abc1234/signal2 | 4}} is sent to the aggregation processor before update
{{abc1234/signal1 | 2}}, the sum is 4 before it becomes 6. It depends on your
cache size specified by {{cache.max.bytes.buffering}}, how many updates you see
downstreams. If you set {{cache.max.bytes.buffering}} to zero, you should see
all updates.
> KGroupedTable.aggregate(...) emits incorrect values
> ---
>
> Key: KAFKA-9611
> URL: https://issues.apache.org/jira/browse/KAFKA-9611
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Neil Green
>Priority: Major
>
> I've run into what appears to be undesirable behaviour in a streams app.
> I have a KTable produced from a topic. The table contains entries like
> "abc1234/signal1" : 1, "abc1234/signal2" : 3
> The key is "id/signal name" and the value is an int. I want to produce a
> aggregate ktable containing the sum all of the
> signals for a given id.
> {{So if source ktable contains:}}
> {{+--+--+}}
> {{| abc1234/signal1 | 2 |}}
> {{| abc1234/signal2 | 4 |}}
> {{| abc4566/signal1 | 3 |}}
> {{+--+--+}}
> {{Then the output should contain}}
> {{+--+--+}}
> {{| abc1234 | 6 |}}
> {{| abc4566 | 3 |}}
> {{+--+--+}}
> {{On a change}}
> {{+--+--+}}
> {{| abc1234/signal1 | 3 |}}
> {{+--+--+}}
> {{```}}
> {{I would expect the change}}
> {{```}}
> {{+--+--+}}
> {{| abc1234 | 7 |}}
> {{+--+--+}}
> {{to be published.}}
> In fact there are two changelog entries published
> {{+--+--+}}
> {{| abc1234 | 4 | // This is incorrect. The sum of the signals is never 4.}}
> {{+--+--+}}
> Then
> {{+--+--+}}
> {{| abc1234 | 7 |}}
> {{+--+--+}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9611) KGroupedTable.aggregate(...) emits incorrect values
[
https://issues.apache.org/jira/browse/KAFKA-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046648#comment-17046648
]
Bruno Cadonna commented on KAFKA-9611:
--
[~mjsax] OK, I see. That would explain the observation. {{abc1234 | 4}} is the
update of the subtractor and {{abc1234 | 7}} is the update of the adder.
[~neilgreen] Do you agree? Can we close this ticket as not a bug?
> KGroupedTable.aggregate(...) emits incorrect values
> ---
>
> Key: KAFKA-9611
> URL: https://issues.apache.org/jira/browse/KAFKA-9611
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Neil Green
>Priority: Major
>
> I've run into what appears to be undesirable behaviour in a streams app.
> I have a KTable produced from a topic. The table contains entries like
> "abc1234/signal1" : 1, "abc1234/signal2" : 3
> The key is "id/signal name" and the value is an int. I want to produce a
> aggregate ktable containing the sum all of the
> signals for a given id.
> {{So if source ktable contains:}}
> {{+--+--+}}
> {{| abc1234/signal1 | 2 |}}
> {{| abc1234/signal2 | 4 |}}
> {{| abc4566/signal1 | 3 |}}
> {{+--+--+}}
> {{Then the output should contain}}
> {{+--+--+}}
> {{| abc1234 | 6 |}}
> {{| abc4566 | 3 |}}
> {{+--+--+}}
> {{On a change}}
> {{+--+--+}}
> {{| abc1234/signal1 | 3 |}}
> {{+--+--+}}
> {{```}}
> {{I would expect the change}}
> {{```}}
> {{+--+--+}}
> {{| abc1234 | 7 |}}
> {{+--+--+}}
> {{to be published.}}
> In fact there are two changelog entries published
> {{+--+--+}}
> {{| abc1234 | 4 | // This is incorrect. The sum of the signals is never 4.}}
> {{+--+--+}}
> Then
> {{+--+--+}}
> {{| abc1234 | 7 |}}
> {{+--+--+}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9611) KGroupedTable.aggregate(...) emits incorrect values
[
https://issues.apache.org/jira/browse/KAFKA-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047318#comment-17047318
]
Bruno Cadonna commented on KAFKA-9611:
--
[~vvcephei] Thank you for the detailed explanation! I agree with you all that
this would be an improvement. Thus, I am in favor of keeping the ticket open
and changing the ticket type from bug to improvement.
> KGroupedTable.aggregate(...) emits incorrect values
> ---
>
> Key: KAFKA-9611
> URL: https://issues.apache.org/jira/browse/KAFKA-9611
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Neil Green
>Priority: Major
>
> I've run into what appears to be undesirable behaviour in a streams app.
> I have a KTable produced from a topic. The table contains entries like
> "abc1234/signal1" : 1, "abc1234/signal2" : 3
> The key is "id/signal name" and the value is an int. I want to produce a
> aggregate ktable containing the sum all of the
> signals for a given id.
> {{So if source ktable contains:}}
> {{+--+--+}}
> {{| abc1234/signal1 | 2 |}}
> {{| abc1234/signal2 | 4 |}}
> {{| abc4566/signal1 | 3 |}}
> {{+--+--+}}
> {{Then the output should contain}}
> {{+--+--+}}
> {{| abc1234 | 6 |}}
> {{| abc4566 | 3 |}}
> {{+--+--+}}
> {{On a change}}
> {{+--+--+}}
> {{| abc1234/signal1 | 3 |}}
> {{+--+--+}}
> {{```}}
> {{I would expect the change}}
> {{```}}
> {{+--+--+}}
> {{| abc1234 | 7 |}}
> {{+--+--+}}
> {{to be published.}}
> In fact there are two changelog entries published
> {{+--+--+}}
> {{| abc1234 | 4 | // This is incorrect. The sum of the signals is never 4.}}
> {{+--+--+}}
> Then
> {{+--+--+}}
> {{| abc1234 | 7 |}}
> {{+--+--+}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9611) KGroupedTable.aggregate(...) emits incorrect values
[
https://issues.apache.org/jira/browse/KAFKA-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-9611:
-
Issue Type: Improvement (was: Bug)
> KGroupedTable.aggregate(...) emits incorrect values
> ---
>
> Key: KAFKA-9611
> URL: https://issues.apache.org/jira/browse/KAFKA-9611
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Neil Green
>Priority: Major
>
> I've run into what appears to be undesirable behaviour in a streams app.
> I have a KTable produced from a topic. The table contains entries like
> "abc1234/signal1" : 1, "abc1234/signal2" : 3
> The key is "id/signal name" and the value is an int. I want to produce a
> aggregate ktable containing the sum all of the
> signals for a given id.
> {{So if source ktable contains:}}
> {{+--+--+}}
> {{| abc1234/signal1 | 2 |}}
> {{| abc1234/signal2 | 4 |}}
> {{| abc4566/signal1 | 3 |}}
> {{+--+--+}}
> {{Then the output should contain}}
> {{+--+--+}}
> {{| abc1234 | 6 |}}
> {{| abc4566 | 3 |}}
> {{+--+--+}}
> {{On a change}}
> {{+--+--+}}
> {{| abc1234/signal1 | 3 |}}
> {{+--+--+}}
> {{```}}
> {{I would expect the change}}
> {{```}}
> {{+--+--+}}
> {{| abc1234 | 7 |}}
> {{+--+--+}}
> {{to be published.}}
> In fact there are two changelog entries published
> {{+--+--+}}
> {{| abc1234 | 4 | // This is incorrect. The sum of the signals is never 4.}}
> {{+--+--+}}
> Then
> {{+--+--+}}
> {{| abc1234 | 7 |}}
> {{+--+--+}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9611) KGroupedTable.aggregate(...) emits incorrect values
[
https://issues.apache.org/jira/browse/KAFKA-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047386#comment-17047386
]
Bruno Cadonna commented on KAFKA-9611:
--
I am glad that we reached an agreement.
> KGroupedTable.aggregate(...) emits incorrect values
> ---
>
> Key: KAFKA-9611
> URL: https://issues.apache.org/jira/browse/KAFKA-9611
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Neil Green
>Priority: Major
>
> I've run into what appears to be undesirable behaviour in a streams app.
> I have a KTable produced from a topic. The table contains entries like
> "abc1234/signal1" : 1, "abc1234/signal2" : 3
> The key is "id/signal name" and the value is an int. I want to produce a
> aggregate ktable containing the sum all of the
> signals for a given id.
> {{So if source ktable contains:}}
> {{+--+--+}}
> {{| abc1234/signal1 | 2 |}}
> {{| abc1234/signal2 | 4 |}}
> {{| abc4566/signal1 | 3 |}}
> {{+--+--+}}
> {{Then the output should contain}}
> {{+--+--+}}
> {{| abc1234 | 6 |}}
> {{| abc4566 | 3 |}}
> {{+--+--+}}
> {{On a change}}
> {{+--+--+}}
> {{| abc1234/signal1 | 3 |}}
> {{+--+--+}}
> {{```}}
> {{I would expect the change}}
> {{```}}
> {{+--+--+}}
> {{| abc1234 | 7 |}}
> {{+--+--+}}
> {{to be published.}}
> In fact there are two changelog entries published
> {{+--+--+}}
> {{| abc1234 | 4 | // This is incorrect. The sum of the signals is never 4.}}
> {{+--+--+}}
> Then
> {{+--+--+}}
> {{| abc1234 | 7 |}}
> {{+--+--+}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9675) RocksDB metrics reported always at zero
[
https://issues.apache.org/jira/browse/KAFKA-9675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-9675:
-
Issue Type: Bug (was: Improvement)
> RocksDB metrics reported always at zero
> ---
>
> Key: KAFKA-9675
> URL: https://issues.apache.org/jira/browse/KAFKA-9675
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Michael Viamari
>Assignee: Bruno Cadonna
>Priority: Major
>
> The rocksdb metrics listed under {{stream-state-metrics}} are reported as
> zero for all metrics and all rocksdb instances. The metrics are present in
> JMX, but are always zero.
> The streams state is configured with {{MetricsRecordingLevel}} at {{debug}}.
> I am able to see metrics with appropriate values in the
> {{stream-rocksdb-window-state-metrics}}, {{stream-record-cache-metrics}},
> {{stream-task-metrics}}, and {{stream-processor-node-metrics}}.
> Additionally, my DEBUG logs show the appropriate messages for recording
> events, i.e.
> {{org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder
> [RocksDB Metrics Recorder for agg-store] Recording metrics for store
> agg-store}}
> It happens that all of my rocksdb instances are windowed stores, not key
> value stores, so I haven't been able to check if this issue is unique to
> windowed stores.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-9675) RocksDB metrics reported always at zero
[
https://issues.apache.org/jira/browse/KAFKA-9675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna reassigned KAFKA-9675:
Assignee: Bruno Cadonna
> RocksDB metrics reported always at zero
> ---
>
> Key: KAFKA-9675
> URL: https://issues.apache.org/jira/browse/KAFKA-9675
> Project: Kafka
> Issue Type: Improvement
>Affects Versions: 2.4.0
>Reporter: Michael Viamari
>Assignee: Bruno Cadonna
>Priority: Major
>
> The rocksdb metrics listed under {{stream-state-metrics}} are reported as
> zero for all metrics and all rocksdb instances. The metrics are present in
> JMX, but are always zero.
> The streams state is configured with {{MetricsRecordingLevel}} at {{debug}}.
> I am able to see metrics with appropriate values in the
> {{stream-rocksdb-window-state-metrics}}, {{stream-record-cache-metrics}},
> {{stream-task-metrics}}, and {{stream-processor-node-metrics}}.
> Additionally, my DEBUG logs show the appropriate messages for recording
> events, i.e.
> {{org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder
> [RocksDB Metrics Recorder for agg-store] Recording metrics for store
> agg-store}}
> It happens that all of my rocksdb instances are windowed stores, not key
> value stores, so I haven't been able to check if this issue is unique to
> windowed stores.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9717) KafkaStreams#metrics() method randomly throws NullPointerException
[
https://issues.apache.org/jira/browse/KAFKA-9717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058929#comment-17058929
]
Bruno Cadonna commented on KAFKA-9717:
--
[~Bataitis] Thank you for the bug report!
Judging from the code you use exactly-once semantics (EOS). Is this correct?
My guess is that it is a data race. With EOS, the call to
{{KafkaStreams#metrics()}} accesses the set of tasks on a stream thread which
can be modified by the stream thread itself during a rebalance.
> KafkaStreams#metrics() method randomly throws NullPointerException
> --
>
> Key: KAFKA-9717
> URL: https://issues.apache.org/jira/browse/KAFKA-9717
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.2.0
> Environment: Kubernetes
>Reporter: Zygimantas
>Priority: Major
>
> We have implemented monitoring tool which monitors Kafka Streams application
> and regularly (every 20s) calls KafkaStreams.metrics() method in that
> application. But metrics() method randomly throws NullPointerException. It
> happens almost every time after application startup, but may also happen at
> random points in time after running application for few hours.
> Stacktrace:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.StreamThread.producerMetrics(StreamThread.java:1320)
> at org.apache.kafka.streams.KafkaStreams.metrics(KafkaStreams.java:379)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9717) KafkaStreams#metrics() method randomly throws NullPointerException
[
https://issues.apache.org/jira/browse/KAFKA-9717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058972#comment-17058972
]
Bruno Cadonna commented on KAFKA-9717:
--
FYI: All metrics are also exposed to JMX.
> KafkaStreams#metrics() method randomly throws NullPointerException
> --
>
> Key: KAFKA-9717
> URL: https://issues.apache.org/jira/browse/KAFKA-9717
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.2.0
> Environment: Kubernetes
>Reporter: Zygimantas
>Priority: Major
>
> We have implemented monitoring tool which monitors Kafka Streams application
> and regularly (every 20s) calls KafkaStreams.metrics() method in that
> application. But metrics() method randomly throws NullPointerException. It
> happens almost every time after application startup, but may also happen at
> random points in time after running application for few hours.
> Stacktrace:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.StreamThread.producerMetrics(StreamThread.java:1320)
> at org.apache.kafka.streams.KafkaStreams.metrics(KafkaStreams.java:379)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9675) RocksDB metrics reported always at zero
[
https://issues.apache.org/jira/browse/KAFKA-9675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-9675:
-
Fix Version/s: 2.5.1
> RocksDB metrics reported always at zero
> ---
>
> Key: KAFKA-9675
> URL: https://issues.apache.org/jira/browse/KAFKA-9675
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Michael Viamari
>Assignee: Bruno Cadonna
>Priority: Blocker
> Fix For: 2.6.0, 2.4.2, 2.5.1
>
>
> The rocksdb metrics listed under {{stream-state-metrics}} are reported as
> zero for all metrics and all rocksdb instances. The metrics are present in
> JMX, but are always zero.
> The streams state is configured with {{MetricsRecordingLevel}} at {{debug}}.
> I am able to see metrics with appropriate values in the
> {{stream-rocksdb-window-state-metrics}}, {{stream-record-cache-metrics}},
> {{stream-task-metrics}}, and {{stream-processor-node-metrics}}.
> Additionally, my DEBUG logs show the appropriate messages for recording
> events, i.e.
> {{org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder
> [RocksDB Metrics Recorder for agg-store] Recording metrics for store
> agg-store}}
> It happens that all of my rocksdb instances are windowed stores, not key
> value stores, so I haven't been able to check if this issue is unique to
> windowed stores.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9738) Add Generics Type Parameters to forwarded() in MockProcessorContext
Bruno Cadonna created KAFKA-9738:
Summary: Add Generics Type Parameters to forwarded() in
MockProcessorContext
Key: KAFKA-9738
URL: https://issues.apache.org/jira/browse/KAFKA-9738
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 2.5.0
Reporter: Bruno Cadonna
The method {{forwarded()}} to capture the forwarded records in
{{MockProcessorContext}} does not have any type parameters although the
corresponding {{forward()}} does have them. To enable type checking at compile
time in tests, generics parameters shall be added to the {{forwarded()}} method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8897) Increase Version of RocksDB
[
https://issues.apache.org/jira/browse/KAFKA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17065799#comment-17065799
]
Bruno Cadonna commented on KAFKA-8897:
--
Hi [~axitkhurana]! Thank you for picking this up.
You can run the Streams tests with {{./gradlew streams:test}}. If you want to
run a single test, you can use {{./gradlew streams:test --test
}}.
Feel free to reassign this ticket to yourself.
> Increase Version of RocksDB
> ---
>
> Key: KAFKA-8897
> URL: https://issues.apache.org/jira/browse/KAFKA-8897
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Blocker
> Fix For: 3.0.0
>
>
> A higher version (6+) of RocksDB is needed for some metrics specified in
> KIP-471.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8897) Increase Version of RocksDB
[ https://issues.apache.org/jira/browse/KAFKA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17065935#comment-17065935 ] Bruno Cadonna commented on KAFKA-8897: -- [~mjsax], [~axitkhurana] could investigate what issues we would incur when we bump the version and document the issues here. I agree that a PR cannot be merged, currently. [~axitkhurana] if you want to do something that results in a merged PR, then I agree with [~mjsax] that you should look for "newbie" and "beginner" tickets. > Increase Version of RocksDB > --- > > Key: KAFKA-8897 > URL: https://issues.apache.org/jira/browse/KAFKA-8897 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Blocker > Fix For: 3.0.0 > > > A higher version (6+) of RocksDB is needed for some metrics specified in > KIP-471. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9770) Caching State Store does not Close Underlying State Store When Exception is Thrown During Flushing
Bruno Cadonna created KAFKA-9770:
Summary: Caching State Store does not Close Underlying State Store
When Exception is Thrown During Flushing
Key: KAFKA-9770
URL: https://issues.apache.org/jira/browse/KAFKA-9770
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.6.0
Reporter: Bruno Cadonna
When a caching state store is closed it calls its {{flush()}} method. If
{{flush()}} throws an exception the underlying state store is not closed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-9770) Caching State Store does not Close Underlying State Store When Exception is Thrown During Flushing
[
https://issues.apache.org/jira/browse/KAFKA-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna reassigned KAFKA-9770:
Assignee: Bruno Cadonna
> Caching State Store does not Close Underlying State Store When Exception is
> Thrown During Flushing
> --
>
> Key: KAFKA-9770
> URL: https://issues.apache.org/jira/browse/KAFKA-9770
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.6.0
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
>
> When a caching state store is closed it calls its {{flush()}} method. If
> {{flush()}} throws an exception the underlying state store is not closed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9770) Caching State Store does not Close Underlying State Store When Exception is Thrown During Flushing
[
https://issues.apache.org/jira/browse/KAFKA-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-9770:
-
Affects Version/s: (was: 2.6.0)
2.5.0
> Caching State Store does not Close Underlying State Store When Exception is
> Thrown During Flushing
> --
>
> Key: KAFKA-9770
> URL: https://issues.apache.org/jira/browse/KAFKA-9770
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.5.0
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
>
> When a caching state store is closed it calls its {{flush()}} method. If
> {{flush()}} throws an exception the underlying state store is not closed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9770) Caching State Store does not Close Underlying State Store When Exception is Thrown During Flushing
[
https://issues.apache.org/jira/browse/KAFKA-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-9770:
-
Affects Version/s: (was: 2.5.0)
2.0.0
> Caching State Store does not Close Underlying State Store When Exception is
> Thrown During Flushing
> --
>
> Key: KAFKA-9770
> URL: https://issues.apache.org/jira/browse/KAFKA-9770
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
>
> When a caching state store is closed it calls its {{flush()}} method. If
> {{flush()}} throws an exception the underlying state store is not closed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (KAFKA-9770) Caching State Store does not Close Underlying State Store When Exception is Thrown During Flushing
[
https://issues.apache.org/jira/browse/KAFKA-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna reopened KAFKA-9770:
--
> Caching State Store does not Close Underlying State Store When Exception is
> Thrown During Flushing
> --
>
> Key: KAFKA-9770
> URL: https://issues.apache.org/jira/browse/KAFKA-9770
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
>
> When a caching state store is closed it calls its {{flush()}} method. If
> {{flush()}} throws an exception the underlying state store is not closed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9770) Caching State Store does not Close Underlying State Store When Exception is Thrown During Flushing
[
https://issues.apache.org/jira/browse/KAFKA-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073468#comment-17073468
]
Bruno Cadonna commented on KAFKA-9770:
--
I will keep this ticket open until you will have tried to cherry-pick the fix
to 2.5 after the upcoming release.
> Caching State Store does not Close Underlying State Store When Exception is
> Thrown During Flushing
> --
>
> Key: KAFKA-9770
> URL: https://issues.apache.org/jira/browse/KAFKA-9770
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
> Fix For: 2.6.0
>
>
> When a caching state store is closed it calls its {{flush()}} method. If
> {{flush()}} throws an exception the underlying state store is not closed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9013) Flaky Test MirrorConnectorsIntegrationTest#testReplication
[
https://issues.apache.org/jira/browse/KAFKA-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165743#comment-17165743
]
Bruno Cadonna commented on KAFKA-9013:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/7576/
{code:java}
13:47:10 java.lang.AssertionError: Connector MirrorCheckpointConnector
tasks did not start in time on cluster:
org.apache.kafka.connect.util.clusters.EmbeddedConnectCluster@7e44a25d
13:47:10 at
org.apache.kafka.connect.util.clusters.EmbeddedConnectClusterAssertions.assertConnectorAndAtLeastNumTasksAreRunning(EmbeddedConnectClusterAssertions.java:296)
13:47:10 at
org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.waitUntilMirrorMakerIsRunning(MirrorConnectorsIntegrationTest.java:191)
13:47:10 at
org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.setup(MirrorConnectorsIntegrationTest.java:184)
13:47:10
{code}
> Flaky Test MirrorConnectorsIntegrationTest#testReplication
> --
>
> Key: KAFKA-9013
> URL: https://issues.apache.org/jira/browse/KAFKA-9013
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Bruno Cadonna
>Priority: Major
> Labels: flaky-test
>
> h1. Stacktrace:
> {code:java}
> java.lang.AssertionError: Condition not met within timeout 2. Offsets not
> translated downstream to primary cluster.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:377)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:354)
> at
> org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testReplication(MirrorConnectorsIntegrationTest.java:239)
> {code}
> h1. Standard Error
> {code}
> Standard Error
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered
> in SERVER runtime does not implement any provider interfaces applicable in
> the SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.
> Oct 09, 2019 11:32:01 PM org.glassfish.jersey.internal.Errors logErrors
> WARNING: The following warnings have been detected: WARNING: The
> (sub)resource method listLoggers in
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectors in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method createConnector in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectorPlugins in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> contains empty path annotation.
> WARNING: The (sub)resource method serverInfo in
> org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty
> path annotation.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the p
[jira] [Commented] (KAFKA-9013) Flaky Test MirrorConnectorsIntegrationTest#testReplication
[
https://issues.apache.org/jira/browse/KAFKA-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165744#comment-17165744
]
Bruno Cadonna commented on KAFKA-9013:
--
https://builds.apache.org/job/kafka-pr-jdk14-scala2.13/1714
{code:java}
13:22:42 org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest >
testReplication STARTED
13:24:09
13:24:09 org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest >
testReplication SKIPPED
13:24:09
13:24:09 > Task :connect:mirror:integrationTest FAILED
{code}
> Flaky Test MirrorConnectorsIntegrationTest#testReplication
> --
>
> Key: KAFKA-9013
> URL: https://issues.apache.org/jira/browse/KAFKA-9013
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Bruno Cadonna
>Priority: Major
> Labels: flaky-test
>
> h1. Stacktrace:
> {code:java}
> java.lang.AssertionError: Condition not met within timeout 2. Offsets not
> translated downstream to primary cluster.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:377)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:354)
> at
> org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testReplication(MirrorConnectorsIntegrationTest.java:239)
> {code}
> h1. Standard Error
> {code}
> Standard Error
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered
> in SERVER runtime does not implement any provider interfaces applicable in
> the SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.
> Oct 09, 2019 11:32:01 PM org.glassfish.jersey.internal.Errors logErrors
> WARNING: The following warnings have been detected: WARNING: The
> (sub)resource method listLoggers in
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectors in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method createConnector in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectorPlugins in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> contains empty path annotation.
> WARNING: The (sub)resource method serverInfo in
> org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty
> path annotation.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered
> in SERVER runtime does not implement any provider interfaces applicable in
> the SER
[jira] [Issue Comment Deleted] (KAFKA-9013) Flaky Test MirrorConnectorsIntegrationTest#testReplication
[
https://issues.apache.org/jira/browse/KAFKA-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-9013:
-
Comment: was deleted
(was: https://builds.apache.org/job/kafka-pr-jdk14-scala2.13/1714
{code:java}
13:22:42 org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest >
testReplication STARTED
13:24:09
13:24:09 org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest >
testReplication SKIPPED
13:24:09
13:24:09 > Task :connect:mirror:integrationTest FAILED
{code}
)
> Flaky Test MirrorConnectorsIntegrationTest#testReplication
> --
>
> Key: KAFKA-9013
> URL: https://issues.apache.org/jira/browse/KAFKA-9013
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Bruno Cadonna
>Priority: Major
> Labels: flaky-test
>
> h1. Stacktrace:
> {code:java}
> java.lang.AssertionError: Condition not met within timeout 2. Offsets not
> translated downstream to primary cluster.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:377)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:354)
> at
> org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testReplication(MirrorConnectorsIntegrationTest.java:239)
> {code}
> h1. Standard Error
> {code}
> Standard Error
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered
> in SERVER runtime does not implement any provider interfaces applicable in
> the SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.
> Oct 09, 2019 11:32:01 PM org.glassfish.jersey.internal.Errors logErrors
> WARNING: The following warnings have been detected: WARNING: The
> (sub)resource method listLoggers in
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectors in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method createConnector in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectorPlugins in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> contains empty path annotation.
> WARNING: The (sub)resource method serverInfo in
> org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty
> path annotation.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered
> in SERVER runtime does not implement any provider interfaces applicable in
> the SERVER runtime. Due to const
[jira] [Commented] (KAFKA-7540) Flaky Test ConsumerBounceTest#testClose
[
https://issues.apache.org/jira/browse/KAFKA-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167720#comment-17167720
]
Bruno Cadonna commented on KAFKA-7540:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/7618
{code:java}
20:08:12 kafka.api.ConsumerBounceTest > testClose FAILED
20:08:12 java.lang.AssertionError: Assignment did not complete on time
20:08:12 at org.junit.Assert.fail(Assert.java:89)
20:08:12 at org.junit.Assert.assertTrue(Assert.java:42)
20:08:12 at
kafka.api.ConsumerBounceTest.checkClosedState(ConsumerBounceTest.scala:486)
20:08:12 at
kafka.api.ConsumerBounceTest.checkCloseWithCoordinatorFailure(ConsumerBounceTest.scala:257)
20:08:12 at
kafka.api.ConsumerBounceTest.testClose(ConsumerBounceTest.scala:220)
{code}
> Flaky Test ConsumerBounceTest#testClose
> ---
>
> Key: KAFKA-7540
> URL: https://issues.apache.org/jira/browse/KAFKA-7540
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 2.2.0
>Reporter: John Roesler
>Assignee: Jason Gustafson
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.7.0, 2.6.1
>
>
> Observed on Java 8:
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/17314/testReport/junit/kafka.api/ConsumerBounceTest/testClose/]
>
> Stacktrace:
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> kafka.integration.KafkaServerTestHarness.killBroker(KafkaServerTestHarness.scala:146)
> at
> kafka.api.ConsumerBounceTest.checkCloseWithCoordinatorFailure(ConsumerBounceTest.scala:238)
> at kafka.api.ConsumerBounceTest.testClose(ConsumerBounceTest.scala:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAda
[jira] [Commented] (KAFKA-9013) Flaky Test MirrorConnectorsIntegrationTest#testReplication
[
https://issues.apache.org/jira/browse/KAFKA-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167726#comment-17167726
]
Bruno Cadonna commented on KAFKA-9013:
--
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3608
{code:java}
java.lang.RuntimeException: Could not find enough records. found 0, expected 100
at
org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.consume(EmbeddedKafkaCluster.java:435)
at
org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testReplication(MirrorConnectorsIntegrationTest.java:221)
{code}
> Flaky Test MirrorConnectorsIntegrationTest#testReplication
> --
>
> Key: KAFKA-9013
> URL: https://issues.apache.org/jira/browse/KAFKA-9013
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Bruno Cadonna
>Priority: Major
> Labels: flaky-test
>
> h1. Stacktrace:
> {code:java}
> java.lang.AssertionError: Condition not met within timeout 2. Offsets not
> translated downstream to primary cluster.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:377)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:354)
> at
> org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testReplication(MirrorConnectorsIntegrationTest.java:239)
> {code}
> h1. Standard Error
> {code}
> Standard Error
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered
> in SERVER runtime does not implement any provider interfaces applicable in
> the SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.
> Oct 09, 2019 11:32:01 PM org.glassfish.jersey.internal.Errors logErrors
> WARNING: The following warnings have been detected: WARNING: The
> (sub)resource method listLoggers in
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectors in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method createConnector in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectorPlugins in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> contains empty path annotation.
> WARNING: The (sub)resource method serverInfo in
> org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty
> path annotation.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered
> in SERVER runtime does not implemen
[jira] [Commented] (KAFKA-10255) Fix flaky testOneWayReplicationWithAutorOffsetSync1 test
[
https://issues.apache.org/jira/browse/KAFKA-10255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167727#comment-17167727
]
Bruno Cadonna commented on KAFKA-10255:
---
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3608
{code:java}
java.lang.AssertionError: consumer record size is not zero expected:<0> but
was:<4>
{code}
> Fix flaky testOneWayReplicationWithAutorOffsetSync1 test
>
>
> Key: KAFKA-10255
> URL: https://issues.apache.org/jira/browse/KAFKA-10255
> Project: Kafka
> Issue Type: Test
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
>
> https://builds.apache.org/blue/rest/organizations/jenkins/pipelines/kafka-trunk-jdk14/runs/279/log/?start=0
> org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest >
> testOneWayReplicationWithAutorOffsetSync1 STARTED
> org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testOneWayReplicationWithAutorOffsetSync1
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-trunk-jdk14/connect/mirror/build/reports/testOutput/org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testOneWayReplicationWithAutorOffsetSync1.test.stdout
> org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest >
> testOneWayReplicationWithAutorOffsetSync1 FAILED
> java.lang.AssertionError: consumer record size is not zero expected:<0> but
> was:<2>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at
> org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testOneWayReplicationWithAutorOffsetSync1(MirrorConnectorsIntegrationTest.java:349)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10137) Clean-up retain Duplicate logic in Window Stores
[
https://issues.apache.org/jira/browse/KAFKA-10137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168521#comment-17168521
]
Bruno Cadonna commented on KAFKA-10137:
---
I also noticed this the last time I looked at
{{ChangeLoggingWindowBytesStore}}. Since the sequence number is not incremented
if {{retainDuplicates}} is false, I think it is OK from a correctness point of
view.
Maybe the reason why we write the sequence number also when
{{retainDuplicates}} is false is that in such a way we do not need to
distinguish whether the key has a sequence number or not when we read the keys
again.
> Clean-up retain Duplicate logic in Window Stores
>
>
> Key: KAFKA-10137
> URL: https://issues.apache.org/jira/browse/KAFKA-10137
> Project: Kafka
> Issue Type: Task
> Components: streams
>Affects Versions: 2.5.0
>Reporter: Bruno Cadonna
>Priority: Minor
>
> Stream-stream joins use the regular `WindowStore` implementation but with
> `retainDuplicates` set to true. To allow for duplicates while using the same
> unique-key underlying stores we just wrap the key with an incrementing
> sequence number before inserting it.
> The logic to maintain and append the sequence number is present in multiple
> locations, namely in the changelogging window store and in its underlying
> window stores. We should consolidate this code to one single location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10357) Handle accidental deletion of repartition-topics as exceptional failure
[ https://issues.apache.org/jira/browse/KAFKA-10357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17174279#comment-17174279 ] Bruno Cadonna commented on KAFKA-10357: --- I have a couple of questions regarding wild idea 1): 1) I guess you mean STARTING and not CREATED, don't you? There is no transition from CREATED to PARTITION_REVOKED or PARTITION_ASSIGNED. 2) I suppose this check on the states of the stream threads is done in the group leader. If a Streams client joined an existing group and a stream thread of this newly added Streams client were elected as the group leader, then we would have the situation where the stream thread is in STARTING but it would not be the first-ever rebalance. Is this correct? > Handle accidental deletion of repartition-topics as exceptional failure > --- > > Key: KAFKA-10357 > URL: https://issues.apache.org/jira/browse/KAFKA-10357 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Guozhang Wang >Assignee: Bruno Cadonna >Priority: Major > > Repartition topics are both written by Stream's producer and read by Stream's > consumer, so when they are accidentally deleted both clients may be notified. > But in practice the consumer would react to it much quicker than producer > since the latter has a delivery timeout expiration period (see > https://issues.apache.org/jira/browse/KAFKA-10356). When consumer reacts to > it, it will re-join the group since metadata changed and during the triggered > rebalance it would auto-recreate the topic silently and continue, causing > data lost silently. > One idea, is to only create all repartition topics *once* in the first > rebalance and not auto-create them any more in future rebalances, instead it > would be treated similar as INCOMPLETE_SOURCE_TOPIC_METADATA error code > (https://issues.apache.org/jira/browse/KAFKA-10355). > The challenge part would be, how to determine if it is the first-ever > rebalance, and there are several wild ideas I'd like to throw out here: > 1) change the thread state transition diagram so that CREATED state would not > transit to PARTITION_REVOKED but only to PARTITION_ASSIGNED, then in the > assign function we can check if the state is still in CREATED and not RUNNING. > 2) augment the subscriptionInfo to encode whether or not this is the first > time ever rebalance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10396) Overall memory of container keep on growing due to kafka stream / rocksdb and OOM killed once limit reached
[ https://issues.apache.org/jira/browse/KAFKA-10396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-10396: -- Component/s: (was: KafkaConnect) > Overall memory of container keep on growing due to kafka stream / rocksdb and > OOM killed once limit reached > --- > > Key: KAFKA-10396 > URL: https://issues.apache.org/jira/browse/KAFKA-10396 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.1, 2.5.0 >Reporter: Vagesh Mathapati >Priority: Critical > > We are observing that overall memory of our container keep on growing and > never came down. > After analysis find out that rocksdbjni.so is keep on allocating 64M chunks > of memory off-heap and never releases back. This causes OOM kill after memory > reaches configured limit. > We use Kafka stream and globalktable for our many kafka topics. > Below is our environment > * Kubernetes cluster > * openjdk 11.0.7 2020-04-14 LTS > * OpenJDK Runtime Environment Zulu11.39+16-SA (build 11.0.7+10-LTS) > * OpenJDK 64-Bit Server VM Zulu11.39+16-SA (build 11.0.7+10-LTS, mixed mode) > * Springboot 2.3 > * spring-kafka-2.5.0 > * kafka-streams-2.5.0 > * kafka-streams-avro-serde-5.4.0 > * rocksdbjni-5.18.3 > Observed same result with kafka 2.3 version. > Below is the snippet of our analysis > from pmap output we took addresses from these 64M allocations (RSS) > Address Kbytes RSS Dirty Mode Mapping > 7f3ce800 65536 65532 65532 rw--- [ anon ] > 7f3cf400 65536 65536 65536 rw--- [ anon ] > 7f3d6400 65536 65536 65536 rw--- [ anon ] > We tried to match with memory allocation logs enabled with the help of Azul > systems team. > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ff7ca0 > @ /tmp/librocksdbjni6564497922441568920.so: > _ZN7rocksdb15BlockBasedTable3GetERKNS_11ReadOptionsERKNS_5SliceEPNS_10GetContextEPKNS_14SliceTransformEb+0x894)[0x7f3e1c898fd4] > - 0x7f3ce8ff9780 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xfa)[0x7f3e1c65d5da] > - 0x7f3ce8ff9750 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ff97c0 > @ > /tmp/librocksdbjni6564497922441568920.so:_Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xfa)[0x7f3e1c65d5da] > - 0x7f3ce8ffccf0 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ffcd10 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xfa)[0x7f3e1c65d5da] > - 0x7f3ce8ffccf0 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ffcd10 > We also identified that content on this 64M is just 0s and no any data > present in it. > I tried to tune the rocksDB configuratino as mentioned but it did not helped. > [https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config] > > Please let me know if you need any more details -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10396) Overall memory of container keep on growing due to kafka stream / rocksdb and OOM killed once limit reached
[ https://issues.apache.org/jira/browse/KAFKA-10396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176892#comment-17176892 ] Bruno Cadonna commented on KAFKA-10396: --- [~vmathapati], thank you for the report. Could you please also post your RocksDB config setter? Do you use time windows or session windows in your topology? Do you have minimal example to reproduces the issue? > Overall memory of container keep on growing due to kafka stream / rocksdb and > OOM killed once limit reached > --- > > Key: KAFKA-10396 > URL: https://issues.apache.org/jira/browse/KAFKA-10396 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.1, 2.5.0 >Reporter: Vagesh Mathapati >Priority: Critical > > We are observing that overall memory of our container keep on growing and > never came down. > After analysis find out that rocksdbjni.so is keep on allocating 64M chunks > of memory off-heap and never releases back. This causes OOM kill after memory > reaches configured limit. > We use Kafka stream and globalktable for our many kafka topics. > Below is our environment > * Kubernetes cluster > * openjdk 11.0.7 2020-04-14 LTS > * OpenJDK Runtime Environment Zulu11.39+16-SA (build 11.0.7+10-LTS) > * OpenJDK 64-Bit Server VM Zulu11.39+16-SA (build 11.0.7+10-LTS, mixed mode) > * Springboot 2.3 > * spring-kafka-2.5.0 > * kafka-streams-2.5.0 > * kafka-streams-avro-serde-5.4.0 > * rocksdbjni-5.18.3 > Observed same result with kafka 2.3 version. > Below is the snippet of our analysis > from pmap output we took addresses from these 64M allocations (RSS) > Address Kbytes RSS Dirty Mode Mapping > 7f3ce800 65536 65532 65532 rw--- [ anon ] > 7f3cf400 65536 65536 65536 rw--- [ anon ] > 7f3d6400 65536 65536 65536 rw--- [ anon ] > We tried to match with memory allocation logs enabled with the help of Azul > systems team. > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ff7ca0 > @ /tmp/librocksdbjni6564497922441568920.so: > _ZN7rocksdb15BlockBasedTable3GetERKNS_11ReadOptionsERKNS_5SliceEPNS_10GetContextEPKNS_14SliceTransformEb+0x894)[0x7f3e1c898fd4] > - 0x7f3ce8ff9780 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xfa)[0x7f3e1c65d5da] > - 0x7f3ce8ff9750 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ff97c0 > @ > /tmp/librocksdbjni6564497922441568920.so:_Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xfa)[0x7f3e1c65d5da] > - 0x7f3ce8ffccf0 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ffcd10 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xfa)[0x7f3e1c65d5da] > - 0x7f3ce8ffccf0 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ffcd10 > We also identified that content on this 64M is just 0s and no any data > present in it. > I tried to tune the rocksDB configuratino as mentioned but it did not helped. > [https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config] > > Please let me know if you need any more details -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-10396) Overall memory of container keep on growing due to kafka stream / rocksdb and OOM killed once limit reached
[ https://issues.apache.org/jira/browse/KAFKA-10396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176892#comment-17176892 ] Bruno Cadonna edited comment on KAFKA-10396 at 8/13/20, 10:07 AM: -- [~vmathapati], thank you for the report. Could you please also post your RocksDB config setter? Do you use time windows or session windows in your topology? Do you have minimal example to reproduce the issue? was (Author: cadonna): [~vmathapati], thank you for the report. Could you please also post your RocksDB config setter? Do you use time windows or session windows in your topology? Do you have minimal example to reproduces the issue? > Overall memory of container keep on growing due to kafka stream / rocksdb and > OOM killed once limit reached > --- > > Key: KAFKA-10396 > URL: https://issues.apache.org/jira/browse/KAFKA-10396 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.1, 2.5.0 >Reporter: Vagesh Mathapati >Priority: Critical > > We are observing that overall memory of our container keep on growing and > never came down. > After analysis find out that rocksdbjni.so is keep on allocating 64M chunks > of memory off-heap and never releases back. This causes OOM kill after memory > reaches configured limit. > We use Kafka stream and globalktable for our many kafka topics. > Below is our environment > * Kubernetes cluster > * openjdk 11.0.7 2020-04-14 LTS > * OpenJDK Runtime Environment Zulu11.39+16-SA (build 11.0.7+10-LTS) > * OpenJDK 64-Bit Server VM Zulu11.39+16-SA (build 11.0.7+10-LTS, mixed mode) > * Springboot 2.3 > * spring-kafka-2.5.0 > * kafka-streams-2.5.0 > * kafka-streams-avro-serde-5.4.0 > * rocksdbjni-5.18.3 > Observed same result with kafka 2.3 version. > Below is the snippet of our analysis > from pmap output we took addresses from these 64M allocations (RSS) > Address Kbytes RSS Dirty Mode Mapping > 7f3ce800 65536 65532 65532 rw--- [ anon ] > 7f3cf400 65536 65536 65536 rw--- [ anon ] > 7f3d6400 65536 65536 65536 rw--- [ anon ] > We tried to match with memory allocation logs enabled with the help of Azul > systems team. > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ff7ca0 > @ /tmp/librocksdbjni6564497922441568920.so: > _ZN7rocksdb15BlockBasedTable3GetERKNS_11ReadOptionsERKNS_5SliceEPNS_10GetContextEPKNS_14SliceTransformEb+0x894)[0x7f3e1c898fd4] > - 0x7f3ce8ff9780 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xfa)[0x7f3e1c65d5da] > - 0x7f3ce8ff9750 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ff97c0 > @ > /tmp/librocksdbjni6564497922441568920.so:_Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xfa)[0x7f3e1c65d5da] > - 0x7f3ce8ffccf0 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ffcd10 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0xfa)[0x7f3e1c65d5da] > - 0x7f3ce8ffccf0 > @ /tmp/librocksdbjni6564497922441568920.so: > _Z18rocksdb_get_helperP7JNIEnv_PN7rocksdb2DBERKNS1_11ReadOptionsEPNS1_18ColumnFamilyHandleEP11_jbyteArrayii+0x261)[0x7f3e1c65d741] > - 0x7f3ce8ffcd10 > We also identified that content on this 64M is just 0s and no any data > present in it. > I tried to tune the rocksDB configuratino as mentioned but it did not helped. > [https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config] > > Please let me know if you need any more details -- This message was sent by Atlassian Jira (v8.3.4#803005)
