date:20200223

Jenkins build is back to normal : kafka-trunk-jdk8 #4265

2020-02-23 Thread Apache Jenkins Server

See

Issue in retention with compact,delete cleanup policy

2020-02-23 Thread Koushik Chitta

Hi,

I have a Topic with following config.

cleanup.policy  =  compact,delete
segment.bytes = 52428800 (~52 mb)
min.compaction.lag.ms = 180 (30 min)
delete.retention.ms = 8640 (1 day)
retention.ms = 25920 (3 days)

Ideally I would want the old records > 3 days to be deleted without producing 
an explicit delete(null value of a key) of the record.
But there can be a case due to continuous compaction, the segments can contain 
a very old record(eg: > 30 days) and new recent record (eg: 1hr) which will 
make the segment ineligible for retention delete.

Currently I don't see a work around for this. Please suggest.
I plan to start a KIP to address this use case.

Thanks,
Koushik

[jira] [Created] (KAFKA-9598) RocksDB exception when grouping dynamically appearing topics into a KTable

2020-02-23 Thread Sergey Menshikov (Jira)

Sergey Menshikov created KAFKA-9598:
---

 Summary: RocksDB exception when grouping dynamically appearing 
topics into a KTable 
 Key: KAFKA-9598
 URL: https://issues.apache.org/jira/browse/KAFKA-9598
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.4.0, 2.2.0
Reporter: Sergey Menshikov
 Attachments: exception-details.txt

A streams application, consumer a number of topics via a whitelisted regex. The 
topics appear dynamically, generated from dynamically appearing MongoDB 
collections by debezium MongoDB source driver.

The development is running on debezium docker images (Debezium 0.9 and Debezium 
1.0 -> Kafka 2.2.0 and 2.4.0), single instance of Kafka, Connect and the 
streams consumer app.

As the MongoDB driver provides only deltas of the changes, to collect full 
record for each key, the code creates KTable which is then transformed into a 
KStream for further joining with other KTables and Global KTables.

The following piece of code results in the exception when a new topic is added:

 
{code:java}
Pattern tResultPattern =
 
Pattern.compile(config.getProperty("mongodb_source_prefix")+".tr[0-9a-fA-F]{32}");
KStream tResultsTempStream = builder.stream(tResultPattern, 
Consumed.with(stringSerde, jsonSerde));
 KTable tResultsTempTable = 
tResultsTempStream.groupByKey(Grouped.with(stringSerde,jsonSerde))
 .reduce((aggValue, newValue) -> mergeNodes(aggValue,newValue)); // mergeNodes 
is a Json traverse/merger procedure
KStream tResults =
 tResultsTempTable.toStream();
 
{code}
kconsumer_1 | Exception in thread "split-reader-client3-StreamThread-1" 
org.apache.kafka.streams.errors.ProcessorStateException: Error opening store 
KSTREAM-REDUCE-STATE-STORE-32 at location 
/tmp/split-reader3/10_0/rocksdb/KSTREAM-REDUCE-STATE-STORE-32

...

kconsumer_1 | Caused by: org.rocksdb.RocksDBException: lock : 
/tmp/split-reader3/10_0/rocksdb/KSTREAM-REDUCE-STATE-STORE-32/LOCK: No 
locks available

Kstore 10_0 contains tr[0-9a-fA-F]\{32} records, I checked.

more details about exception are in the attached file.

The exception is no longer present when I use an intermediate topic instead:

 
{code:java}
Pattern tResultPattern =
 
Pattern.compile(config.getProperty("mongodb_source_prefix")+".tr[0-9a-fA-F]{32}");{code}
 

 
{code:java}
KStream tResultsTempStream = builder.stream(tResultPattern, 
Consumed.with(stringSerde, jsonSerde));
 
tResultsTempStream.transform(trTransformer::new).to(config.getProperty("tr_intermediate_topic_name"),Produced.with(stringSerde,
 jsonSerde)); // trTransformer adds topic name into value Json, in previous 
snippet it was done in the pipeline after grouping/streaming
KStream tResultsTempStream2 = 
builder.stream(config.getProperty("tr_intermediate_topic_name"), 
Consumed.with(stringSerde, jsonSerde));
 KTable tResultsTempTable = 
tResultsTempStream2.groupByKey(Grouped.with(stringSerde,jsonSerde))
 .reduce((aggValue, newValue) -> mergeNodes(aggValue,newValue));
KStream tResults =
 tResultsTempTable.toStream();
{code}
 

 

If making KTable from multiple whitelisted topics is something that is outside 
of scope of Kafka Streams, perhaps it would make sense to mention it in the 
docs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9541) Flaky Test DescribeConsumerGroupTest#testDescribeGroupMembersWithShortInitializationTimeout

2020-02-23 Thread Jason Gustafson (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9541.

Resolution: Fixed

> Flaky Test 
> DescribeConsumerGroupTest#testDescribeGroupMembersWithShortInitializationTimeout
> ---
>
> Key: KAFKA-9541
> URL: https://issues.apache.org/jira/browse/KAFKA-9541
> Project: Kafka
>  Issue Type: Bug
>  Components: core, unit tests
>Affects Versions: 2.4.0
>Reporter: huxihx
>Assignee: huxihx
>Priority: Major
>
> h3. Error Message
> java.lang.AssertionError: assertion failed
> h3. Stacktrace
> java.lang.AssertionError: assertion failed at 
> scala.Predef$.assert(Predef.scala:267) at 
> kafka.admin.DescribeConsumerGroupTest.testDescribeGroupMembersWithShortInitializationTimeout(DescribeConsumerGroupTest.scala:630)
>  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method) at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>  at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>  at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>  at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>  at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at 
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:413) at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
>  at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>  at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>  at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
>  at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>  at jdk.internal.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
>  at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>  at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
>  at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
>  at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
>  at jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
>  at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>  at 
>

[jira] [Resolved] (KAFKA-9530) Flaky Test kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout

2020-02-23 Thread Jason Gustafson (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9530.

Resolution: Fixed

> Flaky Test 
> kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout
> 
>
> Key: KAFKA-9530
> URL: https://issues.apache.org/jira/browse/KAFKA-9530
> Project: Kafka
>  Issue Type: Test
>  Components: core
>Reporter: Bill Bejeck
>Priority: Major
>  Labels: flaky-test, test
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/4570/testReport/junit/kafka.admin/DescribeConsumerGroupTest/testDescribeGroupWithShortInitializationTimeout/]
>  
> {noformat}
> Error Messagejava.lang.AssertionError: assertion 
> failedStacktracejava.lang.AssertionError: assertion failed
>   at scala.Predef$.assert(Predef.scala:267)
>   at 
> kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout(DescribeConsumerGroupTest.scala:585)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at jdk.internal.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
>   at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>   at 
>

Re: [Discuss] KIP-571: Add option to force remove members in StreamsResetter

2020-02-23 Thread Boyang Chen

Hey Feyman,

thanks a lot for the update, the KIP LGTM now. Will let Sophie take a look
again, also a minor API change:
s/setGroupInstanceId/withGroupInstanceId, and similar to setMemberId, as
usually setters are not expected to return an actual object.

Boyang

On Sat, Feb 22, 2020 at 11:05 PM feyman2009  wrote:

> Hi, Boyang
> Thanks for your review, I have updated the KIP page :)
>
> Hi, Sophie
> Thanks for your suggestions!
> 1)  Did you consider an API that just removes *all* remaining members
> from a group?
> We plan to implement the batch removal in StreamsResetter as below:
> 1) adminClient#describeConsumerGroups to get members in each
> group, this part needs no change.
> 2) adminClient#removeMembersFromConsumerGroup to remove all the
> members got from the above call (This involves API change to support the
> dynamic member removal)
> I think your suggestion is feasible but maybe not necessary currently
> since it is a subset of the combination of the above two APIs. Looking at
> the APIs in KafkaAdminClient, the adminClient.deleteXXX always takes a
> collection as the input parameter and the caller does the "query and
> delete" if "delete all" is needed, this leaves more burden on the caller
> side but increases flexibility. Since the KafkaAdminClient's API is still
> evolving, I think it would be reasonable to follow the convention and not
> adding a "removal all members" API.
>
> 2) Thanks to Boyang's correction, broker version >= 2.4 is needed
> since batch members removal is introduced since then(please check KIP-345
> 
>  for
> details).
> If it is used upon the older clusters like 2.3, 
> *UnsupportedVersionException
> *will be thrown.
>
> Thanks!
> Haoran
>
> --
> 发件人：Boyang Chen 
> 发送时间：2020年2月19日(星期三) 11:57
> 收件人：dev 
> 主 题：Re: [Discuss] KIP-571: Add option to force remove members in
> StreamsResetter
>
> Also Feyman, there is one thing I forget which is that the leave group
> change was introduced in 2.4 broker instead of 2.3. Feel free to correct it
> on the KIP.
>
> On Tue, Feb 18, 2020 at 5:44 PM Sophie Blee-Goldman 
> wrote:
>
> > Hey Feyman,
> >
> > Thanks for the KIP! I had two high-level questions:
> >
>
> > It seems like, in the specific case motivating this KIP, we would only ever
> > want to remove *all* the members remaining in the group (and never just a
> > single member at a time). As you mention there is already an admin API to
>
> > remove static members, but we'd still need something new to handle dynamic
> > ones. Did you consider an API that just removes *all* remaining members
> > from a group, rather than requiring the caller to determine and then
> > specify the
> > group.id (static) or member.id (dynamic) for each one? This way we can
> > just
>
> > have a single API exposed that will handle what we need to do regardless of
> > whether static membership is used or not.
> >
>
> > My other question is, will this new option only work for clusters that are
> > on 2.3
> > or higher? Do you have any thoughts about whether it would be possible to
> > implement this feature for older clusters as well, or are we dependent on
> > changes only introduced in 2.3?
> >
> > If so, we should make it absolutely clear what will happen if this used
> > with
> > an older cluster. That is, will the reset tool exit with a clear error
> > message right
> > away, or will it potentially leave the app in a partially reset state?
> >
> > Thanks!
> > Sophie
> >
> > On Tue, Feb 18, 2020 at 4:30 PM Boyang Chen 
> > wrote:
> >
>
> > > Thanks for the update Feyman. The updates look great, except one thing I
> > > would like to be more specific is error cases display. In the "*2)* Add
>
> > > cmdline option" you mention throwing exception when request failed, does
> > > that suggest partial failure or a full failure? How do we deal with
> > > different scenarios?
> > >
> > > Also some minor syntax fix:
>
> > > 1. it only support remove static members -> it only supports the removal
> > of
> > > static members
>
> > > 2. "new constructor is added and the old constructor will be deprecated"
> > > you mean the `new helper` right? Should be `new helper is added`
> > > 3. users should make sure all the stream applications should be are
> > > shutdown
> > >
> > > Other than the above suggestions, I think the KIP is in pretty good
> > shape.
> > >
> > > Boyang
> > >
> > > On Fri, Feb 14, 2020 at 9:29 PM feyman2009 
> > wrote:
> > >
> > > > Hi, Boyang
> > > > You can call me Feyman :)
> > > > Thanks for your quick reply with great advices!
> > > > I have updated the KIP-571 , would you mind to see if it looks
> > good ?
> > > > Thanks !
> > > >
> > > > --
> > > > 发件人：Boyang Chen 
> > >

Jenkins build is back to normal : kafka-trunk-jdk11 #1184

2020-02-23 Thread Apache Jenkins Server

See

Build failed in Jenkins: kafka-2.5-jdk8 #41

2020-02-23 Thread Apache Jenkins Server

See 


Changes:

[matthias] HOTFIX: fix NPE in Kafka Streams IQ (#8158)


--
[...truncated 2.90 MB...]
org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
STARTED

org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task

Build failed in Jenkins: kafka-2.4-jdk8 #151

2020-02-23 Thread Apache Jenkins Server

See 

Changes:


--
[...truncated 2.22 MB...]

kafka.coordinator.group.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition PASSED

kafka.coordinator.group.GroupMetadataTest > 
testFailedTxnOffsetCommitLeavesNoPendingState STARTED

kafka.coordinator.group.GroupMetadataTest > 
testFailedTxnOffsetCommitLeavesNoPendingState PASSED

kafka.coordinator.group.GroupMetadataTest > testNotInvokeJoinCallback STARTED

kafka.coordinator.group.GroupMetadataTest > testNotInvokeJoinCallback PASSED

kafka.coordinator.group.GroupMetadataTest > 
testCanRebalanceWhenCompletingRebalance STARTED

kafka.coordinator.group.GroupMetadataTest > 
testCanRebalanceWhenCompletingRebalance PASSED

kafka.coordinator.group.GroupMetadataTest > 
testDeadToAwaitingRebalanceIllegalTransition STARTED

kafka.coordinator.group.GroupMetadataTest > 
testDeadToAwaitingRebalanceIllegalTransition PASSED

kafka.coordinator.group.GroupMetadataTest > testInvokeJoinCallback STARTED

kafka.coordinator.group.GroupMetadataTest > testInvokeJoinCallback PASSED

kafka.coordinator.group.GroupMetadataTest > testEmptyToDeadTransition STARTED

kafka.coordinator.group.GroupMetadataTest > testEmptyToDeadTransition PASSED

kafka.coordinator.group.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers 
STARTED

kafka.coordinator.group.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers 
PASSED

kafka.coordinator.group.GroupMetadataTest > 
testStableToPreparingRebalanceTransition STARTED

kafka.coordinator.group.GroupMetadataTest > 
testStableToPreparingRebalanceTransition PASSED

kafka.coordinator.group.GroupMetadataTest > 
testSubscribedTopicsNonConsumerGroup STARTED

kafka.coordinator.group.GroupMetadataTest > 
testSubscribedTopicsNonConsumerGroup PASSED

kafka.coordinator.group.GroupMetadataTest > testReplaceGroupInstance STARTED

kafka.coordinator.group.GroupMetadataTest > testReplaceGroupInstance PASSED

kafka.coordinator.group.GroupMetadataTest > 
testTransactionalCommitIsAbortedAndConsumerCommitWins STARTED

kafka.coordinator.group.GroupMetadataTest > 
testTransactionalCommitIsAbortedAndConsumerCommitWins PASSED

kafka.coordinator.group.GroupMetadataTest > 
testAwaitingRebalanceToPreparingRebalanceTransition STARTED

kafka.coordinator.group.GroupMetadataTest > 
testAwaitingRebalanceToPreparingRebalanceTransition PASSED

kafka.coordinator.group.GroupMetadataTest > 
testPreparingRebalanceToDeadTransition STARTED

kafka.coordinator.group.GroupMetadataTest > 
testPreparingRebalanceToDeadTransition PASSED

kafka.coordinator.group.GroupMetadataTest > testStableToStableIllegalTransition 
STARTED

kafka.coordinator.group.GroupMetadataTest > testStableToStableIllegalTransition 
PASSED

kafka.coordinator.group.GroupMetadataTest > 
testOffsetCommitFailureWithAnotherPending STARTED

kafka.coordinator.group.GroupMetadataTest > 
testOffsetCommitFailureWithAnotherPending PASSED

kafka.coordinator.group.GroupMetadataTest > testSubscribedTopics STARTED

kafka.coordinator.group.GroupMetadataTest > testSubscribedTopics PASSED

kafka.coordinator.group.GroupMetadataTest > testDeadToStableIllegalTransition 
STARTED

kafka.coordinator.group.GroupMetadataTest > testDeadToStableIllegalTransition 
PASSED

kafka.coordinator.group.GroupMetadataTest > testOffsetCommit STARTED

kafka.coordinator.group.GroupMetadataTest > testOffsetCommit PASSED

kafka.coordinator.group.GroupMetadataTest > 
testAwaitingRebalanceToStableTransition STARTED

kafka.coordinator.group.GroupMetadataTest > 
testAwaitingRebalanceToStableTransition PASSED

kafka.coordinator.group.GroupMetadataTest > testSupportsProtocols STARTED

kafka.coordinator.group.GroupMetadataTest > testSupportsProtocols PASSED

kafka.coordinator.group.GroupMetadataTest > testEmptyToStableIllegalTransition 
STARTED

kafka.coordinator.group.GroupMetadataTest > testEmptyToStableIllegalTransition 
PASSED

kafka.coordinator.group.GroupMetadataTest > testCanRebalanceWhenStable STARTED

kafka.coordinator.group.GroupMetadataTest > testCanRebalanceWhenStable PASSED

kafka.coordinator.group.GroupMetadataTest > testNotInvokeSyncCallback STARTED

kafka.coordinator.group.GroupMetadataTest > testNotInvokeSyncCallback PASSED

kafka.coordinator.group.GroupMetadataTest > testOffsetCommitWithAnotherPending 
STARTED

kafka.coordinator.group.GroupMetadataTest > testOffsetCommitWithAnotherPending 
PASSED

kafka.coordinator.group.GroupMetadataTest > 
testReplaceGroupInstanceWithEmptyGroupInstanceId STARTED

kafka.coordinator.group.GroupMetadataTest > 
testReplaceGroupInstanceWithEmptyGroupInstanceId PASSED

kafka.coordinator.group.GroupMetadataTest > 
testPreparingRebalanceToPreparingRebalanceIllegalTransition STARTED

kafka.coordinator.group.GroupMetadataTest > 
testPreparingRebalanceToPreparingRebalanceIllegalTransition PASSED

kafka.coordinator.group.GroupMetadataTest > testInvokeSyncCallback STARTED

Build failed in Jenkins: kafka-trunk-jdk8 #4264

2020-02-23 Thread Apache Jenkins Server

See 


Changes:

[github] HOTFIX: fix NPE in Kafka Streams IQ (#8158)


--
[...truncated 2.90 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task

[jira] [Created] (KAFKA-9597) Flaky Test KafkaProducerTest#testInitTransactionTimeout

2020-02-23 Thread Matthias J. Sax (Jira)

Matthias J. Sax created KAFKA-9597:
--

 Summary: Flaky Test KafkaProducerTest#testInitTransactionTimeout
 Key: KAFKA-9597
 URL: https://issues.apache.org/jira/browse/KAFKA-9597
 Project: Kafka
  Issue Type: Bug
  Components: producer , unit tests
Reporter: Matthias J. Sax


[https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/4866/testReport/junit/org.apache.kafka.clients.producer/KafkaProducerTest/testInitTransactionTimeout/]
{quote}java.lang.IllegalStateException: No requests pending for inbound 
response FindCoordinatorResponseData(throttleTimeMs=0, errorCode=0, 
errorMessage='NONE', nodeId=0, host='host1', port=1000) at 
org.apache.kafka.clients.MockClient.respond(MockClient.java:345) at 
org.apache.kafka.clients.MockClient.respond(MockClient.java:319) at 
org.apache.kafka.clients.producer.KafkaProducerTest.testInitTransactionTimeout(KafkaProducerTest.java:770){quote}
STDOUT:
{quote}[2020-02-22 08:52:04,203] ERROR [Producer clientId=producer-9] Uncaught 
error in kafka producer I/O thread: 
(org.apache.kafka.clients.producer.internals.Sender:241) 
java.lang.IllegalStateException: No previous metadata update to use at 
org.apache.kafka.clients.MockClient$DefaultMockMetadataUpdater.updateWithCurrentMetadata(MockClient.java:623)
 at org.apache.kafka.clients.MockClient.poll(MockClient.java:287) at 
org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:324) at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) at 
java.base/java.lang.Thread.run(Thread.java:834) [2020-02-22 08:52:04,863] ERROR 
[Producer clientId=producer-10] Interrupted while joining ioThread 
(org.apache.kafka.clients.producer.KafkaProducer:1197) 
java.lang.InterruptedException at java.base/java.lang.Object.wait(Native 
Method) at java.base/java.lang.Thread.join(Thread.java:1313) at 
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1194) 
at 
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1171) 
at 
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1147) 
at 
org.apache.kafka.clients.producer.KafkaProducerTest.lambda$shouldCloseProperlyAndThrowIfInterrupted$1(KafkaProducerTest.java:295)
 at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:834) [2020-02-22 08:52:05,343] 
ERROR Metadata response reported invalid topics [topic abc] 
(org.apache.kafka.clients.Metadata:295){quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (KAFKA-9441) Refactor commit logic

2020-02-23 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-9441:


The first PR was just part of this ticket. We still need to change the commit 
logic for eos-beta to commit all tasks at once (and not individually as done 
atm).

> Refactor commit logic
> -
>
> Key: KAFKA-9441
> URL: https://issues.apache.org/jira/browse/KAFKA-9441
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Major
>
> Using producer per thread in combination with EOS, it's not possible any 
> longer to commit individual task independently (as done currently).
> We need to refactor StreamsThread, to commit all tasks at the same time for 
> the new model.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk8 #4265

Issue in retention with compact,delete cleanup policy

[jira] [Created] (KAFKA-9598) RocksDB exception when grouping dynamically appearing topics into a KTable

[jira] [Resolved] (KAFKA-9541) Flaky Test DescribeConsumerGroupTest#testDescribeGroupMembersWithShortInitializationTimeout

[jira] [Resolved] (KAFKA-9530) Flaky Test kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout

Re: [Discuss] KIP-571: Add option to force remove members in StreamsResetter

Jenkins build is back to normal : kafka-trunk-jdk11 #1184

Build failed in Jenkins: kafka-2.5-jdk8 #41

Build failed in Jenkins: kafka-2.4-jdk8 #151

Build failed in Jenkins: kafka-trunk-jdk8 #4264

[jira] [Created] (KAFKA-9597) Flaky Test KafkaProducerTest#testInitTransactionTimeout

[jira] [Reopened] (KAFKA-9441) Refactor commit logic

12 matches

Site Navigation

Mail list logo

Footer information