[jira] [Resolved] (KAFKA-15126) Change range queries to accept null lower and upper bounds
[ https://issues.apache.org/jira/browse/KAFKA-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-15126. - Resolution: Fixed [Merged to trunk|https://github.com/apache/kafka/pull/14137] > Change range queries to accept null lower and upper bounds > -- > > Key: KAFKA-15126 > URL: https://issues.apache.org/jira/browse/KAFKA-15126 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Lucia Cerchie >Assignee: Lucia Cerchie >Priority: Minor > Fix For: 3.6.0 > > Original Estimate: 672h > Remaining Estimate: 672h > > {color:#1d1c1d}When web client requests come in with query params, it's > common for those params to be null. We want developers to just be able to > pass in the upper/lower bounds if they want instead of implementing their own > logic to avoid getting the whole range (which will happen if they leave the > params null). {color} > {color:#1d1c1d}An example of the logic they can avoid using after this KIP is > implemented is below:{color} > {code:java} > private RangeQuery> > createRangeQuery(String lower, String upper) { > if (isBlank(lower) && isBlank(upper)) { > return RangeQuery.withNoBounds(); > } else if (!isBlank(lower) && isBlank(upper)) { > return RangeQuery.withLowerBound(lower); > } else if (isBlank(lower) && !isBlank(upper)) { > return RangeQuery.withUpperBound(upper); > } else { > return RangeQuery.withRange(lower, upper); > } > } {code} > > | | -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14539) Simplify StreamsMetadataState by replacing the Cluster metadata with partition info map
[ https://issues.apache.org/jira/browse/KAFKA-14539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-14539. - Resolution: Fixed > Simplify StreamsMetadataState by replacing the Cluster metadata with > partition info map > --- > > Key: KAFKA-14539 > URL: https://issues.apache.org/jira/browse/KAFKA-14539 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Assignee: Danica Fine >Priority: Major > > We can clean up the StreamsMetadataState class a bit by removing the > #onChange invocation that currently occurs within > StreamsPartitionAssignor#assign, which then lets us remove the `Cluster` > parameter in that callback. Instead of building a fake Cluster object from > the map of partition info when we invoke #onChange inside the > StreamsPartitionAssignor#onAssignment method, we can just directly pass in > the `Map` and replace the usage of `Cluster` > everywhere in StreamsMetadataState > (I believe the current system is a historical artifact from when we used to > require passing in a {{Cluster}} for the default partitioning strategy, which > the StreamMetadataState needs to compute the partition for a key. At some > point in the past we provided a better way to get the default partition, so > we no longer need a {{Cluster}} parameter/field at all) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14609) Kafka Streams Processor API cannot use state stores
[ https://issues.apache.org/jira/browse/KAFKA-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-14609. - Resolution: Fixed Fixed by https://issues.apache.org/jira/browse/KAFKA-14388 > Kafka Streams Processor API cannot use state stores > --- > > Key: KAFKA-14609 > URL: https://issues.apache.org/jira/browse/KAFKA-14609 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.3.0 >Reporter: Philipp Schirmer >Priority: Major > > The recently introduced Kafka Streams Processor API (since 3.3, > https://issues.apache.org/jira/browse/KAFKA-13654) likely has a bug with > regards to using state stores. The > [getStateStore|https://javadoc.io/static/org.apache.kafka/kafka-streams/3.3.1/org/apache/kafka/streams/processor/api/ProcessingContext.html#getStateStore-java.lang.String-] > method returns null, even though the store has been registered according to > the docs. The old transformer API still works. I created a small project that > demonstrates the behavior. It uses both methods to register a store for the > transformer, as well as the processor API: > https://github.com/bakdata/kafka-streams-state-store-demo/blob/main/src/test/java/com/bakdata/kafka/StreamsStateStoreTest.java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14388) NPE When Retrieving StateStore with new Processor API
[ https://issues.apache.org/jira/browse/KAFKA-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-14388. - Resolution: Fixed Merged PR to trunk > NPE When Retrieving StateStore with new Processor API > - > > Key: KAFKA-14388 > URL: https://issues.apache.org/jira/browse/KAFKA-14388 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.3.0, 3.3.1 >Reporter: Bill Bejeck >Assignee: Bill Bejeck >Priority: Major > Fix For: 3.4.0, 3.3.2 > > > Using the new Processor API introduced with KIP-820 when adding a state store > to the Processor when executing `context().getStore("store-name")` always > returns `null` as the store is not in the `stores` `HashMap` in the > `ProcessorStateManager`. This occurs even when using the > `ConnectedStoreProvider.stores()` method > I've confirmed the store is associated with the processor by viewing the > `Topology` description. > From some initial triage, it looks like the store is never registered. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14388) NPE When Retrieving StateStore with new Processor API
Bill Bejeck created KAFKA-14388: --- Summary: NPE When Retrieving StateStore with new Processor API Key: KAFKA-14388 URL: https://issues.apache.org/jira/browse/KAFKA-14388 Project: Kafka Issue Type: Bug Affects Versions: 3.3.1, 3.3.0 Reporter: Bill Bejeck Assignee: Bill Bejeck Fix For: 3.4.0 Using the new Processor API introduced with KIP-820 when adding a state store to the Processor when executing `context().getStore("store-name")` always returns `null` as the store is not in the `stores` `HashMap` in the `ProcessorStateManager`. This occurs even when using the `ConnectedStoreProvider.stores()` method I've confirmed the store is associated with the processor by viewing the `Topology` description. >From some initial triage, it looks like the store is never registered. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-8659) SetSchemaMetadata SMT fails on records with null value and schema
[ https://issues.apache.org/jira/browse/KAFKA-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8659. Resolution: Fixed Resovled via https://github.com/apache/kafka/pull/7082 > SetSchemaMetadata SMT fails on records with null value and schema > - > > Key: KAFKA-8659 > URL: https://issues.apache.org/jira/browse/KAFKA-8659 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Marc Löhe >Priority: Minor > Fix For: 3.2.0 > > > If you use the {{SetSchemaMetadata}} SMT with records for which the key or > value and corresponding schema are {{null}} (i.e. tombstone records from > [Debezium|[https://debezium.io/]), the transform will fail. > {code:java} > org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error > handler > at > org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178) > at > org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104) > at > org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:50) > at > org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:293) > at > org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:229) > at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175) > at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.kafka.connect.errors.DataException: Schema required for > [updating schema metadata] > at > org.apache.kafka.connect.transforms.util.Requirements.requireSchema(Requirements.java:31) > at > org.apache.kafka.connect.transforms.SetSchemaMetadata.apply(SetSchemaMetadata.java:67) > at > org.apache.kafka.connect.runtime.TransformationChain.lambda$apply$0(TransformationChain.java:50) > at > org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128) > at > org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162) > ... 11 more > {code} > > I don't see any problem in passing those records as is in favor of failing > and will shortly add this in a PR. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (KAFKA-12336) custom stream naming does not work while calling stream[K, V](topicPattern: Pattern) API with named Consumed parameter
[ https://issues.apache.org/jira/browse/KAFKA-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-12336. - Resolution: Fixed > custom stream naming does not work while calling stream[K, V](topicPattern: > Pattern) API with named Consumed parameter > --- > > Key: KAFKA-12336 > URL: https://issues.apache.org/jira/browse/KAFKA-12336 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.7.0 >Reporter: Ramil Israfilov >Assignee: GeordieMai >Priority: Minor > Labels: easy-fix, newbie > > In our Scala application I am trying to implement custom naming for Kafka > Streams application nodes. > We are using topicPattern for our stream source. > Here is an API which I am calling: > > {code:java} > val topicsPattern="t-[A-Za-z0-9-].suffix" > val operations: KStream[MyKey, MyValue] = > builder.stream[MyKey, MyValue](Pattern.compile(topicsPattern))( > Consumed.`with`[MyKey, MyValue].withName("my-fancy-name") > ) > {code} > Despite the fact that I am providing Consumed with custom name the topology > describe still show "KSTREAM-SOURCE-00" as name for our stream source. > It is not a problem if I just use a name for topic. But our application needs > to get messages from set of topics based on topicname pattern matching. > After checking the kakfa code I see that > org.apache.kafka.streams.kstream.internals.InternalStreamBuilder (on line > 103) has a bug: > {code:java} > public KStream stream(final Pattern topicPattern, >final ConsumedInternal consumed) { > final String name = newProcessorName(KStreamImpl.SOURCE_NAME); > final StreamSourceNode streamPatternSourceNode = new > StreamSourceNode<>(name, topicPattern, consumed); > {code} > node name construction does not take into account the name of consumed > parameter. > For example code for another stream api call with topic name does it > correctly: > {code:java} > final String name = new > NamedInternal(consumed.name()).orElseGenerateWithPrefix(this, > KStreamImpl.SOURCE_NAME); > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12672) Running test-kraft-server-start results in error
Bill Bejeck created KAFKA-12672: --- Summary: Running test-kraft-server-start results in error Key: KAFKA-12672 URL: https://issues.apache.org/jira/browse/KAFKA-12672 Project: Kafka Issue Type: Bug Reporter: Bill Bejeck Assignee: Bill Bejeck Running the {{test-kraft-server-start}} script in the {{raft}} module results in this error {code:java} ERROR Exiting raft server due to fatal exception (kafka.tools.TestRaftServer$) java.lang.IllegalArgumentException: No enum constant org.apache.kafka.common.security.auth.SecurityProtocol. at java.lang.Enum.valueOf(Enum.java:238) at org.apache.kafka.common.security.auth.SecurityProtocol.valueOf(SecurityProtocol.java:26) at org.apache.kafka.common.security.auth.SecurityProtocol.forName(SecurityProtocol.java:72) at kafka.raft.KafkaRaftManager.$anonfun$buildNetworkClient$1(RaftManager.scala:256) at scala.collection.immutable.Map$Map4.getOrElse(Map.scala:530) at kafka.raft.KafkaRaftManager.buildNetworkClient(RaftManager.scala:256) at kafka.raft.KafkaRaftManager.buildNetworkChannel(RaftManager.scala:234) at kafka.raft.KafkaRaftManager.(RaftManager.scala:126) at kafka.tools.TestRaftServer.startup(TestRaftServer.scala:88) at kafka.tools.TestRaftServer$.main(TestRaftServer.scala:442) at kafka.tools.TestRaftServer.main(TestRaftServer.scala) {code} Looks like the listener property in the config is not getting picked up -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12393) Document multi-tenancy considerations
[ https://issues.apache.org/jira/browse/KAFKA-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-12393. - Resolution: Fixed Merged to trunk and cherry-picked to 2.8 > Document multi-tenancy considerations > - > > Key: KAFKA-12393 > URL: https://issues.apache.org/jira/browse/KAFKA-12393 > Project: Kafka > Issue Type: Bug > Components: documentation >Reporter: Michael G. Noll >Assignee: Michael G. Noll >Priority: Minor > Fix For: 3.0.0, 2.8.0 > > > We should provide an overview of multi-tenancy consideration (e.g., user > spaces, security) as the current documentation lacks such information. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KAFKA-10140) Incremental config api excludes plugin config changes
[ https://issues.apache.org/jira/browse/KAFKA-10140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck reopened KAFKA-10140: - I resolved this by mistake, reopening now > Incremental config api excludes plugin config changes > - > > Key: KAFKA-10140 > URL: https://issues.apache.org/jira/browse/KAFKA-10140 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Priority: Critical > Fix For: 2.7.0 > > > I was trying to alter the jmx metric filters using the incremental alter > config api and hit this error: > ``` > java.util.NoSuchElementException: key not found: metrics.jmx.blacklist > at scala.collection.MapLike.default(MapLike.scala:235) > at scala.collection.MapLike.default$(MapLike.scala:234) > at scala.collection.AbstractMap.default(Map.scala:65) > at scala.collection.MapLike.apply(MapLike.scala:144) > at scala.collection.MapLike.apply$(MapLike.scala:143) > at scala.collection.AbstractMap.apply(Map.scala:65) > at kafka.server.AdminManager.listType$1(AdminManager.scala:681) > at > kafka.server.AdminManager.$anonfun$prepareIncrementalConfigs$1(AdminManager.scala:693) > at > kafka.server.AdminManager.prepareIncrementalConfigs(AdminManager.scala:687) > at > kafka.server.AdminManager.$anonfun$incrementalAlterConfigs$1(AdminManager.scala:618) > at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:273) > at scala.collection.immutable.Map$Map1.foreach(Map.scala:154) > at scala.collection.TraversableLike.map(TraversableLike.scala:273) > at scala.collection.TraversableLike.map$(TraversableLike.scala:266) > at scala.collection.AbstractTraversable.map(Traversable.scala:108) > at > kafka.server.AdminManager.incrementalAlterConfigs(AdminManager.scala:589) > at > kafka.server.KafkaApis.handleIncrementalAlterConfigsRequest(KafkaApis.scala:2698) > at kafka.server.KafkaApis.handle(KafkaApis.scala:188) > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:78) > at java.base/java.lang.Thread.run(Thread.java:834) > ``` > It looks like we are only allowing changes to the keys defined in > `KafkaConfig` through this API. This excludes config changes to any plugin > components such as `JmxReporter`. > Note that I was able to use the regular `alterConfig` API to change this > config. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10799) AlterIsr path does not update ISR shrink/expand meters
[ https://issues.apache.org/jira/browse/KAFKA-10799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-10799. - Resolution: Fixed > AlterIsr path does not update ISR shrink/expand meters > -- > > Key: KAFKA-10799 > URL: https://issues.apache.org/jira/browse/KAFKA-10799 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Assignee: David Arthur >Priority: Blocker > Fix For: 2.7.0 > > > We forgot to update the ISR change metrics when we added support for > AlterIsr. These are currently only updated when ISR changes are made through > Zk. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10687) Produce request should be bumped for new error code PRODUCE_FENCED
[ https://issues.apache.org/jira/browse/KAFKA-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-10687. - Resolution: Fixed Fixed for 2.7 via https://github.com/apache/kafka/pull/9613 > Produce request should be bumped for new error code PRODUCE_FENCED > -- > > Key: KAFKA-10687 > URL: https://issues.apache.org/jira/browse/KAFKA-10687 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Blocker > Fix For: 2.7.0 > > > In https://issues.apache.org/jira/browse/KAFKA-9910, we missed a case where > the ProduceRequest needs to be bumped to return the new error code > PRODUCE_FENCED. This gap needs to be addressed as a blocker since it is > shipping in 2.7. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10679) AK documentation changes need to get ported to Kafka/docs
Bill Bejeck created KAFKA-10679: --- Summary: AK documentation changes need to get ported to Kafka/docs Key: KAFKA-10679 URL: https://issues.apache.org/jira/browse/KAFKA-10679 Project: Kafka Issue Type: Bug Components: docs Affects Versions: 2.7.0 Reporter: Bill Bejeck During the update of the Apache Kafka website, changes made to the kafka-site repo were not made to the kafka/docs directory. All the changes made need to get migrated to kafka/docs to keep the website in sync. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9381) Javadocs + Scaladocs not published on maven central
[ https://issues.apache.org/jira/browse/KAFKA-9381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-9381. Resolution: Fixed Resolved via [https://github.com/apache/kafka/pull/9486.] Merged to trunk and cherry-picked to 2.7 > Javadocs + Scaladocs not published on maven central > --- > > Key: KAFKA-9381 > URL: https://issues.apache.org/jira/browse/KAFKA-9381 > Project: Kafka > Issue Type: Bug > Components: documentation, streams >Affects Versions: 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.1.1, 2.0.0, 2.0.1, 2.1.0, > 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2, 2.4.0, 2.3.1, 2.5.0, 2.4.1 >Reporter: Julien Jean Paul Sirocchi >Assignee: Bill Bejeck >Priority: Blocker > Fix For: 2.7.0 > > > As per title, empty (aside for MANIFEST, LICENCE and NOTICE) > javadocs/scaladocs jars on central for any version (kafka nor scala), e.g. > [http://repo1.maven.org/maven2/org/apache/kafka/kafka-streams-scala_2.12/2.3.1/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10454) Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[ https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-10454. - Resolution: Fixed Resolved via https://github.com/apache/kafka/pull/9237 > Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join > partitions don't match > - > > Key: KAFKA-10454 > URL: https://issues.apache.org/jira/browse/KAFKA-10454 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Assignee: Levani Kokhreidze >Priority: Major > Fix For: 2.7.0, 2.6.1, 2.8.0 > > > Here's integration test: [https://github.com/apache/kafka/pull/9237] > > From the first glance, issue is that when one joins stream to table, and > table source topic doesn't have same number of partitions as stream topic, > `StateChangelogReader` tries to recover state from changelog (which in this > case is the same as source topic) for table from partitions that don't exist. > Logs are spammed with: > > {code:java} > [2020-09-01 12:33:07,508] INFO stream-thread > [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] > End offset for changelog > topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; > will retry in the next time. > (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) > [2020-09-01 12:33:07,508] INFO stream-thread > [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] > End offset for changelog > topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; > will retry in the next time. > (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) > [2020-09-01 12:33:07,508] INFO stream-thread > [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] > End offset for changelog > topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; > will retry in the next time. > (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) > [2020-09-01 12:33:07,510] INFO stream-thread > [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] > End offset for changelog > topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; > will retry in the next time. > (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) > [2020-09-01 12:33:07,510] INFO stream-thread > [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] > End offset for changelog > topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; > will retry in the next time. > (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) > [2020-09-01 12:33:07,510] INFO stream-thread > [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] > End offset for changelog > topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; > will retry in the next time. > (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) > [2020-09-01 12:33:07,510] INFO stream-thread > [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] > End offset for changelog > topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; > will retry in the next time. > (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) > [2020-09-01 12:33:07,513] INFO stream-thread > [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] > End offset for changelog > topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; > will retry in the next time. > (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) > [2020-09-01 12:33:07,513] INFO stream-thread > [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] > End offset for changelog > topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; > will retry in the next time. > (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) > [2020-09-01 12:33:07,513] INFO stream-thread > [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] > End offset for changelog > topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; > will retry in the next time. > (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) > [2020-09-01
[jira] [Resolved] (KAFKA-8630) Unit testing a streams processor with a WindowStore throws a ClassCastException
[ https://issues.apache.org/jira/browse/KAFKA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8630. Resolution: Fixed Resolved via [https://github.com/apache/kafka/pull/8927.|https://github.com/apache/kafka/pull/8927] Duplicated ticket of https://issues.apache.org/jira/browse/KAFKA-10200 > Unit testing a streams processor with a WindowStore throws a > ClassCastException > --- > > Key: KAFKA-8630 > URL: https://issues.apache.org/jira/browse/KAFKA-8630 > Project: Kafka > Issue Type: Bug > Components: streams-test-utils >Affects Versions: 2.3.0 >Reporter: Justin Fetherolf >Assignee: John Roesler >Priority: Critical > Fix For: 2.7.0, 2.6.1 > > > I was attempting to write a unit test for a class implementing the > {{Processor}} interface that contained a {{WindowStore}}, but running the > test fails with a {{ClassCastException}} coming out of > {{InMemoryWindowStore.init}} attempting to cast {{MockProcessorContext}} to > {{InternalProcessorContext}}. > Minimal code to reproduce: > {code:java} > package com.cantgetthistowork; > import org.apache.kafka.streams.processor.Processor; > import org.apache.kafka.streams.processor.ProcessorContext; > import org.apache.kafka.streams.state.WindowStore; > public class InMemWindowProcessor implements Processor { > private ProcessorContext context; > private WindowStore windowStore; > @Override > public void init(ProcessorContext context) { > this.context = context; > windowStore = (WindowStore) > context.getStateStore("my-win-store"); > } > @Override > public void process(String key, String value) { > } > @Override > public void close() { > } > } > {code} > {code:java} > package com.cantgetthistowork; > import java.time.Duration; > import java.time.Instant; > import org.apache.kafka.common.serialization.Serdes; > import org.apache.kafka.streams.processor.MockProcessorContext; > import org.apache.kafka.streams.state.Stores; > import org.apache.kafka.streams.state.WindowStore; > import org.junit.Before; > import org.junit.Test; > public class InMemWindowProcessorTest { > InMemWindowProcessor processor = null; > MockProcessorContext context = null; > @Before > public void setup() { > processor = new InMemWindowProcessor(); > context = new MockProcessorContext(); > WindowStore store = > Stores.windowStoreBuilder( > Stores.inMemoryWindowStore( > "my-win-store", > Duration.ofMinutes(10), > Duration.ofSeconds(10), > false > ), > Serdes.String(), > Serdes.String() > ) > .withLoggingDisabled() > .build(); > store.init(context, store); > context.register(store, null); > processor.init(context); > } > @Test > public void testThings() { > Instant baseTime = Instant.now(); > context.setTimestamp(baseTime.toEpochMilli()); > context.setTopic("topic-name"); > processor.process("key1", "value1"); > } > } > {code} > > I was trying this with maven, with mvn --version outputting: > {noformat} > Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; > 2017-04-03T13:39:06-06:00) > Maven home: ~/opt/apache-maven-3.5.0 > Java version: 1.8.0_212, vendor: Oracle Corporation > Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "4.15.0-52-generic", arch: "amd64", family: > "unix"{noformat} > And finally the stack trace: > {noformat} > --- > T E S T S > --- > Running com.cantgetthistowork.InMemWindowProcessorTest > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.076 sec <<< > FAILURE! > testThings(com.cantgetthistowork.InMemWindowProcessorTest) Time elapsed: > 0.05 sec <<< ERROR! > java.lang.ClassCastException: > org.apache.kafka.streams.processor.MockProcessorContext cannot be cast to > org.apache.kafka.streams.processor.internals.InternalProcessorContext > at > org.apache.kafka.streams.state.internals.InMemoryWindowStore.init(InMemoryWindowStore.java:91) > at > org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48) > at > org.apache.kafka.streams.state.internals.MeteredWindowStore.init(MeteredWindowStore.java:90) > at > com.cantgetthistowork.InMemWindowProcessorTest.setup(InMemWindowProcessorTest.java:36) > at
[jira] [Resolved] (KAFKA-9929) Support reverse iterator on WindowStore
[ https://issues.apache.org/jira/browse/KAFKA-9929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-9929. Resolution: Fixed Resolved via !https://github.com/favicon.ico|width=16,height=16! [GitHub Pull Request #9137|https://github.com/apache/kafka/pull/9137] !https://github.com/favicon.ico|width=16,height=16! [GitHub Pull Request #9138|https://github.com/apache/kafka/pull/9138] !https://github.com/favicon.ico|width=16,height=16! [GitHub Pull Request #9139|https://github.com/apache/kafka/pull/9139] !https://github.com/favicon.ico|width=16,height=16! [GitHub Pull Request #9321|https://github.com/apache/kafka/pull/9321] > Support reverse iterator on WindowStore > --- > > Key: KAFKA-9929 > URL: https://issues.apache.org/jira/browse/KAFKA-9929 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Jorge Esteban Quilcate Otoya >Assignee: Jorge Esteban Quilcate Otoya >Priority: Major > Labels: needs-kip > > Currently, WindowStore fetch operations return an iterator sorted from > earliest to latest result: > ``` > * For each key, the iterator guarantees ordering of windows, starting from > the oldest/earliest > * available window to the newest/latest window. > ``` > > We have a use-case where traces are stored in a WindowStore > and use Kafka Streams to create a materialized view of traces. A query > request comes with a time range (e.g. now-1h, now) and want to return the > most recent results, i.e. fetch from this period of time, iterate and pattern > match latest/most recent traces, and if enough results, then reply without > moving further on the iterator. > Same store is used to search for previous traces. In this case, it search a > key for the last day, if found traces, we would also like to iterate from the > most recent. > RocksDb seems to support iterating backward and forward: > [https://github.com/facebook/rocksdb/wiki/Iterator#iterating-upper-bound-and-lower-bound] > > For reference: This in some way extracts some bits from this previous issue: > https://issues.apache.org/jira/browse/KAFKA-4212: > > > The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via > > segment dropping, but it stores multiple items per key, based on their > > timestamp. But this store can be repurposed as a cache by fetching the > > items in reverse chronological order and returning the first item found. > > Would like to know if there is any impediment on RocksDb or WindowStore to > support this. > Adding an argument to reverse in current fetch methods would be great: > ``` > WindowStore.fetch(from,to,Direction.BACKWARD|FORWARD) > ``` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10044) Deprecate ConsumerConfig#addDeserializerToConfig and ProducerConfig#addSerializerToConfig
[ https://issues.apache.org/jira/browse/KAFKA-10044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-10044. - Resolution: Fixed Resolved via https://github.com/apache/kafka/pull/9013 > Deprecate ConsumerConfig#addDeserializerToConfig and > ProducerConfig#addSerializerToConfig > - > > Key: KAFKA-10044 > URL: https://issues.apache.org/jira/browse/KAFKA-10044 > Project: Kafka > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: need-kip > > from [~ijuma] suggestion > (https://github.com/apache/kafka/pull/8605#discussion_r430431086) > {quote} > I think you could submit a KIP for the deprecation of the two methods in this > class, but we can merge the other changes in the meantime. > {quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10509) Add metric to track throttle time due to hitting connection rate quota
[ https://issues.apache.org/jira/browse/KAFKA-10509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-10509. - Resolution: Fixed Resolved via [https://github.com/apache/kafka/pull/9317] merged on 9/28. > Add metric to track throttle time due to hitting connection rate quota > -- > > Key: KAFKA-10509 > URL: https://issues.apache.org/jira/browse/KAFKA-10509 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Anna Povzner >Assignee: Anna Povzner >Priority: Major > Fix For: 2.7.0 > > > See KIP-612. > > kafka.network:type=socket-server-metrics,name=connection-accept-throttle-time,listener=\{listenerName} > * Type: SampledStat.Avg > * Description: Average throttle time due to violating per-listener or > broker-wide connection acceptance rate quota on a given listener. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-6078) Investigate failure of ReassignPartitionsClusterTest.shouldExpandCluster
[ https://issues.apache.org/jira/browse/KAFKA-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-6078. Resolution: Fixed > Investigate failure of ReassignPartitionsClusterTest.shouldExpandCluster > > > Key: KAFKA-6078 > URL: https://issues.apache.org/jira/browse/KAFKA-6078 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.3.0 >Reporter: Dong Lin >Priority: Major > Labels: flaky-test > Fix For: 2.7.0 > > > See https://github.com/apache/kafka/pull/4084 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8940) Flaky Test SmokeTestDriverIntegrationTest.shouldWorkWithRebalance
[ https://issues.apache.org/jira/browse/KAFKA-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8940. Resolution: Fixed > Flaky Test SmokeTestDriverIntegrationTest.shouldWorkWithRebalance > - > > Key: KAFKA-8940 > URL: https://issues.apache.org/jira/browse/KAFKA-8940 > Project: Kafka > Issue Type: Bug > Components: streams, unit tests >Reporter: Guozhang Wang >Assignee: Guozhang Wang >Priority: Major > Labels: flaky-test > > I lost the screen shot unfortunately... it reports the set of expected > records does not match the received records. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10140) Incremental config api excludes plugin config changes
[ https://issues.apache.org/jira/browse/KAFKA-10140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-10140. - Resolution: Fixed > Incremental config api excludes plugin config changes > - > > Key: KAFKA-10140 > URL: https://issues.apache.org/jira/browse/KAFKA-10140 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Priority: Critical > Fix For: 2.7.0 > > > I was trying to alter the jmx metric filters using the incremental alter > config api and hit this error: > ``` > java.util.NoSuchElementException: key not found: metrics.jmx.blacklist > at scala.collection.MapLike.default(MapLike.scala:235) > at scala.collection.MapLike.default$(MapLike.scala:234) > at scala.collection.AbstractMap.default(Map.scala:65) > at scala.collection.MapLike.apply(MapLike.scala:144) > at scala.collection.MapLike.apply$(MapLike.scala:143) > at scala.collection.AbstractMap.apply(Map.scala:65) > at kafka.server.AdminManager.listType$1(AdminManager.scala:681) > at > kafka.server.AdminManager.$anonfun$prepareIncrementalConfigs$1(AdminManager.scala:693) > at > kafka.server.AdminManager.prepareIncrementalConfigs(AdminManager.scala:687) > at > kafka.server.AdminManager.$anonfun$incrementalAlterConfigs$1(AdminManager.scala:618) > at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:273) > at scala.collection.immutable.Map$Map1.foreach(Map.scala:154) > at scala.collection.TraversableLike.map(TraversableLike.scala:273) > at scala.collection.TraversableLike.map$(TraversableLike.scala:266) > at scala.collection.AbstractTraversable.map(Traversable.scala:108) > at > kafka.server.AdminManager.incrementalAlterConfigs(AdminManager.scala:589) > at > kafka.server.KafkaApis.handleIncrementalAlterConfigsRequest(KafkaApis.scala:2698) > at kafka.server.KafkaApis.handle(KafkaApis.scala:188) > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:78) > at java.base/java.lang.Thread.run(Thread.java:834) > ``` > It looks like we are only allowing changes to the keys defined in > `KafkaConfig` through this API. This excludes config changes to any plugin > components such as `JmxReporter`. > Note that I was able to use the regular `alterConfig` API to change this > config. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-6824) Flaky Test DynamicBrokerReconfigurationTest#testAddRemoveSslListener
[ https://issues.apache.org/jira/browse/KAFKA-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-6824. Resolution: Fixed > Flaky Test DynamicBrokerReconfigurationTest#testAddRemoveSslListener > > > Key: KAFKA-6824 > URL: https://issues.apache.org/jira/browse/KAFKA-6824 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Anna Povzner >Assignee: Rajini Sivaram >Priority: Critical > Labels: flaky-test > > Observed two failures of this test (both in PR builds) :( > > *Failure #1: (JDK 7 and Scala 2.11 )* > *17:20:49* kafka.server.DynamicBrokerReconfigurationTest > > testAddRemoveSslListener FAILED > *17:20:49* java.lang.AssertionError: expected:<10> but was:<12> > *17:20:49* at org.junit.Assert.fail(Assert.java:88) > *17:20:49* at org.junit.Assert.failNotEquals(Assert.java:834) > *17:20:49* at org.junit.Assert.assertEquals(Assert.java:645) > *17:20:49* at org.junit.Assert.assertEquals(Assert.java:631) > *17:20:49* at > kafka.server.DynamicBrokerReconfigurationTest.verifyProduceConsume(DynamicBrokerReconfigurationTest.scala:959) > *17:20:49* at > kafka.server.DynamicBrokerReconfigurationTest.verifyRemoveListener(DynamicBrokerReconfigurationTest.scala:784) > *17:20:49* at > kafka.server.DynamicBrokerReconfigurationTest.testAddRemoveSslListener(DynamicBrokerReconfigurationTest.scala:705) > > *Failure #2: (JDK 8)* > *18:46:23* kafka.server.DynamicBrokerReconfigurationTest > > testAddRemoveSslListener FAILED > *18:46:23* java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is > not the leader for that topic-partition. > *18:46:23* at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94) > *18:46:23* at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:77) > *18:46:23* at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29) > *18:46:23* at > kafka.server.DynamicBrokerReconfigurationTest.$anonfun$verifyProduceConsume$3(DynamicBrokerReconfigurationTest.scala:953) > *18:46:23* at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234) > *18:46:23* at scala.collection.Iterator.foreach(Iterator.scala:929) > *18:46:23* at scala.collection.Iterator.foreach$(Iterator.scala:929) > *18:46:23* at > scala.collection.AbstractIterator.foreach(Iterator.scala:1417) > *18:46:23* at > scala.collection.IterableLike.foreach(IterableLike.scala:71) > *18:46:23* at > scala.collection.IterableLike.foreach$(IterableLike.scala:70) > *18:46:23* at > scala.collection.AbstractIterable.foreach(Iterable.scala:54) > *18:46:23* at > scala.collection.TraversableLike.map(TraversableLike.scala:234) > *18:46:23* at > scala.collection.TraversableLike.map$(TraversableLike.scala:227) > *18:46:23* at > scala.collection.AbstractTraversable.map(Traversable.scala:104) > *18:46:23* at > kafka.server.DynamicBrokerReconfigurationTest.verifyProduceConsume(DynamicBrokerReconfigurationTest.scala:953) > *18:46:23* at > kafka.server.DynamicBrokerReconfigurationTest.verifyRemoveListener(DynamicBrokerReconfigurationTest.scala:816) > *18:46:23* at > kafka.server.DynamicBrokerReconfigurationTest.testAddRemoveSslListener(DynamicBrokerReconfigurationTest.scala:705) > *18:46:23* > *18:46:23* Caused by: > *18:46:23* > org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is > not the leader for that topic-partition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8257) Flaky Test DynamicConnectionQuotaTest#testDynamicListenerConnectionQuota
[ https://issues.apache.org/jira/browse/KAFKA-8257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8257. Resolution: Fixed > Flaky Test DynamicConnectionQuotaTest#testDynamicListenerConnectionQuota > > > Key: KAFKA-8257 > URL: https://issues.apache.org/jira/browse/KAFKA-8257 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.3.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3566/tests] > {quote}java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:392) > at kafka.server.BaseRequestTest.receiveResponse(BaseRequestTest.scala:87) > at kafka.server.BaseRequestTest.sendAndReceive(BaseRequestTest.scala:148) > at > kafka.network.DynamicConnectionQuotaTest.verifyConnection(DynamicConnectionQuotaTest.scala:229) > at > kafka.network.DynamicConnectionQuotaTest.$anonfun$testDynamicListenerConnectionQuota$4(DynamicConnectionQuotaTest.scala:133) > at > kafka.network.DynamicConnectionQuotaTest.$anonfun$testDynamicListenerConnectionQuota$4$adapted(DynamicConnectionQuotaTest.scala:133) > at scala.collection.Iterator.foreach(Iterator.scala:941) > at scala.collection.Iterator.foreach$(Iterator.scala:941) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) > at scala.collection.IterableLike.foreach(IterableLike.scala:74) > at scala.collection.IterableLike.foreach$(IterableLike.scala:73) > at scala.collection.AbstractIterable.foreach(Iterable.scala:56) > at > kafka.network.DynamicConnectionQuotaTest.testDynamicListenerConnectionQuota(DynamicConnectionQuotaTest.scala:133){quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8139) Flaky Test SaslSslAdminClientIntegrationTest#testMetadataRefresh
[ https://issues.apache.org/jira/browse/KAFKA-8139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8139. Resolution: Fixed > Flaky Test SaslSslAdminClientIntegrationTest#testMetadataRefresh > > > Key: KAFKA-8139 > URL: https://issues.apache.org/jira/browse/KAFKA-8139 > Project: Kafka > Issue Type: Bug > Components: admin, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/80/testReport/junit/kafka.api/SaslSslAdminClientIntegrationTest/testMetadataRefresh/] > {quote}org.junit.runners.model.TestTimedOutException: test timed out after > 12 milliseconds at java.lang.Object.wait(Native Method) at > java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:334) at > java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:391) at > java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:719) at > scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:379) at > scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:379) at > scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:440) > at > scala.collection.parallel.ForkJoinTasks.executeAndWaitResult(Tasks.scala:423) > at > scala.collection.parallel.ForkJoinTasks.executeAndWaitResult$(Tasks.scala:416) > at > scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:60) > at > scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult(Tasks.scala:555) > at > scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult$(Tasks.scala:555) > at > scala.collection.parallel.ExecutionContextTaskSupport.executeAndWaitResult(TaskSupport.scala:84) > at > scala.collection.parallel.ParIterableLike.foreach(ParIterableLike.scala:465) > at > scala.collection.parallel.ParIterableLike.foreach$(ParIterableLike.scala:464) > at scala.collection.parallel.mutable.ParArray.foreach(ParArray.scala:58) at > kafka.utils.TestUtils$.shutdownServers(TestUtils.scala:201) at > kafka.integration.KafkaServerTestHarness.tearDown(KafkaServerTestHarness.scala:113) > at > kafka.api.IntegrationTestHarness.tearDown(IntegrationTestHarness.scala:134) > at > kafka.api.AdminClientIntegrationTest.tearDown(AdminClientIntegrationTest.scala:87) > at > kafka.api.SaslSslAdminClientIntegrationTest.tearDown(SaslSslAdminClientIntegrationTest.scala:90){quote} > STDOUT > {quote}[2019-03-20 16:30:35,739] ERROR [KafkaServer id=0] Fatal error during > KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer:159) > java.lang.IllegalArgumentException: Could not find a 'KafkaServer' or > 'sasl_ssl.KafkaServer' entry in the JAAS configuration. System property > 'java.security.auth.login.config' is not set at > org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:133) > at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:98) at > org.apache.kafka.common.security.JaasContext.loadServerContext(JaasContext.java:70) > at > org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:121) > at > org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:85) > at kafka.network.Processor.(SocketServer.scala:694) at > kafka.network.SocketServer.newProcessor(SocketServer.scala:344) at > kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:253) > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) at > kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:252) at > kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:216) > at > kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:214) > at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) > at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at > kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:214) > at kafka.network.SocketServer.startup(SocketServer.scala:114) at > kafka.server.KafkaServer.startup(KafkaServer.scala:253) at > kafka.utils.TestUtils$.createServer(TestUtils.scala:140) at > kafka.integration.KafkaServerTestHarness.$anonfun$setUp$1(KafkaServerTestHarness.scala:101) > at scala.collection.Iterator.foreach(Iterator.scala:941) at > scala.collection.Iterator.foreach$(Iterator.scala:941) at > scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at > scala.collection.IterableLike.foreach(IterableLike.scala:74) at
[jira] [Resolved] (KAFKA-8092) Flaky Test GroupAuthorizerIntegrationTest#testSendOffsetsWithNoConsumerGroupDescribeAccess
[ https://issues.apache.org/jira/browse/KAFKA-8092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8092. Resolution: Fixed > Flaky Test > GroupAuthorizerIntegrationTest#testSendOffsetsWithNoConsumerGroupDescribeAccess > -- > > Key: KAFKA-8092 > URL: https://issues.apache.org/jira/browse/KAFKA-8092 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/64/testReport/junit/kafka.api/GroupAuthorizerIntegrationTest/testSendOffsetsWithNoConsumerGroupDescribeAccess/] > {quote}java.lang.AssertionError: Partition [__consumer_offsets,0] metadata > not propagated after 15000 ms at > kafka.utils.TestUtils$.fail(TestUtils.scala:381) at > kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:791) at > kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:880) at > kafka.utils.TestUtils$.$anonfun$createTopic$3(TestUtils.scala:318) at > kafka.utils.TestUtils$.$anonfun$createTopic$3$adapted(TestUtils.scala:317) at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at > scala.collection.immutable.Range.foreach(Range.scala:158) at > scala.collection.TraversableLike.map(TraversableLike.scala:237) at > scala.collection.TraversableLike.map$(TraversableLike.scala:230) at > scala.collection.AbstractTraversable.map(Traversable.scala:108) at > kafka.utils.TestUtils$.createTopic(TestUtils.scala:317) at > kafka.utils.TestUtils$.createOffsetsTopic(TestUtils.scala:375) at > kafka.api.AuthorizerIntegrationTest.setUp(AuthorizerIntegrationTest.scala:242){quote} > STDOUT > {quote}[2019-03-11 16:08:29,319] ERROR [KafkaApi-0] Error when handling > request: clientId=0, correlationId=0, api=UPDATE_METADATA, > body=\{controller_id=0,controller_epoch=1,broker_epoch=25,topic_states=[],live_brokers=[{id=0,end_points=[{port=38324,host=localhost,listener_name=PLAINTEXT,security_protocol_type=0}],rack=null}]} > (kafka.server.KafkaApis:76) > org.apache.kafka.common.errors.ClusterAuthorizationException: Request > Request(processor=0, connectionId=127.0.0.1:38324-127.0.0.1:59458-0, > session=Session(Group:testGroup,/127.0.0.1), > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > buffer=null) is not authorized. [2019-03-11 16:08:29,933] ERROR [Consumer > clientId=consumer-99, groupId=my-group] Offset commit failed on partition > topic-0 at offset 5: Not authorized to access topics: [Topic authorization > failed.] > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:812) > [2019-03-11 16:08:29,933] ERROR [Consumer clientId=consumer-99, > groupId=my-group] Not authorized to commit to topics [topic] > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:850) > [2019-03-11 16:08:31,370] ERROR [KafkaApi-0] Error when handling request: > clientId=0, correlationId=0, api=UPDATE_METADATA, > body=\{controller_id=0,controller_epoch=1,broker_epoch=25,topic_states=[],live_brokers=[{id=0,end_points=[{port=33310,host=localhost,listener_name=PLAINTEXT,security_protocol_type=0}],rack=null}]} > (kafka.server.KafkaApis:76) > org.apache.kafka.common.errors.ClusterAuthorizationException: Request > Request(processor=0, connectionId=127.0.0.1:33310-127.0.0.1:49676-0, > session=Session(Group:testGroup,/127.0.0.1), > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > buffer=null) is not authorized. [2019-03-11 16:08:34,437] ERROR [KafkaApi-0] > Error when handling request: clientId=0, correlationId=0, > api=UPDATE_METADATA, > body=\{controller_id=0,controller_epoch=1,broker_epoch=25,topic_states=[],live_brokers=[{id=0,end_points=[{port=35999,host=localhost,listener_name=PLAINTEXT,security_protocol_type=0}],rack=null}]} > (kafka.server.KafkaApis:76) > org.apache.kafka.common.errors.ClusterAuthorizationException: Request > Request(processor=0, connectionId=127.0.0.1:35999-127.0.0.1:48268-0, > session=Session(Group:testGroup,/127.0.0.1), > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > buffer=null) is not authorized. [2019-03-11 16:08:40,978] ERROR [KafkaApi-0] > Error when handling request: clientId=0, correlationId=0, > api=UPDATE_METADATA, > body=\{controller_id=0,controller_epoch=1,broker_epoch=25,topic_states=[],live_brokers=[{id=0,end_points=[{port=38267,host=localhost,listener_name=PLAINTEXT,security_protocol_type=0}],rack=null}]} > (kafka.server.KafkaApis:76) > org.apache.kafka.common.errors.ClusterAuthorizationException: Request > Request(processor=0, connectionId=127.0.0.1:38267-127.0.0.1:53148-0, > session=Session(Group:testGroup,/127.0.0.1), >
[jira] [Resolved] (KAFKA-8076) Flaky Test ProduceRequestTest#testSimpleProduceRequest
[ https://issues.apache.org/jira/browse/KAFKA-8076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8076. Resolution: Fixed > Flaky Test ProduceRequestTest#testSimpleProduceRequest > -- > > Key: KAFKA-8076 > URL: https://issues.apache.org/jira/browse/KAFKA-8076 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/56/testReport/junit/kafka.server/ProduceRequestTest/testSimpleProduceRequest/] > {quote}java.lang.AssertionError: Partition [topic,0] metadata not propagated > after 15000 ms at kafka.utils.TestUtils$.fail(TestUtils.scala:381) at > kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:791) at > kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:880) at > kafka.utils.TestUtils$.$anonfun$createTopic$3(TestUtils.scala:318) at > kafka.utils.TestUtils$.$anonfun$createTopic$3$adapted(TestUtils.scala:317) at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at > scala.collection.immutable.Range.foreach(Range.scala:158) at > scala.collection.TraversableLike.map(TraversableLike.scala:237) at > scala.collection.TraversableLike.map$(TraversableLike.scala:230) at > scala.collection.AbstractTraversable.map(Traversable.scala:108) at > kafka.utils.TestUtils$.createTopic(TestUtils.scala:317) at > kafka.server.ProduceRequestTest.createTopicAndFindPartitionWithLeader(ProduceRequestTest.scala:91) > at > kafka.server.ProduceRequestTest.testSimpleProduceRequest(ProduceRequestTest.scala:42) > {quote} > STDOUT > {quote}[2019-03-08 01:42:24,797] ERROR [ReplicaFetcher replicaId=0, > leaderId=2, fetcherId=0] Error for partition topic-0 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-08 01:42:38,287] WARN Unable to > read additional data from client sessionid 0x100712b09280002, likely client > has closed socket (org.apache.zookeeper.server.NIOServerCnxn:376) > {quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-7647) Flaky test LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic
[ https://issues.apache.org/jira/browse/KAFKA-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-7647. Resolution: Fixed > Flaky test > LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic > - > > Key: KAFKA-7647 > URL: https://issues.apache.org/jira/browse/KAFKA-7647 > Project: Kafka > Issue Type: Sub-task > Components: core, unit tests >Affects Versions: 2.1.1, 2.3.0 >Reporter: Dong Lin >Priority: Critical > Labels: flaky-test > > {code} > kafka.log.LogCleanerParameterizedIntegrationTest > > testCleansCombinedCompactAndDeleteTopic[3] FAILED > java.lang.AssertionError: Contents of the map shouldn't change > expected: (340,340), 5 -> (345,345), 10 -> (350,350), 14 -> > (354,354), 1 -> (341,341), 6 -> (346,346), 9 -> (349,349), 13 -> (353,353), > 2 -> (342,342), 17 -> (357,357), 12 -> (352,352), 7 -> (347,347), 3 -> > (343,343), 18 -> (358,358), 16 -> (356,356), 11 -> (351,351), 8 -> > (348,348), 19 -> (359,359), 4 -> (344,344), 15 -> (355,355))> but > was: (340,340), 5 -> (345,345), 10 -> (350,350), 14 -> (354,354), > 1 -> (341,341), 6 -> (346,346), 9 -> (349,349), 13 -> (353,353), 2 -> > (342,342), 17 -> (357,357), 12 -> (352,352), 7 -> (347,347), 3 -> > (343,343), 18 -> (358,358), 16 -> (356,356), 11 -> (351,351), 99 -> > (299,299), 8 -> (348,348), 19 -> (359,359), 4 -> (344,344), 15 -> > (355,355))> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:118) > at > kafka.log.LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic(LogCleanerParameterizedIntegrationTest.scala:129) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8137) Flaky Test LegacyAdminClientTest#testOffsetsForTimesWhenOffsetNotFound
[ https://issues.apache.org/jira/browse/KAFKA-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8137. Resolution: Fixed > Flaky Test LegacyAdminClientTest#testOffsetsForTimesWhenOffsetNotFound > -- > > Key: KAFKA-8137 > URL: https://issues.apache.org/jira/browse/KAFKA-8137 > Project: Kafka > Issue Type: Bug > Components: admin, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/80/testReport/junit/kafka.api/LegacyAdminClientTest/testOffsetsForTimesWhenOffsetNotFound/] > {quote}java.lang.AssertionError: Partition [topic,0] metadata not propagated > after 15000 ms at kafka.utils.TestUtils$.fail(TestUtils.scala:381) at > kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:791) at > kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:880) at > kafka.utils.TestUtils$.$anonfun$createTopic$3(TestUtils.scala:318) at > kafka.utils.TestUtils$.$anonfun$createTopic$3$adapted(TestUtils.scala:317) at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at > scala.collection.immutable.Range.foreach(Range.scala:158) at > scala.collection.TraversableLike.map(TraversableLike.scala:237) at > scala.collection.TraversableLike.map$(TraversableLike.scala:230) at > scala.collection.AbstractTraversable.map(Traversable.scala:108) at > kafka.utils.TestUtils$.createTopic(TestUtils.scala:317) at > kafka.integration.KafkaServerTestHarness.createTopic(KafkaServerTestHarness.scala:125) > at > kafka.api.LegacyAdminClientTest.setUp(LegacyAdminClientTest.scala:73){quote} > STDOUT > {quote}[2019-03-20 16:28:10,089] ERROR [ReplicaFetcher replicaId=1, > leaderId=0, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:28:10,093] ERROR > [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Error for partition > __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:28:10,303] ERROR > [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Error for partition > topic-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:28:10,303] ERROR > [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Error for partition > topic-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:28:14,493] ERROR > [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error for partition > __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:28:14,724] ERROR > [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error for partition > topic-1 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:28:21,388] ERROR > [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error for partition > __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:28:21,394] ERROR > [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Error for partition > __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:29:48,224] ERROR > [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error for partition > __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:29:48,249] ERROR > [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Error for partition > __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:29:49,255] ERROR > [ReplicaFetcher replicaId=2, leaderId=1,
[jira] [Resolved] (KAFKA-8084) Flaky Test DescribeConsumerGroupTest#testDescribeMembersOfExistingGroupWithNoMembers
[ https://issues.apache.org/jira/browse/KAFKA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8084. Resolution: Fixed > Flaky Test > DescribeConsumerGroupTest#testDescribeMembersOfExistingGroupWithNoMembers > > > Key: KAFKA-8084 > URL: https://issues.apache.org/jira/browse/KAFKA-8084 > Project: Kafka > Issue Type: Bug > Components: admin, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/62/testReport/junit/kafka.admin/DescribeConsumerGroupTest/testDescribeMembersOfExistingGroupWithNoMembers/] > {quote}java.lang.AssertionError: Partition [__consumer_offsets,0] metadata > not propagated after 15000 ms at > kafka.utils.TestUtils$.fail(TestUtils.scala:381) at > kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:791) at > kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:880) at > kafka.utils.TestUtils$.$anonfun$createTopic$3(TestUtils.scala:318) at > kafka.utils.TestUtils$.$anonfun$createTopic$3$adapted(TestUtils.scala:317) at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at > scala.collection.immutable.Range.foreach(Range.scala:158) at > scala.collection.TraversableLike.map(TraversableLike.scala:237) at > scala.collection.TraversableLike.map$(TraversableLike.scala:230) at > scala.collection.AbstractTraversable.map(Traversable.scala:108) at > kafka.utils.TestUtils$.createTopic(TestUtils.scala:317) at > kafka.utils.TestUtils$.createOffsetsTopic(TestUtils.scala:375) at > kafka.admin.DescribeConsumerGroupTest.testDescribeMembersOfExistingGroupWithNoMembers(DescribeConsumerGroupTest.scala:283){quote} > STDOUT > {quote}TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST > CLIENT-ID foo 0 0 0 0 - - - TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG > CONSUMER-ID HOST CLIENT-ID foo 0 0 0 0 - - - COORDINATOR (ID) > ASSIGNMENT-STRATEGY STATE #MEMBERS localhost:45812 (0) Empty 0{quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8108) Flaky Test kafka.api.ClientIdQuotaTest.testThrottledProducerConsumer
[ https://issues.apache.org/jira/browse/KAFKA-8108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8108. Resolution: Fixed > Flaky Test kafka.api.ClientIdQuotaTest.testThrottledProducerConsumer > > > Key: KAFKA-8108 > URL: https://issues.apache.org/jira/browse/KAFKA-8108 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.3.0 >Reporter: Guozhang Wang >Priority: Critical > Labels: flaky-test > > {code} > java.lang.AssertionError: Client with id=QuotasTestProducer-!@#$%^&*() should > have been throttled > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.assertTrue(Assert.java:42) > at > kafka.api.QuotaTestClients.verifyThrottleTimeMetric(BaseQuotaTest.scala:229) > at > kafka.api.QuotaTestClients.verifyProduceThrottle(BaseQuotaTest.scala:215) > at > kafka.api.BaseQuotaTest.testThrottledProducerConsumer(BaseQuotaTest.scala:82) > {code} > https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/3230/testReport/junit/kafka.api/ClientIdQuotaTest/testThrottledProducerConsumer/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8303) Flaky Test SaslSslAdminClientIntegrationTest#testLogStartOffsetCheckpoint
[ https://issues.apache.org/jira/browse/KAFKA-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8303. Resolution: Fixed > Flaky Test SaslSslAdminClientIntegrationTest#testLogStartOffsetCheckpoint > - > > Key: KAFKA-8303 > URL: https://issues.apache.org/jira/browse/KAFKA-8303 > Project: Kafka > Issue Type: Bug > Components: admin, security, unit tests >Affects Versions: 2.3.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/21274/testReport/junit/kafka.api/SaslSslAdminClientIntegrationTest/testLogStartOffsetCheckpoint/] > {quote}java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.TimeoutException: Aborted due to timeout. at > org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) > at > org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) > at > org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) > at > org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) > at > kafka.api.AdminClientIntegrationTest$$anonfun$testLogStartOffsetCheckpoint$2.apply$mcZ$sp(AdminClientIntegrationTest.scala:820) > at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:789) at > kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpoint(AdminClientIntegrationTest.scala:813){quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-7988) Flaky Test DynamicBrokerReconfigurationTest#testThreadPoolResize
[ https://issues.apache.org/jira/browse/KAFKA-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-7988. Resolution: Fixed > Flaky Test DynamicBrokerReconfigurationTest#testThreadPoolResize > > > Key: KAFKA-7988 > URL: https://issues.apache.org/jira/browse/KAFKA-7988 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0, 2.3.0 >Reporter: Matthias J. Sax >Assignee: Rajini Sivaram >Priority: Critical > Labels: flaky-test > > To get stable nightly builds for `2.2` release, I create tickets for all > observed test failures. > [https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/30/] > {quote}kafka.server.DynamicBrokerReconfigurationTest > testThreadPoolResize > FAILED java.lang.AssertionError: Invalid threads: expected 6, got 5: > List(ReplicaFetcherThread-0-0, ReplicaFetcherThread-0-1, > ReplicaFetcherThread-0-0, ReplicaFetcherThread-0-2, ReplicaFetcherThread-0-1) > at org.junit.Assert.fail(Assert.java:88) at > org.junit.Assert.assertTrue(Assert.java:41) at > kafka.server.DynamicBrokerReconfigurationTest.verifyThreads(DynamicBrokerReconfigurationTest.scala:1260) > at > kafka.server.DynamicBrokerReconfigurationTest.maybeVerifyThreadPoolSize$1(DynamicBrokerReconfigurationTest.scala:531) > at > kafka.server.DynamicBrokerReconfigurationTest.resizeThreadPool$1(DynamicBrokerReconfigurationTest.scala:550) > at > kafka.server.DynamicBrokerReconfigurationTest.reducePoolSize$1(DynamicBrokerReconfigurationTest.scala:536) > at > kafka.server.DynamicBrokerReconfigurationTest.$anonfun$testThreadPoolResize$3(DynamicBrokerReconfigurationTest.scala:559) > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) at > kafka.server.DynamicBrokerReconfigurationTest.verifyThreadPoolResize$1(DynamicBrokerReconfigurationTest.scala:558) > at > kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize(DynamicBrokerReconfigurationTest.scala:572){quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8079) Flaky Test EpochDrivenReplicationProtocolAcceptanceTest#shouldSurviveFastLeaderChange
[ https://issues.apache.org/jira/browse/KAFKA-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8079. Resolution: Fixed > Flaky Test > EpochDrivenReplicationProtocolAcceptanceTest#shouldSurviveFastLeaderChange > - > > Key: KAFKA-8079 > URL: https://issues.apache.org/jira/browse/KAFKA-8079 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.3.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3445/tests] > {quote}java.lang.AssertionError > at org.junit.Assert.fail(Assert.java:87) > at org.junit.Assert.assertTrue(Assert.java:42) > at org.junit.Assert.assertTrue(Assert.java:53) > at > kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest.$anonfun$shouldSurviveFastLeaderChange$2(EpochDrivenReplicationProtocolAcceptanceTest.scala:294) > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) > at > kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest.shouldSurviveFastLeaderChange(EpochDrivenReplicationProtocolAcceptanceTest.scala:273){quote} > STDOUT > {quote}[2019-03-08 01:16:02,452] ERROR [ReplicaFetcher replicaId=101, > leaderId=100, fetcherId=0] Error for partition topic1-0 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. > [2019-03-08 01:16:23,677] ERROR [ReplicaFetcher replicaId=101, leaderId=100, > fetcherId=0] Error for partition topic1-0 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. > [2019-03-08 01:16:35,779] ERROR [Controller id=100] Error completing > preferred replica leader election for partition topic1-0 > (kafka.controller.KafkaController:76) > kafka.common.StateChangeFailedException: Failed to elect leader for partition > topic1-0 under strategy PreferredReplicaPartitionLeaderElectionStrategy > at > kafka.controller.PartitionStateMachine.$anonfun$doElectLeaderForPartitions$9(PartitionStateMachine.scala:390) > at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) > at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) > at > kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:388) > at > kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:315) > at > kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:225) > at > kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:141) > at > kafka.controller.KafkaController.kafka$controller$KafkaController$$onPreferredReplicaElection(KafkaController.scala:649) > at > kafka.controller.KafkaController.$anonfun$checkAndTriggerAutoLeaderRebalance$6(KafkaController.scala:1008) > at scala.collection.immutable.Map$Map1.foreach(Map.scala:128) > at > kafka.controller.KafkaController.kafka$controller$KafkaController$$checkAndTriggerAutoLeaderRebalance(KafkaController.scala:989) > at > kafka.controller.KafkaController$AutoPreferredReplicaLeaderElection$.process(KafkaController.scala:1020) > at > kafka.controller.ControllerEventManager$ControllerEventThread.$anonfun$doWork$1(ControllerEventManager.scala:95) > at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31) > at > kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:95) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89) > Dumping /tmp/kafka-2158669830092629415/topic1-0/.log > Starting offset: 0 > baseOffset: 0 lastOffset: 0 count: 1 baseSequence: -1 lastSequence: -1 > producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: > false isControl: false position: 0 CreateTime: 1552007783877 size: 141 magic: > 2 compresscodec: SNAPPY crc: 2264724941 isvalid: true > baseOffset: 1 lastOffset: 1 count: 1 baseSequence: -1 lastSequence: -1 > producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: > false isControl: false position: 141 CreateTime: 1552007784731 size: 141 > magic: 2 compresscodec: SNAPPY crc: 14988968 isvalid: true > baseOffset: 2 lastOffset: 2 count: 1 baseSequence: -1 lastSequence: -1 > producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: > false isControl: false position: 282 CreateTime: 1552007784734
[jira] [Resolved] (KAFKA-8113) Flaky Test ListOffsetsRequestTest#testResponseIncludesLeaderEpoch
[ https://issues.apache.org/jira/browse/KAFKA-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8113. Resolution: Fixed > Flaky Test ListOffsetsRequestTest#testResponseIncludesLeaderEpoch > - > > Key: KAFKA-8113 > URL: https://issues.apache.org/jira/browse/KAFKA-8113 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.3.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3468/tests] > {quote}java.lang.AssertionError > at org.junit.Assert.fail(Assert.java:87) > at org.junit.Assert.assertTrue(Assert.java:42) > at org.junit.Assert.assertTrue(Assert.java:53) > at > kafka.server.ListOffsetsRequestTest.fetchOffsetAndEpoch$1(ListOffsetsRequestTest.scala:136) > at > kafka.server.ListOffsetsRequestTest.testResponseIncludesLeaderEpoch(ListOffsetsRequestTest.scala:151){quote} > STDOUT > {quote}[2019-03-15 17:16:13,029] ERROR [ReplicaFetcher replicaId=2, > leaderId=1, fetcherId=0] Error for partition topic-0 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. > [2019-03-15 17:16:13,231] ERROR [KafkaApi-0] Error while responding to offset > request (kafka.server.KafkaApis:76) > org.apache.kafka.common.errors.ReplicaNotAvailableException: Partition > topic-0 is not available{quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8087) Flaky Test PlaintextConsumerTest#testConsumingWithNullGroupId
[ https://issues.apache.org/jira/browse/KAFKA-8087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8087. Resolution: Fixed > Flaky Test PlaintextConsumerTest#testConsumingWithNullGroupId > - > > Key: KAFKA-8087 > URL: https://issues.apache.org/jira/browse/KAFKA-8087 > Project: Kafka > Issue Type: Bug > Components: clients, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/62/testReport/junit/kafka.api/PlaintextConsumerTest/testConsumingWithNullGroupId/] > {quote}java.lang.AssertionError: Partition [topic,0] metadata not propagated > after 15000 ms at kafka.utils.TestUtils$.fail(TestUtils.scala:381) at > kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:791) at > kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:880) at > kafka.utils.TestUtils$.$anonfun$createTopic$3(TestUtils.scala:318) at > kafka.utils.TestUtils$.$anonfun$createTopic$3$adapted(TestUtils.scala:317) at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at > scala.collection.immutable.Range.foreach(Range.scala:158) at > scala.collection.TraversableLike.map(TraversableLike.scala:237) at > scala.collection.TraversableLike.map$(TraversableLike.scala:230) at > scala.collection.AbstractTraversable.map(Traversable.scala:108) at > kafka.utils.TestUtils$.createTopic(TestUtils.scala:317) at > kafka.integration.KafkaServerTestHarness.createTopic(KafkaServerTestHarness.scala:125) > at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:69){quote} > STDOUT > {quote}[2019-03-09 08:39:02,022] ERROR [ReplicaFetcher replicaId=1, > leaderId=2, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-09 08:39:02,022] ERROR > [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error for partition > __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-09 08:39:02,202] ERROR > [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Error for partition > topic-1 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-09 08:39:02,204] ERROR > [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Error for partition > topic-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-09 08:39:02,236] ERROR > [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error for partition > topic-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-09 08:39:02,236] ERROR > [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Error for partition > topic-1 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-09 08:39:02,511] ERROR > [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Error for partition > topicWithNewMessageFormat-1 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-09 08:39:02,512] ERROR > [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Error for partition > topicWithNewMessageFormat-1 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-09 08:39:06,568] ERROR > [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Error for partition > topic-1 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-09 08:39:09,582] ERROR > [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Error for partition > __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-09 08:39:09,787] ERROR > [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Error for partition > topic-0 at offset
[jira] [Resolved] (KAFKA-8077) Flaky Test AdminClientIntegrationTest#testConsumeAfterDeleteRecords
[ https://issues.apache.org/jira/browse/KAFKA-8077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8077. Resolution: Fixed > Flaky Test AdminClientIntegrationTest#testConsumeAfterDeleteRecords > --- > > Key: KAFKA-8077 > URL: https://issues.apache.org/jira/browse/KAFKA-8077 > Project: Kafka > Issue Type: Bug > Components: admin, unit tests >Affects Versions: 2.0.1 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/237/tests] > {quote}java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. > at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94) > at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64) > at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29) > at > kafka.api.AdminClientIntegrationTest$$anonfun$sendRecords$1.apply(AdminClientIntegrationTest.scala:994) > at > kafka.api.AdminClientIntegrationTest$$anonfun$sendRecords$1.apply(AdminClientIntegrationTest.scala:994) > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > kafka.api.AdminClientIntegrationTest.sendRecords(AdminClientIntegrationTest.scala:994) > at > kafka.api.AdminClientIntegrationTest.testConsumeAfterDeleteRecords(AdminClientIntegrationTest.scala:909) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: > This server does not host this topic-partition.{quote} > STDERR > {quote}Exception in thread "Thread-1638" > org.apache.kafka.common.errors.InterruptException: > java.lang.InterruptedException > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:504) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:287) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1247) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1187) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115) > at > kafka.api.AdminClientIntegrationTest$$anon$1.run(AdminClientIntegrationTest.scala:1132) > Caused by: java.lang.InterruptedException > ... 7 more{quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8075) Flaky Test GroupAuthorizerIntegrationTest#testTransactionalProducerTopicAuthorizationExceptionInCommit
[ https://issues.apache.org/jira/browse/KAFKA-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8075. Resolution: Fixed > Flaky Test > GroupAuthorizerIntegrationTest#testTransactionalProducerTopicAuthorizationExceptionInCommit > -- > > Key: KAFKA-8075 > URL: https://issues.apache.org/jira/browse/KAFKA-8075 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/56/testReport/junit/kafka.api/GroupAuthorizerIntegrationTest/testTransactionalProducerTopicAuthorizationExceptionInCommit/] > {quote}org.apache.kafka.common.errors.TimeoutException: Timeout expired while > initializing transactional state in 3000ms.{quote} > STDOUT > {quote}[2019-03-08 01:48:45,226] ERROR [Consumer clientId=consumer-99, > groupId=my-group] Offset commit failed on partition topic-0 at offset 5: Not > authorized to access topics: [Topic authorization failed.] > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:812) > [2019-03-08 01:48:45,227] ERROR [Consumer clientId=consumer-99, > groupId=my-group] Not authorized to commit to topics [topic] > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:850) > [2019-03-08 01:48:57,870] ERROR [KafkaApi-0] Error when handling request: > clientId=0, correlationId=0, api=UPDATE_METADATA, > body=\{controller_id=0,controller_epoch=1,broker_epoch=25,topic_states=[],live_brokers=[{id=0,end_points=[{port=43610,host=localhost,listener_name=PLAINTEXT,security_protocol_type=0}],rack=null}]} > (kafka.server.KafkaApis:76) > org.apache.kafka.common.errors.ClusterAuthorizationException: Request > Request(processor=0, connectionId=127.0.0.1:43610-127.0.0.1:44870-0, > session=Session(Group:testGroup,/127.0.0.1), > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > buffer=null) is not authorized. [2019-03-08 01:49:14,858] ERROR [KafkaApi-0] > Error when handling request: clientId=0, correlationId=0, > api=UPDATE_METADATA, > body=\{controller_id=0,controller_epoch=1,broker_epoch=25,topic_states=[],live_brokers=[{id=0,end_points=[{port=44107,host=localhost,listener_name=PLAINTEXT,security_protocol_type=0}],rack=null}]} > (kafka.server.KafkaApis:76) > org.apache.kafka.common.errors.ClusterAuthorizationException: Request > Request(processor=0, connectionId=127.0.0.1:44107-127.0.0.1:38156-0, > session=Session(Group:testGroup,/127.0.0.1), > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > buffer=null) is not authorized. [2019-03-08 01:49:21,984] ERROR [KafkaApi-0] > Error when handling request: clientId=0, correlationId=0, > api=UPDATE_METADATA, > body=\{controller_id=0,controller_epoch=1,broker_epoch=25,topic_states=[],live_brokers=[{id=0,end_points=[{port=39025,host=localhost,listener_name=PLAINTEXT,security_protocol_type=0}],rack=null}]} > (kafka.server.KafkaApis:76) > org.apache.kafka.common.errors.ClusterAuthorizationException: Request > Request(processor=0, connectionId=127.0.0.1:39025-127.0.0.1:41474-0, > session=Session(Group:testGroup,/127.0.0.1), > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > buffer=null) is not authorized. [2019-03-08 01:49:39,438] ERROR [KafkaApi-0] > Error when handling request: clientId=0, correlationId=0, > api=UPDATE_METADATA, > body=\{controller_id=0,controller_epoch=1,broker_epoch=25,topic_states=[],live_brokers=[{id=0,end_points=[{port=44798,host=localhost,listener_name=PLAINTEXT,security_protocol_type=0}],rack=null}]} > (kafka.server.KafkaApis:76) > org.apache.kafka.common.errors.ClusterAuthorizationException: Request > Request(processor=0, connectionId=127.0.0.1:44798-127.0.0.1:58496-0, > session=Session(Group:testGroup,/127.0.0.1), > listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, > buffer=null) is not authorized. Error: Consumer group 'my-group' does not > exist. [2019-03-08 01:49:55,502] WARN Ignoring unexpected runtime exception > (org.apache.zookeeper.server.NIOServerCnxnFactory:236) > java.nio.channels.CancelledKeyException at > sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at > sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87) at > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:205) > at java.lang.Thread.run(Thread.java:748) [2019-03-08 01:50:02,720] WARN > Unable to read additional data from client sessionid 0x1007131d81c0001, > likely client has closed socket > (org.apache.zookeeper.server.NIOServerCnxn:376) [2019-03-08 01:50:03,855] > ERROR [KafkaApi-0] Error when handling request:
[jira] [Resolved] (KAFKA-8138) Flaky Test PlaintextConsumerTest#testFetchRecordLargerThanFetchMaxBytes
[ https://issues.apache.org/jira/browse/KAFKA-8138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8138. Resolution: Fixed > Flaky Test PlaintextConsumerTest#testFetchRecordLargerThanFetchMaxBytes > --- > > Key: KAFKA-8138 > URL: https://issues.apache.org/jira/browse/KAFKA-8138 > Project: Kafka > Issue Type: Bug > Components: clients, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/80/testReport/junit/kafka.api/PlaintextConsumerTest/testFetchRecordLargerThanFetchMaxBytes/] > {quote}java.lang.AssertionError: Partition [topic,0] metadata not propagated > after 15000 ms at kafka.utils.TestUtils$.fail(TestUtils.scala:381) at > kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:791) at > kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:880) at > kafka.utils.TestUtils$.$anonfun$createTopic$3(TestUtils.scala:318) at > kafka.utils.TestUtils$.$anonfun$createTopic$3$adapted(TestUtils.scala:317) at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at > scala.collection.immutable.Range.foreach(Range.scala:158) at > scala.collection.TraversableLike.map(TraversableLike.scala:237) at > scala.collection.TraversableLike.map$(TraversableLike.scala:230) at > scala.collection.AbstractTraversable.map(Traversable.scala:108) at > kafka.utils.TestUtils$.createTopic(TestUtils.scala:317) at > kafka.integration.KafkaServerTestHarness.createTopic(KafkaServerTestHarness.scala:125) > at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:69){quote} > STDOUT (truncated) > {quote}[2019-03-20 16:10:19,759] ERROR [ReplicaFetcher replicaId=2, > leaderId=0, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:10:19,760] ERROR > [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Error for partition > __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:10:19,963] ERROR > [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Error for partition > topic-1 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:10:19,964] ERROR > [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Error for partition > topic-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. [2019-03-20 16:10:19,975] ERROR > [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error for partition > topic-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition.{quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8141) Flaky Test FetchRequestDownConversionConfigTest#testV1FetchWithDownConversionDisabled
[ https://issues.apache.org/jira/browse/KAFKA-8141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8141. Resolution: Fixed > Flaky Test > FetchRequestDownConversionConfigTest#testV1FetchWithDownConversionDisabled > - > > Key: KAFKA-8141 > URL: https://issues.apache.org/jira/browse/KAFKA-8141 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > Fix For: 2.7.0, 2.6.1 > > > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/80/testReport/junit/kafka.server/FetchRequestDownConversionConfigTest/testV1FetchWithDownConversionDisabled/] > {quote}java.lang.AssertionError: Partition [__consumer_offsets,0] metadata > not propagated after 15000 ms at > kafka.utils.TestUtils$.fail(TestUtils.scala:381) at > kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:791) at > kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:880) at > kafka.utils.TestUtils$.$anonfun$createTopic$3(TestUtils.scala:318) at > kafka.utils.TestUtils$.$anonfun$createTopic$3$adapted(TestUtils.scala:317) at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at > scala.collection.immutable.Range.foreach(Range.scala:158) at > scala.collection.TraversableLike.map(TraversableLike.scala:237) at > scala.collection.TraversableLike.map$(TraversableLike.scala:230) at > scala.collection.AbstractTraversable.map(Traversable.scala:108) at > kafka.utils.TestUtils$.createTopic(TestUtils.scala:317) at > kafka.utils.TestUtils$.createOffsetsTopic(TestUtils.scala:375) at > kafka.api.IntegrationTestHarness.doSetup(IntegrationTestHarness.scala:95) at > kafka.api.IntegrationTestHarness.setUp(IntegrationTestHarness.scala:73){quote} > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8269) Flaky Test TopicCommandWithAdminClientTest#testDescribeUnderMinIsrPartitionsMixed
[ https://issues.apache.org/jira/browse/KAFKA-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8269. Resolution: Duplicate > Flaky Test > TopicCommandWithAdminClientTest#testDescribeUnderMinIsrPartitionsMixed > - > > Key: KAFKA-8269 > URL: https://issues.apache.org/jira/browse/KAFKA-8269 > Project: Kafka > Issue Type: Bug > Components: admin, unit tests >Affects Versions: 2.3.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > > [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3573/tests] > {quote}java.lang.AssertionError > at org.junit.Assert.fail(Assert.java:87) > at org.junit.Assert.assertTrue(Assert.java:42) > at org.junit.Assert.assertTrue(Assert.java:53) > at > kafka.admin.TopicCommandWithAdminClientTest.testDescribeUnderMinIsrPartitionsMixed(TopicCommandWithAdminClientTest.scala:659){quote} > It's a long LOG. This might be interesting: > {quote}[2019-04-20 21:30:37,936] ERROR [ReplicaFetcher replicaId=4, > leaderId=5, fetcherId=0] Error for partition > testCreateWithReplicaAssignment-0cpsXnG35w-0 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. > [2019-04-20 21:30:48,600] WARN Unable to read additional data from client > sessionid 0x10510a59d3c0004, likely client has closed socket > (org.apache.zookeeper.server.NIOServerCnxn:376) > [2019-04-20 21:30:48,908] WARN Unable to read additional data from client > sessionid 0x10510a59d3c0003, likely client has closed socket > (org.apache.zookeeper.server.NIOServerCnxn:376) > [2019-04-20 21:30:48,919] ERROR [RequestSendThread controllerId=0] Controller > 0 fails to send a request to broker localhost:43520 (id: 5 rack: rack3) > (kafka.controller.RequestSendThread:76) > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) > at kafka.utils.ShutdownableThread.pause(ShutdownableThread.scala:75) > at > kafka.controller.RequestSendThread.backoff$1(ControllerChannelManager.scala:224) > at > kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:252) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89) > [2019-04-20 21:30:48,920] ERROR [RequestSendThread controllerId=0] Controller > 0 fails to send a request to broker localhost:33570 (id: 4 rack: rack3) > (kafka.controller.RequestSendThread:76) > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) > at kafka.utils.ShutdownableThread.pause(ShutdownableThread.scala:75) > at > kafka.controller.RequestSendThread.backoff$1(ControllerChannelManager.scala:224) > at > kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:252) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89) > [2019-04-20 21:31:28,942] ERROR [ReplicaFetcher replicaId=3, leaderId=1, > fetcherId=0] Error for partition under-min-isr-topic-0 at offset 0 > (kafka.server.ReplicaFetcherThread:76) > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server > does not host this topic-partition. > [2019-04-20 21:31:28,973] ERROR [ReplicaFetcher replicaId=0, leaderId=1, > fetcherId=0] Error for partition under-min-isr-topic-0 at offset 0 > (kafka.server.ReplicaFetcherThread:76){quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KAFKA-10017) Flaky Test EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta
[ https://issues.apache.org/jira/browse/KAFKA-10017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck reopened KAFKA-10017: - > Flaky Test EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta > --- > > Key: KAFKA-10017 > URL: https://issues.apache.org/jira/browse/KAFKA-10017 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Blocker > Labels: flaky-test, unit-test > Fix For: 2.6.0 > > > Creating a new ticket for this since the root cause is different than > https://issues.apache.org/jira/browse/KAFKA-9966 > With injectError = true: > h3. Stacktrace > java.lang.AssertionError: Did not receive all 20 records from topic > multiPartitionOutputTopic within 6 ms Expected: is a value equal to or > greater than <20> but: <15> was less than <20> at > org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at > org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinKeyValueRecordsReceived$1(IntegrationTestUtils.java:563) > at > org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:429) > at > org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:397) > at > org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:559) > at > org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:530) > at > org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.readResult(EosBetaUpgradeIntegrationTest.java:973) > at > org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.verifyCommitted(EosBetaUpgradeIntegrationTest.java:961) > at > org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta(EosBetaUpgradeIntegrationTest.java:427) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9273) Refactor AbstractJoinIntegrationTest and Sub-classes
[ https://issues.apache.org/jira/browse/KAFKA-9273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-9273. Resolution: Fixed > Refactor AbstractJoinIntegrationTest and Sub-classes > > > Key: KAFKA-9273 > URL: https://issues.apache.org/jira/browse/KAFKA-9273 > Project: Kafka > Issue Type: Test > Components: streams >Affects Versions: 2.7.0 >Reporter: Bill Bejeck >Assignee: Albert Lowis >Priority: Major > Labels: newbie > Fix For: 2.7.0 > > > The AbstractJoinIntegrationTest uses an embedded broker, but not all the > sub-classes require the use of an embedded broker anymore. Additionally, > there are two test remaining that require an embedded broker, but they don't > perform joins, the are tests validating other conditions, so ideally those > tests should move into a separate test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10405) Flaky Test org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState
Bill Bejeck created KAFKA-10405: --- Summary: Flaky Test org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState Key: KAFKA-10405 URL: https://issues.apache.org/jira/browse/KAFKA-10405 Project: Kafka Issue Type: Test Components: streams Reporter: Bill Bejeck >From build [https://builds.apache.org/job/kafka-pr-jdk14-scala2.13/1979/] {noformat} org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest > shouldRestoreState FAILED 14:25:19 java.lang.AssertionError: Condition not met within timeout 6. Repartition topic restore-test-KSTREAM-AGGREGATE-STATE-STORE-02-repartition not purged data after 6 ms. 14:25:19 at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:26) 14:25:19 at org.apache.kafka.test.TestUtils.lambda$waitForCondition$6(TestUtils.java:401) 14:25:19 at org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:449) 14:25:19 at org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:417) 14:25:19 at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:398) 14:25:19 at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:388) 14:25:19 at org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState(PurgeRepartitionTopicIntegrationTest.java:206){noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10404) Flaky Test kafka.api.SaslSslConsumerTest.testCoordinatorFailover
Bill Bejeck created KAFKA-10404: --- Summary: Flaky Test kafka.api.SaslSslConsumerTest.testCoordinatorFailover Key: KAFKA-10404 URL: https://issues.apache.org/jira/browse/KAFKA-10404 Project: Kafka Issue Type: Test Components: core, unit tests Reporter: Bill Bejeck >From build [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3829/] {noformat} kafka.api.SaslSslConsumerTest > testCoordinatorFailover FAILED 11:27:15 java.lang.AssertionError: expected: but was: 11:27:15 at org.junit.Assert.fail(Assert.java:89) 11:27:15 at org.junit.Assert.failNotEquals(Assert.java:835) 11:27:15 at org.junit.Assert.assertEquals(Assert.java:120) 11:27:15 at org.junit.Assert.assertEquals(Assert.java:146) 11:27:15 at kafka.api.AbstractConsumerTest.sendAndAwaitAsyncCommit(AbstractConsumerTest.scala:195) 11:27:15 at kafka.api.AbstractConsumerTest.ensureNoRebalance(AbstractConsumerTest.scala:302) 11:27:15 at kafka.api.BaseConsumerTest.testCoordinatorFailover(BaseConsumerTest.scala:76) 11:27:15 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 11:27:15 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 11:27:15 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 11:27:15 at java.lang.reflect.Method.invoke(Method.java:498) 11:27:15 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) 11:27:15 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) 11:27:15 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) 11:27:15 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) 11:27:15 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 11:27:15 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 11:27:15 at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) 11:27:15 at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) 11:27:15 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) 11:27:15 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) 11:27:15 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) 11:27:15 at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) 11:27:15 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) 11:27:15 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) 11:27:15 at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) 11:27:15 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) 11:27:15 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 11:27:15 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 11:27:15 at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) 11:27:15 at org.junit.runners.ParentRunner.run(ParentRunner.java:413) 11:27:15 at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) 11:27:15 at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) 11:27:15 at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) 11:27:15 at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) 11:27:15 at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) 11:27:15 at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) 11:27:15 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 11:27:15 at java.lang.reflect.Method.invoke(Method.java:498) 11:27:15 at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) 11:27:15 at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) 11:27:15 at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) 11:27:15 at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) 11:27:15 at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) 11:27:15 at
[jira] [Created] (KAFKA-9976) Aggregates should reuse repartition nodes
Bill Bejeck created KAFKA-9976: -- Summary: Aggregates should reuse repartition nodes Key: KAFKA-9976 URL: https://issues.apache.org/jira/browse/KAFKA-9976 Project: Kafka Issue Type: Improvement Components: streams Reporter: Bill Bejeck Assignee: Bill Bejeck The `GroupedStreamAggregateBuilder` will re-use the repartition node if the user provides a name via `Grouped`, otherwise it will create a new repartition node. The fix in KAFKA-9298 results in reusing the repartition node for KStream objects performing multiple joins, so the `KGroupedStream` should follow the same pattern and reuse the repartition node when a `KGroupedStream` needs repartitioning and performs multiple aggregates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9798) org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldAllowConcurrentAccesses
Bill Bejeck created KAFKA-9798: -- Summary: org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldAllowConcurrentAccesses Key: KAFKA-9798 URL: https://issues.apache.org/jira/browse/KAFKA-9798 Project: Kafka Issue Type: Test Components: streams Reporter: Bill Bejeck -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9533) ValueTransform forwards `null` values
[ https://issues.apache.org/jira/browse/KAFKA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-9533. Resolution: Fixed > ValueTransform forwards `null` values > - > > Key: KAFKA-9533 > URL: https://issues.apache.org/jira/browse/KAFKA-9533 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.4.0 >Reporter: Michael Viamari >Assignee: Michael Viamari >Priority: Minor > > According to the documentation for `KStream#transformValues`, nulls returned > from `ValueTransformer#transform` are not forwarded. (see > [KStream#transformValues|https://kafka.apache.org/24/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-] > However, this does not appear to be the case. In > `KStreamTransformValuesProcessor#process` the result of the transform is > forwarded directly. > {code:java} > @Override > public void process(final K key, final V value) { > context.forward(key, valueTransformer.transform(key, value)); > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9530) Flaky Test kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout
Bill Bejeck created KAFKA-9530: -- Summary: Flaky Test kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout Key: KAFKA-9530 URL: https://issues.apache.org/jira/browse/KAFKA-9530 Project: Kafka Issue Type: Test Components: core Reporter: Bill Bejeck [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/4570/testReport/junit/kafka.admin/DescribeConsumerGroupTest/testDescribeGroupWithShortInitializationTimeout/] {noformat} Error Messagejava.lang.AssertionError: assertion failedStacktracejava.lang.AssertionError: assertion failed at scala.Predef$.assert(Predef.scala:267) at kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout(DescribeConsumerGroupTest.scala:585) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at jdk.internal.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at
[jira] [Resolved] (KAFKA-9152) Improve Sensor Retrieval
[ https://issues.apache.org/jira/browse/KAFKA-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-9152. Resolution: Fixed > Improve Sensor Retrieval > - > > Key: KAFKA-9152 > URL: https://issues.apache.org/jira/browse/KAFKA-9152 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Assignee: highluck >Priority: Minor > Labels: newbie, tech-debt > Fix For: 2.5.0 > > > This ticket shall improve two aspects of the retrieval of sensors: > 1. Currently, when a sensor is retrieved with {{*Metrics.*Sensor()}} (e.g. > {{ThreadMetrics.createTaskSensor()}}) after it was created with the same > method {{*Metrics.*Sensor()}}, the sensor is added again to the corresponding > queue in {{*Sensors}} (e.g. {{threadLevelSensors}}) in > {{StreamsMetricsImpl}}. Those queues are used to remove the sensors when > {{removeAll*LevelSensors()}} is called. Having multiple times the same > sensors in this queue is not an issue from a correctness point of view. > However, it would reduce the footprint to only store a sensor once in those > queues. > 2. When a sensor is retrieved, the current code attempts to create a new > sensor and to add to it again the corresponding metrics. This could be > avoided. > > Both aspects could be improved by checking whether a sensor already exists by > calling {{getSensor()}} on the {{Metrics}} object and checking the return > value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-7317) Use collections subscription for main consumer to reduce metadata
[ https://issues.apache.org/jira/browse/KAFKA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-7317. Resolution: Fixed > Use collections subscription for main consumer to reduce metadata > - > > Key: KAFKA-7317 > URL: https://issues.apache.org/jira/browse/KAFKA-7317 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Assignee: Sophie Blee-Goldman >Priority: Major > Fix For: 2.5.0 > > > In KAFKA-4633 we switched from "collection subscription" to "pattern > subscription" for `Consumer#subscribe()` to avoid triggering auto topic > creating on the broker. In KAFKA-5291, the metadata request was extended to > overwrite the broker config within the request itself. However, this feature > is only used in `KafkaAdminClient`. KAFKA-7320 adds this feature for the > consumer client, too. > This ticket proposes to use the new feature within Kafka Streams to allow the > usage of collection based subscription in consumer and admit clients to > reduce the metadata response size than can be very large for large number of > partitions in the cluster. > Note, that Streams need to be able to distinguish if it connects to older > brokers that do not support the new metadata request and still use pattern > subscription for this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8821) Avoid pattern subscription to allow for stricter ACL settings
[ https://issues.apache.org/jira/browse/KAFKA-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8821. Resolution: Fixed Resolved via [https://github.com/apache/kafka/pull/7969] > Avoid pattern subscription to allow for stricter ACL settings > - > > Key: KAFKA-8821 > URL: https://issues.apache.org/jira/browse/KAFKA-8821 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Assignee: Sophie Blee-Goldman >Priority: Minor > Fix For: 2.5.0 > > > To avoid triggering auto topic creation (if `auto.create.topic.enable=true` > on the brokers), Kafka Streams uses consumer pattern subscription. For this > case, the consumer requests all metadata from the brokers and does client > side filtering. > However, if users want to set ACL to restrict a Kafka Streams application, > this may results in broker side ERROR logs that some metadata cannot be > provided. The only way to avoid those broker side ERROR logs is to grant > corresponding permissions. > As of 2.3 release it's possible to disable auto topic creation client side > (via https://issues.apache.org/jira/browse/KAFKA-7320). Kafka Streams should > use this new feature (note, that broker version 0.11 is required) to allow > users to set strict ACLs without getting flooded with ERROR logs on the > broker. > The proposal is that by default Kafka Streams disables auto-topic create > client side (optimistically) and uses regular subscription (not pattern > subscription). If an older broker is used, users need to explicitly enable > `allow.auto.create.topic` client side. If we detect this setting, we switch > back to pattern based subscription. > If users don't enable auto topic create client side and run with an older > broker, we would just rethrow the exception to the user, adding some context > information on how to fix the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9398) Kafka Streams main thread may not exit even after close timeout has passed
Bill Bejeck created KAFKA-9398: -- Summary: Kafka Streams main thread may not exit even after close timeout has passed Key: KAFKA-9398 URL: https://issues.apache.org/jira/browse/KAFKA-9398 Project: Kafka Issue Type: Improvement Components: streams Reporter: Bill Bejeck Assignee: Bill Bejeck Fix For: 2.5.0 Kafka Streams offers the {{KafkaStreams.close()}} method when shutting down a Kafka Streams application. There are two overloads to this method, one that takes no parameters and another taking a {{Duration}} specifying how long the {{close()}} method should block waiting for streams shutdown operations to complete. The no-arg version of {{close()}} sets the timeout to {{Long.MAX_VALUE}}. The issue is that if a {{StreamThread}} is some how hung or if one of the {{Consumer}} or {{Producer}} clients are in a hung state, the Kafka Streams application won't exit even after the specified timeout has expired. For example consider this scenario: # A sink topic gets deleted by accident # The {{Producer max.block.ms}} config is set to high value In this case the {{Producer}} will issue a {{WARN}} logging statement and will continue to make metadata requests looking for the expected topic. This will continue up until the {{max.block.ms}} expires. If this value is high enough, calling {{close()}} with a timeout won't fix the issue as when the timeout expires, the Kafka Streams application main thread won't exit. To prevent this type of issue, we should call {{Thread.interrupt()}} on all {{StreamThread}} instances once the {{close()}} timeout has expired. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9294) Enhance DSL Naming Guide to Include All Naming Rules
Bill Bejeck created KAFKA-9294: -- Summary: Enhance DSL Naming Guide to Include All Naming Rules Key: KAFKA-9294 URL: https://issues.apache.org/jira/browse/KAFKA-9294 Project: Kafka Issue Type: Improvement Components: streams Reporter: Bill Bejeck Assignee: Bill Bejeck We already have a naming guide in the docs, but we should expand it to cover how all components of the DSL get named. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9273) Refactor AbstractJoinIntegrationTest and Sub-classes
Bill Bejeck created KAFKA-9273: -- Summary: Refactor AbstractJoinIntegrationTest and Sub-classes Key: KAFKA-9273 URL: https://issues.apache.org/jira/browse/KAFKA-9273 Project: Kafka Issue Type: Test Components: streams Reporter: Bill Bejeck Assignee: Bill Bejeck The AbstractJoinIntegrationTest uses an embedded broker, but not all the sub-classes require the use of an embedded broker anymore. Additionally, there are two test remaining that require an embedded broker, but they don't perform joins, the are tests validating other conditions, so ideally those tests should move into a separate test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9086) Refactor Processor Node Streams Metrics
[ https://issues.apache.org/jira/browse/KAFKA-9086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-9086. Resolution: Fixed > Refactor Processor Node Streams Metrics > --- > > Key: KAFKA-9086 > URL: https://issues.apache.org/jira/browse/KAFKA-9086 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > Fix For: 2.5.0 > > > Refactor processor node metrics as described in KIP-444. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9188) Flaky Test SslAdminClientIntegrationTest.testSynchronousAuthorizerAclUpdatesBlockRequestThreads
Bill Bejeck created KAFKA-9188: -- Summary: Flaky Test SslAdminClientIntegrationTest.testSynchronousAuthorizerAclUpdatesBlockRequestThreads Key: KAFKA-9188 URL: https://issues.apache.org/jira/browse/KAFKA-9188 Project: Kafka Issue Type: Test Components: core Reporter: Bill Bejeck Failed in [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/9373/testReport/junit/kafka.api/SslAdminClientIntegrationTest/testSynchronousAuthorizerAclUpdatesBlockRequestThreads/] {noformat} Error Messagejava.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Aborted due to timeout.Stacktracejava.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Aborted due to timeout. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) at kafka.api.SslAdminClientIntegrationTest.$anonfun$testSynchronousAuthorizerAclUpdatesBlockRequestThreads$1(SslAdminClientIntegrationTest.scala:201) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at kafka.api.SslAdminClientIntegrationTest.testSynchronousAuthorizerAclUpdatesBlockRequestThreads(SslAdminClientIntegrationTest.scala:201) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.kafka.common.errors.TimeoutException: Aborted due to timeout. Standard Output[2019-11-14 15:13:51,489] ERROR [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Error for partition mytopic1-1 at offset 0 (kafka.server.ReplicaFetcherThread:76) org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. [2019-11-14 15:13:51,490] ERROR [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Error for partition mytopic1-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. [2019-11-14 15:14:04,686] ERROR [KafkaApi-2] Error when handling request: clientId=adminclient-644, correlationId=4, api=CREATE_ACLS, version=1, body={creations=[{resource_type=2,resource_name=foobar,resource_pattern_type=3,principal=User:ANONYMOUS,host=*,operation=3,permission_type=3},{resource_type=5,resource_name=transactional_id,resource_pattern_type=3,principal=User:ANONYMOUS,host=*,operation=4,permission_type=3}]} (kafka.server.KafkaApis:76) org.apache.kafka.common.errors.ClusterAuthorizationException: Request Request(processor=1, connectionId=127.0.0.1:41993-127.0.0.1:34770-0, session=Session(User:ANONYMOUS,/127.0.0.1), listenerName=ListenerName(SSL), securityProtocol=SSL, buffer=null) is not authorized. [2019-11-14 15:14:04,689] ERROR [KafkaApi-2] Error when handling request: clientId=adminclient-644, correlationId=5, api=DELETE_ACLS, version=1, body={filters=[{resource_type=2,resource_name=foobar,resource_pattern_type_filter=3,principal=User:ANONYMOUS,host=*,operation=3,permission_type=3},{resource_type=5,resource_name=transactional_id,resource_pattern_type_filter=3,principal=User:ANONYMOUS,host=*,operation=4,permission_type=3}]} (kafka.server.KafkaApis:76)
[jira] [Created] (KAFKA-9187) kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
Bill Bejeck created KAFKA-9187: -- Summary: kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl Key: KAFKA-9187 URL: https://issues.apache.org/jira/browse/KAFKA-9187 Project: Kafka Issue Type: Test Components: core Reporter: Bill Bejeck Failed in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/26593/] {noformat} Error Messageorg.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout instead of the expected 1 recordsStacktraceorg.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout instead of the expected 1 records at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:530) at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389) at org.scalatest.Assertions$class.fail(Assertions.scala:1091) at org.scalatest.Assertions$.fail(Assertions.scala:1389) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:842) at kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:793) at kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1334) at kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1343) at kafka.api.EndToEndAuthorizationTest.consumeRecords(EndToEndAuthorizationTest.scala:530) at kafka.api.EndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl(EndToEndAuthorizationTest.scala:369) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.ParentRunner.run(ParentRunner.java:412) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at
[jira] [Created] (KAFKA-9182) Flaky Test org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosEnabled
Bill Bejeck created KAFKA-9182: -- Summary: Flaky Test org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosEnabled Key: KAFKA-9182 URL: https://issues.apache.org/jira/browse/KAFKA-9182 Project: Kafka Issue Type: Test Components: streams Reporter: Bill Bejeck Failed in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/26571/testReport/junit/org.apache.kafka.streams.integration/KTableSourceTopicRestartIntegrationTest/shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosEnabled/] {noformat} Error Messagejava.lang.AssertionError: Condition not met within timeout 3. Table did not read all valuesStacktracejava.lang.AssertionError: Condition not met within timeout 3. Table did not read all values at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:24) at org.apache.kafka.test.TestUtils.lambda$waitForCondition$4(TestUtils.java:369) at org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:417) at org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:385) at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:368) at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:356) at org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.assertNumberValuesRead(KTableSourceTopicRestartIntegrationTest.java:187) at org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosEnabled(KTableSourceTopicRestartIntegrationTest.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.ParentRunner.run(ParentRunner.java:412) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at
[jira] [Created] (KAFKA-9181) Flaky test kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe
Bill Bejeck created KAFKA-9181: -- Summary: Flaky test kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe Key: KAFKA-9181 URL: https://issues.apache.org/jira/browse/KAFKA-9181 Project: Kafka Issue Type: Test Components: core Reporter: Bill Bejeck Failed in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/26571/testReport/junit/kafka.api/SaslGssapiSslEndToEndAuthorizationTest/testNoConsumeWithoutDescribeAclViaSubscribe/] {noformat} Error Messageorg.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [topic2]Stacktraceorg.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [topic2] Standard OutputAdding ACLs for resource `ResourcePattern(resourceType=CLUSTER, name=kafka-cluster, patternType=LITERAL)`: (principal=User:kafka, host=*, operation=CLUSTER_ACTION, permissionType=ALLOW) Current ACLs for resource `Cluster:LITERAL:kafka-cluster`: User:kafka has Allow permission for operations: ClusterAction from hosts: * Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, patternType=LITERAL)`: (principal=User:kafka, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `Topic:LITERAL:*`: User:kafka has Allow permission for operations: Read from hosts: * Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /tmp/kafka6494439724844851846.tmp refreshKrb5Config is false principal is kafka/localh...@example.com tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is kafka/localh...@example.com Will use keytab Commit Succeeded [2019-11-13 04:43:16,187] ERROR [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. [2019-11-13 04:43:16,191] ERROR [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Error for partition __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. [2019-11-13 04:43:16,384] ERROR [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Error for partition e2etopic-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. [2019-11-13 04:43:16,384] ERROR [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error for partition e2etopic-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=e2etopic, patternType=LITERAL)`: (principal=User:client, host=*, operation=WRITE, permissionType=ALLOW) (principal=User:client, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:client, host=*, operation=CREATE, permissionType=ALLOW) Current ACLs for resource `Topic:LITERAL:e2etopic`: User:client has Allow permission for operations: Describe from hosts: * User:client has Allow permission for operations: Write from hosts: * User:client has Allow permission for operations: Create from hosts: * Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=e2etopic, patternType=LITERAL)`: (principal=User:client, host=*, operation=READ, permissionType=ALLOW) (principal=User:client, host=*, operation=DESCRIBE, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=group, patternType=LITERAL)`: (principal=User:client, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `Topic:LITERAL:e2etopic`: User:client has Allow permission for operations: Read from hosts: * User:client has Allow permission for operations: Describe from hosts: * User:client has Allow permission for operations: Write from hosts: * User:client has Allow permission for operations: Create from hosts: * Current ACLs for resource `Group:LITERAL:group`: User:client has Allow permission for operations: Read from hosts: * Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /tmp/kafka3083328529571706878.tmp refreshKrb5Config is false principal is cli...@example.com tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is cli...@example.com Will use keytab Commit
[jira] [Resolved] (KAFKA-9072) Add Section to Streams Developer Guide for Topology Naming (KIP-307)
[ https://issues.apache.org/jira/browse/KAFKA-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-9072. Resolution: Fixed > Add Section to Streams Developer Guide for Topology Naming (KIP-307) > > > Key: KAFKA-9072 > URL: https://issues.apache.org/jira/browse/KAFKA-9072 > Project: Kafka > Issue Type: Task > Components: streams >Affects Versions: 2.4.0 >Reporter: Bill Bejeck >Assignee: Bill Bejeck >Priority: Major > Labels: docs > Fix For: 2.5.0 > > > WIth KIP-307 users can name operators in a topology. Naming is important as > it can help with pinning state store, changelog topic, and repartition topic > names keeping the topology robust in the face of adding/removing operators in > a Kafka Streams DSL. We should add a section to the developer guide to > explain why this is important. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KAFKA-9011) Add KStream#flatTransform and KStream#flatTransformValues to Scala API
[ https://issues.apache.org/jira/browse/KAFKA-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck reopened KAFKA-9011: There's still something left to address on this PR [https://github.com/apache/kafka/pull/7520#discussion_r345374820] So I'm reopening the ticket. > Add KStream#flatTransform and KStream#flatTransformValues to Scala API > -- > > Key: KAFKA-9011 > URL: https://issues.apache.org/jira/browse/KAFKA-9011 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Alex Kokachev >Assignee: Alex Kokachev >Priority: Major > Labels: scala, streams > Fix For: 2.5.0 > > > Part of KIP-313: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-313%3A+Add+KStream.flatTransform+and+KStream.flatTransformValues] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8980) Refactor State-Store-level Metrics
[ https://issues.apache.org/jira/browse/KAFKA-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8980. Resolution: Fixed > Refactor State-Store-level Metrics > -- > > Key: KAFKA-8980 > URL: https://issues.apache.org/jira/browse/KAFKA-8980 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.5.0 >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > Fix For: 2.5.0 > > > Refactor state-store-level metrics as proposed in KIP-444. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9077) System Test Failure: StreamsSimpleBenchmarkTest
[ https://issues.apache.org/jira/browse/KAFKA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-9077. Resolution: Fixed > System Test Failure: StreamsSimpleBenchmarkTest > --- > > Key: KAFKA-9077 > URL: https://issues.apache.org/jira/browse/KAFKA-9077 > Project: Kafka > Issue Type: Bug > Components: streams, system tests >Affects Versions: 2.4.0 >Reporter: Manikumar >Assignee: Bruno Cadonna >Priority: Minor > Fix For: 2.5.0 > > > StreamsSimpleBenchmarkTest tests are failing on 2.4 and trunk. > http://confluent-kafka-2-4-system-test-results.s3-us-west-2.amazonaws.com/2019-10-21--001.1571716233--confluentinc--2.4--cb4944f/report.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8968) Refactor Task-level Metrics
[ https://issues.apache.org/jira/browse/KAFKA-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8968. Resolution: Fixed > Refactor Task-level Metrics > --- > > Key: KAFKA-8968 > URL: https://issues.apache.org/jira/browse/KAFKA-8968 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.5.0 >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > Fix For: 2.5.0 > > > Refactor task-level metrics as proposed in KIP-444. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9098) Name Repartition Filter, Source, and Sink Processors
Bill Bejeck created KAFKA-9098: -- Summary: Name Repartition Filter, Source, and Sink Processors Key: KAFKA-9098 URL: https://issues.apache.org/jira/browse/KAFKA-9098 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 2.3.0, 2.2.0 Reporter: Bill Bejeck Assignee: Bill Bejeck When users provide a name for repartition topics, we should the same name as the base for the filter, source and sink operators. While this does not break a topology, users providing names for all processors in a DSL topology may find the generated names for the repartition topics filter, source, and sink operators as inconsistent with the naming approach. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9072) Add Section to Streams Developer Guide for Topology Naming (KIP-307)
Bill Bejeck created KAFKA-9072: -- Summary: Add Section to Streams Developer Guide for Topology Naming (KIP-307) Key: KAFKA-9072 URL: https://issues.apache.org/jira/browse/KAFKA-9072 Project: Kafka Issue Type: Task Affects Versions: 2.4.0 Reporter: Bill Bejeck Assignee: Bill Bejeck Fix For: 2.4.0 WIth KIP-307 users can name operators in a topology. Naming is important as it can help with pinning state store, changelog topic, and repartition topic names keeping the topology robust in the face of adding/removing operators in a Kafka Streams DSL. We should add a section to the developer guide to explain why this is important. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9053) AssignmentInfo#encode hardcodes the LATEST_SUPPORTED_VERSION
[ https://issues.apache.org/jira/browse/KAFKA-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-9053. Resolution: Fixed > AssignmentInfo#encode hardcodes the LATEST_SUPPORTED_VERSION > - > > Key: KAFKA-9053 > URL: https://issues.apache.org/jira/browse/KAFKA-9053 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Blocker > Fix For: 2.1.2, 2.2.2, 2.4.0, 2.3.2 > > > We should instead encode the commonlySupportedVersion field. This affects > version probing with a subscription change -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9019) Flaky Test kafka.api.SslAdminClientIntegrationTest.testCreateTopicsResponseMetadataAndConfig
Bill Bejeck created KAFKA-9019: -- Summary: Flaky Test kafka.api.SslAdminClientIntegrationTest.testCreateTopicsResponseMetadataAndConfig Key: KAFKA-9019 URL: https://issues.apache.org/jira/browse/KAFKA-9019 Project: Kafka Issue Type: Test Components: core Reporter: Bill Bejeck Seen in [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/2492/testReport/junit/kafka.api/SslAdminClientIntegrationTest/testCreateTopicsResponseMetadataAndConfig/] {noformat} Error Messagejava.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.Stacktracejava.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) at kafka.api.SaslSslAdminClientIntegrationTest.testCreateTopicsResponseMetadataAndConfig(SaslSslAdminClientIntegrationTest.scala:452) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. Standard Output[2019-10-10 05:41:46,134] ERROR [KafkaApi-2] Error when handling request: clientId=adminclient-123, correlationId=4, api=CREATE_ACLS, version=1, body={creations=[{resource_type=2,resource_name=foobar,resource_pattern_type=3,principal=User:ANONYMOUS,host=*,operation=3,permission_type=3},{resource_type=5,resource_name=transactional_id,resource_pattern_type=3,principal=User:ANONYMOUS,host=*,operation=4,permission_type=3}]} (kafka.server.KafkaApis:76) org.apache.kafka.common.errors.ClusterAuthorizationException: Request Request(processor=1, connectionId=127.0.0.1:34475-127.0.0.1:42444-0, session=Session(User:ANONYMOUS,/127.0.0.1), listenerName=ListenerName(SSL), securityProtocol=SSL, buffer=null) is not authorized. [2019-10-10 05:41:46,136] ERROR [KafkaApi-2] Error when handling request: clientId=adminclient-123, correlationId=5, api=DELETE_ACLS, version=1, body={filters=[{resource_type=2,resource_name=foobar,resource_pattern_type_filter=3,principal=User:ANONYMOUS,host=*,operation=3,permission_type=3},{resource_type=5,resource_name=transactional_id,resource_pattern_type_filter=3,principal=User:ANONYMOUS,host=*,operation=4,permission_type=3}]} (kafka.server.KafkaApis:76) org.apache.kafka.common.errors.ClusterAuthorizationException: Request Request(processor=1, connectionId=127.0.0.1:34475-127.0.0.1:42444-0, session=Session(User:ANONYMOUS,/127.0.0.1), listenerName=ListenerName(SSL), securityProtocol=SSL, buffer=null) is not authorized. [2019-10-10 05:41:46,241] ERROR [KafkaApi-2] Error when handling request: clientId=adminclient-123, correlationId=7, api=DESCRIBE_ACLS, version=1, body={resource_type=2,resource_name=*,resource_pattern_type_filter=3,principal=User:*,host=*,operation=2,permission_type=3} (kafka.server.KafkaApis:76) org.apache.kafka.common.errors.ClusterAuthorizationException: Request Request(processor=1, connectionId=127.0.0.1:34475-127.0.0.1:42444-0, session=Session(User:ANONYMOUS,/127.0.0.1), listenerName=ListenerName(SSL), securityProtocol=SSL, buffer=null) is not authorized. [2019-10-10 05:41:46,243] ERROR [KafkaApi-2] Error
[jira] [Created] (KAFKA-9009) Flaky Test kafka.integration.MetricsDuringTopicCreationDeletionTest.testMetricsDuringTopicCreateDelete
Bill Bejeck created KAFKA-9009: -- Summary: Flaky Test kafka.integration.MetricsDuringTopicCreationDeletionTest.testMetricsDuringTopicCreateDelete Key: KAFKA-9009 URL: https://issues.apache.org/jira/browse/KAFKA-9009 Project: Kafka Issue Type: Test Components: core Reporter: Bill Bejeck Failure seen in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/25644/testReport/junit/kafka.integration/MetricsDuringTopicCreationDeletionTest/testMetricsDuringTopicCreateDelete/] {noformat} Error Messagejava.lang.AssertionError: assertion failed: UnderReplicatedPartitionCount not 0: 1Stacktracejava.lang.AssertionError: assertion failed: UnderReplicatedPartitionCount not 0: 1 at scala.Predef$.assert(Predef.scala:170) at kafka.integration.MetricsDuringTopicCreationDeletionTest.testMetricsDuringTopicCreateDelete(MetricsDuringTopicCreationDeletionTest.scala:123) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.ParentRunner.run(ParentRunner.java:412) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
[jira] [Created] (KAFKA-9008) Flaky Test kafka.api.SaslSslAdminClientIntegrationTest.testDescribeConfigsForTopic
Bill Bejeck created KAFKA-9008: -- Summary: Flaky Test kafka.api.SaslSslAdminClientIntegrationTest.testDescribeConfigsForTopic Key: KAFKA-9008 URL: https://issues.apache.org/jira/browse/KAFKA-9008 Project: Kafka Issue Type: Test Components: core Reporter: Bill Bejeck Failure seen in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/25644/testReport/junit/kafka.api/SaslSslAdminClientIntegrationTest/testDescribeConfigsForTopic/] {noformat} Error Messageorg.junit.runners.model.TestTimedOutException: test timed out after 12 millisecondsStacktraceorg.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds at java.io.FileDescriptor.sync(Native Method) at jdbm.recman.TransactionManager.sync(TransactionManager.java:385) at jdbm.recman.TransactionManager.commit(TransactionManager.java:368) at jdbm.recman.RecordFile.commit(RecordFile.java:320) at jdbm.recman.PageManager.commit(PageManager.java:289) at jdbm.recman.BaseRecordManager.commit(BaseRecordManager.java:419) at jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:350) at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.sync(JdbmTable.java:976) at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.close(JdbmTable.java:961) at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.close(JdbmIndex.java:571) at org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.doDestroy(AbstractBTreePartition.java:524) at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmPartition.doDestroy(JdbmPartition.java:744) at org.apache.directory.server.core.api.partition.AbstractPartition.destroy(AbstractPartition.java:153) at org.apache.directory.server.core.shared.partition.DefaultPartitionNexus.removeContextPartition(DefaultPartitionNexus.java:886) at org.apache.directory.server.core.shared.partition.DefaultPartitionNexus.doDestroy(DefaultPartitionNexus.java:287) at org.apache.directory.server.core.api.partition.AbstractPartition.destroy(AbstractPartition.java:153) at org.apache.directory.server.core.DefaultDirectoryService.shutdown(DefaultDirectoryService.java:1313) at kafka.security.minikdc.MiniKdc.stop(MiniKdc.scala:278) at kafka.api.SaslSetup$class.closeSasl(SaslSetup.scala:120) at kafka.api.SaslSslAdminClientIntegrationTest.closeSasl(SaslSslAdminClientIntegrationTest.scala:38) at kafka.api.SaslSslAdminClientIntegrationTest.tearDown(SaslSslAdminClientIntegrationTest.scala:100) at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Standard OutputDebug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /tmp/kafka5680437396247072679.tmp refreshKrb5Config is false principal is kafka/localh...@example.com tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is kafka/localh...@example.com Will use keytab Commit Succeeded Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /tmp/kafka308994997839802486.tmp refreshKrb5Config is false principal is clie...@example.com tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is clie...@example.com Will use keytab Commit Succeeded Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /tmp/kafka4996638752651080021.tmp refreshKrb5Config is false principal is kafka/localh...@example.com tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is kafka/localh...@example.com Will use keytab
[jira] [Created] (KAFKA-9007) Flaky Test kafka.api.SaslPlaintextConsumerTest.testCoordinatorFailover
Bill Bejeck created KAFKA-9007: -- Summary: Flaky Test kafka.api.SaslPlaintextConsumerTest.testCoordinatorFailover Key: KAFKA-9007 URL: https://issues.apache.org/jira/browse/KAFKA-9007 Project: Kafka Issue Type: Test Components: core Reporter: Bill Bejeck Failed in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/25644/testReport/junit/kafka.api/SaslPlaintextConsumerTest/testCoordinatorFailover/] {noformat} Error Messagejava.lang.AssertionError: expected: but was:Stacktracejava.lang.AssertionError: expected: but was: at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:120) at org.junit.Assert.assertEquals(Assert.java:146) at kafka.api.AbstractConsumerTest.sendAndAwaitAsyncCommit(AbstractConsumerTest.scala:195) at kafka.api.AbstractConsumerTest.ensureNoRebalance(AbstractConsumerTest.scala:302) at kafka.api.BaseConsumerTest.testCoordinatorFailover(BaseConsumerTest.scala:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.ParentRunner.run(ParentRunner.java:412) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at
[jira] [Reopened] (KAFKA-8460) Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
[ https://issues.apache.org/jira/browse/KAFKA-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck reopened KAFKA-8460: Test failed again in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/25644/testReport/junit/kafka.api/PlaintextConsumerTest/testLowMaxFetchSizeForRequestAndPartition/] {noformat} Error Messageorg.scalatest.exceptions.TestFailedException: Timed out before consuming expected 2700 records. The number consumed was 1560.Stacktraceorg.scalatest.exceptions.TestFailedException: Timed out before consuming expected 2700 records. The number consumed was 1560. at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:530) at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389) at org.scalatest.Assertions$class.fail(Assertions.scala:1091) at org.scalatest.Assertions$.fail(Assertions.scala:1389) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:841) at kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:792) at kafka.api.AbstractConsumerTest.consumeRecords(AbstractConsumerTest.scala:158) at kafka.api.PlaintextConsumerTest.testLowMaxFetchSizeForRequestAndPartition(PlaintextConsumerTest.scala:802) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.ParentRunner.run(ParentRunner.java:412) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at
[jira] [Created] (KAFKA-9006) Flaky Test KTableKTableForeignKeyJoinIntegrationTest.doLeftJoinFilterOutRapidlyChangingForeignKeyValues
Bill Bejeck created KAFKA-9006: -- Summary: Flaky Test KTableKTableForeignKeyJoinIntegrationTest.doLeftJoinFilterOutRapidlyChangingForeignKeyValues Key: KAFKA-9006 URL: https://issues.apache.org/jira/browse/KAFKA-9006 Project: Kafka Issue Type: Improvement Components: streams Reporter: Bill Bejeck h3. {noformat} Error Message array lengths differed, expected.length=2 actual.length=1; arrays first differed at element [0]; expected: but was: Stacktrace array lengths differed, expected.length=2 actual.length=1; arrays first differed at element [0]; expected: but was: at org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:78) at org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:28) at org.junit.Assert.internalArrayEquals(Assert.java:534) at org.junit.Assert.assertArrayEquals(Assert.java:285) at org.junit.Assert.assertArrayEquals(Assert.java:300) at org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.doLeftJoinFilterOutRapidlyChangingForeignKeyValues(KTableKTableForeignKeyJoinIntegrationTest.java:585) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.ParentRunner.run(ParentRunner.java:412) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.ParentRunner.run(ParentRunner.java:412) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at
[jira] [Resolved] (KAFKA-8944) Compiler Warning
[ https://issues.apache.org/jira/browse/KAFKA-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8944. Resolution: Fixed > Compiler Warning > > > Key: KAFKA-8944 > URL: https://issues.apache.org/jira/browse/KAFKA-8944 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Matthias J. Sax >Assignee: huxihx >Priority: Minor > Labels: scala > > When building Kafka Streams, we get the following compiler warning that we > should fix: > {code:java} > scala/src/main/scala/org/apache/kafka/streams/scala/kstream/KTable.scala:24: > imported `Suppressed' is permanently hidden by definition of object > Suppressed in package kstream import > org.apache.kafka.streams.kstream.{Suppressed, > ValueTransformerWithKeySupplier, KTable => KTableJ} > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9002) org.apache.kafka.streams.integration.RegexSourceIntegrationTest.testRegexMatchesTopicsAWhenCreated
Bill Bejeck created KAFKA-9002: -- Summary: org.apache.kafka.streams.integration.RegexSourceIntegrationTest.testRegexMatchesTopicsAWhenCreated Key: KAFKA-9002 URL: https://issues.apache.org/jira/browse/KAFKA-9002 Project: Kafka Issue Type: Improvement Components: streams Reporter: Bill Bejeck [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/25603/testReport/junit/org.apache.kafka.streams.integration/RegexSourceIntegrationTest/testRegexMatchesTopicsAWhenCreated/] {noformat} Error Messagejava.lang.AssertionError: Condition not met within timeout 15000. Stream tasks not updatedStacktracejava.lang.AssertionError: Condition not met within timeout 15000. Stream tasks not updated at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:376) at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:336) at org.apache.kafka.streams.integration.RegexSourceIntegrationTest.testRegexMatchesTopicsAWhenCreated(RegexSourceIntegrationTest.java:175) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.ParentRunner.run(ParentRunner.java:412) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at
[jira] [Resolved] (KAFKA-3705) Support non-key joining in KTable
[ https://issues.apache.org/jira/browse/KAFKA-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-3705. Resolution: Fixed > Support non-key joining in KTable > - > > Key: KAFKA-3705 > URL: https://issues.apache.org/jira/browse/KAFKA-3705 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Guozhang Wang >Assignee: Adam Bellemare >Priority: Major > Labels: api, kip > Fix For: 2.4.0 > > > KIP-213: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable] > Today in Kafka Streams DSL, KTable joins are only based on keys. If users > want to join a KTable A by key {{a}} with another KTable B by key {{b}} but > with a "foreign key" {{a}}, and assuming they are read from two topics which > are partitioned on {{a}} and {{b}} respectively, they need to do the > following pattern: > {code:java} > tableB' = tableB.groupBy(/* select on field "a" */).agg(...); // now tableB' > is partitioned on "a" > tableA.join(tableB', joiner); > {code} > Even if these two tables are read from two topics which are already > partitioned on {{a}}, users still need to do the pre-aggregation in order to > make the two joining streams to be on the same key. This is a draw-back from > programability and we should fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KAFKA-8807) Flaky Test GlobalThreadShutDownOrderTest.shouldFinishGlobalStoreOperationOnShutDown
[ https://issues.apache.org/jira/browse/KAFKA-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck reopened KAFKA-8807: Reopening this as I am going to refactor the test to check that close is only called once during shutdown. > Flaky Test > GlobalThreadShutDownOrderTest.shouldFinishGlobalStoreOperationOnShutDown > --- > > Key: KAFKA-8807 > URL: https://issues.apache.org/jira/browse/KAFKA-8807 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Sophie Blee-Goldman >Assignee: Bill Bejeck >Priority: Major > Labels: flaky-test > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/24229/testReport/junit/org.apache.kafka.streams.integration/GlobalThreadShutDownOrderTest/shouldFinishGlobalStoreOperationOnShutDown/] > > h3. Error Message > java.lang.AssertionError: expected:<[1, 2, 3, 4]> but was:<[1, 2, 3, 4, 1, 2, > 3, 4]> > h3. Stacktrace > java.lang.AssertionError: expected:<[1, 2, 3, 4]> but was:<[1, 2, 3, 4, 1, 2, > 3, 4]> at org.junit.Assert.fail(Assert.java:89) at > org.junit.Assert.failNotEquals(Assert.java:835) at > org.junit.Assert.assertEquals(Assert.java:120) at > org.junit.Assert.assertEquals(Assert.java:146) at > org.apache.kafka.streams.integration.GlobalThreadShutDownOrderTest.shouldFinishGlobalStoreOperationOnShutDown(GlobalThreadShutDownOrderTest.java:138) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at > org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at > org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) at > org.junit.rules.RunRules.evaluate(RunRules.java:20) at > org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at > org.junit.runners.ParentRunner.run(ParentRunner.java:412) at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) > at > org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) > at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at > org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) > at
[jira] [Resolved] (KAFKA-6958) Allow to define custom processor names with KStreams DSL
[ https://issues.apache.org/jira/browse/KAFKA-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-6958. Resolution: Fixed Thanks again [~fhussonnois] for your hard work and persistence in seeing this valuable contribution through to completion! > Allow to define custom processor names with KStreams DSL > > > Key: KAFKA-6958 > URL: https://issues.apache.org/jira/browse/KAFKA-6958 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 1.1.0 >Reporter: Florian Hussonnois >Assignee: Florian Hussonnois >Priority: Minor > Labels: kip > Fix For: 2.4.0 > > > Currently, while building a new Topology through the KStreams DSL the > processors are automatically named. > The genarated names are prefixed depending of the operation (i.e > KSTREAM-SOURCE, KSTREAM-FILTER, KSTREAM-MAP, etc). > To debug/understand a topology it is possible to display the processor > lineage with the method Topology#describe(). However, a complex topology with > dozens of operations can be hard to understand if the processor names are not > relevant. > It would be useful to be able to set more meaningful names. For example, a > processor name could describe the business rule performed by a map() > operation. > [KIP-307|https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8859) Refactor Cache-level Streams Metrics
[ https://issues.apache.org/jira/browse/KAFKA-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8859. Resolution: Fixed > Refactor Cache-level Streams Metrics > > > Key: KAFKA-8859 > URL: https://issues.apache.org/jira/browse/KAFKA-8859 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.4.0 >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > Fix For: 2.4.0 > > > Refactoring of cache-level Streams metrics according KIP-444. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8878) Flaky Test AssignedStreamsTasksTest#shouldCloseCleanlyWithSuspendedTaskAndEOS
[ https://issues.apache.org/jira/browse/KAFKA-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8878. Resolution: Fixed > Flaky Test AssignedStreamsTasksTest#shouldCloseCleanlyWithSuspendedTaskAndEOS > - > > Key: KAFKA-8878 > URL: https://issues.apache.org/jira/browse/KAFKA-8878 > Project: Kafka > Issue Type: Bug > Components: streams, unit tests >Affects Versions: 2.4.0 >Reporter: Matthias J. Sax >Assignee: Chris Pettitt >Priority: Major > Labels: flaky-test > Fix For: 2.4.0 > > > [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3887/tests] > {quote}java.lang.AssertionError: Expected no ERROR message while closing > assignedTasks, but got 1. First error: [AdminClient clientId=adminclient-67] > Connection to node -1 (localhost/127.0.0.1:8080) could not be established. > Broker may not be available.. Cause: N/A > at org.junit.Assert.fail(Assert.java:89) > at > org.apache.kafka.streams.processor.internals.AssignedStreamsTasksTest.shouldCloseCleanlyWithSuspendedTaskAndEOS(AssignedStreamsTasksTest.java:555){quote} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KAFKA-8744) Add Support to Scala API for KIP-307
Bill Bejeck created KAFKA-8744: -- Summary: Add Support to Scala API for KIP-307 Key: KAFKA-8744 URL: https://issues.apache.org/jira/browse/KAFKA-8744 Project: Kafka Issue Type: Task Components: streams Affects Versions: 2.4.0 Reporter: Bill Bejeck Assignee: Florian Hussonnois Fix For: 2.4.0 With the ability to provide names for all operators in a Kafka Streams topology ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL]) coming in the 2.4 release, we also need to add this new feature to the Streams Scala API. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KAFKA-8692) Transient failure in kafka.api.SaslScramSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
Bill Bejeck created KAFKA-8692: -- Summary: Transient failure in kafka.api.SaslScramSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl Key: KAFKA-8692 URL: https://issues.apache.org/jira/browse/KAFKA-8692 Project: Kafka Issue Type: Bug Components: core, unit tests Reporter: Bill Bejeck Failed in build [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/420/] {noformat} Error Message org.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout instead of the expected 1 records Stacktrace org.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout instead of the expected 1 records at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530) at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529) at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389) at org.scalatest.Assertions.fail(Assertions.scala:1091) at org.scalatest.Assertions.fail$(Assertions.scala:1087) at org.scalatest.Assertions$.fail(Assertions.scala:1389) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:822) at kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:781) at kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1309) at kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1317) at kafka.api.EndToEndAuthorizationTest.consumeRecords(EndToEndAuthorizationTest.scala:522) at kafka.api.EndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl(EndToEndAuthorizationTest.scala:361) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.ParentRunner.run(ParentRunner.java:412) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at jdk.internal.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at
[jira] [Resolved] (KAFKA-8689) Cannot Name Join State Store Topics
[ https://issues.apache.org/jira/browse/KAFKA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8689. Resolution: Duplicate Duplicate of https://issues.apache.org/jira/browse/KAFKA-8558 > Cannot Name Join State Store Topics > --- > > Key: KAFKA-8689 > URL: https://issues.apache.org/jira/browse/KAFKA-8689 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.0 >Reporter: Simon Dean >Priority: Major > > Performing a join on two KStreams, produces two state store topics. > Currently the names state store topics are auto generated and cannot be > overridden. > Example code: > > {code:java} > import org.apache.kafka.clients.producer.KafkaProducer; > import org.apache.kafka.clients.producer.ProducerConfig; > import org.apache.kafka.clients.producer.ProducerRecord; > import org.apache.kafka.common.serialization.LongSerializer; > import org.apache.kafka.common.serialization.Serde; > import org.apache.kafka.common.serialization.Serdes; > import org.apache.kafka.common.serialization.StringSerializer; > import org.apache.kafka.streams.KafkaStreams; > import org.apache.kafka.streams.StreamsBuilder; > import org.apache.kafka.streams.StreamsConfig; > import org.apache.kafka.streams.kstream.Consumed; > import org.apache.kafka.streams.kstream.JoinWindows; > import org.apache.kafka.streams.kstream.Joined; > import org.apache.kafka.streams.kstream.KStream; > import java.time.Duration; > import java.util.HashMap; > import java.util.Map; > import java.util.Properties; > import java.util.concurrent.TimeUnit; > public class JoinTopicNamesExample { > public static void main(final String[] args) throws InterruptedException { > new Thread(() -> { > produce(args); > }).run(); > new Thread(() -> { > try { > streams(args); > } catch (InterruptedException e) { > e.printStackTrace(); > } > }).run(); > } > private static void produce(String[] args) { > Map props = new HashMap<>(); > props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); > props.put(ProducerConfig.RETRIES_CONFIG, 0); > props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384); > props.put(ProducerConfig.LINGER_MS_CONFIG, 1); > props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 33554432); > props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, > StringSerializer.class); > props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, > LongSerializer.class); > KafkaProducer producer = new KafkaProducer<>(props); > for (long i = 0; i < 10; i++) { > producer.send(new ProducerRecord("left", Long.toString(i), i)); > } > for (long i = 0; i < 10; i++) { > producer.send(new ProducerRecord("right", Long.toString(i), i)); > } > } > private static void streams(String[] args) throws InterruptedException { > final String bootstrapServers = args.length > 0 ? args[0] : > "localhost:9092"; > final Properties streamsConfiguration = new Properties(); > // Give the Streams application a unique name. The name must be > unique in the Kafka cluster > // against which the application is run. > streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, > "join-topic-names-example"); > streamsConfiguration.put(StreamsConfig.CLIENT_ID_CONFIG, > "user-region-lambda-example-client"); > // Where to find Kafka broker(s). > streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, > bootstrapServers); > // Specify default (de)serializers for record keys and for record > values. > > streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, > Serdes.String().getClass().getName()); > > streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, > Serdes.Long().getClass().getName()); > // Records should be flushed every 10 seconds. This is less than the > default > // in order to keep this example interactive. > streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 10 > * 1000); > final Serde stringSerde = Serdes.String(); > final Serde longSerde = Serdes.Long(); > final StreamsBuilder builder = new StreamsBuilder(); > final KStream left = builder.stream("left", > Consumed.with(stringSerde, longSerde)); > final KStream right = builder.stream("right", > Consumed.with(stringSerde, longSerde)); > left.join( > right, > (value1, value2) -> value1 + value2, > JoinWindows.of(Duration.ofHours(1)), > Joined.as("sum")); >
[jira] [Resolved] (KAFKA-8602) StreamThread Dies Because Restore Consumer is not Subscribed to Any Topic
[ https://issues.apache.org/jira/browse/KAFKA-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8602. Resolution: Fixed > StreamThread Dies Because Restore Consumer is not Subscribed to Any Topic > - > > Key: KAFKA-8602 > URL: https://issues.apache.org/jira/browse/KAFKA-8602 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.1 >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Critical > > StreamThread dies with the following exception: > {code:java} > java.lang.IllegalStateException: Consumer is not subscribed to any topics or > assigned any partitions > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1199) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1126) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:923) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774) > {code} > The reason is that the restore consumer is not subscribed to any topic. This > happens when a StreamThread gets assigned standby tasks for sub-topologies > with just state stores with disabled logging. > To reproduce the bug start two applications with one StreamThread and one > standby replica each and the following topology. The input topic should have > two partitions: > {code:java} > final StreamsBuilder builder = new StreamsBuilder(); > final String stateStoreName = "myTransformState"; > final StoreBuilder> keyValueStoreBuilder = > > Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore(stateStoreName), > Serdes.Integer(), > Serdes.Integer()) > .withLoggingDisabled(); > builder.addStateStore(keyValueStoreBuilder); > builder.stream(INPUT_TOPIC, Consumed.with(Serdes.Integer(), Serdes.Integer())) > .transform(() -> new Transformer Integer>>() { > private KeyValueStore state; > @SuppressWarnings("unchecked") > @Override > public void init(final ProcessorContext context) { > state = (KeyValueStore) > context.getStateStore(stateStoreName); > } > @Override > public KeyValue transform(final Integer key, > final Integer value) { > final KeyValue result = new KeyValue<>(key, > value); > return result; > } > @Override > public void close() {} > }, stateStoreName) > .to(OUTPUT_TOPIC); > {code} > Both StreamThreads should die with the above exception. > The root cause is that standby tasks are created although all state stores of > the sub-topology have logging disabled. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KAFKA-8666) Improve Documentation on usage of Materialized config object
Bill Bejeck created KAFKA-8666: -- Summary: Improve Documentation on usage of Materialized config object Key: KAFKA-8666 URL: https://issues.apache.org/jira/browse/KAFKA-8666 Project: Kafka Issue Type: Improvement Components: documentation, streams Reporter: Bill Bejeck When using the Materialized object if the user wants to name the statestore with {code:java} Materialized.as("MyStoreName"){code} then subsequently provide the key and value serde the calls to do so must take the form of {code:java} Materialized.as("MyStoreName").withKeySerde(keySerde).withValueSerde(valueSerde) {code} If users do the following {code:java} Materialized.as("MyStoreName").with(keySerde, valueSerde) {code} the Materialized instance created by the "as(storeName)" call is replaced by a new Materialized instance resulting from the "with(...)" call and any configuration on the first Materialized instance is lost. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (KAFKA-8198) KStreams testing docs use non-existent method "pipe"
[ https://issues.apache.org/jira/browse/KAFKA-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8198. Resolution: Fixed > KStreams testing docs use non-existent method "pipe" > > > Key: KAFKA-8198 > URL: https://issues.apache.org/jira/browse/KAFKA-8198 > Project: Kafka > Issue Type: Bug > Components: documentation, streams >Affects Versions: 1.1.1, 2.0.1, 2.2.0, 2.1.1 >Reporter: Michael Drogalis >Assignee: Slim Ouertani >Priority: Minor > Labels: documentation, newbie > > In [the testing docs for > KStreams|https://kafka.apache.org/20/documentation/streams/developer-guide/testing.html], > we use the following code snippet: > {code:java} > ConsumerRecordFactory factory = new > ConsumerRecordFactory<>("input-topic", new StringSerializer(), new > IntegerSerializer()); > testDriver.pipe(factory.create("key", 42L)); > {code} > We should correct the docs to use the pipeInput method. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (KAFKA-5998) /.checkpoint.tmp Not found exception
[ https://issues.apache.org/jira/browse/KAFKA-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-5998. Resolution: Fixed > /.checkpoint.tmp Not found exception > > > Key: KAFKA-5998 > URL: https://issues.apache.org/jira/browse/KAFKA-5998 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0, 0.11.0.1, 2.1.1 >Reporter: Yogesh BG >Assignee: John Roesler >Priority: Critical > Attachments: 5998.v1.txt, 5998.v2.txt, Kafka5998.zip, Topology.txt, > exc.txt, props.txt, streams.txt > > > I have one kafka broker and one kafka stream running... I am running its > since two days under load of around 2500 msgs per second.. On third day am > getting below exception for some of the partitions, I have 16 partitions only > 0_0 and 0_1 gives this error > {{09:43:25.955 [ks_0_inst-StreamThread-6] WARN > o.a.k.s.p.i.ProcessorStateManager - Failed to write checkpoint file to > /data/kstreams/rtp-kafkastreams/0_1/.checkpoint: > java.io.FileNotFoundException: > /data/kstreams/rtp-kafkastreams/0_1/.checkpoint.tmp (No such file or > directory) > at java.io.FileOutputStream.open(Native Method) ~[na:1.7.0_111] > at java.io.FileOutputStream.(FileOutputStream.java:221) > ~[na:1.7.0_111] > at java.io.FileOutputStream.(FileOutputStream.java:171) > ~[na:1.7.0_111] > at > org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:73) > ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.ProcessorStateManager.checkpoint(ProcessorStateManager.java:324) > ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:267) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:260) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:254) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.AssignedTasks$1.apply(AssignedTasks.java:322) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:415) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.AssignedTasks.commit(AssignedTasks.java:314) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:700) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:683) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:523) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > 09:43:25.974 [ks_0_inst-StreamThread-15] WARN > o.a.k.s.p.i.ProcessorStateManager - Failed to write checkpoint file to > /data/kstreams/rtp-kafkastreams/0_0/.checkpoint: > java.io.FileNotFoundException: > /data/kstreams/rtp-kafkastreams/0_0/.checkpoint.tmp (No such file or > directory) > at java.io.FileOutputStream.open(Native Method) ~[na:1.7.0_111] > at java.io.FileOutputStream.(FileOutputStream.java:221) > ~[na:1.7.0_111] > at java.io.FileOutputStream.(FileOutputStream.java:171) > ~[na:1.7.0_111] > at > org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:73) > ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.ProcessorStateManager.checkpoint(ProcessorStateManager.java:324) > ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:267) >
[jira] [Created] (KAFKA-8615) Change to track partition time breaks TimestampExtractor
Bill Bejeck created KAFKA-8615: -- Summary: Change to track partition time breaks TimestampExtractor Key: KAFKA-8615 URL: https://issues.apache.org/jira/browse/KAFKA-8615 Project: Kafka Issue Type: Bug Affects Versions: 2.3.0 Reporter: Bill Bejeck Assignee: Bill Bejeck >From the users mailing list {noformat} am testing the new version 2.3 for Kafka Streams specifically. I have noticed that now, the implementation of the method extract from the interface org.apache.kafka.streams.processor.TimestampExtractor *public* *long* extract(ConsumerRecord record, *long* previousTimestamp) is always returning -1 as value. Previous version 2.2.1 was returning the correct value for the record partition. {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8558) KIP-479 - Add Materialized Overload to KStream#Join
Bill Bejeck created KAFKA-8558: -- Summary: KIP-479 - Add Materialized Overload to KStream#Join Key: KAFKA-8558 URL: https://issues.apache.org/jira/browse/KAFKA-8558 Project: Kafka Issue Type: Improvement Components: streams Reporter: Bill Bejeck Assignee: Bill Bejeck Fix For: 2.4.0 To prevent a topology incompatibility with the release of 2.4 and the naming of Join operations we'll add an overloaded KStream#join method accepting a Materialized parameter. The overloads will apply to all flavors of KStream#join (inner, left, and right). Additionally, new methods withQueryingEnabled and withQueryingDisabled are going to be added to Materialized -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6874) Add Configuration Allowing for Optional Topology Optimization
[ https://issues.apache.org/jira/browse/KAFKA-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-6874. Resolution: Duplicate Fix Version/s: 2.0.0 Duplicate of https://issues.apache.org/jira/browse/KAFKA-6935 Resolved with PR [https://github.com/apache/kafka/pull/5071] > Add Configuration Allowing for Optional Topology Optimization > -- > > Key: KAFKA-6874 > URL: https://issues.apache.org/jira/browse/KAFKA-6874 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bill Bejeck >Assignee: Bill Bejeck >Priority: Major > Fix For: 2.0.0 > > > With the release of 2.0 Streams will introduce topology optimization. We > should provide a config with a default value of false allowing users to > enable/disable optimization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8501) Remove key and value from exception message
[ https://issues.apache.org/jira/browse/KAFKA-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8501. Resolution: Fixed > Remove key and value from exception message > --- > > Key: KAFKA-8501 > URL: https://issues.apache.org/jira/browse/KAFKA-8501 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Badai Aqrandista >Assignee: Carlos Manuel Duclos Vergara >Priority: Major > Labels: easy-fix, newbie > Fix For: 2.4.0 > > > KAFKA-7510 moves the WARN messages that contain key and value to TRACE level. > But the exceptions still contain key and value. These are the two in > RecordCollectorImpl: > > [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java#L185-L196] > > [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java#L243-L254] > > Can these be modified as well to remove key and value from the error message, > which we don't know what log level it will be printed in? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8331) Add system test for enabling static membership on KStream
[ https://issues.apache.org/jira/browse/KAFKA-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8331. Resolution: Fixed > Add system test for enabling static membership on KStream > - > > Key: KAFKA-8331 > URL: https://issues.apache.org/jira/browse/KAFKA-8331 > Project: Kafka > Issue Type: Sub-task > Components: streams >Affects Versions: 2.4.0 >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Major > Fix For: 2.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8469) Named Suppress Operator Needs to increment Name Counter
[ https://issues.apache.org/jira/browse/KAFKA-8469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8469. Resolution: Not A Problem > Named Suppress Operator Needs to increment Name Counter > --- > > Key: KAFKA-8469 > URL: https://issues.apache.org/jira/browse/KAFKA-8469 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Bill Bejeck >Assignee: Bill Bejeck >Priority: Major > > The {{KTable#suppress}} operator can take a user-supplied name for the > operator. If the user does supply a name, the code should still increment > the name counter index to ensure any downstream operators that use the > generated name don't change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8469) Named Suppress Operator Needs to increment Name Counter
Bill Bejeck created KAFKA-8469: -- Summary: Named Suppress Operator Needs to increment Name Counter Key: KAFKA-8469 URL: https://issues.apache.org/jira/browse/KAFKA-8469 Project: Kafka Issue Type: Bug Components: streams Reporter: Bill Bejeck Assignee: Bill Bejeck The {{KTable#suppress}} operator can take a user-supplied name for the operator. If the user does supply a name, the code should still increment the name counter index to ensure any downstream operators that use the generated name don't change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8187) State store record loss across multiple reassignments when using standby tasks
[ https://issues.apache.org/jira/browse/KAFKA-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8187. Resolution: Fixed merged to trunk and cherry-picked to 2.3, 2.2 and 2.1 > State store record loss across multiple reassignments when using standby tasks > -- > > Key: KAFKA-8187 > URL: https://issues.apache.org/jira/browse/KAFKA-8187 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.0.1 >Reporter: William Greer >Assignee: Lifei Chen >Priority: Major > Fix For: 2.3.0, 2.1.2, 2.2.1 > > > Overview: > There is a race condition that can cause a partitioned state store to be > missing records up to an offset when using standby tasks. > When a reassignment occurs and a task is migrated to a StandbyTask in another > StreamThread/TaskManager on the same JVM, there can be lock contention that > prevents the StandbyTask on the currently assigned StreamThread from > acquiring the lock and to not retry acquiring the lock because all of the > active StreamTasks are running for that StreamThread. If the StandbyTask does > not acquire the lock before the StreamThread enters into the RUNNING state, > then the StandbyTask will not consume any records. If there is no subsequent > reassignment before the second execution of the stateDirCleaner Thread, then > the task directory for the StandbyTask will be deleted. When the next > reassignment occurs the offset that was read by the StandbyTask at creation > time before acquiring the lock will be written back to the state store > directory, this re-creates the state store directory. > An example: > StreamThread(A) and StreamThread(B) are running on the same JVM in the same > streams application. > StreamThread(A) has StandbyTask 1_0 > StreamThread(B) has no tasks > A reassignment is triggered by another host in the streams application fleet. > StreamThread(A) is notified with a PARTITIONS_REVOKED event of the threads > one task > StreamThread(B) is notified with a PARTITIONS_ASSIGNED event of a standby > task for 1_0 > Here begins the race condition. > StreamThread(B) creates the StandbyTask which reads the current checkpoint > from disk. > StreamThread(B) then attempts to updateNewAndRestoringTasks() for it's > assigned tasks. [0] > StreamThread(B) initializes the new tasks for the active and standby tasks. > [1] [2] > StreamThread(B) attempts to lock the state directory for task 1_0 but fails > with a LockException [3], since StreamThread(A) still holds the lock. > StreamThread(B) returns true from updateNewAndRestoringTasks() due to the > check at [4] which only checks that the active assigned tasks are running. > StreamThread(B) state is set to RUNNING > StreamThread(A) closes the previous StandbyTask specifically calling > closeStateManager() [5] > StreamThread(A) state is set to RUNNING > Streams application for this host has completed re-balancing and is now in > the RUNNING state. > State at this point is the following: State directory exists for 1_0 and all > data is present. > Then at a period that is 1 to 2 intervals of [6](which is default of 10 > minutes) after the reassignment had completed the stateDirCleaner thread will > execute [7]. > The stateDirCleaner will then do [8], which finds the directory 1_0, finds > that there isn't an active lock for that directory, acquire the lock, and > deletes the directory. > State at this point is the following: State directory does not exist for 1_0. > When the next reassignment occurs. The offset that was read by > StreamThread(B) during construction of the StandbyTask for 1_0 will be > written back to disk. This write re-creates the state store directory and > writes the .checkpoint file with the old offset. > State at this point is the following: State directory exists for 1_0 with a > '.checkpoint' file in it, but there is no other state store data in the > directory. > If this host is assigned the active task for 1_0 then all the history in the > state store will be missing from before the offset that was read at the > previous reassignment. > If this host is assigned the standby task for 1_0 then the lock will be > acquired and the standby will start to consume records, but it will still be > missing all records from before the offset that was read at the previous > reassignment. > If this host is not assigned 1_0, then the state directory will get cleaned > up by the stateDirCleaner thread 10 to 20 minutes later and the record loss > issue will be hidden. > [0] > https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L865-L869 > [1] >
[jira] [Resolved] (KAFKA-8413) Add possibility to do repartitioning on KStream
[ https://issues.apache.org/jira/browse/KAFKA-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8413. Resolution: Not A Problem > Add possibility to do repartitioning on KStream > --- > > Key: KAFKA-8413 > URL: https://issues.apache.org/jira/browse/KAFKA-8413 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Priority: Minor > Attachments: topology-1.png, topology-2.png > > > Consider following code: > {code:java} > final KStream streamByProfileId = streamsBuilder >.stream("input-topic", Consumed.with(Serdes.String(), Serdes.String())) >.selectKey((key, value) -> value); > streamByProfileId >.groupByKey() >.aggregate( > () -> 0d, > (key, value, aggregate) -> aggregate, > Materialized.as("store-1") >); > streamByProfileId >.groupByKey() >.aggregate( > () -> 0d, > (key, value, aggregate) -> aggregate, > Materialized.as("store-2") >); > {code} > > This code will generate following topology: > {code:java} > Topologies: > Sub-topology: 0 > Source: KSTREAM-SOURCE-00 (topics: [input-topic]) > --> KSTREAM-KEY-SELECT-01 > Processor: KSTREAM-KEY-SELECT-01 (stores: []) > --> KSTREAM-FILTER-04, KSTREAM-FILTER-08 > <-- KSTREAM-SOURCE-00 > Processor: KSTREAM-FILTER-04 (stores: []) > --> KSTREAM-SINK-03 > <-- KSTREAM-KEY-SELECT-01 > Processor: KSTREAM-FILTER-08 (stores: []) > --> KSTREAM-SINK-07 > <-- KSTREAM-KEY-SELECT-01 > Sink: KSTREAM-SINK-03 (topic: store-1-repartition) > <-- KSTREAM-FILTER-04 > Sink: KSTREAM-SINK-07 (topic: store-2-repartition) > <-- KSTREAM-FILTER-08 > Sub-topology: 1 > Source: KSTREAM-SOURCE-05 (topics: [store-1-repartition]) > --> KSTREAM-AGGREGATE-02 > Processor: KSTREAM-AGGREGATE-02 (stores: [store-1]) > --> none > <-- KSTREAM-SOURCE-05 > Sub-topology: 2 > Source: KSTREAM-SOURCE-09 (topics: [store-2-repartition]) > --> KSTREAM-AGGREGATE-06 > Processor: KSTREAM-AGGREGATE-06 (stores: [store-2]) > --> none > <-- KSTREAM-SOURCE-09 > > {code} > Kafka Streams creates two repartition topics for each `groupByKey` operation. > In this example, two repartition topics are not really necessary and > processing can be done with one sub-topology. > > Kafka Streams user, in DSL, may specify repartition topic manually using > *KStream#through* method: > {code:java} > final KStream streamByProfileId = streamsBuilder >.stream("input-topic") >.selectKey((key, value) -> value) >.through("repartition-topic"); > streamByProfileId >.groupByKey() >.aggregate( > () -> 0d, > (key, value, aggregate) -> aggregate, > Materialized.as("store-1") >); > streamByProfileId >.groupByKey() >.aggregate( > () -> 0d, > (key, value, aggregate) -> aggregate, > Materialized.as("store-2") >); > {code} > > > {code:java} > Topologies: > Sub-topology: 0 > Source: KSTREAM-SOURCE-00 (topics: [input-topic]) > --> KSTREAM-KEY-SELECT-01 > Processor: KSTREAM-KEY-SELECT-01 (stores: []) > --> KSTREAM-SINK-02 > <-- KSTREAM-SOURCE-00 > Sink: KSTREAM-SINK-02 (topic: repartition-topic) > <-- KSTREAM-KEY-SELECT-01 > Sub-topology: 1 > Source: KSTREAM-SOURCE-03 (topics: [repartition-topic]) > --> KSTREAM-AGGREGATE-04, KSTREAM-AGGREGATE-05 > Processor: KSTREAM-AGGREGATE-04 (stores: [store-1]) > --> none > <-- KSTREAM-SOURCE-03 > Processor: KSTREAM-AGGREGATE-05 (stores: [store-2]) > --> none > <-- KSTREAM-SOURCE-03 > {code} > > While this gives possibility to optimizes Kafka Streams application, user > still has to manually create repartition topic with correct number of > partitions based on input topic. It would be great if in DSL we could have > something like *repartition()* operation on *KStream* which can generate > repartition topic based on user command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8416) Improve Documentation for Enabling Optimizations
Bill Bejeck created KAFKA-8416: -- Summary: Improve Documentation for Enabling Optimizations Key: KAFKA-8416 URL: https://issues.apache.org/jira/browse/KAFKA-8416 Project: Kafka Issue Type: Improvement Reporter: Bill Bejeck To enable optimizations, users need to set the {{StreamsConfig.TOPOLOGY_OPTIMIZATION}} setting to "all". But in addition to setting the config users need to pass in the {{Properties}} object to the {{StreamBuilder#build()}} method as well. We should make a pass over the existing documentation and Javadoc to make sure this required step is clear. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7201) Optimize repartition operations
[ https://issues.apache.org/jira/browse/KAFKA-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-7201. Resolution: Fixed Resolved via https://issues.apache.org/jira/browse/KAFKA-6761 > Optimize repartition operations > --- > > Key: KAFKA-7201 > URL: https://issues.apache.org/jira/browse/KAFKA-7201 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bill Bejeck >Assignee: Bill Bejeck >Priority: Major > > When the topology has a key changing operation, any downstream processors > using the key will automatically create a repartition topic. In most cases > these multiple repartition topics can be collapsed into one repartition > operation occurring immediately after the key changing operation, thus > reducing streams overall footprint. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8399) Add back `internal.leave.group.on.close` config for KStreams
[ https://issues.apache.org/jira/browse/KAFKA-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8399. Resolution: Fixed > Add back `internal.leave.group.on.close` config for KStreams > > > Key: KAFKA-8399 > URL: https://issues.apache.org/jira/browse/KAFKA-8399 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Major > > The behavior for KStream rebalance default has changed from no leave group to > leave group. We should add it back for system test pass, reduce the risk of > being detected not working in other public cases. > Reference: [https://github.com/apache/kafka/pull/6673] -- This message was sent by Atlassian JIRA (v7.6.3#76005)