Can we get a 2.3.1 release?
Hi all, 2.3.0 was released June 25, 2019. There are 22 fixes scheduled for inclusion in 2.3.1 so far, 18 of which are resolved/closed. https://issues.apache.org/jira/browse/KAFKA-8736?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.3.1 Do folks have a sense as to when they would like to see a release? There's at least two show-stoppers that are preventing my team from upgrading, which we're very eager to do. The hotfixes we've tried are very promising. Thanks, Jonathan
[jira] [Created] (KAFKA-8731) InMemorySessionStore throws NullPointerException on startup
Jonathan Gordon created KAFKA-8731:
--
Summary: InMemorySessionStore throws NullPointerException on
startup
Key: KAFKA-8731
URL: https://issues.apache.org/jira/browse/KAFKA-8731
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.3.0
Reporter: Jonathan Gordon
I receive a NullPointerException on startup when trying to use the new
InMemorySessionStore via Stores.inMemorySessionStore(...) using the DSL.
Here's the stack trace:
{{ERROR [2019-07-29 21:56:52,246]
org.apache.kafka.streams.processor.internals.StreamThread: stream-thread
[trace_indexer-c8439020-12af-4db2-ad56-3e58cd56540f-StreamThread-1] Encountered
the following error during processing:}}
{{! java.lang.NullPointerException: null}}
{{! at
org.apache.kafka.streams.state.internals.InMemorySessionStore.remove(InMemorySessionStore.java:123)}}
{{! at
org.apache.kafka.streams.state.internals.InMemorySessionStore.put(InMemorySessionStore.java:115)}}
{{! at
org.apache.kafka.streams.state.internals.InMemorySessionStore.lambda$init$0(InMemorySessionStore.java:93)}}
{{! at
org.apache.kafka.streams.processor.internals.StateRestoreCallbackAdapter.lambda$adapt$1(StateRestoreCallbackAdapter.java:47)}}
{{! at
org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89)}}
{{! at
org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:92)}}
{{! at
org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:317)}}
{{! at
org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:92)}}
{{! at
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:328)}}
{{! at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:867)}}
{{! at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)}}
{{! at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)}}
Here's the Slack thread:
[https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1564438647169600]
Here's a current PR aimed at fixing the issue:
[https://github.com/apache/kafka/pull/7132]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (KAFKA-7748) Add wall clock TimeDefinition for suppression of intermediate events
Jonathan Gordon created KAFKA-7748:
--
Summary: Add wall clock TimeDefinition for suppression of
intermediate events
Key: KAFKA-7748
URL: https://issues.apache.org/jira/browse/KAFKA-7748
Project: Kafka
Issue Type: New Feature
Components: streams
Affects Versions: 2.1.0
Reporter: Jonathan Gordon
Currently, Kafka Streams offers the ability to suppress intermediate events
based on either RecordTime or WindowEndTime, which are in turn defined by
stream time:
{{Suppressed.untilTimeLimit(final Duration timeToWaitForMoreEvents, final
BufferConfig bufferConfig)}}
It would be helpful to have another option that would allow suppression of
intermediate events based on wall clock time. This would allow us to only
produce a limited number of aggregates independent of their stream time (which
in our case is event time).
For reference, here's the relevant KIP:
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables#KIP-328:AbilitytosuppressupdatesforKTables-Best-effortratelimitperkey]
And here's the relevant Confluent Slack thread:
https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1544468349201700
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7652) Kafka Streams Session store performance degradation from 0.10.2.2 to 0.11.0.0
Jonathan Gordon created KAFKA-7652: -- Summary: Kafka Streams Session store performance degradation from 0.10.2.2 to 0.11.0.0 Key: KAFKA-7652 URL: https://issues.apache.org/jira/browse/KAFKA-7652 Project: Kafka Issue Type: Bug Affects Versions: 2.0.1, 2.0.0, 1.1.1, 0.11.0.3, 0.11.0.2, 0.11.0.1, 0.11.0.0 Reporter: Jonathan Gordon Attachments: kafka_10_2_1_flushes.txt, kafka_11_0_3_flushes.txt Here's the original thread from the mailing list: https://lists.apache.org/thread.html/97d620f4fd76be070ca4e2c70e2fda53cafe051e8fc4505dbcca0321@%3Cusers.kafka.apache.org%3E We are attempting to upgrade our Kafka Streams application from 0.10.2.1 but experience a severe performance degradation. The highest amount of CPU time seems spent in retrieving from the local cache. Here's an example with 0.11.0.0: [https://i.imgur.com/l5VEsC2.png] When things are running smoothly we're gated by retrieving from the state store with acceptable performance. Here's an example with 0.10.2.1: [https://i.imgur.com/IHxC2cZ.png] Some investigation reveals that it appears we're performing about 3 orders magnitude more lookups on the NamedCache over a comparable time period. I've attached logs of the NamedCache flush logs for 0.10.2.1 and 0.11.0.3. We're using session windows and have the app configured for commit.interval.ms of 30 * 1000 and cache.max.bytes.buffering = 10485760 I'm happy to share more details if they would be helpful. Also happy to run tests on our data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
