Re: Subject: [VOTE] 2.4.1 RC0

Hi Bill, I built from source and ran unit and integration tests. They passed. There was a large number of skipped tests, but I'm assuming that is intentional. Cheers Eno On Tue, Mar 3, 2020 at 8:42 PM Eric Lalonde wrote: > > Hi, > > I ran: > $ https://github.com/elalonde/kafka/blob/master/bin/

Operationalizing Zookeeper and common gotchas

Hi folks, The team here has come up with a couple of clarifying tips for operationalizing Zookeeper for Kafka that we found missing from the official documentation, and passed them along to share. If you find them useful, I'm thinking of putting on https://cwiki.apache.org/confluence/display/KAFKA

Re: [VOTE] 2.1.1 RC2

+1 passes unit + integration tests. Eno On Fri, Feb 8, 2019 at 11:10 PM Magnus Edenhill wrote: > +1 > > Passes librdkafka test suite. > > Den fre 8 feb. 2019 kl 21:02 skrev Colin McCabe : > > > Hi all, > > > > This is the third candidate for release of Apache Kafka 2.1.1. This > > release inclu

Re: [VOTE] 2.1.1 RC1

n the Jenkins build we did here: > https://builds.apache.org/job/kafka-2.1-jdk8/118/ > > Perhaps there is an environment issue at play here? Do you get the same > failures running those tests on the 2.1 release? > > Best, > Colin > > On Wed, Jan 30, 2019, at 09:11, Eno Theresk

Re: [VOTE] 2.1.1 RC1

Hi Colin, I've been running the tests and so far I get the following failures. Are they known? kafka.server.ReplicaManagerQuotasTest > shouldGetBothMessagesIfQuotasAllow FAILED kafka.server.ReplicaManagerQuotasTest > testCompleteInDelayedFetchWithReplicaThrottling FAILED kafka.server.ReplicaManag

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

The general answer depends on what control plane software is taking care of your Kafka deployment. You probably have a layer that launches Kafka instances and monitors their health, right? If so, that layer should take care of the mapping between instances and broker IDs and keep that in a table pe

Re: [VOTE] 2.1.0 RC1

Built code and ran tests. Getting a single integration test failure: kafka.log.LogCleanerParameterizedIntegrationTest > testCleansCombinedCompactAndDeleteTopic[3] FAILED java.lang.AssertionError: Contents of the map shouldn't change expected: (340,340), 5 -> (345,345), 10 -> (350,350), 14 -> (

Re: [VOTE] 2.0.1 RC0

Anything else holding this up? Thanks Eno On Thu, Nov 1, 2018 at 10:27 AM Jakub Scholz wrote: > +1 (non-binding) ... I used the staged binaries and run tests with > different clients. > > On Fri, Oct 26, 2018 at 4:29 AM Manikumar > wrote: > > > Hello Kafka users, developers and client-develope

Re: Problems trying to make kafka 'rack-aware'

Hi Bryan, I did a simple check with starting a broker with no rack id and then restarting with a rack id and I can confirm I could get the rack id from zookeeper after the restart. This was on trunk. Does that basic check work for you (i.e., without reassigning partitions)? Thanks Eno On Fri, Se

Re: Continue to consume messages when exception occurs in Kafka Stream

Hi Duy, What kind of exception are you getting? With KIP-161 (checked in trunk) we allow log-and-skip type exception handlers for deserialization errors: https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers

Re: Forwarding consumer with kafka streams

Hi Ricardo, Kafka Streams should handle that case as well. What streams config are you using, could you share it? There is one parameter that is called “ConsumerConfig.AUTO_OFFSET_RESET_CONFIG” and by default it’s set to “earliest”. Any chance your app has changed it to “latest”? Thanks Eno >

Re: Kafka Streams not auto-creating the state store changelog topic

s. > Also, if I have regex for the input topic on kafka streams and a new topic > is added to kafka matching the regex, the application will break? > > On Fri, Aug 4, 2017, 8:33 PM Eno Thereska <mailto:eno.there...@gmail.com>> wrote: > >> Hi, >> >>

Re: Kafka Streams not auto-creating the state store changelog topic

rg.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582) > at > org.apache.kafka.streams.processor.interna

Re: Kafka Streams not auto-creating the state store changelog topic

Hi Anish, Could you give more info on how you create the state stores in your code? Also could you copy-paste the exact error message from the log? Thanks Eno > On Aug 4, 2017, at 9:05 AM, Anish Mashankar wrote: > > I have a new application, call it streamsApp with state stores S1 and S2. > So

Re: Do we have to query localWindowStore in same java instance we are creating the store

Hi Sachin, 1. You can run a remote query and we provide some example code (https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/ ) however by default A

Re: Windows OS platform support

Hi Harish, I believe many people/orgs use it on Windows. We rely on the community to test/fix/answer any Windows questions, same as with Linux or MacOS. However, based on what I've observed, perhaps there are more people answering Linux-related questions. Eno > On 14 Jul 2017, at 13:24, haris

Re: State management & restore functionality

tp://docs.confluent.io/current/streams/developer-guide.html#streams-developer-guide-standby-replicas <http://docs.confluent.io/current/streams/developer-guide.html#streams-developer-guide-standby-replicas>. Hope this helps, Eno > regards. > > On Fri, Jul 14, 2017 at 6:23 PM, Eno T

Re: Kafka streams: Record linkage / collecting all messages linked to one entity

onnect the > messages with the match-keys using edges. Now the message nodes are connected > through the match-key nodes. Each entity is defined by the graph that > connects all messages that are linked together. > > Kind regards, > > Wladislaw > > Eno Thereska

Re: State management & restore functionality

Hi Debasish, Your intuition about the first part is correct. Kafka Streams automatically assigns a partition of a topic to a task in an instance. It will never be the case that the same partition is assigned to two tasks. About the merging or changing of partitions part, it would help if we kn

Re: Is this a decent use case for Kafka Streams?

From just looking at your description of the problem, I'd say yes, this looks like a typical scenario for Kafka Streams. Kafka Streams supports exactly once semantics too in 0.11. Cheers Eno > On 12 Jul 2017, at 17:06, Stephen Powis wrote: > > Hey! I was hoping I could get some input from peo

Re: Kafka streams: Record linkage / collecting all messages linked to one entity

Hi Wladislaw, Would splitting the one topic into multiple topics be acceptable at all? E.g., you could use the "branch" function in the DSL to split the messages and send to different topics. Then, once you have multiple topics you can do the joins etc. Thoughts? Thanks Eno > On 11 Jul 2017,

Re: kafka-streams app(s) stopped consuming new events

It’s hard to tell, the logs do not contain much, I agree. It could be a number of things. If it’s happening as you say on restart as well (so it’s reproducible), any chance you could start streaming with DEBUG logs on and collect those logs? I’m hoping something shows up there. Thanks, Eno >

Re: Kafka Stream invalid partitions

se this API as follows: > > ... > .groupByKey() > .aggregate(...) > .toStream(...); > > > Please let me know if you need addiotional information. > > Thanks, > > > 2017-06-27 11:39 GMT+02:00 Eno Thereska : > >> Hi there, >> >> Thanks for th

Re: Kafka Stream invalid partitions

Hi there, Thanks for the report. What version of Kafka are you using? Also, between runs do you change the number of partitions for your topics? I’m trying to figure out how this problem happens, any information on what is changing in between runs is appreciated. Thanks, Eno > On Jun 27, 2017

Re: Kafka 10.2.1 KStreams rebalancing issue

Hi Sameer, Could you elaborate on your question? Are you concerned that machine2 does not have any tasks in the beginning? Could you share your streams configuration? In particular how many threads does each stream instance have? Also how many topics and partitions do you have? Thanks, Eno

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

is no need for the response config option. Thanks Eno > > Guozhang > > > > > > > > > On Wed, Jun 21, 2017 at 1:39 AM, Eno Thereska <mailto:eno.there...@gmail.com>> > wrote: > >> Thanks Guozhang, >> >> I’ve updated the K

Re: [DISCUSS] Streams DSL/StateStore Refactoring

roposed > APIs? > > Meanwhile Damian could you provide a formal set of APIs for people to > exercise on them? Also could you briefly describe how custom storage > engines could be swapped in with the above APIs? > > > > Guozhang > > > On Wed, Jun 21, 2017 at 9:0

Re: [DISCUSS] Streams DSL/StateStore Refactoring

. >> >> Yes i agree. We could apply this same approach to most of the operations >> where we have multiple overloads, i.e., we have a single method for each >> operation that takes the required parameters and everything else is >> specified as you have done above. &

Re: [DISCUSS] Streams DSL/StateStore Refactoring

(cc’ing user-list too) Given that we already have StateStoreSuppliers that are configurable using the fluent-like API, probably it’s worth discussing the other examples with joins and serdes first since those have many overloads and are in need of some TLC. So following your example, I guess yo

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

probably would like to not rename the > config again. Also from the experience of `default partitioner` and > `default timestamp extractor` we may also make sure that the passed in > object can be either a string "class name" or a class object? > > > Guozhang >

Re: KStream Usage spikes memory consumption and breaks Kafka

Could you provide some configuration information and more context? What application are you running, when is it running out of memory? Otherwise it's hard to tell. Eno > On 20 Jun 2017, at 22:15, IT Consultant <0binarybudd...@gmail.com> wrote: > > Hi All , > > Kafka instance is breaking down w

Re: [VOTE] 0.11.0.0 RC1

+1 (non-binding) passes Kafka Streams tests. Thanks, Eno > On 19 Jun 2017, at 06:49, Magnus Edenhill wrote: > > +1 (non-binding) > > Passes librdkafka integration tests (v0.9.5 and master) > > > 2017-06-19 0:32 GMT+02:00 Ismael Juma : > >> Hello Kafka users, developers and client-developers,

Re: Scala type mismatch after upgrade to 0.10.2.1

Hi there, Yeah with 0.10.2 some Scala applications need to explicitly declare the type of certain variables. See this: http://docs.confluent.io/current/streams/upgrade-guide.html#scala Thanks Eno > On Jun 17, 2017, at 10:03

Re: Slow Consumer Group Startup

While I do have some logs its not trivial to share since the logs are >> across 16 JVMs and a few different hosts. >> >> On Wed, Jun 14, 2017 at 10:34 AM Eno Thereska >> wrote: >> >>> The delay in that KIP is just 3 seconds, not minutes though, right? Would >

Re: Single Key Aggregation

I'm not sure if I fully understand this but let me check: - if you start 2 instances, one instance will process half of the partitions, the other instance will process the other half - for any given key, like key 100, it will only be processed on one of the instances, not both. Does this help?

Re: KStream and KTable different behaviour on filter() operation

Yeah the semantics are slightly different. For a KTable, a null value just means that the record is a tombstone, and will be anyways ignored by subsequent processing: http://docs.confluent.io/current/streams/javadocs/org/apache/kafka/streams/kstream/KTable.html#filter-org.apache.kafka.streams.kst

Re: Kafka Streams vs Spark Streaming : reduce by window

olo. > > > Paolo Patierno > Senior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Windows Embedded & IoT > Microsoft Azure Advisor > > Twitter : @ppatierno<http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatiern

Re: Kafka Streams vs Spark Streaming : reduce by window

Hi Paolo, That is indeed correct. We don’t believe in closing windows in Kafka Streams. You could reduce the number of downstream records by using record caches: http://docs.confluent.io/current/streams/developer-guide.html#record-caches-in-the-dsl

Re: Question on Support provision

Hi Sofia, Thank you for your recent enquiry for Kafka support services. Confluent employs some of the world’s foremost Apache Kafka experts, and that expertise shows in the level of support we can provide. The subscription offers a scali

Re: Slow Consumer Group Startup

The delay in that KIP is just 3 seconds, not minutes though, right? Would you have any logs to share? Thanks Eno > On 14 Jun 2017, at 16:14, Bryan Baugher wrote: > > Our consumer group isn't doing anything stateful and we've seen this > behavior for existing groups as well. It seems like timing

Re: getting intermittent TimeoutException at producer side in streams application

Hi Sachin, As Damian mentioned it'd be useful to see some logs from both broker and streams. One thing that comes to mind is whether your topics are replicated at all. You could try setting the replication factor of streams topics (e.g., changelogs and repartition topics) to 2 or 3 using Strea

Re: Kafka Streams Failed to rebalance error

; >>>>> On Wed, Jun 7, 2017 at 3:18 PM, João Peixoto >>> >>>>> wrote: >>>>> >>>>>> Eno before I do so I just want to be sure this would not be a >>>> duplicate. >>>>> I >>>>>&

Re: Kafka Streams Failed to rebalance error

Hi there, This might be a bug, would you mind opening a JIRA (copy-pasting below is sufficient). Thanks Eno > On 7 Jun 2017, at 21:38, João Peixoto wrote: > > I'm using Kafka Streams 0.10.2.1 and I still see this error > > 2017-06-07 20:28:37.211 WARN 73 --- [ StreamThread-1] > o.a.k.s.p.int

Re: Reliably implementing global KeyValueStore#get

ave a very fast replication > setup (assume it takes zero time, for the sake of argument) -- I'm fairly > sure there's still a race here as this exception only fires *during a > migration* > not *after a migration that may have invalidated your metadata lookup > complet

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

Comments inline: > On 5 Jun 2017, at 18:19, Jan Filipiak wrote: > > Hi > > just my few thoughts > > On 05.06.2017 11:44, Eno Thereska wrote: >> Hi there, >> >> Sorry for the late reply, I was out this past week. Looks like good progress >> was

Re: Reliably implementing global KeyValueStore#get

Hi Steven, Do you know beforehand if a key exists? If you know that and are getting null() the code will have to retry by refreshing the metadata and going to the new instance. If you don’t know beforehand if a key exists or not you might have to check all instances of a store to make sure. E

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

Hi there, Sorry for the late reply, I was out this past week. Looks like good progress was made with the discussions either way. Let me recap a couple of points I saw into one big reply: 1. Jan mentioned CRC errors. I think this is a good point. As these happen in Kafka, before Kafka Streams g

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

izer to a method that creates a state store. In which case it would > be a user error and probably should just abort? > > Thanks, > Damian > > On Fri, 26 May 2017 at 16:32 Eno Thereska wrote: > >> See latest reply to Jan's note. I think I unnecessarily broadened

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

handlers instead of one? One for deserialization exception and one for > all other exceptions from user code? > > Just a thought. > > > -Matthias > > On 5/26/17 7:49 AM, Jim Jagielski wrote: >> >>> On May 26, 2017, at 5:13 AM, Eno Thereska wrote: >>

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

esponse handle(that guy here >>>>>>> ConsumerRecord<||byte||[], > ||byte||[]> record, Exception exception);| > |}| > || > |public| |enum| |HandlerResponse {| > |||/* continue with processing */| > |||CONTINUE(||1||), | > |||/* fail the processing and stop */| > |

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

processing. > > would it not make more sense to get the ProcessorName + object object for > processing and > statestore or topic name + byte[] byte[] for serializers? maybe passing in > the used serdes? > > Best Jan > > > > On 25.05.2017 11:47, Eno The

Re: Streams error handling

> to the error and handle it - possibly by replacing the message, sending a > message to a new stream, or even putting it back on the original stream for > retry. > > Regards > MikeG > > On Wed, 24 May 2017 at 10:09, Eno Thereska <mailto:eno.there...@gmail.com>> w

[DISCUSS]: KIP-161: streams record processing exception handlers

Hi there, I’ve added a KIP on improving exception handling in streams: KIP-161: streams record processing exception handlers. https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+record+processing+exception+handlers

Re: Streams error handling

thods would be able to examine the error with the key/value prior > to the error and handle it - possibly by replacing the message, sending a > message to a new stream, or even putting it back on the original stream for > retry. > > Regards > MikeG > > On Wed, 24 May 2

Re: Streams error handling

Just a heads up that we're tracking this and other improvements in exception handling at https://issues.apache.org/jira/browse/KAFKA-5156 . Thanks Eno > On 23 May 2017, at 17:31, Mike Gould wrote: > > That's great for the value but not the key

Re: Kafka Streams RocksDB permanent StateStoreSupplier seemingly never deletes data

't see how it could be CPU starved. Still I will of > course try your suggestion. > > Thanks again, > V. > > > On Tue, May 16, 2017 at 5:15 PM, Eno Thereska > wrote: > >> Which version of Kafka are you using? It might be that RocksDb doesn't get >>

Re: Kafka Streams RocksDB permanent StateStoreSupplier seemingly never deletes data

ularly. > I can certainly try streams 0.10.2.1 if it's compatible with Kafka 0.10.1. > I have not checked the compatibility matrix yet. > > Thanks for the help, > V. > > On Tue, 16 May 2017 at 17:57, Eno Thereska wrote: > >> Thanks. Which RocksDb files are growi

Re: Kafka Streams RocksDB permanent StateStoreSupplier seemingly never deletes data

so I don't see how it could be CPU starved. Still I will of >> course try your suggestion. >> >> Thanks again, >> V. >> >> >> On Tue, May 16, 2017 at 5:15 PM, Eno Thereska >> wrote: >> >>> Which version of Kafka are you using

Re: KafkaStreams reports RUNNING even though all StreamThreads has crashed

Hi Andreas, Thanks for reporting. This sounds like a bug, but could also be a semantic thing. Couple of questions: - which version of Kafka are you using? - what is the nature of the failure of the threads, e.g., how come they have all crashed? If all threads crash, was there an exception they

Re: Can state stores function as a caching layer for persistent storage

t;>>> >>>> There's very little information about the latter, at least that I could >>>> find, but an entry seems to be added whenever a record enters the >>>> "aggregate", but the state store "consumer" of this topic only updates >

Re: Kafka Streams RocksDB permanent StateStoreSupplier seemingly never deletes data

Which version of Kafka are you using? It might be that RocksDb doesn't get enough resources to compact the data fast enough. If that's the case you can try increasing the number of background compaction threads for RocksDb through the RocksDbConfigSetter class (see http://docs.confluent.io/curr

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

+1 thanks. Eno > On 16 May 2017, at 04:20, BigData dev wrote: > > Hi All, > Given the simple and non-controversial nature of the KIP, I would like to > start the voting process for KIP-157: Add consumer config options to > streams reset tool > > *https://cwiki.apache.org/confluence/display/KAFK

Re: Kafka-streams process stopped processing messages

time I restart this process it gets >> into >>> rebalancing state for several hours. >>> That mean that every time we need to deploy a new version of our app >> (which >>> can be several times a day) we have a down time of hours. >>> >&

Re: Kafka Streams reports: "The timestamp of the message is out of acceptable range"

Hi Frank, Could you confirm that you're using 0.10.2.1? This error was fixed ad part of this JIRA I believe: https://issues.apache.org/jira/browse/KAFKA-4861 Thanks Eno > On 14 May 2017, at 23:09, Frank Lyaruu wrote: > > Hi Kafka people... >

Re: Can state stores function as a caching layer for persistent storage

Hi João, Some answers inline: > On 12 May 2017, at 18:27, João Peixoto wrote: > > Thanks for the comments, here are some clarifications: > > I did look at interactive queries, if I understood them correctly it means > that my state store must hold all the results in order for it to be > querie

Re: Can state stores function as a caching layer for persistent storage

Hi there, A couple of general comments, plus some answers: - general comment: have you thought of using Interactive Queries to directly query the aggregate data, without needing to store them to an external database (see this blog: https://www.confluent.io/blog/unifying-stream-processing-and-i

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

+1 for me. I’m not sure we even need a KIP for this but it’s better to be safe I guess. Eno > On May 9, 2017, at 8:41 PM, BigData dev wrote: > > Hi, Everyone, > > Since this is a relatively simple change, I would like to start the voting > process for KIP-156: Add option "dry run" to Streams

Re: Kafka Stream stops polling new messages

ncreased to guard against failure for long stat recreation phases. > > Any thoughts? > > > -Matthias > > > On 5/3/17 8:48 AM, João Peixoto wrote: > > That'd be great as I'm not familiar with the protocol there > > On Wed, May 3, 2017 at 8:41 AM Eno There

Re: Large Kafka Streams deployment takes a long time to bootstrap

Hi there, I wanted to add something: how many CPU cores does each of your Kubernetes instance have? In 0.10.2.1 we noticed a regression in environments with 1 core as described in https://issues.apache.org/jira/browse/KAFKA-5174 . If you have

Re: Kafka-streams process stopped processing messages

ti wrote: > > Thanks Eno, > I already set the the recurve buffer size to 1MB > I will also try producer > > What about session timeout and heart beat timeout? Do you think it should > be increased? > > Thanks, > Shimi > > On Sat, 6 May 2017 at 0:21 Eno Thereska wr

Re: Kafka-streams process stopped processing messages

Sending metadata request (type=MetadataRequest, topics=) to node 2 > 2017-05-04 14:31:06,200 [StreamThread-1] DEBUG > o.a.k.c.c.i.AbstractCoordinator: Received GroupCoordinator response > ClientResponse(receivedTimeMs=1493908266200, latencyMs=5, > disconnected=false, requestHeader=

Re: Windowed aggregations memory requirements

This is a timely question and we've updated the documentation here on capacity planning and sizing for Kafka Streams jobs: http://docs.confluent.io/current/streams/sizing.html . Any feedback welcome. It has scenarios with windowed stores too

Re: Kafka Streams Failed to rebalance error

Hi, Which version of Kafka are you using? This should be fixed in 0.10.2.1, any chance you could try that release? Thanks Eno > On 3 May 2017, at 14:04, Sameer Kumar wrote: > > Hi, > > > I ran two nodes in my streams compute cluster, they were running fine for few > hours before outputting

Re: Kafka Stream stops polling new messages

x27;t actually crash. I > performed thread dumps on stale and not-stale instances which revealed the > common behavior and I collect and plot several Kafka metrics, including > "punctuate" durations, therefore I know it took a long time and eventually > finished. > > Jo

Re: Setting up Kafka & Kafka Streams for loading real-time and 'older' data concurrently

Just to add to this, there is a JIRA that tracks the fact that we don’t have an in-memory windowed store. https://issues.apache.org/jira/browse/KAFKA-4730 Eno > On May 3, 2017, at 12:42 PM, Damian Guy wrote: > > The windowed state store is onl

Re: Kafka Stream stops polling new messages

Hi there, Thanks for double checking. Does RocksDB actually crash or produce a crash dump? I’m curious how you know that the issue is https://github.com/facebook/rocksdb/issues/1121 , so just double checking with you. If that’s indeed the case,

Re: session window bug not fixed in 0.10.2.1?

Hi Ara, The PR https://github.com/apache/kafka/pull/2645 has gone to both trunk and 0.10.2.1, I just checked. What error are you seeing, could you give us an update? Thanks Eno On Fri, Apr 28, 2017 at 7:10 PM, Ara Ebrahimi wrote: > Hi, > > I upgraded to 0.10.2.1 yesterday, enabled caching for

Re: Kafka Streams 0.10.0.1 - multiple consumers not receiving messages

20170502_instancea_1234" > brokers - "localhost:9092,localhost:9093,localhost:9094" > zookeepers - "localhost:2181,localhost:2182,localhost:2183" > maxMessageBytes - 3000 > ssl - true > > Thanks, > Henry > -- > Henry Thacker > > On 2 May 2017 at 10:16:

Re: Failure on timestamp extraction for kafka streams 0.10.2.0

Hi Sachin, This should be fixed in 0.10.2.1, could you upgrade to that release? Here is JIRA: https://issues.apache.org/jira/browse/KAFKA-4861 . Thanks Eno > On May 2, 2017, at 8:43 AM, Sachin Mittal wrote: > > The timestamp of the message is

Re: Kafka Streams 0.10.0.1 - multiple consumers not receiving messages

Hi Henry, Could you share the streams configuration for your apps? I.e., the part where you assign application id and all the rest of the configs (just configs, not code). Thanks Eno > On May 2, 2017, at 8:53 AM, Henry Thacker wrote: > > Thanks all for your replies - I have checked out the do

Re: Kafka-streams process stopped processing messages

0.1.1 > After a LONG startup time (more than an hour) where the status of the group > was rebalancing, all the 3 processes started processing messages again. > > This all thing started after we hit a bug in our code (NPE) that crashed > the stream processing thread. > So now af

Re: Kafka-streams process stopped processing messages

Hi Shimi, Could you provide more info on your setup? How many kafka streams processes do you have and from how many partitions are they consuming from. If you have more processes than partitions some of the processes will be idle and won’t do anything. Eno > On Apr 30, 2017, at 5:58 PM, Shimi

Re: Kafka Streams 0.10.0.1 - multiple consumers not receiving messages

quot;).withStringKeys().withByteArrayValues().persistent().build(); >> >> KStream stream = builder.stream(topic); >> >> builder.addStateStore(stateStore); >> >> stream.process(this::buildStreamProcessor, "mystore"); >> >> stream.to(outputTopic); >>

Re: Kafka Streams 0.10.0.1 - multiple consumers not receiving messages

Hi Henry, Could you share the code that builds your topology so we see how the topics are passed in? Also, this would depend on what the streaming logic is doing with the topics, e.g., if you're joining them then both partitions need to be consumed by the same instance. Eno > On 28 Apr 2017, a

Re: Debugging Kafka Streams Windowing

Hi Mahendra, We are currently looking at the skipped-records-rate metric as part of https://issues.apache.org/jira/browse/KAFKA-5055 . Could you let us know if you use any special TimeStampExtractor class, or if it is the default? Thanks Eno >

Re: Stream applications dying on broker ISR change

nning in the hope it would recover; After > ~1hour it's still just continuously spilling out the same exception and not > managing to continue processing. > > On 25 April 2017 at 16:24, Eno Thereska wrote: > >> Hi Ian, >> >> Retries are sometimes expected

Re: Stream applications dying on broker ISR change

p(StreamThread.java:592) at org.apache.kafka.streams.processor.internals.StreamThread. run(StreamThread.java:361) On 24 April 2017 at 16:02, Eno Thereska wrote: > Hi Sachin, > > In KIP-62 a background heartbeat thread was introduced to deal with the > group protocol arrivals an

Re: [ANNOUNCE] New committer: Rajini Sivaram

Congrats! Eno > On Apr 25, 2017, at 12:17 PM, Rajini Sivaram wrote: > > Thanks everyone! > > It has been a pleasure working with all of you in the Kafka community. Many > thanks to the PMC for this exciting opportunity. > > Regards, > > Rajini > > On Tue, Apr 25, 2017 at 10:51 AM, Damian Guy

Re: Stream applications dying on broker ISR change

ped an > instance and needs to be booted out of the group. > > Thanks > Sachin > > > On Mon, Apr 24, 2017 at 5:55 PM, Eno Thereska > wrote: > >> Hi Ian, >> >> >> This is now fixed in 0.10.2.1. The default configuration need tweaking. If &g

Re: Stream applications dying on broker ISR change

Hi Ian, This is now fixed in 0.10.2.1. The default configuration need tweaking. If you can't pick that up (it's currently being voted), make sure you have these two parameters set as follows in your streams config: final Properties props = new Properties(); ... props.put(ProducerConfig.RETRIES

Re: [VOTE] 0.10.2.1 RC3

+1 tested the usual streams tests as before. Thanks Eno > On 21 Apr 2017, at 17:56, Gwen Shapira wrote: > > Hello Kafka users, developers, friends, romans, countrypersons, > > This is the fourth (!) candidate for release of Apache Kafka 0.10.2.1. > > It is a bug fix release, so we have lots of

Re: Calculating time elapsed using event start / stop notification messages

Hi Ali, One starting point would be the low level Processor API, where you get each event and process it. You can also use a persistent state store to keep track of the events seen so far, it can probably be an in-memory store. An an entry can probably be deleted once both start and stop events

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

at org.apache.kafka.common.config.AbstractConfig. > getConfiguredInstances(AbstractConfig.java:220) >at org.apache.kafka.clients.consumer.KafkaConsumer.( > KafkaConsumer.java:673) >... 6 more > > > > On Tue, Apr 18, 2017 at 5:47 AM, Mahendra Kariya > w

Re: Kafka Streams - Join synchronization issue

locationActivityJoiner(JsonObject activity, String > loc) { > JsonObject join = activity.copy(); > join.put("city" , loc); > return join; > } > > hum... your question is letting me think... are you telling me that since > both are kstreams, they actually need to

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

as a part of > build pipeline? > > On Mon, Apr 17, 2017 at 9:18 PM, Eno Thereska > wrote: > >> Not yet, but as soon as 0.10.2 is voted it should be. Hopefully this week. >> >> Eno >>> On 17 Apr 2017, at 13:25, Mahendra Kariya >> wrote: >>&

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

Not yet, but as soon as 0.10.2 is voted it should be. Hopefully this week. Eno > On 17 Apr 2017, at 13:25, Mahendra Kariya wrote: > > Are the bug fix releases published to Maven central repo? > > On Sat, Apr 1, 2017 at 12:26 PM, Eno Thereska > wrote: > >> Hi Sa

Re: Kafka Streams - Join synchronization issue

Hi Marco, Could you share a bit of your code, or at a minimum provide some info on: - is userActivitiesStream and geoDataStream a KStream of KTable? - what is the length of "timewindow"? Thanks Eno > On 16 Apr 2017, at 19:44, Marco Abitabile wrote: > > Hi All! > > I need a little hint to unde

Re: Kafka-Streams: Cogroup

Hi Kyle, (cc-ing user list as well) This could be an interesting scenario. Two things to help us think through it some more: 1) it seems you attached a figure, but I cannot seem to open it. 2) what about using the low level processor API instead of the DSL as approach 3? Do you have any thought

Re: [VOTE] 0.10.2.1 RC1

+1 (non-binding) Built sources, ran all unit and integration tests, checked new documentation, esp with an eye on the streams library. Thanks Gwen Eno > On 12 Apr 2017, at 17:25, Gwen Shapira wrote: > > Hello Kafka users, developers, client-developers, friends, romans, > citizens, etc, > >

Re: Kafka Streams Application does not start after 10.1 to 10.2 update if topics need to be auto-created

No, internal topics do not need to be manually created. Eno > On 13 Apr 2017, at 10:00, Shimi Kiviti wrote: > > Is that (manual topic creation) also true for internal topics? > > On Thu, 13 Apr 2017 at 19:14 Matthias J. Sax wrote: > >> Hi, >> >> thanks for reporting this issue. We are aware

  1   2   3   >