Re: Spread log segment deletion over a couple hours

2018-05-02 Thread Jonathan Bethune
Howdy Vincent. Sounds like a painful situation! I have experienced similar drama with Kafka so maybe I can offer some advice. You said you decreased the retention time on 4 topics. I wonder, was this done on all 4 topics at the same time? Depending on broker and partition config, that can be ver

Re: KStreams API Usage

2018-05-02 Thread Guozhang Wang
Hello Pradeep, If you just want to close the Streams app based on the first observation of a specific value, you can consider using a shutdown latch, in which the `transformValues()` will decrement and the main thread starting the streams will listen on, and once the latch has been decremented the

Re: kafka error after upgrading to 1.1.0: “the state store…may have migrated to another instance”

2018-05-02 Thread Guozhang Wang
Hello, Sorry for the late reply. I cannot think of a direct reason that you might see this issue in 1.1. Could you upload your code snippet especially your code building the topology and the code around calling the KafkaStreams.store(), so that I can further look into it? Guozhang On Thu, Apr

Spread log segment deletion over a couple hours

2018-05-02 Thread Vincent Rischmann
Hi, I'm wondering if there is a way to tell Kafka to spread the log file deletion when decreasing the retention time of a topic, and if not, if it would make sense. I'm asking because this afternoon, after decreasing the retention time from 2 months to 1 month on 4 of my topics, the whole cluster

Re: ClassCastException in KStreams job for SessionWindow aggregation

2018-05-02 Thread Conrad Crampton
Thanks Damian for taking the time to point out my stupidity - that was it :-) Case of Occams Razor!! Thanks again On Wed, 2 May 2018 at 16:56 Conrad Crampton wrote: > Looks like I may be victim of that copy and paste gremlin!! > Without checking I think you might be onto something - I'll check i

Re: ClassCastException in KStreams job for SessionWindow aggregation

2018-05-02 Thread Conrad Crampton
Looks like I may be victim of that copy and paste gremlin!! Without checking I think you might be onto something - I'll check it out later and repost back. On one hand I really hope it's this, on the other I can't believe I've been so stupid! Thanks On Wed, 2 May 2018 at 16:30 Damian Guy wrote:

Re: ClassCastException in KStreams job for SessionWindow aggregation

2018-05-02 Thread Damian Guy
Hi, I think it **might** be related to this: final Serializer httpSessionSerializer = new JsonPOJOSerializer<>(); serdeProps.put("JsonPOJOClass", Http.class); httpSessionSerializer.configure(serdeProps, false); final Deserializer httpSessionDeserializer = new JsonPOJODe

ClassCastException in KStreams job for SessionWindow aggregation

2018-05-02 Thread Conrad Crampton
I'm trying to window over http logs and create an HttpSession i.e. a list of http requests (and some other properties). However when in my aggregate Merger part (I think) I'm getting a classcastexception I think in when my sessions are being merged and cannot for the life of me work out why. The ex

Re: Using Kafka CLI without specifying the URLs every single time?

2018-05-02 Thread Horváth Péter Gergely
Hi All, Thanks for the inputs: apparently this is an issue for which everyone tries to come up with a solution. I think it should be done in the core Kafka CLI; it cries for a feature request/improvement. I've created a JIRA issue for it; if you think it would be helpful for you as well, please