Re: Monitor the state of an event-based choreography

2018-07-19 Thread Christopher Smith
This sounds like a pretty good use case for opentracing.org. There is some nice integration with Kafka now too. --Chris On Thu, Jul 19, 2018, 8:26 AM Dan Rosanova wrote: > Are you looking more for an Actor or orchestration layer and visibility? I > don’t know of one per se, it would also be

Re: Batch size, linger-ms not working as intended

2020-05-18 Thread Christopher Smith
Honestly, I'm not sure why you'd want to have it send only one message at a time, but yes, this does not guarantee that it will not batch. It only guarantees that it won't wait before sending, which is not the same thing. --Chris On Mon, May 18, 2020 at 4:03 PM Bibek Shrestha wrote: > Hi All,

Re: KSQL to search for data in Kafka Topics

2020-05-19 Thread Christopher Smith
I'm not sure what you mean by "not using stream topology". What does that mean to you that you'd rather avoid? However, you can indeed use KSQL to define streams & tables that process data from a number of topics. However, I think you may have the misimpression that KSQL is designed so you can

Re: GDPR compliance

2020-08-19 Thread Christopher Smith
Yup. The crypto-shredding approach tends to be the most practical. Basically do payload encryption of your PI and with a unique per-user key. Throw away the per user key, and the data is "deleted" from a CCPA perspective. The alternative is to have the relevant topic have tight retention SLAs,

Re: Kafka retention policy per topic not working as expected

2020-05-24 Thread Christopher Smith
It's a bit weird having the default policy for your brokers being compact, but yes, the policy for the topic overrides the broker policy. What you are seeing on with your example of the segment files makes sense to me. With a 121000ms retention policy you've actually got 400 seconds beyond

Re: Send several Serialized Java objects under one Kafka Topic

2021-01-31 Thread Christopher Smith
Use Avro and a discriminated union / sum type to combine all the types. On Sun, Jan 31, 2021, 6:49 AM Peter Penzov wrote: > Hello All, > I'm working on proof of concept for sending several Java Objects > under one Kafka Topic. More about the requirements: > >