Kafka logo on website

Hello, My company, Dattell, is a consulting agency that helps businesses with big data messaging, storage and analysis using open source tools. I am writing to let you know that we listed your logo on our website, here . Please let us know if you would like us to

Re: Finding StreamsMetadata with value-dependent partitioning

Thanks Steven, interesting use case. The current streams state store metadata discovery is assuming the `DefaultStreamPartitioner` is used, which is a limitation for such cases. Another workaround that I can think of is, that you can first partition on D in the first stage to let the workers to

Re: Reliably implementing global KeyValueStore#get

> On Jun 6, 2017, at 2:52 PM, Damian Guy wrote: > > Steven, > > In practice, data shouldn't be migrating that often. If it is then you > probably have bigger problems. Understood and agreed, but when designing distributed systems, it usually helps to model for the worst

Re: Reliably implementing global KeyValueStore#get

Steven, In practice, data shouldn't be migrating that often. If it is then you probably have bigger problems. You should be able to use the metadata api to find the instance the key should be on and then when you check that node you can also check with the metadata api that the key should still

Can a SourceTask run out of things to do?

Hi, I'm creating a Kafka source connector that load's some data that is in the form of individual files that are being created continuously. I was planning initially to create one task per file - that would allow the framework to balance the work across all workers in a straightforward way. In

Re: Reliably implementing global KeyValueStore#get

> On Jun 6, 2017, at 6:16 AM, Eno Thereska wrote: > > Hi Steven, > > Do you know beforehand if a key exists? If you know that and are getting > null() the code will have to retry by refreshing the metadata and going to > the new instance. If you don’t know beforehand

Turning on rack awareness

I am practicing turning on rack awareness on a test cluster. With a cluster of 3 nodes that currently does not use rack awareness (broker.rack is not set). During a rolling restart of the brokers where broker.rack was filled in one by one, on each node I got the error [2017-06-06

Re: Finding StreamsMetadata with value-dependent partitioning

Happy to hear you found a working solution, Steven! -Michael On Sat, Jun 3, 2017 at 12:53 AM, Steven Schlansker < sschlans...@opentable.com> wrote: > > > > On Jun 2, 2017, at 3:32 PM, Matthias J. Sax > wrote: > > > > Thanks. That helps to understand the use case

Re: Reliably implementing global KeyValueStore#get

Hi Steven, Do you know beforehand if a key exists? If you know that and are getting null() the code will have to retry by refreshing the metadata and going to the new instance. If you don’t know beforehand if a key exists or not you might have to check all instances of a store to make sure.