Yeah ! It does seem a bit hackish - but I think this approach promises less config/operation errors.
Although I think some of these checks can be built within Samza - assuming Kafka has a metadata store in the near future - the Samza container can validate the #topics against this store. On Wed, Mar 18, 2015 at 6:16 PM, Chris Riccomini <criccom...@apache.org> wrote: > Hey Chinmay, > > Cool, this is good feedback. I didn't think I was *that* crazy. :) > > Cheers, > Chris > > On Wed, Mar 18, 2015 at 6:10 PM, Chinmay Soman <chinmay.cere...@gmail.com> > wrote: > > > Thats what we're doing as well - appending partition count to the kafka > > topic name. This actually helps keep track of the #partitions for each > > topic (since Kafka doesn't have a Metadata store yet). > > > > In case of topic expansion - we actually just resort to creating a new > > topic. Although that is an overhead - the thought process is that this > will > > minimize operational errors. Also, this is necessary to do in case we're > > doing some kind of joins. > > > > > > On Wed, Mar 18, 2015 at 5:59 PM, Jakob Homan <jgho...@gmail.com> wrote: > > > > > On 18 March 2015 at 17:48, Chris Riccomini <criccom...@apache.org> > > wrote: > > > > One thing I haven't seen, but might be relevant, is including > partition > > > > counts in the topic. > > > > > > Yeah, but then if you change the partition count later on, you've got > > > incorrect information forever. Or you need to create a new stream, > > > which might be a nice forcing function to make sure your join isn't > > > screwed up. There'd need to be something somewhere to enforce that > > > though. > > > > > > > > > > > -- > > Thanks and regards > > > > Chinmay Soman > > > -- Thanks and regards Chinmay Soman
