Glad to help. I will say, as you probably got from my interest/questions, that is definitely outside of normal use (that I have seen). Why do you need dynamic logical partitioning?
On Fri, Aug 7, 2015 at 1:20 PM, Gelinas, Chiara <cgeli...@illumina.com> wrote: > Thank you! We are new to Kafka, so this makes complete sense - the > metadata refresh age. > > Yes, incrementally assigning 1 partitioner key - we are tracking it in a > relational DB along with offset, etc, for the consumers. > > We haven¹t yet implemented the dynamic consumer side but there are several > approaches from the brute/force quick/dirty (e.g. Poll DB at interval for > updates), to using another Kafka topic for new partition event > notifications (of course, we favor the latter, but timelines will likely > mean that the first option is what initially gets implemented). > > On 8/7/15, 10:56 AM, "Grant Henke" <ghe...@cloudera.com> wrote: > > >Interesting use case. I would be interested to hear more. Are you > >assigning > >1 partition per key incrementally? How does your consumer know which > >partition has which key? > > > >I don't think there is a way to manually invalidate the cached metadata in > >the public producer api (I could be wrong), but the longest you should > >have > >to wait is whatever is configured for metadata.max.age.ms. > > > >metadata.max.age.ms is the period of time in milliseconds after which we > >force a refresh of metadata even if we haven't seen any partition > >leadership changes to proactively discover any new brokers or partitions. > >http://kafka.apache.org/documentation.html#newproducerconfigs > > > > > > > > > >On Fri, Aug 7, 2015 at 12:34 PM, Gelinas, Chiara <cgeli...@illumina.com> > >wrote: > > > >> Hi All, > >> > >> We are looking to dynamically create partitions when we see a new piece > >>of > >> data that requires logical partitioning (so partitioning from a logical > >> perspective rather than partitioning solely for load-based reasons). I > >> noticed that when I create a partition via AdminUtils.addPartition, and > >> then send the message (within milliseconds since it¹s all happening on > >>one > >> thread execution), I get the following error: > >> > >> Invalid partition given with record: 3 is not in the range [0...3]. > >> > >> Basically, the Producer can¹t see the new partition. When I set a > >> breakpoint just before the send (which essentially sleeps the thread), > >>then > >> all is well, and it pushes the message to the new partition with no > >>issues. > >> > >> I am running just one zookeeper, one kafka (no replicas) this is all > >> local on my dev environment. > >> > >> Is this normal behavior or is there possibly some issue with how we are > >> using addPartition? Also, once we have replicas in a more realistic > >> production environment, should we expect this lag to increase? > >> > >> The only workaround I can envision for this is to have the thread check > >> the partition count via AdminUtils and only move on when the partition > >> count comes back as expected. > >> > >> Thanks, > >> Chiara > >> > >> > >> > > > > > >-- > >Grant Henke > >Software Engineer | Cloudera > >gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke > > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
