Since the partitions of the Kafka input operator are based on (topic, partition) combination, can we increase the number of Kafka partitions per topic?
~ Bhupesh On Mon, Jul 25, 2016 at 10:21 PM, Munagala Ramanath <[email protected]> wrote: > One way is to have a pass-through operator X that is parallel partitioned > like your B currently. > Then, connect the output port of X to B and use a suitable partitioner for > B to create as many > partitions as you want: A -> X -> B -> C. > > Ram > > On Mon, Jul 25, 2016 at 9:41 AM, Yogi Devendra <[email protected]> > wrote: > >> Hi, >> >> I have a DAG A->B->C. >> >> 1. A is kafka input operator reading from 4 different topics configured >> with ONE_TO_ONE strategy. Thus creating 4 partitons of A. >> 2. B and C are configured to have parallel partitions w.r.t. their input >> port. Thus, currently both B, C have 4 partitions. >> >> I am observing B significant latency in B operator. Thus, would like to >> have 2 partitions for B per partition of A. Since, application is latency >> intensive, I want to avoid unifiers as far as possible. >> >> How to achieve this partitioning? >> >> >> ~ Yogi >> > >
