Any thoughts? On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian <alinazem...@gmail.com> wrote:
> Hi, > > We have noticed recently that no matter what batch size we use for Metron > indexing feeds, as long as we start using different batch size for > different Metron feeds, indexing topology throughput will start dropping > due to the high error rate! So I was wondering whether based on the current > indexing topology design, we have to choose the same batch size for all the > feeds or not. Otherwise, throughout will be dropped. I assume since it is > acceptable to use different batch sizes for different feeds, it is not > expected by design. > > Moreover, I have noticed in practice that even if we change the batch > size, it will not affect the messages that are already in enrichments or > indexing topics, and it will only affect the new messages that are coming > to the parser. Therefore, we need to let all the messages pass the indexing > topology so that we can change the batch size! > > It would be great if we can have more details regarding the design of this > section so we can understand our observations are based on the design or > some kind of bug. > > Regards, > Ali > -- A.Nazemian
