I would assume that in those cases where you see the "rebalance... max poll()" message that topology is unable to keep up with the input throughput. The messages are not ack'd quickly enough, they fail, and are retried. I would not focus on HBase because that is not likely your bottleneck.
You need to tune the Profiler topology to keep up with your peak incoming throughput. The same ideas for tuning any Storm topology apply here. Start with 1 worker and increase the number of executors first. You will also want to explore reducing your window lag and tuning other parameters before increasing the number of workers. Try to max out the performance of a single worker before adding more workers. You will probably first start to see performance issues on the Splitter bolt that has to consume every message and determine if that message is needed by any of the profilers. Using the Storm UI watch the metrics generated for that bolt first. If asking for more in-depth help, these are the types of questions that I would ask. - What is your peak input throughput to the Profiler? - What are your Profiler properties? - How many profiles do you have and what are they doing? Provide the profile definitions. On Mon, Jul 1, 2019 at 3:04 AM tkg_cangkul <[email protected]> wrote: > Hi, i've a problem about hbase profiler on metron. i've found if there > are some inconsistentcy data that insert to hbase. sometimes insert > normally but sometimes doesn't inserted. if not inserted, there are an > error msg on storm (rebalance... Max poll()). > > I've set 2 worker for profiler. profiler.hbase.batch 10. > profiler.hbase.flush.interval.seconds 30 > > any suggest about this? pls help >
