AFAIK, there can be only one master-batch-coordinator for each batch group. You should look at increasing parallelism of trident operations to get better latencies.
Thanks, Satish. On Wed, Jul 13, 2016 at 12:26 PM, hong mao <[email protected]> wrote: > I get the resaon, there is only one MasterBatchCoordinator for each > trident spout, so setting the thrid parameter (parallelism) takes no > effect. > > > 2016-07-12 0:08 GMT+08:00 hong mao <[email protected]>: > >> Hi all, >> We are using trident topology to pull messages from kafka and store >> into hbase using partitionPersist. Recently we meet with a following >> situation, the latency of the topology is increasing, as is shown in >> stormUI, large portion of the latency is taken by $mastercoord-bg0 >> <http://10.139.19.81:8080/component.html?id=%24mastercoord-bg0&topology_id=biztracer-product-73-1468239144> >> , >> which corresponds to MasterBatchCoordinator, and its task parallelism is 1. >> By checking source code TridentTopologyBuilder.java, I find that there is >> no way to configure the task parallelism of MasterBatchCoordinator (MBC). >> >> for(String batch: batchesToCommitIds.keySet()) { >>> List<String> commitIds = batchesToCommitIds.get(batch); >>> builder.setSpout(masterCoordinator(batch), new >>> MasterBatchCoordinator(commitIds, batchesToSpouts.get(batch))); >>> } >>> >> >> Should I just write a new Builder which overrides TridentTopologyBuilder >> so that I can increase the task parallelism of MasterBatchCoordinator and >> imporve the throughput of MBC ? >> >> If it is ok, may I open an issue to solve this problem ? >> >> >> Thanks a lot! >> >> > >
