I get the resaon, there is only one MasterBatchCoordinator for each trident spout, so setting the thrid parameter (parallelism) takes no effect.
2016-07-12 0:08 GMT+08:00 hong mao <[email protected]>: > Hi all, > We are using trident topology to pull messages from kafka and store > into hbase using partitionPersist. Recently we meet with a following > situation, the latency of the topology is increasing, as is shown in > stormUI, large portion of the latency is taken by $mastercoord-bg0 > <http://10.139.19.81:8080/component.html?id=%24mastercoord-bg0&topology_id=biztracer-product-73-1468239144> > , > which corresponds to MasterBatchCoordinator, and its task parallelism is 1. > By checking source code TridentTopologyBuilder.java, I find that there is > no way to configure the task parallelism of MasterBatchCoordinator (MBC). > > for(String batch: batchesToCommitIds.keySet()) { >> List<String> commitIds = batchesToCommitIds.get(batch); >> builder.setSpout(masterCoordinator(batch), new >> MasterBatchCoordinator(commitIds, batchesToSpouts.get(batch))); >> } >> > > Should I just write a new Builder which overrides TridentTopologyBuilder > so that I can increase the task parallelism of MasterBatchCoordinator and > imporve the throughput of MBC ? > > If it is ok, may I open an issue to solve this problem ? > > > Thanks a lot! > >
