GGraziadei opened a new issue, #2131: URL: https://github.com/apache/stormcrawler/issues/2131
The `crawler-conf.yaml` generated by the archetypes leaves Storm's inter-executor batching at its default (`topology.producer.batch.size: 1`), so no batching happens between the spout, fetcher, parser, indexer and status updater executors. Storm 3.0 adds `topology.producer.batch.dynamic`: when enabled together with a `topology.producer.batch.size` greater than 1, the effective batch size adapts at runtime between 1 and the configured size with AIMD, staying small under light load to keep latency low and growing under sustained load to preserve throughput. That removes the usual objection to batching in a crawl topology, where traffic is bursty and latency to the status updater matters. Proposal: enable it in the archetype configurations with a conservative batch size below `topology.max.spout.pending`, and document the Storm topology settings that affect a crawl (`topology.message.timeout.secs`, `topology.max.spout.pending`, the producer batch settings and the flush interval), which are currently not covered in the configuration documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
