rzo1 commented on PR #2261: URL: https://github.com/apache/stormcrawler/pull/2261#issuecomment-6387423646
Thanks for picking this up, the documentation of the Storm settings is a good addition on its own. Before we change the defaults in the archetypes, though, I'd like to see some numbers. Neither the issue nor the PR measures the effect so far, and the archetype config is what most new users start from. Could one of you (@GitMehdi-sys or @GGraziadei) run the archetype topology with batch size 1 and with `32` + `dynamic: true`, and compare: - throughput (pages fetched per second, acks per second) - complete latency of the spout tuples, and whether anything times out - CPU per worker ideally once under light load (a few hosts, politeness-bound) and once under sustained load (many hosts). That would also tell us whether 32 is a good value or whether something smaller works as well. jnioche's StatusUpdaterBenchmark (#2213) might help for the status updater part. Small things: the `THIRD-PARTY.txt` change looks unrelated and can be dropped, and it would be nice to fill in the PR description with a short summary and the issue number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
