jnioche opened a new pull request, #2226:
URL: https://github.com/apache/stormcrawler/pull/2226
Companion to `StatusUpdaterBenchmark` (#2213, #2222), for the read side.
`SpoutBenchmark` runs one or more instances of a spout against a backend
populated beforehand (e.g. with `StatusUpdaterBenchmark`) for a given number of
minutes. Every URL emitted is passed straight to instances of a status updater
bolt as FETCHED, so that it is rescheduled in the backend and acked to the
spout. The number of URLs emitted, the number of unique ones among them and the
unique URLs/sec are reported every minute and at the end.
```
storm local target/crawler-1.0-SNAPSHOT.jar \
org.apache.stormcrawler.persistence.SpoutBenchmark \
org.apache.stormcrawler.opensearch.persistence.AggregationSpout \
org.apache.stormcrawler.opensearch.persistence.StatusUpdaterBolt 10 4 4 \
backend.yaml
```
The arguments are the spout class, the bolt class, the duration in minutes,
optionally the number of spout then bolt instances (1 by default), and one or
more configuration files.
- Each instance gets its own task in the `TopologyContext` and its own
thread, as a Storm executor would. A spout is only called from its thread, for
`nextTuple` as well as for `ack` and `fail`, and is not asked for more tuples
while it has `topology.max.spout.pending` in flight, if set.
- The URLs are routed to the status updaters based on their hash, as a
fields grouping on the URL does.
- The tuples go through `execute()`, so the next fetch date comes from the
scheduler (`fetchInterval.default`, one day by default), which keeps the URLs
from becoming due again during the run.
- URLs which the spout refuses because of their scheme are sent on the
status stream as ERROR, as in a topology, and are not counted.
- Once the time is up, the spouts are no longer asked for tuples and the
utility waits for those in flight to be acked or failed before closing the
components.
The benchmark utilities are also excluded from the JaCoCo coverage, as they
are run manually against a live backend.
### Notes for reviewers
- Emitted minus unique is the number of URLs handed out again, e.g. by the
OpenSearch spouts when the updates are not visible yet by the time the URLs
leave the purgatory.
- The unique URLs are kept in memory, so long runs against large backends
need a large enough heap.
- Unlike in Storm, the tuples never time out.
- Tested with `MemorySpout` and `MemoryStatusUpdater`, and against the
URLFrontier service with its spout and status updater.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
