The GitHub Actions job "Java CI with Maven" on stormcrawler.git/issue-2062-urlfrontier-batch-discovered has failed. Run started by GitHub user abhinav-phi (triggered by rzo1).
Head commit for run: 4addf93d9a9696cc3c71c9a0d173df7105b0213d / abhinav-phi <[email protected]> Use batched PutDiscovered endpoint for DISCOVERED URLs in URLFrontier StatusUpdaterBolt Discovered URLs, which are the bulk of what a crawl writes, are now grouped into batches and sent on the PutDiscovered endpoint introduced in URLFrontier 2.6, amortising the per-message cost that limits the ingestion rate. Known URLs keep using the streaming PutURLs endpoint. - new config key urlfrontier.batch.size (default 100, 0 disables batching) - partially filled batches are flushed after 1s by a scheduled flusher, and whenever a known URL arrives (natural end-of-page boundary) or permits run short - batches are acked as a whole through BatchAck: one status per URL in the order they were sent, each resolved against the waitAck cache - frontiers without PutDiscovered (pre-2.6) are detected via UNIMPLEMENTED and the bolt falls back to sending discovered URLs individually - flow control follows the PutURLs client shipped with URLFrontier: throttled sends wait on a monitor woken by acks and by the transports' on-ready notifications instead of polling - requires urlfrontier-API 2.6 (wire compatible with 2.5 servers) Fixes #2062 Report URL: https://github.com/apache/stormcrawler/actions/runs/33688655282 With regards, GitHub Actions via GitBox
