abhinav-phi commented on PR #2123:
URL: https://github.com/apache/stormcrawler/pull/2123#issuecomment-5588335860
Thanks — pushed f8564fae addressing both blockers:
**1. Comma-separated protocols parsing** — confirmed and fixed.
`loadListFromConf` returned the shipped string as one entry, so `protocols:
"http,https,file"` produced a single unmatchable scheme and every stored URL
(https included) was rejected with ERROR. The key is now split on commas with
surrounding whitespace tolerated (`split(" *, *")`), matching how
`ProtocolFactory` parses the same key. Tests added: the shipped comma string
(https emitted, ftp rejected), an explicit YAML list (`https` only → http
rejected), and whitespace forms.
**2. Status stream wiring** — the three archetype `crawler.flux` files
(urlfrontier, opensearch, solr) now connect the spout to the status updater on
`Constants.StatusStreamName` with the same FIELDS/`url` grouping the other
status producers use. The updater consumes `(url, metadata, status)`, which is
exactly what the spout emits on rejection. `internals.adoc` gained a paragraph
documenting that declaring the stream is only half the story and that
hand-built topologies must add the same connection when upgrading.
**Topology-level test** — `SpoutStatusStreamWiringTest` loads the archetype
`crawler.flux`, and fails if the spout→status-updater stream on
`Constants.StatusStreamName` is missing, so the wiring cannot silently regress.
Local verification: checkstyle 0 on the full reactor, format validation
clean, and the full core suite passes (the only errors in one run were WireMock
port-contention from parallel local test processes — the affected suite passes
in isolation). The workflow run for f8564fae will need a maintainer's workflow
approval again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
