ErnestasVaiciukeviciusAdform opened a new pull request, #9175: URL: https://github.com/apache/storm/pull/9175
## What is the purpose of the change To fix https://github.com/apache/storm/issues/9174. When `topology.upstream.feedback.enable=true`, workers flood their logs with `WARN o.a.s.utils.TransferDrainer - No remote destination available for task <id>`, and — more importantly — the upstream-feedback tuples that trigger those warnings are **dropped**. Upstream feedback is therefore only delivered to tasks on the *same* worker; feedback destined for upstream tasks on *other* workers never arrives. **Root cause.** Upstream feedback is routed directly to upstream (source) component task ids via `Task.sendUnanchoredFeedback`, bypassing stream grouping (`Executor.sendUpstreamFeedback` / `Executor.computeUpstreamTaskIds`). But a worker's routable remote destinations (`WorkerState.cachedTaskToNodePort`) and its connections are built only from `outboundTasks`, which `WorkerState.workerOutboundTasks()` derives purely from the *downstream-target* components of local tasks. `addUpstreamFeedback` declares `FEEDBACK_STREAM_ID` only as an output stream (no grouping/input edge), so feedback recipients are never added to `outboundTasks`. For a remote recipient, `TransferDrainer.send` finds no node, logs the warning, and drops the tuple. **Why it looks intermittent / misleading.** `outboundTasks` membership is decided per component: if a component is a downstream target of *any* local task, all of its tasks become routable. So a worker emits no warning for feedback to upstream component `X` whenever `X` *also* happens to be a downstream target of some co-located task — the feedback rides the existing connection and looks healthy. The same feedback edge warns and drops on a worker that hosts upstream tasks of `X` but none of `X`'s downstream consumers. Consequently **"no warnings" does not mean feedback is delivered**; it only reflects task placement (e.g. the warnings vanish entirely if every worker happens to host a task of every component), which can mislead operators into thinking the feature works. **Fix.** When upstream feedback is enabled, also include the non-system source (upstream) components of the locally-hosted components when computing `outboundTasks`, mirroring `Executor.computeUpstreamTaskIds()` at worker granularity. This makes every feedback recipient (bolt or spout) a routable remote destination with a connection, so feedback is delivered correctly and the spurious warning disappears — independent of task placement. The change is confined to `WorkerState.workerOutboundTasks()` and only takes effect when `topology.upstream.feedback.enable=true` (off by default). ## How was the change tested - Added `WorkerOutboundTasksTest` (storm-client) covering a `spout -> bolt1 -> bolt2` topology: - feedback disabled → only downstream-target tasks are routable (upstream spout tasks excluded); - feedback enabled → upstream source (spout) tasks are additionally routable, downstream tasks preserved; - the misleading placement-dependent case → when a worker co-locates a spout and a `bolt2` task, `bolt1` is already routable as the spout's downstream target even with feedback off, documenting that absence of warnings does not imply correct delivery. - Commands run (Maven 3.10, Java 25): - `mvn -pl storm-client -am install -DskipTests` — **BUILD SUCCESS** - `mvn -pl storm-client test -Dtest=WorkerOutboundTasksTest` — **Tests run: 3, Failures: 0, Errors: 0** - `mvn -pl storm-client test -Dtest=WorkerStateTest` (existing tests, no regression) — **Tests run: 3, Failures: 0, Errors: 0** - `mvn -pl storm-client checkstyle:check@validate` — **0 Checkstyle violations** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
