Doris-Breakwater commented on issue #66398: URL: https://github.com/apache/doris/issues/66398#issuecomment-5167980461
## Initial triage: confirmed BE runtime-filter lifecycle bug This is a valid bug with high confidence from source inspection at both exact build commits reported in the issue (`39dbf23ff3d` and `737218b90e2`). The observed 900-second pause is a timeout-bounded liveness/latency failure; there is no evidence here of an incorrect query result. ### Verified code path 1. `RuntimeState::enable_join_spill()` returns true when either `enable_force_spill` or `enable_join_spill` is true. 2. For a non-broadcast hash join, `PipelineFragmentContext::_create_operator()` then selects `PartitionedHashJoin*` and explicitly executes `tnode_.runtime_filters.clear()` under the comment `TODO: support rf in partitioned hash join`. 3. In these affected revisions, that RF-cleared `tnode_` is also used to initialize the build-side inner `HashJoinBuildSinkOperatorX`. Consequently, the BE registers no RF producer for this join, while the RF consumer descriptors already attached to the scan node are unaffected. 4. With no registered producer, there is nothing to compute, publish, or disable. This directly explains `RuntimeFilterComputeTime = 0ns`, `PublishRuntimeFilterTime = 0ns`, consumer state `NOT_READY`, `Disabled = false`, and waiting until `runtime_filter_wait_time_ms` expires. 5. This happens when the partitioned operator is constructed, before any runtime spill decision. Therefore, the all-zero spill counters are expected and do not contradict the diagnosis. The zero-row second build instance is **not the primary trigger**: RF descriptors are removed for every instance. The code predicts that the same orphaned consumer can occur with `parallel_fragment_exec_instance_num=1`, provided the join remains non-broadcast/partitioned and the RF target is remote. `LIMIT 100` mainly makes it easy to demonstrate the non-spilling branch. The scope is also wider than the debug flag alone: `enable_join_spill=true` selects the same path even when `enable_force_spill=false`. ### Recommended fix Please backport the complete RF lifecycle used by newer partitioned-hash-join code, rather than only removing `runtime_filters.clear()`: 1. Preserve the original join `tnode` (including RF descriptors) for the build-side inner `HashJoinBuildSinkOperatorX`; use the RF-cleared copy only for per-partition recovery/probe-side inner operators. 2. On the no-spill path, wire the wrapper's finish dependency to the inner build sink and have `PartitionedHashJoinSinkLocalState::close()` close the inner sink, so its normal RF size/build/publish path runs. 3. On the first transition to the spilled path, call `HashJoinBuildSinkLocalState::disable_runtime_filters()` before any empty-build early return. That method sends size zero, marks the filters disabled, and publishes the terminal state so remote consumers are released. Only doing step 1 is insufficient: in the affected code, the wrapper does not close the inner sink on the no-spill path, and `_revoke_unpartitioned_block()` does not disable RFs on the spill path. Conversely, merely clearing BE producer descriptors is unsafe unless FE also removes every corresponding consumer descriptor. ### Regression coverage Please cover all of the following: - Partitioned join selected, no actual spill, one build instance: RF reaches `READY` and the scan does not wait for the RF timeout. - Two build instances with one receiving zero rows: every producer participates in size synchronization/publication. - Actual build spill: RF reaches a terminal disabled/ignored state and the scan starts immediately. - Both selection modes independently: `enable_join_spill=true` with force spill off, and `enable_force_spill=true`. - A remote RF target, since a local-only target may not expose this failure. ### Additional information useful for the regression test The current issue body says the full profiles are attached, but its raw Markdown contains no attachment links. Please reattach the exported profiles if the upload was lost. A minimal DDL/data generator plus `EXPLAIN VERBOSE` would also make the exact RIGHT OUTER JOIN/LIMIT/remote-RF topology deterministic in regression testing. These items are not blockers for the root-cause determination above. Current workaround: keep both `enable_force_spill` and `enable_join_spill` disabled for affected builds (or reduce `runtime_filter_wait_time_ms` only as a mitigation; that does not resolve the orphaned RF lifecycle). The issue currently has no labels; it should be triaged under BE execution, spill, and runtime filter. Breakwater-GitHub-Analysis-Slot: slot_49f75bc8b5f0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
