tuantran0910 opened a new issue, #69902: URL: https://github.com/apache/doris/issues/69902
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. Related but different: #67997 / #68002 (UNINITED filter serialized as disabled). In this case the producer's filter is genuinely `DISABLED` by `skip_process()`, so #68002 does not fix it. ### Version 4.1.5 (cloud mode, 2 BEs). The same code path is still on `master` (`HashJoinBuildSinkLocalState::terminate` → `skip_process`). ### What's Wrong? A shuffled (`PARTITIONED`) `RIGHT OUTER JOIN` with a very small build side intermittently loses its global runtime filter. The probe side then scans the whole table. Results stay correct; only latency regresses (in our case from about 0.2 s to 10–30 s, reading 74M rows instead of 1). Query shape (anonymized): ```sql SELECT a.id, a.entity, a.action_time, b.params FROM actions a -- UNIQUE KEY(id), HASH(id), 174M rows LEFT JOIN audit_info b -- UNIQUE KEY(action_id), HASH(action_id), 74M rows ON b.action_id = a.id WHERE a.entity = ? -- selective, often matches a single row ORDER BY a.action_time DESC, a.id DESC LIMIT 21; ``` Plan: the `actions` TopN is the build side of a `RIGHT OUTER JOIN(PARTITIONED)` with 2 join instances (one per BE). `RF000[min_max]` and `RF001[in_or_bloom]` on `id` target the `audit_info` scan. Profile of a slow run. Both filters arrive well before the wait timeout, but as disabled: ``` RF0 Info: Consumer: ([id: 0, state: [DISABLED], type: MINMAX_FILTER, column_type: VARCHAR, reason: get disabled from remote], mode: GLOBAL, stage: 0, state: APPLIED, reached_timeout: false, timeout_limit: 86400000ms) RF1 Info: Consumer: ([id: 1, state: [DISABLED], type: IN_OR_BLOOM_FILTER(IN_FILTER), column_type: VARCHAR, reason: get disabled from remote], ...) RF0 WaitTime: 175.0ms audit_info scan: RowsRead 36.965154M per instance (full table) ``` A fast run of the same query shows `state: [READY], size: 1` and reads 0–1 rows. ### Root cause When the build matches one row, one of the two join instances has an empty build side. The sequence is: 1. At `eos` the build sink releases the probe side (`_dependency->set_ready_to_read()`, `hashjoin_build_sink.cpp`, end of `sink()`). The runtime filter is only published later, in `close()`. Before that, the task waits on `_finish_dependency` for the global filter-size sync (`send_filter_size`). 2. In the empty instance, `init_short_circuit_for_probe()` sets `short_circuit_for_probe = true`: the build is empty and the join is `RIGHT_OUTER_JOIN`. `HashJoinProbeOperatorX::pull` then returns `eos` immediately. 3. The probe pipeline closes. `PipelineFragmentContext::decrement_running_task` → `Pipeline::make_all_runnable` wakes the upstream build pipeline early and terminates it, even though the build already reached `eos` and is only waiting to publish. 4. `HashJoinBuildSinkLocalState::terminate` → `RuntimeFilterProducerHelper::skip_process()` publishes this instance's filters as `DISABLED` ("skip all rf process"). `close()` then skips `build()`/`publish()` because `_terminated` is set. 5. `RuntimeFilterWrapper::merge` treats a `DISABLED` input as absorbing, so the merged global filter becomes `DISABLED`. Every consumer gets "get disabled from remote", and every probe-side scanner reads the full table. The race is between step 3 and the filter-size sync reply plus `close()`. So it is intermittent, and the same input can be fast or slow. `skip_process()` in `terminate()` comes from #55942: it stops other instances from hanging on the size sync when a task is woken early. That is still needed when the build never finished. But when the build already reached `eos`, the instance has a valid filter. For an empty build that filter is empty, which correctly matches nothing, and disabling it degrades the whole query. ### What You Expected? A build sink that has already processed its whole input should publish its real filter (READY; empty for an empty build) when it is terminated early, instead of `DISABLED`. Only a build that is terminated before `eos` should fall back to `skip_process()`. ### How to Reproduce? Timing dependent; a reproduction with high hit rate: 1. A cluster with 2 BEs. Two `UNIQUE KEY` tables distributed by their join key: a large probe table (tens of millions of rows) and a build-side table. 2. Use a filter on the build table that matches exactly one row, and the query shape above (`LEFT JOIN`, which the planner turns into a `RIGHT OUTER JOIN(PARTITIONED)` with the build side as the right child, plus `ORDER BY ... LIMIT`). 3. Run it repeatedly with `enable_profile=true`. In our environment some single-row values hit it on 3 out of 3 runs, others intermittently. Under concurrent load (4 sessions) about 1–6% of all queries were affected. 4. Check the probe-side scan in the profile: `RF* Info ... state: [DISABLED] ... reason: get disabled from remote`, `reached_timeout: false`, full `RowsRead`. Observations: - `runtime_filter_wait_time_ms=5000` does not help (5/92 vs 6/95 queries affected at 1000 ms), because the filter is not late. - `enable_sync_runtime_filter_size=false` removes the size-sync wait. It cut the problem from 9/9 to 0/9 runs sequentially, but 1 of ~107 queries was still affected under concurrent load. That fits the window between `set_ready_to_read` and `close()` still existing. ### Anything Else? Possible fix directions: 1. In `HashJoinBuildSinkLocalState::terminate`, if the sink already reached `eos` and the hash table was built, run the normal `_runtime_filter_producer_helper->build()` + `publish()` (or skip `terminate` for the filter producers) instead of `skip_process()`. 2. Alternatively, do not release the probe dependency of a short-circuited empty instance until its filters have been published. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
