924060929 commented on code in PR #64793:
URL: https://github.com/apache/doris/pull/64793#discussion_r3479115546
##########
be/src/exec/exchange/vdata_stream_recvr.h:
##########
@@ -198,7 +198,17 @@ class VDataStreamRecvr::SenderQueue {
void close();
- void set_dependency(std::shared_ptr<Dependency> dependency) {
_source_dependency = dependency; }
+ void set_dependency(std::shared_ptr<Dependency> dependency) {
Review Comment:
Good catch — fixed. Moved the `_source_dependency = dependency;` assignment
inside the `_lock` scope so it can no longer race with the locked read in
`set_source_ready()` (reached via `decrement_senders`/`cancel`/`close` on other
threads). The non-atomic `shared_ptr` write is now serialized with those
readers.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -1636,6 +1640,27 @@ public enum IgnoreSplitType {
"Whether to force to local shuffle on pipelineX
engine."})
private boolean forceToLocalShuffle = false;
+ @VarAttrDef.VarAttr(
+ name = LOCAL_SHUFFLE_BUCKET_UPGRADE_RATIO, fuzzy = false, varType
= VariableAnnotation.EXPERIMENTAL,
+ description = {"FE规划Local Shuffle时, 当池化bucket
join所在fragment的每BE实例数大于"
+ + "每BE有数据分桶数的该倍数时, 将join两侧的桶分布本地重分发为hash分布以突破桶数并发上限。"
+ + "必须大于1才生效; 小于等于1(含0和负数)时关闭该优化",
+ "When FE plans local shuffle and a pooled bucket join
fragment has more instances"
+ + " per BE than (buckets-with-data per BE) * this ratio,
re-distribute both join"
+ + " sides with local hash instead of bucket hash so join
parallelism is no longer"
+ + " capped at bucket count. Only takes effect when > 1;
values <= 1 (including 0"
+ + " and negatives) disable the upgrade."}, needForward =
true)
+ private double localShuffleBucketUpgradeRatio = 1.5;
Review Comment:
Thanks — the code value (1.5) is the intended one; the PR description was
stale. Updated the description (Part 2 + release note) to say 1.5, and added a
note on why: with the default `bucket_shuffle_downgrade_ratio=0.8` a single
bucket join only forms when instances/buckets <= 1.25, so a 1.5 upgrade
threshold stays off for single joins by default and engages only for stacked /
wide bucket-domain shapes. Keeping it conservative (opt-in for single joins) on
purpose for this PR.
##########
regression-test/suites/nereids_p0/local_shuffle/test_local_shuffle_rqg_bugs.groovy:
##########
@@ -513,7 +513,7 @@ suite("test_local_shuffle_rqg_bugs") {
// ============================================================
// Bug 10: GLOBAL_HASH_SHUFFLE Rows mismatched — self-join + NLJ
- // RQG case: 906784672 (build 184181)
+ // RQG regression case
// Root cause: HashJoinNode used requireGlobalExecutionHash() → GLOBAL
local exchange
// inserted when use_serial_exchange=true; shuffle_idx_to_instance_idx
map has only
Review Comment:
Fixed all three occurrences ("RQG RQG testing" -> "RQG testing"). It was
exactly the find-and-replace artifact you guessed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
