avamingli opened a new pull request, #1311:
URL: https://github.com/apache/cloudberry/pull/1311
Previously in CBDB, we had to disable parallel execution for UNION ALL
queries containing Motion nodes due to a subtle but critical correctness issue.
The problem occurred when Parallel Append workers marked subnodes as
completed, causing other workers to skip them. While normally harmless, this
became critical in MPP databases where Motion nodes are ubiquitous. This
limitation forced us to disable parallel plans for most UNION ALL queries
involving distributed tables, missing significant optimization opportunities.
As a result, we fell back to serial execution:
```sql
explain(costs off) select b, count(*) from t1 group by b union all select b,
count(*) from t2 group by b;
QUERY PLAN
------------------------------------------------------------------
Gather Motion 3:1 (slice1; segments: 3)
-> Append
-> HashAggregate
Group Key: t1.b
-> Redistribute Motion 3:3 (slice2; segments: 3)
Hash Key: t1.b
-> Seq Scan on t1
-> HashAggregate
Group Key: t2.b
-> Redistribute Motion 3:3 (slice3; segments: 3)
Hash Key: t2.b
-> Seq Scan on t2
Optimizer: Postgres query optimizer
(13 rows)
```
The commit makes plan parallel by first attempting a parallel-aware Append
when it's safe to do so, but crucially, we now have a robust fallback path:
when Motion hazards are detected, we switch to using a parallel-oblivious
Append. This works because while Parallel Append might skip slices containing
Motions, regular Append doesn't have this problem - it will reliably execute
all subnodes regardless of whether they contain Motion nodes or not. Moreover,
since CBDB's Motion nodes are designed to handle tuples individually, we don't
need to worry about coordination between workers when processing these Motion
nodes.
This approach unlocks powerful new optimization opportunities, as shown in
this example where we can now execute the query with different levels of
parallelism for each subplan (2 workers for t1 and 3 workers for t2):
```sql
explain(costs off) select b, count(*) from t1 group by b union all select b,
count(*) from t2 group by b;
QUERY PLAN
------------------------------------------------------------------
Gather Motion 9:1 (slice1; segments: 9)
-> Append
-> HashAggregate
Group Key: t1.b
-> Redistribute Motion 6:9 (slice2; segments: 6)
Hash Key: t1.b
Hash Module: 3
-> Parallel Seq Scan on t1
-> HashAggregate
Group Key: t2.b
-> Redistribute Motion 9:9 (slice3; segments: 9)
Hash Key: t2.b
Hash Module: 3
-> Parallel Seq Scan on t2
Optimizer: Postgres query optimizer
(15 rows)
```
This change represents a significant improvement in CBDB's query optimizer,
allowing UNION ALL queries to benefit from parallel execution even when they
contain Motion nodes, while maintaining correctness and supporting flexible
parallelism configurations across different parts of the query. The
optimization is particularly valuable for complex queries like TPC-DS tests
where UNION ALL operations are common.
Authored-by: Zhang Mingli avamin...@gmail.com
<!-- Thank you for your contribution to Apache Cloudberry (Incubating)! -->
Fixes #ISSUE_Number
### What does this PR do?
<!-- Brief overview of the changes, including any major features or fixes -->
### Type of Change
- [ ] Bug fix (non-breaking change)
- [ ] New feature (non-breaking change)
- [ ] Breaking change (fix or feature with breaking changes)
- [ ] Documentation update
### Breaking Changes
<!-- Remove if not applicable. If yes, explain impact and migration path -->
### Test Plan
<!-- How did you test these changes? -->
- [ ] Unit tests added/updated
- [ ] Integration tests added/updated
- [ ] Passed `make installcheck`
- [ ] Passed `make -C src/test installcheck-cbdb-parallel`
### Impact
<!-- Remove sections that don't apply -->
**Performance:**
<!-- Any performance implications? -->
**User-facing changes:**
<!-- Any changes visible to users? -->
**Dependencies:**
<!-- New dependencies or version changes? -->
### Checklist
- [ ] Followed [contribution
guide](https://cloudberry.apache.org/contribute/code)
- [ ] Added/updated documentation
- [ ] Reviewed code for security implications
- [ ] Requested review from [cloudberry
committers](https://github.com/orgs/apache/teams/cloudberry-committers)
### Additional Context
<!-- Any other information that would help reviewers? Remove if none -->
### CI Skip Instructions
<!--
To skip CI builds, add the appropriate CI skip identifier to your PR title.
The identifier must:
- Be in square brackets []
- Include the word "ci" and either "skip" or "no"
- Only use for documentation-only changes or when absolutely necessary
-->
---
<!-- Join our community:
- Mailing list:
[d...@cloudberry.apache.org](https://lists.apache.org/list.html?d...@cloudberry.apache.org)
(subscribe: dev-subscr...@cloudberry.apache.org)
- Discussions: https://github.com/apache/cloudberry/discussions -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cloudberry.apache.org
For additional commands, e-mail: commits-h...@cloudberry.apache.org
