GitHub user yjhjstz added a comment to the discussion: [Ideas] Adaptive join:
correct a wrong Broadcast decision at runtime
Today, a broadcast motion replicates the inner side of a join: every segment
receives all R rows and builds its own complete copy of the hash table. The
total footprint of a broadcast is therefore N× the data — N× bytes on the
wire, N× deserialize CPU, N× hash-table build, and N× memory per join.
The alternative this idea explores: transmit and materialize the inner side
only once, as a single shared copy, and let all segments read from that shared
copy when probing, rather than each maintaining a private full-size hash table.
In other words:
- Broadcast (today): copy the data N times, one private copy per segment. Each
segment's probe is local and fast, but replication multiplies every cost by N.
- Broadcast-once-and-share (proposed): copy the data once to a shared
location. Replication cost disappears — the wire carries 1× the bytes, the hash
table is built 1×, memory holds 1× the data — and segments share access to it
during the probe.
GitHub link:
https://github.com/apache/cloudberry/discussions/1962#discussioncomment-18335683
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
