ajaypadwal73 commented on PR #1026: URL: https://github.com/apache/sedona-db/pull/1026#issuecomment-4975472854
Thanks, that is the right baseline to compare against. I ran a local follow-up comparison to separate partition-level parallelism from probe-row chunking. This is still a `query_batch()`-level benchmark, not a full `ProbeShuffleExec` physical-plan benchmark: it simulates partition-level parallelism by running N concurrent `query_batch()` calls over row ranges against the same shared index. For the skewed workload, partitioning alone removes a lot of the original gap, but probe-row chunking still adds value once multiple probe partitions are already active: | Probe partitions | Partition only | + chunk 64 | + chunk 256 | Best chunk gain | | ---: | ---: | ---: | ---: | ---: | | 1 | 182.3 ms | 53.3 ms | 47.3 ms | ~3.9x | | 2 | 174.8 ms | 52.2 ms | 51.5 ms | ~3.4x | | 4 | 61.9 ms | 49.1 ms | 50.8 ms | ~1.3x | | 8 | 60.7 ms | 45.9 ms | 60.8 ms | ~1.3x | | 16 | 54.1 ms | 38.8 ms | 52.4 ms | ~1.4x | For the uniform workload, the picture is different: once partition count is high enough, partition-level parallelism mostly catches up, and combining both can regress from overhead. So I agree this should not be framed as a broad replacement for `ProbeShuffleExec` or a default-on speedup. The narrower case where this still seems useful is when partition-level parallelism is present but a probe stream still has expensive row-local refinement work. In that case, this gives a second level of parallelism inside the remaining hot `query_batch()` work. I am happy to rework the PR around that narrower framing, add this benchmark matrix, or drop the public config knob if you think the added tuning surface is not worth the narrower skewed-workload gain. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
