ajaypadwal73 commented on PR #1026: URL: https://github.com/apache/sedona-db/pull/1026#issuecomment-4975326747
@james-willis Thanks, that’s the right comparison. The current benchmark isolates `query_batch()` directly, so it proves the local refinement path benefits from row chunking, but it does not yet prove incremental value over `ProbeShuffleExec` / partition-level parallelism. My hypothesis is that this helps in cases where the expensive work is trapped inside one probe stream or one large/hot probe batch: low input partition counts, intra-batch skew, or tail/drain when only one expensive partition remains. Since `ProbeShuffleExec` round-robins batches across the existing output partitions, increasing partition count may cover many cases, and I agree that should be the baseline. I’ll add a benchmark/comparison that varies probe partition count and batch layout, comparing: - partition-level parallelism only - probe-row chunking only - partition-level parallelism plus probe-row chunking If partition-level parallelism closes the gap, I’m happy to narrow this, make it internal/automatic, or drop the public option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
