ajaypadwal73 commented on PR #1026: URL: https://github.com/apache/sedona-db/pull/1026#issuecomment-4981352613
@paleolimbot Thanks, that framing makes sense. I think this PR was useful as an exploration, but I agree it may not be the right shape to merge as-is. Between James’s point about `ProbeShuffleExec`/partition-level parallelism and your point about accumulating candidates across rows before refinement, I think the core issue is probably better addressed at the refinement batching layer rather than by adding a second public parallelism knob. The current approach is deliberately low-invasive, but that also means it works around the existing row-local refinement shape instead of improving it. I’m happy to close this PR and take the learnings forward. If useful, I can open a smaller follow-up issue/PR exploring candidate accumulation across probe rows, with benchmarks comparing: - current per-row refinement - partition-level parallelism - candidate accumulation across rows - this probe-chunk approach as a reference point Unless you’d prefer otherwise, I’ll close this one rather than push on the current design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
