ajaypadwal73 opened a new issue, #1054: URL: https://github.com/apache/sedona-db/issues/1054
Follow-up from #1026. `query_batch()` currently refines candidates mostly at probe-row granularity. That works well when a single probe row produces enough candidates to trigger existing intra-row refinement parallelism, but it can underuse CPU when many probe rows each produce moderate candidate counts. In #1026, I explored splitting probe rows into parallel chunks. That showed useful speedups on skewed workloads, but it also adds another parallelism layer/config knob and overlaps conceptually with `ProbeShuffleExec` / partition-level parallelism. An alternative worth exploring is batch-level refinement: - collect candidates across multiple probe rows - trigger refinement once a candidate threshold is reached - preserve row-major output and existing `max_result_size` / resume semantics - reuse or align with the existing `parallel_refinement_chunk_size` behavior if possible Suggested benchmark comparisons: - current per-row refinement - partition-level parallelism via `ProbeShuffleExec` - probe-row chunking from #1026 as a reference point - candidate accumulation across rows This should help decide whether the right fix is a new probe-row parallel path, a refinement batching change, or simply better benchmark coverage around the existing partition-level behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
