sunchao commented on PR #5449: URL: https://github.com/apache/datafusion-comet/pull/5449#issuecomment-5401307887
I agree with the retry concern, and I think the performance improvement makes this worth pursuing. One extra detail is that the batch boundaries need to stay the same too. Even with identical row order, `[a,b] [c,d]` and `[a] [b,c,d]` can send rows to different partitions. Could we initially enable this only for cases where we’ve verified that retries produce the same batches in the same order, and use the existing hashing approach elsewhere? Another option is to count rows across incoming batches and send each fixed-size group to the next partition. That would make the assignment independent of how the reader divides rows into batches. We would still need repeatable row order. There’s also a separate issue with the starting partition: the counter receives zero for every task, rather than the Spark input partition ID. If each task produces one batch, everything goes to partition 0. Passing the actual input partition ID through should fix that. Before enabling this more broadly, maybe we should add a test that fails and retries a task after some output has already been consumed, then checks for missing or duplicated rows. I’d also check partition sizes and whole-job runtime, so we know the faster writes aren’t offset by uneven work downstream. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
