peterxcli commented on PR #5725: URL: https://github.com/apache/datafusion-comet/pull/5725#issuecomment-5578031237
@sunchao Added the requested old-dispatcher/native/pure-Spark comparison in 7e8f30a37 and published both runs in the PR description. The long-array fixture, release library, batch size and JVM settings are matched. The old arm uses the original serializer from the PR's parent, and the benchmark checks dispatcher activity as well as physical plans. For literal separators, native averaged 171/172 ms versus the old dispatcher's 176/185 ms. For column separators, it averaged 133/140 ms versus 173/170 ms. The first native literal run had a 75 ms standard deviation, so the small literal improvement is uncertain. I propose retaining native array routing based on this comparison with the old default. Pure Spark still performs better on several long-array measurements; its controls and the timing variance are included in the table. For the remaining gap to pure Spark, the plan is to profile allocation and copying, then test an upstream output-capacity estimate against the current reservation on both short and long inputs. Profiling will establish whether reallocations explain the gap; scalar expansion and list-row slicing are the next candidates to examine. The benchmark path checks, compilation and semantic lint passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
