andygrove commented on issue #4440: URL: https://github.com/apache/datafusion-comet/issues/4440#issuecomment-5152258736
Closing this — the questions here have been answered, and the answer has been acted on. On the GC framing: @mbutrovich's SF1000 cluster comparison found total GC at 33.7 min native vs 31.3 min JVM (~0.3% of aggregate task time in both), peak on-heap and off-heap indistinguishable, and aggregate task time within noise (178.1h vs 177.1h). @parthchandra's earlier native C2R experiments reached the same no-speedup result. So the GC-pressure rationale from #3221 does not hold up, and it stopped being applicable at all once #3367 made the converter deep-copy every row to the JVM heap. On the cost/benefit question in item 4: #5114 changed the default of `spark.comet.exec.columnarToRow.native.enabled` to `false` and updated the config doc to state the per-batch JNI cost. Isolated benchmarking in #5112 measured the native converter at 3.7x slower per row than the JVM `CometColumnarToRowExec` at the default 8192-row batch size, and up to 15.7x slower for small batches. Remaining work is tracked elsewhere: #5112 for making the native path competitive, and #5119 for optimizing the JVM converter. Neither needs this issue open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
