andygrove commented on issue #6855: URL: https://github.com/apache/datafusion-comet/issues/6855#issuecomment-6099432768
On the "JVM shuffle with 201 partitions" item: a profile of the fuzz schema case (max depth 6) shows the per-partition cost is not in the native writer. `libcomet` takes about 10% of the samples. The time goes to the reduce side, in each of the 201 tasks: - `CometColumnarToRowExec.doExecute` calls `UnsafeProjection.create` in every partition, which regenerates the projection's Java source. The schema is over `spark.sql.codegen.maxFields`, so the plan is never under whole-stage codegen. That was about 19 ms per partition at depth 6 and 2 ms at depth 2. #6858 generates it once per executor, which halves those cases (827 to 404 ms, and 7,798 to 3,992 ms). - Most of what remains is importing the wide schema's Arrow arrays for each batch (`ArrowImporter.importVector`, about 15% of the executor thread), and wrapping and closing the CometVectors (about 15%). Also, in these cases and in the ten nested shapes #6856 adds, the "Comet (native Shuffle)" arm runs Spark's shuffle. `repartition(n)` is round-robin, which native shuffle declines by default (`spark.comet.shuffle.native.partitioning.roundrobin.enabled`), and the fuzz schema is read by Spark's scan. Its parity with Spark says nothing about native shuffle. For the ten shapes, enabling round-robin in that arm, or shuffling by hash on a key, would measure native shuffle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
