andygrove commented on issue #6855: URL: https://github.com/apache/datafusion-comet/issues/6855#issuecomment-6100134726
Follow-up to the comment above: - #6856 now measures native shuffle in the ten nested shapes (36acd72e8). On the same 7950X3D, native shuffle runs them at 1.1x to 3.8x Spark's time at both 5 and 201 partitions, while JVM shuffle is at about parity at 5 partitions and 0.7x to 0.9x at 201. The fuzz-schema cases skip the native arm, since Spark's scan reads that schema. - #6860, stacked on #6858, does the same for Comet's in-memory cache reader (`CachedBatchRowIterator`), which also generated its source in every partition when Spark reads a wide cached relation as rows. Reading the depth-6 fuzz schema from a 201-partition cache went from 16,971 ms to 1,390 ms, and a 1,500-column table at 200 partitions from 9,938 ms to 5,802 ms. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
