andygrove commented on issue #5112: URL: https://github.com/apache/datafusion-comet/issues/5112#issuecomment-5112912032
Follow-up analysis: TPC-H at 1TB on a cluster can show the opposite result (native C2R faster than the JVM implementation), so I extended the isolated benchmark with schema scenarios and heap-allocation reporting in #5118 to understand when each implementation wins. The original benchmark schema (long, int, double, string) turns out to be close to the JVM path's best case. Results (M3 Ultra, Spark 4.1, per-row ns, JVM vs native at batchSize=8192): | Scenario | JVM | Native | |---|---|---| | long, int, double, string (original) | 8.5 | 32.7 | | 4 x decimal(12,2), date, 2 x string (tpch-like) | 35.5 | 52.6 | | 4 x decimal(12,2), date, long (all fixed-width) | 20.4 | **16.7** | | 2 x decimal(38,10), long | 58.7 | 70.5 | | 16 x long | 23.7 | 31.4 | Key findings: - **Data types move the ratio.** The JVM accessor allocates a `Decimal` object per non-null decimal value (and a byte[] / BigInteger / BigDecimal chain for precision > 18), while the native converter writes the unscaled value directly. Adding four compact decimals shrinks the JVM lead from 3.8x to 1.5x. - **The native column-wise fast path only engages when every column is fixed-width**, and it wins outright there (16.7 vs 20.4 ns/row). Any var-length column forces the row-at-a-time general path, so the original benchmark schema never exercised it. TPC-H intermediate schemas (keys + decimals + dates) frequently qualify. - **Small batches remain a JVM win regardless of schema**, and the native fixed cost is per column, not just per batch: 16 columns at 32 rows costs roughly 30us/batch of FFI export/import vs roughly 10us for 4 columns. - **Allocation profiles differ by schema.** The native path is not allocation-free: the per-row `unsafeRow.copy()` from #3367 allocates 100-230 bytes/row. But for decimal schemas the JVM path allocates as much or up to 3x more (392 vs 137 bytes/row for decimal(38,10)), and that cost is nearly invisible on an idle microbenchmark heap while being real GC time on a loaded executor. Conclusion: the two implementations win different regimes. JVM wins small batches and narrow primitive/string schemas; native wins full batches of fixed-width/decimal-heavy data, with the gap widening under real heap pressure and on weaker cluster cores. This favors a per-batch dynamic fallback (#5115) over a blanket default choice. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
