coderfender opened a new issue, #7476: URL: https://github.com/apache/datafusion-comet/issues/7476
#6938 runs `CartesianProductExec` natively behind `spark.comet.exec.cartesianProduct.enabled` (default `false`), only without a join condition and when each input reports exactly one partition. Remaining work: - **Multi-partition inputs.** Spark crosses every partition pair through `CartesianRDD`. A non-bucketed Parquet scan reports `UnknownPartitioning(0)`, so today the native path needs a `COALESCE(1)` or similar. - **Spilling.** `CrossJoinExec` buffers the whole left input and cannot spill (apache/datafusion#26383). Its `concat_batches` copy of the left side is also unreserved. - **Large string and binary columns.** That concatenation is exposed to the 2 GiB offset limit (#7273). - **Output batching.** One output batch per left row for each right batch, which explains the 3.1× large-left benchmark result against 18.6× for the square case. - **Row order.** Native output is right-batch-major, Spark's is left-row-major. Documented, since Spark reports no output ordering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
