andygrove commented on issue #5796: URL: https://github.com/apache/datafusion-comet/issues/5796#issuecomment-5603880350
Correcting this issue: the original version of it was wrong and I have rewritten the description above. I first measured with stock Comet settings and reported the result as "Comet's native JSON readers are ~9x slower than Velox". They are not — the native readers never ran. `CometGetJsonObject` and `CometJsonToStructs` are both `CometCodegenDispatch` with `NativeOptIn`, so by default they execute Spark's own implementation inside a `CometProject` node. What I measured was Spark, correctly, under Comet's name. With the two `allowIncompatible` keys set, native `get_json_object` is 62 ms against Velox's 87 ms on this query — faster, not 9x slower — and native `from_json` is 156 ms against 90 ms. Worth recording the reason I got it wrong, since it affects anyone benchmarking Comet: this tier is invisible in the plan. The node still prints `CometProject [get_json_object(...)]` while Spark's implementation runs inside it, so plan inspection alone cannot distinguish "Comet ran this natively" from "Comet dispatched this to Spark". Gluten's equivalent is structural — a rejected expression leaves a vanilla `Project` behind — so the same check works there and does not work here. Apologies for the noise on the original. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
