andygrove commented on PR #5381: URL: https://github.com/apache/datafusion-comet/pull/5381#issuecomment-5540370515
Thanks @comphead — added, as a new `Explode - nested input` group over 100K rows. I had to change one thing from your example, and it turned out to be the interesting part. `events` in your query is a `map<string, array<struct>>`, and `CometExplodeExec` refuses map inputs (#2837), so the first `explode` falls back to Spark's `GenerateExec` and the Comet arm stops measuring Comet at all — it becomes Spark behind a columnar-to-row transition. So the outer container is an `array<struct<platform, entries>>` here instead, which keeps everything else: the event list still sits at `profile.account.settings.preferences.notifications.activity.sessions.events`, each element still holds a second array of four-field structs, and the query still chains two generators and carries `customer_id` and `region` through both. Worth adding the map case as soon as #2837 lands, and I'd expect it to be the worst of the lot. Three cases: the array at depth 1, the same array eight struct accessors down, and the chained second explode. Depth 1 and depth 8 hold the same array written twice into one file, so the pair isolates what the struct chain costs. The whole event struct is counted rather than one of its fields, otherwise nested schema pruning narrows the exploded element and the case quietly stops measuring a wide row. Pruning does drop the siblings nothing reads — confirmed on the plan, the depth-8 scan projects `profile` as `struct<account:struct<...sessions:struct<events:array<...>>>>` with `billing`, `addresses` and `device` gone. Results (Apple M3 Max, `local[1]`, Spark 4.1, best ms): | Case | Spark | Comet | Relative | | ---- | ----- | ----- | -------- | | depth 1 | 69 | 156 | 0.4X | | depth 8 | 84 | 167 | 0.5X | | depth 8, then inner array | 96 | 181 | 0.5X | So this shape is a clear loss for us — Comet is about 2x slower than Spark on it, and the reproducible part is the element being a wide struct rather than the nesting itself: depth 8 costs Comet only 11ms over depth 1, where it costs Spark 15ms. The plain `array<struct<bigint, string>>` case at fan-out 10 is 0.7X for the same reason. Good call asking for this; it is the largest gap the benchmark found and I think it is where the operator work should start. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
