andygrove commented on issue #4859: URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5547193582
There is now a reproducible microbenchmark for this on `main`, which may be useful for validating #5262 without needing a production dataset. `CometExplodeBenchmark` (#5381) writes a `nested` view holding `array<struct<platform string, entries array<struct<type, ts, page, source>>>>`, four elements per row, 100K rows. Querying that view directly reproduces this issue in a few seconds. Apple M3 Ultra, JDK 17, `local[1]`, Spark 4.1 / Scala 2.13, release native build, Best ms: | Query | Spark | Comet on `main` | Comet on `df55` | | ----- | ----- | --------------- | --------------- | | `count(k)`, flat column, control | 17 / 21 | 10 (1.6X) | 11 (1.9X) | | `count(events)`, whole struct, 5 leaves | 60 / 65 | 92 (0.7X) | **59 (1.1X)** | | `count(events.platform)`, 1 leaf | 22 / 25 | 93 (0.2X) | **13 (1.9X)** | | `count(events.entries)`, inner array | 61 / 63 | 93 (0.7X) | **57 (1.1X)** | | `explode(events)` then `count(col.platform)` | 23 / 34 | 95 (0.2X) | **21 (1.6X)** | The two Spark columns are the two branches' runs of the same query, given here so the Comet columns can be read against a control rather than against each other. The one-leaf row is the clean statement of the bug. Spark drops to about a third of its whole-struct time when only one leaf is needed; Comet on `main` does not move at all, 92ms to 93ms. Both executed plans print the identical pruned `ReadSchema: struct<events:array<struct<platform:string>>>`, so the pruning reaches the plan and then changes nothing about what gets read. Comet's time is flat at 92ms to 95ms across every subset of the column requested. On `df55` that row goes to 13ms, which is 1.9X Spark, so this confirms @mixermt's production finding on a dataset anyone can regenerate locally. Worth noting separately: the whole-struct row also improves, 92ms to 59ms, where there is no pruning available at all, so the DataFusion bump helps the nested reader beyond leaf selection. A caveat for anyone rerunning this: `df55` is currently 114 commits behind `main` and has no `explode.rs`, so it predates `CometExplodeExec` and a generator query there falls back to Spark's `GenerateExec`. Only the scan-shaped queries above compare meaningfully across the two branches. Cross-referencing #5685: this issue turns out to be the dominant term there. In that issue's `depth 1` case, 92ms of Comet's 113ms is spent before the generator exists, so this is what to fix first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
