andygrove opened a new issue, #5582: URL: https://github.com/apache/datafusion-comet/issues/5582
Part of #5572. `ArraysBase.isTypeSupported` (`spark/src/main/scala/org/apache/comet/serde/arrays.scala:888`) rejects `BinaryType` outright and `StructType` pending #1307, and `childTypesSupportLevel` turns that into `Unsupported`: ```scala .map(dt => Unsupported(Some(s"data type not supported: $dt"))) ``` `arrays.scala:911` Seven serdes mix in `ArraysBase`: `CometArrayRemove` (`:36`), `CometFlatten` (`:700`), `CometArrayPosition` (`:791`), `CometShuffle` (`serde/collectionOperations.scala:64`), `CometArrayInsert` (`:409`), `CometCreateArray` (`:503`) and `CometArrayReverse` (`:572`). None routes the declined types through the dispatcher, so `array<struct<...>>` and `array<binary>` — both very ordinary shapes in nested Parquet data — fail the whole projection back to Spark. `docs/source/user-guide/latest/expressions.md` already records this for three of them: `array_position`, `flatten` and `shuffle` all carry "Binary/struct/map elements fall back". Both element types are admitted by `CometBatchKernelCodegen.isSupportedDataType`, which handles `ArrayType`, `StructType` and `MapType` recursively and accepts `BinaryType` as a leaf. One important carve-out: the "null elements" half of those doc notes is **not** dispatchable. `NullType` is absent from `isSupportedDataType`, so `array<null>` will still fall back and the doc notes should be updated to say so rather than being deleted wholesale. `CometShuffle` is nondeterministic, which is fine — `canHandle` explicitly admits nondeterministic and stateful expressions, and each cache entry gets a single `init(partitionIndex)` call. Worth a test that confirms the shuffled permutation still matches Spark through the dispatcher. `CometArrayInsert`, `CometCreateArray` and `CometArrayReverse` also decline inside `convert`, which the dispatcher cannot see; see the prerequisite issue on `convert`-side declines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
