rishvin commented on issue #1941: URL: https://github.com/apache/datafusion-comet/issues/1941#issuecomment-3120621876
> > managed to scan the map-type by setting `CometConf.COMET_NATIVE_SCAN_IMPL.key -> native_datafusion `. Added `map_sort` UDF with return type as `Map`. > > Right. `native_comet` does not support complex types. You can also try with `native_iceberg_compat`. For `map_sort` you could try an `order by` instead of a `group by` in your test query Thanks @parthchandra. We cannot use `order by` because spark applies `MapSort` either for groupby or repartitioning using the following rules, [InsertMapSortInRepartitioningExpression](https://github.com/apache/spark/blob/6c6ca1f0b6f43a2034f4d6bcf1af1f07e6b7ccc7/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InsertMapSortExpression.scala#L90-L90) [InsertMapSortInGroupingExpression](https://github.com/apache/spark/blob/6c6ca1f0b6f43a2034f4d6bcf1af1f07e6b7ccc7/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InsertMapSortExpression.scala#L38) I doubled-checked applying orderby on mapType in Spark-4.0 and got, ``` The `sortorder` does not support ordering on type "MAP<STRING, INT>". SQLSTATE: 42K09; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
