philo-he commented on code in PR #12675: URL: https://github.com/apache/gluten/pull/12675#discussion_r3703244358
########## docs/velox-backend-limitations.md: ########## @@ -144,17 +144,10 @@ Gluten's. Only reading with INT96 and dictionary encoding is supported. When reading INT64 represented millisecond/microsecond timestamps, or INT96 represented timestamps of other encodings, exceptions can occur. -- Complex types - - Parquet scan of nested array with struct or array as element type is not supported in Velox (fallback behavior). - - Parquet scan of nested map with struct as key type, or array type as value type is not supported in Velox (fallback behavior). - -### CSV Read -The header option should be true. And now we only support DatasourceV1, i.e., user should set `spark.sql.sources.useV1SourceList=csv`. User defined read option is not supported, which will make CSV read fall back to vanilla Spark in most case. -CSV read will also fall back to vanilla Spark and log warning when user specifies schema is different with file schema. - ### Utilizing Map Type as Hash Keys in ColumnarShuffleExchange Review Comment: Arrow based CSV reader has been removed. So we can remove it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
