alamb commented on issue #75: URL: https://github.com/apache/parquet-testing/issues/75#issuecomment-2800087150
I looked into [primitive.parquet.txt](https://github.com/user-attachments/files/19639256/primitive.parquet.txt) and [array_simple.parquet.txt](https://github.com/user-attachments/files/19639257/array_simple.parquet.txt) from @aihuaxu above ❤ # `primitive.parquet`: ```sql CREATE TEMPORARY VIEW parquetTable USING org.apache.spark.sql.parquet OPTIONS ( path "/Users/andrewlamb/Downloads/primitive.parquet" ); DESCRIBE parquetTable; SELECT * from parquetTable ``` Yields this output (somehow spark doesn't see the file as variant 🤔 ) ```sql spark-sql (default)> DESCRIBE parquetTable; id int var struct<metadata:binary,value:binary> Time taken: 0.051 seconds, Fetched 2 row(s) spark-sql (default)> SELECT * from parquetTable; 1 {"metadata":,"value": "} Time taken: 0.066 seconds, Fetched 1 row(s) ``` # `array_simple.parquet`: ```sql CREATE TEMPORARY VIEW parquetTable USING org.apache.spark.sql.parquet OPTIONS ( path "/Users/andrewlamb/Downloads/array_simple.parquet" ); DESCRIBE parquetTable; SELECT * from parquetTable ``` Yields this output (somehow spark doesn't see the file as variant 🤔 ) ```sql spark-sql (default)> DESCRIBE parquetTable; id int var struct<metadata:binary,value:binary> Time taken: 0.051 seconds, Fetched 2 row(s) spark-sql (default)> SELECT * from parquetTable; 1 {"metadata":,"value": "} Time taken: 0.066 seconds, Fetched 1 row(s) ``` spark-sql (default)> CREATE TEMPORARY VIEW parquetTable > USING org.apache.spark.sql.parquet > OPTIONS ( > path "/Users/andrewlamb/Downloads/array_simple.parquet" > ); Time taken: 0.806 seconds spark-sql (default)> DESCRIBE parquetTable; id int var struct<metadata:binary,value:binary,typed_value:struct<a:struct<value:binary,typed_value:array<struct<value:binary,typed_value:string>>>>> Time taken: 0.108 seconds, Fetched 2 row(s) spark-sql (default)> SELECT * from parquetTable > ; 1 {"metadata":abcde,"value":null,"typed_value":{"a":{"value":null,"typed_value":[{"value":null,"typed_value":"comedy"},{"value":null,"typed_value":"drama"}]}}} Time taken: 0.313 seconds, Fetched 1 row(s) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
