alamb commented on code in PR #15537: URL: https://github.com/apache/datafusion/pull/15537#discussion_r2045205455
########## datafusion/sqllogictest/test_files/information_schema.slt: ########## @@ -296,6 +297,7 @@ datafusion.execution.parquet.bloom_filter_fpp NULL (writing) Sets bloom filter f datafusion.execution.parquet.bloom_filter_ndv NULL (writing) Sets bloom filter number of distinct values. If NULL, uses default parquet writer setting datafusion.execution.parquet.bloom_filter_on_read true (writing) Use any available bloom filters when reading parquet files datafusion.execution.parquet.bloom_filter_on_write false (writing) Write bloom filters for all columns when creating parquet files +datafusion.execution.parquet.coerce_int96 NULL (reading) If true, parquet reader will read columns of physical type int96 as originating from a different resolution than nanosecond. This is useful for reading data from systems like Spark which stores microsecond resolution timestamps in an int96 allowing it to write values with a larger date range than 64-bit timestamps with nanosecond resolution. Review Comment: While reviewiew this PR again, I think this text is not quite right -- it isn't `true`, instead it takes a string value `ms`, `ns`, `us`, etc for the timestamp resolution ``` External error: task 17 panicked with message "called `Result::unwrap()` on an `Err` value: Configuration(\"Unknown or unsupported parquet coerce_int96: true. Valid values are: ns, us, ms, and s.\")" ``` I will file a ticket ########## datafusion/sqllogictest/test_files/information_schema.slt: ########## @@ -296,6 +297,7 @@ datafusion.execution.parquet.bloom_filter_fpp NULL (writing) Sets bloom filter f datafusion.execution.parquet.bloom_filter_ndv NULL (writing) Sets bloom filter number of distinct values. If NULL, uses default parquet writer setting datafusion.execution.parquet.bloom_filter_on_read true (writing) Use any available bloom filters when reading parquet files datafusion.execution.parquet.bloom_filter_on_write false (writing) Write bloom filters for all columns when creating parquet files +datafusion.execution.parquet.coerce_int96 NULL (reading) If true, parquet reader will read columns of physical type int96 as originating from a different resolution than nanosecond. This is useful for reading data from systems like Spark which stores microsecond resolution timestamps in an int96 allowing it to write values with a larger date range than 64-bit timestamps with nanosecond resolution. Review Comment: Strangely setting it to `true` seems to work in datafusion-cli 🤔 ```sql DataFusion CLI v46.0.1 > set datafusion.execution.parquet.coerce_int96 = true; 0 row(s) fetched. Elapsed 0.000 seconds. > show all; ... | datafusion.execution.parquet.coerce_int96 | true | ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
