awdavidson commented on code in PR #38312:
URL: https://github.com/apache/spark/pull/38312#discussion_r1004137291
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala:
##########
@@ -198,6 +205,31 @@ abstract class ParquetSchemaTest extends ParquetTest with
SharedSparkSession {
}
class ParquetSchemaInferenceSuite extends ParquetSchemaTest {
+ testSchemaInference[Tuple1[Long]](
Review Comment:
So I've been looking further into it, it's because the message is different
between `1.10.1` and `1.12.3`.
In `1.10.1` the message is
```
message schema {
required int64 attribute;
}
```
where as `1.12.3` the message is the same as the unit test
```
message schema {
required int64 attribute (TIMESTAMP(NANOS,true));
}
```
So in Spark 3.1.0 you end up hitting this block with returns a `LongType`
https://github.com/apache/spark/blob/branch-3.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L146
where as since 3.2 you hit
https://github.com/apache/spark/blob/branch-3.2/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L174
because a case for `TimeUnit.NANOS` is not covered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
