Github user squito commented on the issue:
https://github.com/apache/spark/pull/19702
ok sorry for taking a bit of time to look at this. tl;dr I think what you
are proposing here is fine.
So to be honest, I'm a bit confused about the parquet versioning -- the
types I was mentioned were added here:
https://github.com/apache/parquet-format/commit/863875e0be3237c6aa4ed71733d54c91a51deabe#diff-0f9d1b5347959e15259da7ba8f4b6252
to parquet-format v2.4.0
I can't really tell what version of parquet-mr that corresponds to. I can
at least see that parquet-mr v1.8.2 doesn't have the new types. I don't even
think its master. But IIUC, the latest version of parquet-format introduces
new types, independent of the old TIMESTAMP_MILLIS and TIMESTAMP_MICROS. So
there won't be any compatibility issues when `isAdjustedToUTC` is added, as
that will be in an entirely new type.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
