Github user zivanfi commented on the issue:
https://github.com/apache/spark/pull/19250
@attilajeges has just found a problem with the behavior specified in the
requirements:
* Partitions of a table can use different file formats.
* As a result, a single table can have data files of different file formats
at the same time.
* Timestamps are already handled differently in these formats (this was our
original problem to begin with).
* As a result, **no uniform adjustment can fix timestamps for all file
formats of the same table at the same time.**
We can still solve the issue using a file-format-specific table property
though. @rxin, I would like to ask you opinion of such a solution:
* It **is** implemented in the analyzer, as you asked.
* It **is** writer-agnostic, as you asked.
* It **is not** file-format-agnostic, but Parquet-specific instead for the
time being.
Would you find such a soltion be acceptable, given that a
file-format-agnostic fix seems infeasible at this point?
Thanks,
Zoltan
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
