Github user npoberezkin commented on the issue:
https://github.com/apache/spark/pull/22255
Hello, @dbtsai, @HyukjinKwon . I added test on reading writer.model.name to
PR. Justification for this change is below.
This is original jira:
https://issues.apache.org/jira/browse/SPARK-25102
and it was referring to this one:
https://issues.apache.org/jira/browse/PARQUET-352
where the justification was given (it will be possible to identify files
written by object models incorrectly). Also here is the link to Parquet
repository with corresponding code changes (justification is also provided
there):
https://github.com/apache/parquet-mr/commit/dcd1c33f0dba247b43418b922c1c3a2fc432dc11
And i found another case in which possibly this change can be useful:
https://github.com/dask/fastparquet/issues/352
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
