Hyukjin Kwon created SPARK-61223:
------------------------------------
Summary: DataFrameReader.parquet applies every keyword option,
where PySpark applies only its named ones
Key: SPARK-61223
URL: https://issues.apache.org/jira/browse/SPARK-61223
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
DataFrameReader.parquet(*paths, **options) set every keyword option on the
reader, where PySpark sets only mergeSchema, pathGlobFilter,
recursiveFileLookup, modifiedBefore, modifiedAfter, datetimeRebaseMode and
int96RebaseMode and ignores the others. With ignoreCorruptFiles=True, PySpark
still fails on a corrupt footer (FAILED_READ_FILE.CANNOT_READ_FILE_FOOTER) but
this client skipped the file and failed later with UNABLE_TO_INFER_SCHEMA, and
the reader kept the ignored keys in its options.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
