dejankrak-db commented on PR #59305: URL: https://github.com/apache/spark/pull/59305#issuecomment-6200834754
@HyukjinKwon, on the Parquet footer question: I would accept this and record it. JSON decoding stays ungated on purpose, so that a persisted schema (a table property, a Delta log) stays readable with the flag off, and gating it would turn that into an `INVALID_JSON_DATA_TYPE` failure. A Parquet footer that lists a `"file"` type is now decoded as a FILE column, and `spark.read.parquet` on it fails with `UNSUPPORTED_FILE_TYPE` instead of reading the column as a struct. I updated the user-facing change section of the description to say so. Parquet support for FILE is a later part of the SPIP (it waits for a parquet-java release), and that part decides how footers carry FILE. I could not confirm that any writer records FILE columns in a released footer, so the exposure may be nil today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
