LuciferYang opened a new pull request, #10744: URL: https://github.com/apache/paimon/pull/10744
### Purpose close #10743 With `format-table.implementation = engine`, Spark's `FallBackFileSourceV2` turns `INSERT INTO` a format table into a V1 `InsertIntoHadoopFsRelationCommand` that carries only the statement's options. The table's format options never reached the writer, so a CSV table with `csv.field-delimiter = ';'` was written with commas and read back as NULLs, and `file.compression` had no effect. - The file indexes of engine format tables keep the table options, and a post-hoc rule `PaimonFormatTableWriteOptions` adds them to the V1 write of a CSV or TEXT table, under the statement's options. These are the two formats whose reads use the table options. - The TEXT branch of `FormatTableCatalog` now sets `compression` before copying the options. - `compression` is passed to Spark as the Hadoop codec class from `HadoopCompressionType`, the same codec the Paimon implementation uses, since Spark has no short name for zstd. V2 writes such as `df.writeTo(...).append()` on Spark 3.x still use the write's options only and are not changed here. ### Tests `FormatTableTestBase` "Format table: engine writes with the format options of the table": with the engine implementation, a gzip TEXT table writes `.txt.gz` files, a `;`-delimited gzip CSV table reads back a value containing a comma and writes `.csv.gz` files, and CSV and TEXT tables with `file.compression = 'zstd'` can be read. On master it fails at the first check: the TEXT table writes an uncompressed `.txt` file. ### API and Format no ### Documentation no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
