plusplusjiajia opened a new pull request, #11007:
URL: https://github.com/apache/paimon/pull/11007
### Purpose
PyPaimon read and wrote format table CSV files with pyarrow's defaults. When
writing, it:
- added a header row;
- ignored the `csv.*` table options;
- wrote BINARY values as raw bytes.
When reading, it took column names and types from the first line of each
file.
Java `CsvFormatWriter` writes no header by default, and `CsvParser` maps
columns by position. This caused failures in both directions:
- PyPaimon took the first row of a Java file as its header. For example,
Spark wrote 3 rows and PyPaimon read 1 row with the columns `2` and `b`.
- A file written with another delimiter read as no rows, without an error.
- Java could not read PyPaimon's files.
As in Java, the writer now:
- writes the non-partition columns in schema order;
- applies the `csv.*` options: field and line delimiter, quote and escape
character, null literal and header;
- formats values with Java's casts;
- encodes BINARY values as Base64;
- quotes and escapes values like `CsvFormatWriter`;
- refuses values that contain the row separator.
The reader now:
- splits lines and fields like `CsvParser`, including its handling of quotes
and escapes;
- parses values with the table's types;
- applies `csv.mode` (PERMISSIVE, DROPMALFORMED or FAILFAST) to the
projected columns, and `csv.skip-empty-lines`.
Both reject the types and option combinations that Java rejects, as in
`CsvFileFormat.validateDataFields` and the `CsvParser` constructor. A failed
read now raises instead of returning no rows.
Java's string formatting of partition values moves out of
`_canonical_partition` into a function that the CSV writer shares. That
function also accepts pandas timestamps for TIMESTAMP_LTZ: pandas'
`astimezone()` requires an argument, so the old code failed on them.
Compatibility: CSV files written by earlier PyPaimon versions start with a
header and may contain partition columns. They now read the way Java reads
them, so the header is read as a data row.
### Tests
- `format_table_csv_test.py`, 21 tests:
- the writer's output for every scalar type, null, empty strings, quotes,
escapes, the header and custom options, compared with what Java writes;
- Java's quoting rules on read;
- the header, BOM and empty lines;
- the three parse modes, including only parsing the projected columns;
- option and type validation;
- a partitioned table written and read back, and a file in Java's layout
read through the table.
- The two table-level tests fail with master's reader and writer.
- Cross-client check against a DLF catalog. This was run with this change
together with the partition path fix. Spark with Paimon 2.1.0, PyPaimon and
pypaimon-rust 0.4.0 each wrote CSV tables with commas, quotes, backslashes,
tabs, empty strings and binary values, and every client read every table back
with the same rows.
- The format table tests pass: `format_table_csv_test.py`,
`format_write_execution_test.py`, `rest_format_table_test.py` and
`cli_table_test.py`, 120 tests. flake8 is clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]