plusplusjiajia opened a new pull request, #11018:
URL: https://github.com/apache/paimon/pull/11018
### Purpose
PyPaimon's format table writer did not build partition directories the way
Java does:
- Keys and values were not escaped, so a value `x/y` was written to
`dt=x/y`, two directory levels. No client, PyPaimon included, could read the
row back.
- A null value ended the path early instead of being written as
`__DEFAULT_PARTITION__`, so Spark and Flink did not see the row.
- Some values were formatted with Python's `str()`.
PyPaimon also read Java's directories without unescaping them: `dt=x%2Fy`
read as `x%2Fy`. An INT partition at `__DEFAULT_PARTITION__` failed the whole
read with a cast error.
As in Java `FormatTableFileWriter` and `PartitionPathUtils`, the writer now:
- formats values like Java's partition computer;
- escapes keys and values;
- writes null values as the default partition name;
- rejects values that a value-only layout would hide;
- leaves partition columns out of the data files, as Java `FormatTableWrite`
does;
- refuses tables whose columns are all partition columns, since their data
files would have no columns. With pyarrow 22, a Parquet file without columns
reads back no rows, and writing an ORC file without columns crashes.
The reader now:
- unescapes keys and values;
- reads the default partition name as null;
- skips directories that are not partition directories, as Java does.
Older files that contain partition columns still read their own values.
Equality partition filters compare typed values, so they match the new
directory names.
A static overwrite also compares typed values. Before, it compared `str()`
of the row's value with the spec, so `overwrite({'pt': 'true'})` did not clear
the partition written for `True`, and the old rows stayed. The same happened
for `'1.0'` with a DECIMAL value 1.00, and for the default partition name with
a null value.
The value formatting is shared with the path factory of normal tables.
#11007 and #11008 carry the same change to `file_store_path_factory.py`.
Directories that earlier PyPaimon versions wrote with unescaped special
characters, such as `pt=a=b`, are skipped, as Java skips them. An overwrite
does not clear a partition written under an older spelling such as `pt=True`.
#11007 writes CSV files without the partition columns, as Java does, so it
needs these paths to keep null and BINARY partition values. Whichever of the
two is merged second needs two adjustments:
- #11007 supports CSV tables whose columns are all partition columns, so the
check above should then refuse them for Parquet and ORC only.
- The test cases here that read CSV files in the old PyPaimon layout, with a
header and the partition columns, no longer apply: #11007 reads CSV the way
Java does.
### Tests
- `format_write_execution_test.py`:
- round trip of escaped values, null values and the value-only layout,
with and without legacy partition names;
- `file:` roots whose names need escaping;
- typed equality filters;
- LTZ partitions in the local time zone;
- unparseable partition names;
- static overwrites with BOOLEAN, DECIMAL, DOUBLE, DATE and null
partitions. 4 of these cases fail without this change;
- reading files written before and after this change.
- Cross-client check against a DLF catalog: Spark with Paimon 2.1.0,
PyPaimon with this change and pypaimon-rust 0.4.0 each wrote Parquet and ORC
tables with partition values `a b`, `x/y` and null. Every client read every
table back with the same rows.
- The format table and partition tests pass:
`format_write_execution_test.py`, `rest_format_table_test.py`,
`cli_table_test.py`, `filesystem_catalog_test.py` and
`native_commit_partition_spec_test.py`, 258 tests. 94 tests are skipped because
they need pypaimon-rust. flake8 is clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]