plusplusjiajia opened a new pull request, #11008:
URL: https://github.com/apache/paimon/pull/11008
This is based on #11007, which adds the CSV reader and writer that this
change shares line splitting and number parsing with. Only the last commit
belongs to this PR.
### Purpose
PyPaimon wrote format table JSON files with `json.dumps` of Python values,
which fails for DECIMAL, DATE, TIMESTAMP and BINARY columns. It read each file
with types inferred from its values.
Java `JsonFormatWriter` writes every scalar as a JSON string. As a result,
PyPaimon:
- read Java's files with string columns;
- failed with `Schema at index N was different` when the files of one table
were inferred differently.
As in Java, the writer now:
- writes the non-partition columns;
- writes scalar values as strings formatted with Java's casts;
- writes map keys as the string of their value;
- writes BINARY values as Base64;
- writes nested values as arrays and objects;
- uses `json.line-delimiter`, or its fallback key `lineSep`.
As in `JsonFileReader`, the reader now:
- converts the projected columns with the table's types;
- accepts JSON numbers and booleans as well as strings;
- reads floating-point numbers as decimals when a projected column is
DECIMAL;
- keeps the last of the map keys that convert to the same value;
- skips null lines;
- applies `json.ignore-parse-errors`.
Like Java, a blank line is a parse error unless `json.ignore-parse-errors`
is set. Jackson reads a blank line as a missing node, which `JsonFileReader`
cannot convert.
The reader and the writer reject the types that
`JsonFileFormat.validateDataFields` rejects, such as BLOB and VARIANT.
### Tests
- `format_table_json_test.py`, 13 tests:
- the writer's output for scalar, binary, map and nested values, compared
with what Java writes;
- a round trip of every type;
- native JSON values read with the table's types;
- parsing only the projected columns;
- parse errors with and without `json.ignore-parse-errors`;
- map key conversion and duplicates;
- VECTOR lengths;
- type validation;
- both line delimiter keys;
- a partitioned table written and read back.
- The table-level test fails without this change: writing a DECIMAL column
raises `Object of type Decimal is not JSON serializable`.
- Cross-client check against a DLF catalog. This was run with this change
together with the partition path fix. Spark with Paimon 2.1.0, PyPaimon and
pypaimon-rust 0.4.0 each wrote JSON tables with DECIMAL, DATE, TIMESTAMP and
BINARY values, and every client read every table back with the same rows.
- The format table tests pass: `format_table_json_test.py`,
`format_table_csv_test.py`, `format_write_execution_test.py`,
`rest_format_table_test.py` and `cli_table_test.py`, 133 tests. flake8 is clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]