wombatu-kun opened a new pull request, #8842:
URL: https://github.com/apache/paimon/pull/8842
### Purpose
This closes a TODO in `pypaimon/table/system/manifests_table.py`:
```python
# TODO: render min/max_partition_stats by casting partition
# rows to their string form. pypaimon
# has SimpleStats but no shared partition-row-to-string
# helper yet; emit NULL to preserve the column shape.
min_partition_stats.append(None)
max_partition_stats.append(None)
```
So `$manifests` returns NULL for `min_partition_stats` and
`max_partition_stats` in pypaimon while Java renders them, and inspecting which
manifests cover which partitions gives nothing on the Python side. The data is
already parsed: `ManifestListManager` builds `SimpleStats` from the manifest
list, only the rendering was missing.
This adds `cast_row_to_string`, the helper the TODO asks for: a port of
`RowToStringCastRule` and the per type `*ToStringCastRule` rules of
`paimon-common`, with fields joined by `", "` inside braces, the literal `null`
for a null field, and `{}` for an unpartitioned table. The types that need more
than `str()` are boolean (lower case), binary (decoded as UTF-8), date, time,
timestamp and float. Timestamps follow `DateTimeUtils.formatTimestamp`, which
separates date and time with a space and strips trailing zeros of the fraction
down to the declared precision, so `datetime.isoformat()` can not be used.
Floats print the shortest text that round trips as float32, otherwise a float32
`0.1` read into a Python float prints as `0.10000000149011612`.
One point for a maintainer opinion, deliberately not changed here: pypaimon
renders the `partition` column of `$files` and `$buckets` as `pt=v/pt2=v2`,
while Java renders those two with the brace form and keeps `pt=v` only for
`$partitions`. Aligning them would change values these tables already return,
so it is out of scope for this PR.
### Tests
New `pypaimon/tests/row_to_string_test.py` covers the rendering per type,
including a null field, an empty row, the timestamp fraction at precision 0/3/6
and the float32 shortest form.
`pypaimon/tests/system/manifests_table_test.py` replaces the assertions that
pinned the NULL placeholder: an unpartitioned table now yields `{}`, and a
table partitioned by INT and STRING yields `{1, 2024-01-01}` and `{2,
2024-01-02}`.
Cross checked against Java on a single warehouse: a table written by
pypaimon and partitioned by INT, STRING, DATE, BOOLEAN and DECIMAL renders
identically on both sides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
