JingsongLi commented on code in PR #11031:
URL: https://github.com/apache/paimon/pull/11031#discussion_r4522860444
##########
paimon-python/pypaimon/execution/python/read/reader/aggregate/aggregators.py:
##########
@@ -1280,9 +1314,17 @@ def _merge_input_map(self, result_map: Dict[Any, Any],
input_field: Any):
continue
existing_ts = self._get_ts_field(existing_row)
- if existing_ts is None or new_ts > existing_ts:
+ if existing_ts is None or self._is_newer_timestamp(new_ts,
existing_ts):
result_map[key] = new_row
+ def _is_newer_timestamp(self, new_ts: Any, existing_ts: Any) -> bool:
+ # CHAR/VARCHAR keep Java String.compareTo (UTF-16 code units).
+ # Other supported types, including TIMESTAMP values that still carry a
+ # nanosecond tail, compare by their native order.
+ if self._string_timestamp:
+ return _utf16_code_unit_greater(new_ts, existing_ts)
Review Comment:
**[P2] Verify nanosecond ordering through the table read path**
I wrote two PK updates for the same map entry with timestamp tails of 1 ns
and 2 ns. Parquet preserves the two distinct values. The public PyPaimon read
passes the nested values through `record_batch_reader.py`'s Polars `iter_rows`
conversion, which rounds both to the same microsecond before they reach this
comparator. The older value wins. I reproduced this with TIMESTAMP(9) and
TIMESTAMP_LTZ(9). The new BinaryRow test bypasses that conversion. Could we
preserve nanosecond values for this path or scope the Python precision
guarantee to what the table reader supports and add a public read regression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]