PyRSA commented on code in PR #11031:
URL: https://github.com/apache/paimon/pull/11031#discussion_r4535015565
##########
paimon-python/pypaimon/execution/python/read/reader/aggregate/aggregators.py:
##########
@@ -1280,9 +1314,17 @@ def _merge_input_map(self, result_map: Dict[Any, Any],
input_field: Any):
continue
existing_ts = self._get_ts_field(existing_row)
- if existing_ts is None or new_ts > existing_ts:
+ if existing_ts is None or self._is_newer_timestamp(new_ts,
existing_ts):
result_map[key] = new_row
+ def _is_newer_timestamp(self, new_ts: Any, existing_ts: Any) -> bool:
+ # CHAR/VARCHAR keep Java String.compareTo (UTF-16 code units).
+ # Other supported types, including TIMESTAMP values that still carry a
+ # nanosecond tail, compare by their native order.
+ if self._string_timestamp:
+ return _utf16_code_unit_greater(new_ts, existing_ts)
Review Comment:
Thanks for pointing this out! I've fixed the nanosecond precision loss in
the record batch reader and added regression tests covering the public table
read path for TIMESTAMP(9) and TIMESTAMP_LTZ(9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]