JingsongLi commented on code in PR #11031:
URL: https://github.com/apache/paimon/pull/11031#discussion_r4581725952
##########
paimon-python/pypaimon/execution/native/read/table_read.py:
##########
@@ -40,6 +40,54 @@
_NATIVE_READ_FILE_SUFFIXES = tuple('.%s' % file_format for file_format in
_NATIVE_READ_FILE_FORMATS)
_NATIVE_BLOB_FILE_SUFFIXES = ('.blob', '.video')
+# merge_map_with_keytime always uses the Python aggregator for now:
+# * STRING/CHAR/VARCHAR: Rust UTF-8 ``&str`` order diverges from
+# Java/Python UTF-16 (e.g. U+E000 vs U+10000).
+# * Every other timestamp type historically fell back already (the old
+# Native gate only cleared STRING-family via StringArray), and there
+# is no end-to-end-correct Native replacement without a Rust compare
+# fix. Fall back before Native opens; do not catch mid-read.
+_MERGE_MAP_WITH_KEYTIME = "merge_map_with_keytime"
+
+
+def _field_agg_func_name(field_name, primary_keys, options_map,
sequence_fields):
+ # Same precedence as Python ``resolve_agg_func_name`` / Java
+ # ``AggregateMergeFunction.getAggFuncName``, inlined to avoid a
+ # Native → Python execution import.
+ if field_name in sequence_fields:
+ return "last_value"
+ if field_name in primary_keys:
+ return "primary_key"
+ return (
+ options_map.get("fields.{}.aggregate-function".format(field_name))
+ or options_map.get("fields.default-aggregate-function")
+ or "last_non_null_value"
+ )
+
+
+def _native_merge_map_keytime_timestamp_supported(table) -> bool:
+ """Whether Native can run ``merge_map_with_keytime`` for this table.
+
+ Always ``False`` when any field uses ``merge_map_with_keytime`` on a
+ PK aggregation table. Non-PK / non-AGGREGATE / other aggregators are
+ unchanged and still clear this gate.
+ """
+ if not getattr(table, 'is_primary_key_table', False):
+ return True
+ options = table.options
+ if options.merge_engine() != MergeEngine.AGGREGATE:
Review Comment:
[P2] Cover partial-update aggregators in the Native capability check
`merge_map_with_keytime` can also be used by Java partial-update tables with
a sequence group, so returning true for every non-aggregation engine still
permits the same comparator mismatch. I wrote a real Parquet PK table through
the reviewed Java writer with top-level seq 1/2,
`fields.seq.sequence-group=updates`, and nested string timestamps U+E000 then
U+10000. Java reads `first`. Loading those exact persisted files in PyPaimon
with Native enabled returns `second`, with Python fallback forbidden; this gate
returns true.
Please account for configured field aggregators in the partial-update path
too. If the supported Native runtime cannot match these semantics and Python
cannot execute the configuration, fail explicitly before opening a Native read
or require a matching runtime rather than returning a different value.
##########
paimon-python/pypaimon/execution/python/read/reader/aggregate/aggregators.py:
##########
@@ -1339,10 +1364,23 @@ def _merge_input_map(self, result_map: Dict[Any, Any],
input_field: Any):
continue
existing_ts = self._get_ts_field(existing_row)
- if (existing_ts is None or new_ts > existing_ts
- or (self.overwrite_on_equal_ts and new_ts == existing_ts)):
+ if existing_ts is None or
self._should_replace_by_timestamp(new_ts, existing_ts):
result_map[key] = new_row
+ def _should_replace_by_timestamp(self, new_ts: Any, existing_ts: Any) ->
bool:
+ # CHAR/VARCHAR keep Java String.compareTo (UTF-16 code units).
+ # Other supported types compare by their native order, including
+ # TIMESTAMP(9) / TIMESTAMP_LTZ(9) values that still carry a
+ # nanosecond tail after Arrow row materialization. Equal timestamps
+ # replace only when overwrite-on-equal-ts is enabled.
+ if self._string_timestamp:
+ if _utf16_code_unit_greater(new_ts, existing_ts):
Review Comment:
[P2] Gate Native writes for the new string comparison too
The new UTF-16 comparison is applied in Python and Java, but Native writes
remain eligible and reduce map entries with Rust's UTF-8 string ordering before
any read fallback. On a real PK aggregation table, one public `write_arrow`
batch contains the same PK/map key first at timestamp U+E000 (`first`), then
U+10000 (`second`). The actual Rust 0.5.0 NativeBatchTableWrite persists only
`second`; reopening via the explicitly Python reader cannot recover the UTF-16
winner `first`. Two single-record Python commits retain `first`. The exact
merge-base selects `second` consistently, and current ASCII controls agree, so
this is introduced by the comparator change rather than a tie/ordering artifact.
Please gate Native write reduction for this aggregator until its comparator
matches (or require a compatible runtime), and add a persisted-table write/read
regression. Reader fallback alone cannot repair a value already discarded
during writing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]