JingsongLi commented on code in PR #1378:
URL: https://github.com/apache/paimon-rust/pull/1378#discussion_r4656641599


##########
crates/paimon/src/table/aggregator/merge_map.rs:
##########
@@ -225,7 +260,11 @@ impl MergeMapAgg {
             .collect();
         if !self.seen_input {
             // Java returns the first non-null map unchanged, including its
-            // entry order and any duplicate keys.
+            // entry order and any duplicate keys. Still reject NaN key-times
+            // so they never become accumulator state.
+            if self.timestamp_index.is_some() {
+                self.reject_nan_in_entries(&incoming)?;

Review Comment:
   **[P1] Reject NaN before the singleton write can publish it**
   
   This check runs only when the field aggregator runs. The singleton PK fast 
paths in `kv_file_writer.rs` can bypass it. I wrote one unique primary key with 
a NaN time to separate FLOAT and DOUBLE tables. Both committed snapshot 1 and 
could read back the row. A finite update for the same key then committed 
snapshot 2, after which the scan failed with `rejects NaN`. The current 
`write_or_merge_err` helper also accepts a scan failure after a successful 
commit. Validate configured key-time cells on the normal ingestion path before 
the singleton shortcut, and assert that rejection leaves the published snapshot 
unchanged.



##########
crates/paimon/src/table/aggregator/merge_map.rs:
##########
@@ -150,17 +155,37 @@ impl MergeMapAgg {
         if column.is_null(0) {
             return Ok(None);
         }
-        let string = column
-            .as_any()
-            .downcast_ref::<StringArray>()
-            .ok_or_else(|| Error::DataInvalid {
-                message: format!(
-                    "merge_map_with_keytime timestamp for '{}' requires 
STRING",
-                    self.field_name
-                ),
-                source: None,
-            })?;
-        Ok(Some(string.value(0)))
+        Ok(Some(column.as_ref()))
+    }
+
+    fn should_replace_by_timestamp(
+        &self,
+        new_ts: &dyn Array,
+        old_ts: &dyn Array,
+    ) -> crate::Result<bool> {
+        let kind = self.keytime_kind.as_ref().expect("timestamp mode");
+        let cmp = compare_keytime(kind, new_ts, old_ts, &self.field_name)?;
+        Ok(match cmp {
+            KeytimeCmp::Greater => true,
+            KeytimeCmp::Equal => self.overwrite_on_equal_ts,

Review Comment:
   **[P1] Preserve the winning sequence for equal key-times across flushes**
   
   With INT key-time, `sequence.field=seq`, and `overwrite-on-equal-ts=true`, I 
wrote the same three rows for one PK and map key in both cases. The `(seq, 
payload, key-time)` values were `(1,11,5)`, `(3,33,0)`, and `(2,22,5)`. A 
single flush returns payload `22`. Flushing the first two together and the last 
separately returns `11`. The first file retains payload `11` with the group's 
sequence `3`, so it wins an equal-time comparison against sequence `2`. 
Preserve the winning entry's sequence across partial reductions or reject this 
option combination until it can be represented.



##########
crates/paimon/src/spec/aggregation.rs:
##########
@@ -298,11 +304,12 @@ fn parse_field_scoped_option_key(key: &str) -> 
Option<(&str, FieldScopedOptionKi
 /// `SchemaManager.applyRenameColumnsToOptions` (Java's case-2 list; the
 /// aggregation/partial-update engines do not accept every entry at create
 /// time, but the keys can still arrive in Java-written metadata).
-const FIELD_SCOPED_RENAMEABLE_SUFFIXES: [&str; 7] = [
+const FIELD_SCOPED_RENAMEABLE_SUFFIXES: [&str; 8] = [
     AGG_FUNCTION_SUFFIX,
     IGNORE_RETRACT_SUFFIX,
     DISTINCT_SUFFIX,
     LIST_AGG_DELIMITER_SUFFIX,
+    OVERWRITE_ON_EQUAL_TS_SUFFIX,

Review Comment:
   **[P2] Migrate the explicit time-field selector with the map column**
   
   The new equal-time option follows the map through rename, but 
`fields.<column>.ts-field` still stays under the old name. I created a table 
with a non-last INT field `payload` as the explicit key-time, wrote 
`(payload=10,ts=1)`, renamed the map through the filesystem catalog, then 
appended `(payload=1,ts=2)`. ALTER and both writes succeed, and the scan 
changes the winner to `1` because the renamed field falls back to the trailing 
`ts`. Rename and clean up `.ts-field` alongside `.overwrite-on-equal-ts`, and 
cover a non-default selector in the catalog round trip.



##########
docs/src/sql.md:
##########
@@ -3554,12 +3554,31 @@ Field-scoped options used by these functions:
   reject integer overflow instead of wrapping. Floating-point addition is
   unaffected; Decimal overflow still yields NULL as in Java.
 - `fields.<field>.ts-field` selects the key-time field for
-  `merge_map_with_keytime` (defaults to the last ROW field).
+  `merge_map_with_keytime` (defaults to the last ROW field). Supported
+  key-time types are `CHAR`/`VARCHAR`, `TINYINT`/`SMALLINT`/`INT`/`BIGINT`,
+  `FLOAT`/`DOUBLE`, `DATE`, `TIME`, `DECIMAL`, `TIMESTAMP`, and
+  `TIMESTAMP_LTZ`.
+- `fields.<field>.overwrite-on-equal-ts` (default `false`) for
+  `merge_map_with_keytime`: when `true`, an equal key-time replaces the
+  existing ROW; when `false`, the existing ROW is kept.
 - `fields.<field>.nested-key`, `fields.<field>.nested-sequence-field`,
   `fields.<field>.nested-key-null-strategy`, and `fields.<field>.count-limit`
   for nested array aggregators. `nested_partial_update` requires
   `nested-key`.
 
+For `merge_map_with_keytime`, a newer key-time replaces the existing ROW for
+that map key. `CHAR`/`VARCHAR` key-times use unsigned UTF-8 byte
+lexicographical order (Java `BinaryString.compareTo`). `FLOAT`/`DOUBLE`
+compare as ordinary numbers with `+0.0` equal to `-0.0`; NaN key-times are
+rejected and never enter aggregation. `TIMESTAMP` / `TIMESTAMP_LTZ` compare
+by ticks under a unified time semantics (timezone metadata does not affect
+ordering). A NULL key-time ignores the incoming update and leaves any
+existing key unchanged. A NULL ROW value removes that key. Rust Native
+supports the key-time types listed above, including `FLOAT`/`DOUBLE`. The
+Python Native eligibility gate allows the same Rust-supported set (and
+falls back only for types Rust cannot compare). Java still rejects
+`FLOAT`/`DOUBLE` key-times.

Review Comment:
   **[P2] Document the actual Java key-time type gate**
   
   The Java caveat needs to include DECIMAL. I ran the production Java factory 
in Paimon 2.2 snapshot `20261118.220748-71`. It accepts INT, TIME(3), and 
TIMESTAMP(9), and rejects DECIMAL(10,2), FLOAT, and DOUBLE. I also wrote, 
scanned, and compacted a DECIMAL table using this Rust head successfully. 
Readers of mixed-engine tables need the version-specific Java type restriction 
before enabling the new types.



##########
docs/src/sql.md:
##########
@@ -3554,12 +3554,31 @@ Field-scoped options used by these functions:
   reject integer overflow instead of wrapping. Floating-point addition is
   unaffected; Decimal overflow still yields NULL as in Java.
 - `fields.<field>.ts-field` selects the key-time field for
-  `merge_map_with_keytime` (defaults to the last ROW field).
+  `merge_map_with_keytime` (defaults to the last ROW field). Supported
+  key-time types are `CHAR`/`VARCHAR`, `TINYINT`/`SMALLINT`/`INT`/`BIGINT`,
+  `FLOAT`/`DOUBLE`, `DATE`, `TIME`, `DECIMAL`, `TIMESTAMP`, and
+  `TIMESTAMP_LTZ`.
+- `fields.<field>.overwrite-on-equal-ts` (default `false`) for
+  `merge_map_with_keytime`: when `true`, an equal key-time replaces the
+  existing ROW; when `false`, the existing ROW is kept.
 - `fields.<field>.nested-key`, `fields.<field>.nested-sequence-field`,
   `fields.<field>.nested-key-null-strategy`, and `fields.<field>.count-limit`
   for nested array aggregators. `nested_partial_update` requires
   `nested-key`.
 
+For `merge_map_with_keytime`, a newer key-time replaces the existing ROW for
+that map key. `CHAR`/`VARCHAR` key-times use unsigned UTF-8 byte
+lexicographical order (Java `BinaryString.compareTo`). `FLOAT`/`DOUBLE`
+compare as ordinary numbers with `+0.0` equal to `-0.0`; NaN key-times are
+rejected and never enter aggregation. `TIMESTAMP` / `TIMESTAMP_LTZ` compare
+by ticks under a unified time semantics (timezone metadata does not affect
+ordering). A NULL key-time ignores the incoming update and leaves any
+existing key unchanged. A NULL ROW value removes that key. Rust Native

Review Comment:
   **[P1] Preserve a key tombstone across partial aggregation**
   
   I used a newly supported INT key-time and wrote three versions of the same 
key: payload `11` at time `9`, payload `22` at time `1`, then a NULL ROW for 
that map key. Flushing all three together leaves the map empty. Flushing the 
old version separately and grouping the lower-time update with the tombstone 
leaves payload `11` readable. The second partial aggregate persists an empty 
map and loses the deletion marker, so it cannot remove the older key. Preserve 
that deletion information across materialization or restrict this newly 
supported path to a safe merge.



##########
crates/paimon/src/table/aggregator/merge_map.rs:
##########
@@ -72,41 +101,15 @@ impl MergeMapAgg {
         data_type: &DataType,
         options: &HashMap<String, String>,
     ) -> crate::Result<Self> {
-        let DataType::Map(map_type) = data_type else {
-            return Err(unsupported_type_error(
-                "merge_map_with_keytime",
-                field_name,
-                data_type,
-            ));
-        };
-        let DataType::Row(value_type) = map_type.value_type() else {
-            return Err(unsupported_type_error(
-                "merge_map_with_keytime",
-                field_name,
-                data_type,
-            ));
-        };
-        if value_type.fields().len() < 2 {
-            return Err(unsupported_type_error(
-                "merge_map_with_keytime",
-                field_name,
-                data_type,
-            ));
-        }
-        let timestamp_index = match 
options.get(&format!("fields.{field_name}.ts-field")) {
-            Some(name) => value_type
-                .fields()
-                .iter()
-                .position(|field| field.name() == name)
-                .ok_or_else(|| Error::ConfigInvalid {
-                    message: format!(
-                        "Timestamp field '{name}' not found in ROW type for 
field '{field_name}'"
-                    ),
-                })?,
-            None => value_type.fields().len() - 1,
-        };
+        let (timestamp_index, keytime_kind) =
+            resolve_keytime_config(field_name, data_type, options)?;
+        let overwrite_on_equal_ts = options
+            .get(&format!("fields.{field_name}.overwrite-on-equal-ts"))
+            .is_some_and(|value| value.eq_ignore_ascii_case("true"));

Review Comment:
   **[P2] Reject invalid values for the new boolean option**
   
   Rust CREATE accepts `fields.updates.overwrite-on-equal-ts=treu` and silently 
interprets it as false. Java Paimon 2.2 snapshot `20261118.220748-71` rejects 
this same option when constructing the aggregator, while `true` and `false` 
both work. An accepted schema with a typo can therefore select the wrong tie 
winner in Rust and fail to initialize in Java. Parse and validate the boolean 
in the shared schema/runtime configuration code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to