JingsongLi commented on code in PR #1378:
URL: https://github.com/apache/paimon-rust/pull/1378#discussion_r4656641599
##########
crates/paimon/src/table/aggregator/merge_map.rs:
##########
@@ -225,7 +260,11 @@ impl MergeMapAgg {
.collect();
if !self.seen_input {
// Java returns the first non-null map unchanged, including its
- // entry order and any duplicate keys.
+ // entry order and any duplicate keys. Still reject NaN key-times
+ // so they never become accumulator state.
+ if self.timestamp_index.is_some() {
+ self.reject_nan_in_entries(&incoming)?;
Review Comment:
**[P1] Reject NaN before the singleton write can publish it**
This check runs only when the field aggregator runs. The singleton PK fast
paths in `kv_file_writer.rs` can bypass it. I wrote one unique primary key with
a NaN time to separate FLOAT and DOUBLE tables. Both committed snapshot 1 and
could read back the row. A finite update for the same key then committed
snapshot 2, after which the scan failed with `rejects NaN`. The current
`write_or_merge_err` helper also accepts a scan failure after a successful
commit. Validate configured key-time cells on the normal ingestion path before
the singleton shortcut, and assert that rejection leaves the published snapshot
unchanged.
##########
crates/paimon/src/table/aggregator/merge_map.rs:
##########
@@ -150,17 +155,37 @@ impl MergeMapAgg {
if column.is_null(0) {
return Ok(None);
}
- let string = column
- .as_any()
- .downcast_ref::<StringArray>()
- .ok_or_else(|| Error::DataInvalid {
- message: format!(
- "merge_map_with_keytime timestamp for '{}' requires
STRING",
- self.field_name
- ),
- source: None,
- })?;
- Ok(Some(string.value(0)))
+ Ok(Some(column.as_ref()))
+ }
+
+ fn should_replace_by_timestamp(
+ &self,
+ new_ts: &dyn Array,
+ old_ts: &dyn Array,
+ ) -> crate::Result<bool> {
+ let kind = self.keytime_kind.as_ref().expect("timestamp mode");
+ let cmp = compare_keytime(kind, new_ts, old_ts, &self.field_name)?;
+ Ok(match cmp {
+ KeytimeCmp::Greater => true,
+ KeytimeCmp::Equal => self.overwrite_on_equal_ts,
Review Comment:
**[P1] Preserve the winning sequence for equal key-times across flushes**
With INT key-time, `sequence.field=seq`, and `overwrite-on-equal-ts=true`, I
wrote the same three rows for one PK and map key in both cases. The `(seq,
payload, key-time)` values were `(1,11,5)`, `(3,33,0)`, and `(2,22,5)`. A
single flush returns payload `22`. Flushing the first two together and the last
separately returns `11`. The first file retains payload `11` with the group's
sequence `3`, so it wins an equal-time comparison against sequence `2`.
Preserve the winning entry's sequence across partial reductions or reject this
option combination until it can be represented.
##########
crates/paimon/src/spec/aggregation.rs:
##########
@@ -298,11 +304,12 @@ fn parse_field_scoped_option_key(key: &str) ->
Option<(&str, FieldScopedOptionKi
/// `SchemaManager.applyRenameColumnsToOptions` (Java's case-2 list; the
/// aggregation/partial-update engines do not accept every entry at create
/// time, but the keys can still arrive in Java-written metadata).
-const FIELD_SCOPED_RENAMEABLE_SUFFIXES: [&str; 7] = [
+const FIELD_SCOPED_RENAMEABLE_SUFFIXES: [&str; 8] = [
AGG_FUNCTION_SUFFIX,
IGNORE_RETRACT_SUFFIX,
DISTINCT_SUFFIX,
LIST_AGG_DELIMITER_SUFFIX,
+ OVERWRITE_ON_EQUAL_TS_SUFFIX,
Review Comment:
**[P2] Migrate the explicit time-field selector with the map column**
The new equal-time option follows the map through rename, but
`fields.<column>.ts-field` still stays under the old name. I created a table
with a non-last INT field `payload` as the explicit key-time, wrote
`(payload=10,ts=1)`, renamed the map through the filesystem catalog, then
appended `(payload=1,ts=2)`. ALTER and both writes succeed, and the scan
changes the winner to `1` because the renamed field falls back to the trailing
`ts`. Rename and clean up `.ts-field` alongside `.overwrite-on-equal-ts`, and
cover a non-default selector in the catalog round trip.
##########
docs/src/sql.md:
##########
@@ -3554,12 +3554,31 @@ Field-scoped options used by these functions:
reject integer overflow instead of wrapping. Floating-point addition is
unaffected; Decimal overflow still yields NULL as in Java.
- `fields.<field>.ts-field` selects the key-time field for
- `merge_map_with_keytime` (defaults to the last ROW field).
+ `merge_map_with_keytime` (defaults to the last ROW field). Supported
+ key-time types are `CHAR`/`VARCHAR`, `TINYINT`/`SMALLINT`/`INT`/`BIGINT`,
+ `FLOAT`/`DOUBLE`, `DATE`, `TIME`, `DECIMAL`, `TIMESTAMP`, and
+ `TIMESTAMP_LTZ`.
+- `fields.<field>.overwrite-on-equal-ts` (default `false`) for
+ `merge_map_with_keytime`: when `true`, an equal key-time replaces the
+ existing ROW; when `false`, the existing ROW is kept.
- `fields.<field>.nested-key`, `fields.<field>.nested-sequence-field`,
`fields.<field>.nested-key-null-strategy`, and `fields.<field>.count-limit`
for nested array aggregators. `nested_partial_update` requires
`nested-key`.
+For `merge_map_with_keytime`, a newer key-time replaces the existing ROW for
+that map key. `CHAR`/`VARCHAR` key-times use unsigned UTF-8 byte
+lexicographical order (Java `BinaryString.compareTo`). `FLOAT`/`DOUBLE`
+compare as ordinary numbers with `+0.0` equal to `-0.0`; NaN key-times are
+rejected and never enter aggregation. `TIMESTAMP` / `TIMESTAMP_LTZ` compare
+by ticks under a unified time semantics (timezone metadata does not affect
+ordering). A NULL key-time ignores the incoming update and leaves any
+existing key unchanged. A NULL ROW value removes that key. Rust Native
+supports the key-time types listed above, including `FLOAT`/`DOUBLE`. The
+Python Native eligibility gate allows the same Rust-supported set (and
+falls back only for types Rust cannot compare). Java still rejects
+`FLOAT`/`DOUBLE` key-times.
Review Comment:
**[P2] Document the actual Java key-time type gate**
The Java caveat needs to include DECIMAL. I ran the production Java factory
in Paimon 2.2 snapshot `20261118.220748-71`. It accepts INT, TIME(3), and
TIMESTAMP(9), and rejects DECIMAL(10,2), FLOAT, and DOUBLE. I also wrote,
scanned, and compacted a DECIMAL table using this Rust head successfully.
Readers of mixed-engine tables need the version-specific Java type restriction
before enabling the new types.
##########
docs/src/sql.md:
##########
@@ -3554,12 +3554,31 @@ Field-scoped options used by these functions:
reject integer overflow instead of wrapping. Floating-point addition is
unaffected; Decimal overflow still yields NULL as in Java.
- `fields.<field>.ts-field` selects the key-time field for
- `merge_map_with_keytime` (defaults to the last ROW field).
+ `merge_map_with_keytime` (defaults to the last ROW field). Supported
+ key-time types are `CHAR`/`VARCHAR`, `TINYINT`/`SMALLINT`/`INT`/`BIGINT`,
+ `FLOAT`/`DOUBLE`, `DATE`, `TIME`, `DECIMAL`, `TIMESTAMP`, and
+ `TIMESTAMP_LTZ`.
+- `fields.<field>.overwrite-on-equal-ts` (default `false`) for
+ `merge_map_with_keytime`: when `true`, an equal key-time replaces the
+ existing ROW; when `false`, the existing ROW is kept.
- `fields.<field>.nested-key`, `fields.<field>.nested-sequence-field`,
`fields.<field>.nested-key-null-strategy`, and `fields.<field>.count-limit`
for nested array aggregators. `nested_partial_update` requires
`nested-key`.
+For `merge_map_with_keytime`, a newer key-time replaces the existing ROW for
+that map key. `CHAR`/`VARCHAR` key-times use unsigned UTF-8 byte
+lexicographical order (Java `BinaryString.compareTo`). `FLOAT`/`DOUBLE`
+compare as ordinary numbers with `+0.0` equal to `-0.0`; NaN key-times are
+rejected and never enter aggregation. `TIMESTAMP` / `TIMESTAMP_LTZ` compare
+by ticks under a unified time semantics (timezone metadata does not affect
+ordering). A NULL key-time ignores the incoming update and leaves any
+existing key unchanged. A NULL ROW value removes that key. Rust Native
Review Comment:
**[P1] Preserve a key tombstone across partial aggregation**
I used a newly supported INT key-time and wrote three versions of the same
key: payload `11` at time `9`, payload `22` at time `1`, then a NULL ROW for
that map key. Flushing all three together leaves the map empty. Flushing the
old version separately and grouping the lower-time update with the tombstone
leaves payload `11` readable. The second partial aggregate persists an empty
map and loses the deletion marker, so it cannot remove the older key. Preserve
that deletion information across materialization or restrict this newly
supported path to a safe merge.
##########
crates/paimon/src/table/aggregator/merge_map.rs:
##########
@@ -72,41 +101,15 @@ impl MergeMapAgg {
data_type: &DataType,
options: &HashMap<String, String>,
) -> crate::Result<Self> {
- let DataType::Map(map_type) = data_type else {
- return Err(unsupported_type_error(
- "merge_map_with_keytime",
- field_name,
- data_type,
- ));
- };
- let DataType::Row(value_type) = map_type.value_type() else {
- return Err(unsupported_type_error(
- "merge_map_with_keytime",
- field_name,
- data_type,
- ));
- };
- if value_type.fields().len() < 2 {
- return Err(unsupported_type_error(
- "merge_map_with_keytime",
- field_name,
- data_type,
- ));
- }
- let timestamp_index = match
options.get(&format!("fields.{field_name}.ts-field")) {
- Some(name) => value_type
- .fields()
- .iter()
- .position(|field| field.name() == name)
- .ok_or_else(|| Error::ConfigInvalid {
- message: format!(
- "Timestamp field '{name}' not found in ROW type for
field '{field_name}'"
- ),
- })?,
- None => value_type.fields().len() - 1,
- };
+ let (timestamp_index, keytime_kind) =
+ resolve_keytime_config(field_name, data_type, options)?;
+ let overwrite_on_equal_ts = options
+ .get(&format!("fields.{field_name}.overwrite-on-equal-ts"))
+ .is_some_and(|value| value.eq_ignore_ascii_case("true"));
Review Comment:
**[P2] Reject invalid values for the new boolean option**
Rust CREATE accepts `fields.updates.overwrite-on-equal-ts=treu` and silently
interprets it as false. Java Paimon 2.2 snapshot `20261118.220748-71` rejects
this same option when constructing the aggregator, while `true` and `false`
both work. An accepted schema with a typo can therefore select the wrong tie
winner in Rust and fail to initialize in Java. Parse and validate the boolean
in the shared schema/runtime configuration code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]