JingsongLi commented on code in PR #1009:
URL: https://github.com/apache/paimon-rust/pull/1009#discussion_r4173422997
##########
crates/paimon/src/table/kv_file_reader.rs:
##########
@@ -392,6 +394,33 @@ impl KeyValueFileReader {
.map(|s| s.as_str())
.collect();
+ // File readers run before PK deduplication. Keep Variant values in
their
+ // logical storage type until the visible row has been selected: a
strict
+ // extraction must not fail on an older, overwritten version.
+ let merge_read_type: Vec<DataField> = self
+ .config
+ .read_type
+ .iter()
+ .map(|field| {
+ if !is_variant_extraction_row_type(field.data_type()) {
+ return Ok(field.clone());
+ }
+ self.config
+ .table_fields
+ .iter()
+ .find(|source| source.id() == field.id())
+ .filter(|source| matches!(source.data_type(),
PaimonDataType::Variant(_)))
+ .cloned()
Review Comment:
[P2] Preserve the requested field name when restoring the Variant storage
type
Cloning the complete table field here replaces the name supplied through
`with_read_type`. That API can resolve historical fields by their stable field
IDs when reading older files, but `reorder_map` later still looks up the
original requested name in `value_fields` and calls `unwrap()`.
I reproduced this by writing two versions of the same primary key, renaming
`payload` to `renamed_payload`, then requesting a FLOAT32 Variant extraction
with the original name `payload` and the unchanged field ID through the Rust
`with_read_type` API. The splits require sort-merge. Base commit `69118de8`
returns the expected value `1.5`; this head panics at line 578 with
`Option::unwrap()` on `None`. This is specific to the Rust historical-name
read-type path; the Python builder uses canonical names.
Please retain the requested field's ID and name and replace only its data
type with the source Variant type, or resolve the output mapping by field ID.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
