JingsongLi commented on code in PR #1042:
URL: https://github.com/apache/paimon-rust/pull/1042#discussion_r4386203017


##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -2459,6 +2520,42 @@ fn resolve_field_ids(file: &DataFileMeta, fields: 
&[DataField]) -> crate::Result
     }
 }
 
+/// The key of the vector bunch a vector file joins: its schema, format and
+/// normalized write columns.
+fn vector_bunch_key(
+    file: &DataFileMeta,
+    normalized_write_cols: &[String],
+) -> (i64, String, Vec<String>) {
+    (
+        file.schema_id,
+        vector_format_suffix(&file.file_name),
+        normalized_write_cols.to_vec(),
+    )
+}
+
+/// Whether some VECTOR field is written by files of two vector bunch keys,
+/// such as a rewrite after `ALTER TABLE`; the source plan cannot combine
+/// their bunches.
+fn vector_field_has_two_layouts(files: &[DataFileMeta], infos: 
&[ResolvedFileInfo]) -> bool {
+    let mut layouts = HashMap::new();
+    for (file, info) in files.iter().zip(infos) {
+        let Some(columns) = info
+            .normalized_write_cols
+            .as_deref()
+            .filter(|_| is_vector_store_file_name(&file.file_name))

Review Comment:
   [P1] Include inline VECTOR providers in the layout fallback
   
   This filter ignores normal Parquet files that actually carry an inline 
VECTOR column. Removing vector.file.format is accepted: after an initial 
dedicated-vector write, a subsequent embedding rewrite produces a newer normal 
partial-column file covering the same row IDs. With just the old 
.vector.parquet counted here, the combined group is approved for merging, but 
build_source_plan prefers the dedicated provider and silently drops the new 
inline embedding. I reproduced this through public write, persisted 
RemoveOption, rewrite, combined Delta scan and projected read APIs: the new 
[9,9] update disappears on this head, while the unchanged test retains it on 
exact base ecbf5806. The ordinary snapshot-reader limitation pre-exists; this 
PR newly exposes it to to_incremental_arrow by routing that API through the 
evolution reader. Include actual inline VECTOR providers when deciding whether 
layouts can merge, and use the existing file-by-file fallback for mixed 
inline/dedicated groups.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to