lxy-9602 commented on code in PR #268: URL: https://github.com/apache/paimon-cpp/pull/268#discussion_r3909816377
########## docs/source/user_guide/data_types.rst: ########## @@ -197,9 +197,14 @@ and `Arrow DataTypes <https://arrow.apache.org/docs/format/Columnar.html#data-ty Paimon C++ currently supports VECTOR columns only in append-only tables backed by Parquet data files. They use the standard Parquet LIST representation on disk and are restored as Arrow ``FixedSizeList`` - values on read. Primary-key tables and data-evolution tables containing - VECTOR fields are rejected. VECTOR columns also cannot be partition or - bucket keys. Dedicated vector storage is not included yet. + values on read. Data-evolution tables may add or drop VECTOR columns and + continue reading files written with older table schemas. A VECTOR field's + element type and dimension cannot be changed through schema evolution; + an incompatible next schema is rejected before it is persisted. + Primary-key tables containing VECTOR fields are rejected. VECTOR columns + also cannot be partition or bucket keys, or comparator-based ordering + fields such as sequence and sequence-group fields. Dedicated vector + storage is not included yet. Review Comment: It seems there may be some confusion here between schema evolution and data evolution. Schema evolution refers to changes to the schema itself, such as adding, dropping, or renaming fields, as well as changing field types. Paimon has supported this for quite a long time. Data evolution, on the other hand, is a newer mode in Paimon. It allows newly added columns to be populated dynamically and efficiently without rewriting existing data. Since, in data evolution mode, different fields of the same row may be stored in different files, each row has a globally unique row ID so that fields from different files can be stitched together correctly at query time. The current PR seems to conflate these two features. Could you please confirm which functionality this PR is intended to support and adjust the implementation accordingly? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
