zhuxiangyi commented on code in PR #8334:
URL: https://github.com/apache/paimon/pull/8334#discussion_r3467987768
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/DataEvolutionMergeIntoAction.java:
##########
@@ -526,7 +671,17 @@ private void checkSchema(Table source) {
.getTypeRoot()
.getFamilies()
.contains(DataTypeFamily.BINARY_STRING);
+ // For sub-field-level data evolution, a struct column's
source value is a partial
+ // (subset) ROW carrying only the updated sub-fields, which is
not directly
+ // cast-compatible with the full target struct. Accept it when
every source
+ // sub-field exists in the target struct with a compatible
cast.
+ boolean partialStructCompatible =
Review Comment:
Agreed — fixed in ef144accb. The partial-struct check is now gated on
`isSubFieldWrite(column)` (i.e. the column actually has dotted write paths like
`nest.a`). Whole-column assignments such as `T.nest=S.nest` stay on the
full-type compatibility check, so a narrower source struct is rejected instead
of being written as an incomplete whole-struct file.
##########
paimon-api/src/main/java/org/apache/paimon/types/RowType.java:
##########
@@ -312,6 +312,144 @@ public RowType project(String... names) {
return project(Arrays.asList(names));
}
+ /**
+ * Project this row type by a list of (possibly nested) dotted paths, e.g.
{@code ["f0",
+ * "nest.a"]}. A path without a dot selects the whole top-level field
(same as {@link
+ * #project(List)}); a dotted path selects only the addressed sub-field of
a nested {@link
+ * RowType}, preserving field ids and nullability of every level. Schema
field order is
+ * preserved. This is used by data evolution to reconstruct the partial
nested schema of a
+ * column-group file from its {@code writeCols}.
+ */
+ public RowType projectByPaths(List<String> paths) {
+ return projectTypeByPaths(this, paths);
+ }
+
+ private static RowType projectTypeByPaths(RowType type, List<String>
paths) {
+ // group paths by their immediate child name; a child appearing
without a tail (or also with
+ // a tail) is selected as a whole field
+ Map<String, List<String>> childToSubPaths = new HashMap<>();
+ Set<String> wholeChildren = new HashSet<>();
+ Set<String> fieldNames = new HashSet<>();
+ for (DataField field : type.getFields()) {
+ fieldNames.add(field.name());
+ }
+ for (String path : paths) {
+ int dot = path.indexOf('.');
+ // Prefer an exact field-name match so a column whose name itself
contains a dot (and
+ // any
+ // plain top-level name) is selected whole; only split into
head.tail for genuine nested
+ // sub-field paths that do not name a field directly. This keeps
backward compatibility
+ // with the legacy exact-name project(List).
+ if (dot < 0 || fieldNames.contains(path)) {
+ childToSubPaths.computeIfAbsent(path, k -> new ArrayList<>());
+ wholeChildren.add(path);
+ } else {
+ String head = path.substring(0, dot);
+ String tail = path.substring(dot + 1);
+ childToSubPaths.computeIfAbsent(head, k -> new
ArrayList<>()).add(tail);
+ }
+ }
+
+ Set<String> matched = new HashSet<>();
+ List<DataField> result = new ArrayList<>();
+ for (DataField field : type.getFields()) {
+ List<String> subPaths = childToSubPaths.get(field.name());
+ if (subPaths == null) {
+ continue;
+ }
+ matched.add(field.name());
+ if (wholeChildren.contains(field.name())
+ || subPaths.isEmpty()
+ || !(field.type() instanceof RowType)) {
Review Comment:
Good point — fixed in ef144accb. `projectByPaths` now throws
`IllegalArgumentException` when a dotted path's head field is not a ROW (e.g.
`id.a`), instead of silently returning the whole `id`. Exact top-level matches
(including column names that themselves contain a dot) are still selected
whole. Added coverage in `DataTypesTest#testProjectByPaths`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
