JingsongLi commented on code in PR #8334:
URL: https://github.com/apache/paimon/pull/8334#discussion_r3464539214
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/DataEvolutionMergeIntoAction.java:
##########
@@ -526,7 +671,17 @@ private void checkSchema(Table source) {
.getTypeRoot()
.getFamilies()
.contains(DataTypeFamily.BINARY_STRING);
+ // For sub-field-level data evolution, a struct column's
source value is a partial
+ // (subset) ROW carrying only the updated sub-fields, which is
not directly
+ // cast-compatible with the full target struct. Accept it when
every source
+ // sub-field exists in the target struct with a compatible
cast.
+ boolean partialStructCompatible =
Review Comment:
This relaxation should be scoped to sub-field writes. Today it also accepts
whole-column assignments, e.g. `--matched_update_set T.nest=S.nest` where the
source `S.nest` is `ROW<a>` and the target is `ROW<a,b>`.
`partialStructCompatible` returns true here, but `writePaths` is still just
`nest`, so `sourceType` is built as the full target struct and the partial
RowData is sent to a whole-struct write. That can fail at runtime or create an
incomplete whole-struct file. Please keep whole-struct assignments on the
normal full-type compatibility check, and only allow this subset check when the
column is actually being written through dotted paths such as `nest.a`.
##########
paimon-api/src/main/java/org/apache/paimon/types/RowType.java:
##########
@@ -312,6 +312,144 @@ public RowType project(String... names) {
return project(Arrays.asList(names));
}
+ /**
+ * Project this row type by a list of (possibly nested) dotted paths, e.g.
{@code ["f0",
+ * "nest.a"]}. A path without a dot selects the whole top-level field
(same as {@link
+ * #project(List)}); a dotted path selects only the addressed sub-field of
a nested {@link
+ * RowType}, preserving field ids and nullability of every level. Schema
field order is
+ * preserved. This is used by data evolution to reconstruct the partial
nested schema of a
+ * column-group file from its {@code writeCols}.
+ */
+ public RowType projectByPaths(List<String> paths) {
+ return projectTypeByPaths(this, paths);
+ }
+
+ private static RowType projectTypeByPaths(RowType type, List<String>
paths) {
+ // group paths by their immediate child name; a child appearing
without a tail (or also with
+ // a tail) is selected as a whole field
+ Map<String, List<String>> childToSubPaths = new HashMap<>();
+ Set<String> wholeChildren = new HashSet<>();
+ Set<String> fieldNames = new HashSet<>();
+ for (DataField field : type.getFields()) {
+ fieldNames.add(field.name());
+ }
+ for (String path : paths) {
+ int dot = path.indexOf('.');
+ // Prefer an exact field-name match so a column whose name itself
contains a dot (and
+ // any
+ // plain top-level name) is selected whole; only split into
head.tail for genuine nested
+ // sub-field paths that do not name a field directly. This keeps
backward compatibility
+ // with the legacy exact-name project(List).
+ if (dot < 0 || fieldNames.contains(path)) {
+ childToSubPaths.computeIfAbsent(path, k -> new ArrayList<>());
+ wholeChildren.add(path);
+ } else {
+ String head = path.substring(0, dot);
+ String tail = path.substring(dot + 1);
+ childToSubPaths.computeIfAbsent(head, k -> new
ArrayList<>()).add(tail);
+ }
+ }
+
+ Set<String> matched = new HashSet<>();
+ List<DataField> result = new ArrayList<>();
+ for (DataField field : type.getFields()) {
+ List<String> subPaths = childToSubPaths.get(field.name());
+ if (subPaths == null) {
+ continue;
+ }
+ matched.add(field.name());
+ if (wholeChildren.contains(field.name())
+ || subPaths.isEmpty()
+ || !(field.type() instanceof RowType)) {
Review Comment:
Could we reject dotted paths when the selected head is not a ROW? With the
current branch, `projectByPaths(Collections.singletonList("id.a"))` falls into
this arm and returns the whole `id` field. That makes invalid dotted
`writeCols` look valid to callers such as the conflict checker, and in the
Flink action an invalid SET target under a scalar can pass path resolution
before failing later with a less helpful error. Since dotted paths now encode
physical sub-fields, this should throw unless the head field is a ROW, or the
whole path matched an exact top-level field name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
