JingsongLi commented on code in PR #9532:
URL: https://github.com/apache/paimon/pull/9532#discussion_r3909659423
##########
paimon-common/src/main/java/org/apache/paimon/data/variant/InferVariantShreddingSchema.java:
##########
@@ -654,6 +658,37 @@ private DataType finalizeAdaptiveSchema(
return selectScalarType(combined, current, previousSelected);
}
+ /**
+ * Carries a previously selected schema forward for a node the current
file has no evidence for.
+ * The selection is already final, so nothing is re-thresholded, but its
nodes still consume the
+ * shared width budget and are dropped once it runs out, on the same terms
as the
+ * evidence-driven walk above: one unit per node, VARIANT once the budget
is gone.
+ */
+ private DataType retainSelectedSchema(DataType selected, MaxFields
maxFields) {
+ if (selected instanceof RowType) {
+ List<DataField> fields = new ArrayList<>();
+ for (DataField field : ((RowType) selected).getFields()) {
+ maxFields.remaining--;
+ if (maxFields.remaining <= 0) {
Review Comment:
[P2] Preserve the retained field/container at the last budget unit
This does not match the evidence-driven walk described above. When one unit
remains, that walk enters the child, decrements the unit to zero, returns
`VARIANT`, and still adds the parent `DataField`; this helper decrements first
and breaks before adding it. If it was the only field, line 678 then collapses
the whole retained `RowType` to the root `VARIANT`. The Array branch similarly
collapses the array instead of retaining `ARRAY<VARIANT>`.
I reproduced this with `maxSchemaWidth = 7` and two Variant columns. Round 1
is `(1, 5)`, round 2 is `(1, {q:1})`, and round 3 is `({x:1,y:1}, null)`.
Column `a` consumes five units, leaving exactly the root plus one field unit
for retained column `b`. The updated code returns an untyped root for `b` and
loses `q`; a focused assertion that `b` still contains `q` fails.
Please mirror `finalizeAdaptiveSchema` at exhaustion: retain the row
field/array container and downgrade the exhausted child to `VARIANT`, rather
than breaking/returning before preserving the parent shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
