nsivabalan commented on code in PR #19205:
URL: https://github.com/apache/hudi/pull/19205#discussion_r3654576590
##########
hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java:
##########
@@ -124,7 +125,11 @@ private Option<HoodieSchema>
getTableSchemaFromDataFileInternal() {
* @throws Exception
*/
public HoodieSchema getTableSchema() throws Exception {
- return getTableSchema(metaClient.getTableConfig().populateMetaFields());
+ // Include meta fields whenever the table's meta-fields mode populates any
of them. Under
+ // selective modes (COMMIT_TIME_ONLY / FILE_NAME_ONLY /
COMMIT_TIME_AND_FILE_NAME) the meta
+ // columns exist as physical nullable Parquet columns even though
populateMetaFields() is false,
+ // and read paths (e.g. incremental relations) must see them in the
projected schema.
+ return getTableSchema(metaClient.getTableConfig().getMetaFieldsMode() !=
MetaFieldsMode.NONE);
Review Comment:
Not compatibility — it reflects what is physically on disk.
Under a selective mode all five meta columns are still written to the
Parquet file; the ones the mode doesn't populate simply hold `null`.
`HoodieDatasetBulkInsertHelper` prepends nullable null stubs for every meta
field, and `HoodieRowCreateHandle` / the parquet writers then fill only the
opted-in ones. Parquet stores those nulls as definition-level flags, so the
cost is bookkeeping bits rather than data — that's what makes a selective mode
cheaper than `ALL` without needing a different column layout per mode.
Given the columns exist, the schema has to describe them. If
`getTableSchema()` omitted them for selective modes, any reader projecting
`_hoodie_commit_time` on a `COMMIT_TIME_ONLY` table would fail against a file
that actually has the column — and incremental queries depend on exactly that
projection. The predicate is `mode != NONE` rather than `populateMetaFields()`
for that reason: `populateMetaFields()` is false for every selective mode,
which would give the wrong answer here.
`NONE` is the genuinely different case: no meta columns are written, so none
appear in the schema — same as today's `populate.meta.fields=false`.
One consequence worth stating explicitly, since it's a behavior change for
existing tables: because the stubs are now nullable literals rather than empty
strings, meta columns on `populate.meta.fields=false` tables materialize as SQL
`NULL` instead of `""`. That's required for the columns to be `OPTIONAL` in
Parquet, but it is visible to anything filtering on `col = ''`. I called it out
in a separate comment on the PR so it gets a deliberate sign-off rather than
riding in unnoticed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
