yihua opened a new issue, #19738: URL: https://github.com/apache/hudi/issues/19738
Upgrading a table that is still at table version 1 leaves it with no ordering field recorded in `hoodie.properties`, even when the writer is configured with one. `hoodie.table.precombine.field` is only written at table creation, by `HoodieTableMetaClient.PropertyBuilder`, and only started being written in 0.8.0 (commit `0d8a4d0a56dc`, "Honor ordering field for MOR Spark datasource reader"). A table created before that has no ordering field in its table config, and none of the handlers in the version 1 to 6 upgrade chain adds one: | Handler | Table properties written | | --- | --- | | `OneToTwoUpgradeHandler` | `PARTITION_FIELDS`, `RECORDKEY_FIELDS`, `BASE_FILE_FORMAT` | | `TwoToThreeUpgradeHandler` | `URL_ENCODE_PARTITIONING`, `HIVE_STYLE_PARTITIONING_ENABLE`, `KEY_GENERATOR_CLASS_NAME` | | `ThreeToFourUpgradeHandler` | `DATABASE_NAME`, `TABLE_CHECKSUM`, `TABLE_METADATA_PARTITIONS` | | `FourToFiveUpgradeHandler` | none | | `FiveToSixUpgradeHandler` | none | `OneToTwoUpgradeHandler` is the handler that backfills the other key-schema properties from the write config, so the ordering field looks like an omission from that list. Nothing reports it afterwards. `HoodieWriterUtils.validateTableConfig` flags an ordering field only when the datasource value and the table config value are both non-null and differ, unlike the record key check just above it, which fails when the write config has a key and the table config does not. The write path itself is unaffected while a writer is configured, since `ConfigUtils.getOrderingField` prefers `hoodie.payload.ordering.field` and then `hoodie.datasource.write.precombine.field` before falling back to the table config. What breaks is everything that only has `hoodie.properties` to go on: - Copy-on-write updates: `HoodieMergeHandle` calls `DefaultHoodieRecordPayload.combineAndGetUpdateValue`, whose `needUpdatingPersistedRecord` gets a null ordering field and stops comparing, so an older record can overwrite a newer one and the wrong value is written into the base file. - Spark SQL DML: `ProvidesHoodieConfig` resolves the ordering field from `HoodieTableConfig.PRECOMBINE_FIELD` first, then the datasource key, then `""`, and a SQL session has no write config to fall back on. - Merge-on-read snapshot reads, which lose ordering in the log merge. - Flink, through `HoodieTableFactory` and `CompactionUtil.setPreCombineField`. On copy-on-write the failure is silent: there is no wrong query result to notice, only the wrong record winning an update. Note that this is not simply "table version 1 predates the property". 0.8.0 still stamped new tables as `HoodieTableVersion.ONE`, so version 1 straddles the change: a version 1 table created by 0.6.0 or 0.7.0 has no ordering field, one created by 0.8.0 does. The presence of the key has to be checked rather than inferred from the table version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
