wombatu-kun opened a new issue, #20345:
URL: https://github.com/apache/hudi/issues/20345
### Bug Description
**What happened:**
Since 1.1.0, `HoodieTableMetaClient.TableBuilder` records the ordering
fields of a table only as `hoodie.table.ordering.fields`, at every table
version. Readers of a table below version 9 only know
`hoodie.table.precombine.field`: Hudi 0.x for version 6 and Hudi 1.0.x for
version 8. So a table that a 1.x writer creates with
`hoodie.write.table.version` 6 or 8 (Spark datasource or SQL, Hudi Streamer,
Flink, Kafka Connect) has no ordering field for those readers. The 9 to 8
downgrade writes `hoodie.table.precombine.field` for exactly that reason, but
table creation does not.
`ALTER TABLE ... RENAME TO` does the same to tables that carry only the
legacy key, such as tables written by 0.x. It re-creates `hoodie.properties`
from the table's own properties through `TableBuilder.fromProperties`, which
reads `hoodie.table.precombine.field` through the alternative key and writes it
back only as `hoodie.table.ordering.fields`.
The key generator class has the same problem below version 8 (#20275).
**What you expected:**
A table below version 9 records its ordering fields under
`hoodie.table.precombine.field` as well, as tables written by 0.x and 1.0.x do,
and keeps that key when it is re-created from its own properties.
**Steps to reproduce:**
1. Create a table below version 9, for example with Spark SQL: `create table
t (id int, name string, ts long) using hudi tblproperties (primaryKey = 'id',
orderingFields = 'ts', hoodie.write.table.version = '6')`.
2. `.hoodie/hoodie.properties` holds `hoodie.table.ordering.fields=ts` and
no `hoodie.table.precombine.field`.
3. Loading that file into the `HoodieTableConfig` of Hudi 0.15.0 gives
`getPreCombineField() = null`. For a version 8 table, Hudi 1.0.2 gives the same.
Reproducing tests in `TestHoodieTableMetaClient`, which fail on master for
versions 6 and 8:
```java
@ParameterizedTest
@EnumSource(value = HoodieTableVersion.class, names = {"SIX", "EIGHT",
"NINE", "TEN"})
void testOrderingFieldsKeysFollowTableVersion(HoodieTableVersion version)
throws IOException {
HoodieTableMetaClient created = orderingFieldsTableBuilder(version)
.setOrderingFields("ts")
.initTable(this.metaClient.getStorageConf(), tempDir.toAbsolutePath()
+ Path.SEPARATOR + "ordering-v" + version.versionCode());
assertOrderingFieldsKeys(version, created.getTableConfig());
}
@ParameterizedTest
@EnumSource(value = HoodieTableVersion.class, names = {"SIX", "EIGHT"})
void testReInitFromPropertiesKeepsLegacyOrderingFieldsKey(HoodieTableVersion
version) throws IOException {
String path = tempDir.toAbsolutePath() + Path.SEPARATOR +
"legacy-ordering-v" + version.versionCode();
HoodieTableMetaClient created =
orderingFieldsTableBuilder(version).initTable(this.metaClient.getStorageConf(),
path);
Properties legacyOrderingFields = new Properties();
legacyOrderingFields.setProperty(HoodieTableConfig.PRECOMBINE_FIELD.key(),
"ts");
HoodieTableConfig.update(created.getStorage(), created.getMetaPath(),
legacyOrderingFields);
created.reloadTableConfig();
assertFalse(created.getTableConfig().getProps().containsKey(HoodieTableConfig.ORDERING_FIELDS.key()));
HoodieTableMetaClient reInitialised =
HoodieTableMetaClient.newTableBuilder()
.fromProperties(created.getTableConfig().getProps())
.initTable(this.metaClient.getStorageConf(), path);
assertOrderingFieldsKeys(version, reInitialised.getTableConfig());
}
private static HoodieTableMetaClient.TableBuilder
orderingFieldsTableBuilder(HoodieTableVersion version) {
return HoodieTableMetaClient.newTableBuilder()
.setTableType(HoodieTableType.MERGE_ON_READ.name())
.setTableName("table")
.setTableVersion(version)
.setRecordKeyFields("id");
}
private static void assertOrderingFieldsKeys(HoodieTableVersion version,
HoodieTableConfig tableConfig) {
assertEquals("ts",
tableConfig.getProps().getProperty(HoodieTableConfig.ORDERING_FIELDS.key()));
assertEquals(version.lesserThan(HoodieTableVersion.NINE) ? "ts" : null,
tableConfig.getProps().getProperty(HoodieTableConfig.PRECOMBINE_FIELD.key()),
"precombine key at version " + version.versionCode());
}
```
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT), and every release since 1.1.0
**Query engine:** any writer that creates a table below version 9 (Spark,
Flink, Hudi Streamer, Kafka Connect)
**Relevant configs:** `hoodie.write.table.version` = 6 or 8 with ordering
fields set
### Logs and Stack Trace
The reproducing tests on master:
```
[ERROR] Run 1:
TestHoodieTableMetaClient.testOrderingFieldsKeysFollowTableVersion:527->assertOrderingFieldsKeys:562
precombine key at version 6 ==> expected: <ts> but was: <null>
[ERROR] Run 1:
TestHoodieTableMetaClient.testOrderingFieldsKeysFollowTableVersion:527->assertOrderingFieldsKeys:562
precombine key at version 8 ==> expected: <ts> but was: <null>
[ERROR] Run 1:
TestHoodieTableMetaClient.testReInitFromPropertiesKeepsLegacyOrderingFieldsKey:549->assertOrderingFieldsKeys:562
precombine key at version 6 ==> expected: <ts> but was: <null>
[ERROR] Run 1:
TestHoodieTableMetaClient.testReInitFromPropertiesKeepsLegacyOrderingFieldsKey:549->assertOrderingFieldsKeys:562
precombine key at version 8 ==> expected: <ts> but was: <null>
```
`hoodie.properties` written by the table builder on master, loaded into the
`HoodieTableConfig` of the released readers. `reinit-v6` is a table that
carried `hoodie.table.precombine.field=ts`, re-created from its own properties
the way `ALTER TABLE RENAME` does:
```
== Hudi 0.15.0 (table version 6 reader)
tables-master/created-v6/.hoodie/hoodie.properties ->
hoodie.table.version=6, getPreCombineField()=null
tables-master/reinit-v6/.hoodie/hoodie.properties -> hoodie.table.version=6,
getPreCombineField()=null
== Hudi 1.0.2 (table version 8 reader)
tables-master/created-v8/.hoodie/hoodie.properties ->
hoodie.table.version=8, getPreCombineField()=null
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
