Indian-boult opened a new pull request, #20738:
URL: https://github.com/apache/hudi/pull/20738

   
   ### Describe the issue this Pull Request addresses
   
   Closes #19766
   
   On a schema-on-read table with at least one commit, `ALTER TABLE ... RENAME 
COLUMN` and `ALTER TABLE ... DROP COLUMN` on a top-level column fail under 
default configs with `MissingSchemaFieldException`. #13595 added 
`HoodieTable.validateSchema()` to `AlterTableCommand.commitWithSchema` for its 
secondary index check. `validateSchema` is the write path check: with 
`hoodie.datasource.write.schema.allow.auto.evolution.column.drop=false` (the 
default) it rejects any table column that is missing from the incoming schema, 
and a rename or a drop always produces one.
   
   Two more problems come from the same call:
   
   - It runs after `client.startCommit`, so every rejected ALTER leaves a 
REQUESTED instant on the timeline. Once a later ALTER commits after it, the 
next write fails while rolling it back: `HoodieRollbackException: Found commits 
after time :<instant>, please rollback greater commits first`, and leaves a 
rollback INFLIGHT behind. Reproduced on COW and MOR with a rejected rename, a 
successful `ADD COLUMNS`, then an `INSERT`.
   - With the column drop config set to `true` (and 
`hoodie.avro.schema.validate` at its default), `validateSchema` returns before 
any check, so the secondary index check of #13595 is skipped too: `ALTER COLUMN 
... TYPE` on a column with a secondary index goes through.
   
   ### Summary and Changelog
   
   ALTER TABLE is an explicit schema change, so it now runs only the secondary 
index check, before the commit starts. Renaming and dropping columns under 
default configs works again, a rejected ALTER leaves nothing on the timeline, 
and the secondary index check no longer depends on the column drop config.
   
   - `HoodieTable#validateAlterSchema`: new. Runs 
`validateSecondaryIndexSchemaEvolution` against the writer schema, with the 
same skip conditions as `validateSchema` (no index metadata, no completed 
commit, no writer schema), and does not read 
`hoodie.datasource.write.schema.allow.auto.evolution.column.drop` or 
`hoodie.avro.schema.validate`. `validateSchema` and the write path are 
unchanged.
   - `HoodieTable#validateSecondaryIndexSchemaEvolution`: new 
`allowMissingColumns` argument; the existing 3-argument form passes `true` and 
behaves as before. `validateAlterSchema` passes `false`, so renaming or 
dropping a column that has a secondary index is rejected with "Column 'q' has 
secondary index 'secondary_index_idx_q' and cannot be renamed or dropped". 
Under default configs that ALTER was already rejected on master (by the 
`MissingSchemaFieldException` above), so this keeps that outcome with an exact 
message. Without it, the fix would let the rename through and leave the index 
stale: on master with the column drop config on, a renamed indexed column gets 
null secondary keys for new rows and keeps the old keys on updates, and the 
index definition keeps the old name.
   - `AlterTableCommand.commitWithSchema`: calls `validateAlterSchema` before 
`client.startCommit` instead of `validateSchema` after it. It is the only ALTER 
path that called `validateSchema`: with schema-on-read, ADD COLUMNS, RENAME 
COLUMN, DROP COLUMN, REPLACE COLUMNS and ALTER COLUMN on Spark 3.x 
(`AlterColumn`) and 4.x (`AlterColumns`) all resolve to this command through 
`BaseResolveHudiAlterTableCommand`, and the schema-on-write ALTER commands 
never called it.
   - Tests:
     - `TestHoodieTableSchemaEvolution` (now on `HoodieCommonTestHarness`): 
`testValidateAlterSchemaOnlyChecksSecondaryIndexColumns` builds a table with a 
commit and a secondary index on `age`. A rename plus a drop of other columns 
fails `validateSchema` with `MissingSchemaFieldException` and passes 
`validateAlterSchema`. Retyping `age` with the column drop config on passes 
`validateSchema` and fails `validateAlterSchema` with the exact message. 
Renaming or dropping `age` fails with the exact message. 
`testIndexedColumnMissingFromWriterSchema` pins both values of 
`allowMissingColumns`.
     - `TestSpark3DDL`: removed `allow.auto.evolution.column.drop=true` from 
"Test Chinese table", "Test alter column by add rename and drop", "Test alter 
column multiple times" and "Test alter column with complex schema", so they run 
under defaults again. New "Test rename and drop column under default write 
configs" (COW and MOR): asserts the config is unset, the resulting columns and 
rows, one ALTER_SCHEMA commit per DDL, and no pending instant.
     - `TestSecondaryIndex` "Column Type Change Should Fail": the same ALTER is 
also rejected with the column drop config on; RENAME and DROP of the indexed 
column are rejected; no pending instant is left after the rejected ALTERs.
     - Removed the same workaround from `TestTimeTravelTable`, 
`TestHiveTableSchemaEvolution` (two tests), 
`TestNestedSchemaPruningOptimization` and `TestVariantShreddingMixedLayouts`, 
with the comments that explained it. `TestVariantShreddingMixedLayouts` is 
gated to Spark 4.1, so it compiled but did not run locally (Spark 3.5).
   
   Not changed here: Flink `ALTER TABLE` and the write client's column change 
API commit through `BaseHoodieWriteClient#commitTableChange`, which never 
called `validateSchema`, so they were not affected by this bug and do not run 
the secondary index check either.
   
   ### Impact
   
   `ALTER TABLE ... RENAME COLUMN` and `DROP COLUMN` work under default configs 
on schema-on-read tables, as before #13595. ALTER TABLE no longer depends on 
`hoodie.datasource.write.schema.allow.auto.evolution.column.drop` or 
`hoodie.avro.schema.validate`. One visible change for users who set 
`hoodie.avro.schema.validate=true`: ALTER TABLE no longer runs the Avro 
reader/writer compatibility check, which refused for example `ALTER COLUMN d 
TYPE string` on a `date` column, a change schema-on-read supports. Before 
#13595 ALTER TABLE did not run it either. `HoodieTable#validateAlterSchema` is 
a new public method.
   
   ### Risk Level
   
   low. The write path check is unchanged. On the ALTER path the only checks 
removed are the missing-field check and the Avro compatibility check described 
above; the secondary index check now also runs when the column drop config is 
on, and rejects renaming or dropping an indexed column.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] fix(spark): allow ALTER TA... Indian-boult (via GitHub) via commits

Reply via email to