voonhous opened a new issue, #19766:
URL: https://github.com/apache/hudi/issues/19766
## Bug Description
**What happened:**
On a schema-on-read table with at least one commit, `ALTER TABLE ... RENAME
COLUMN` and `ALTER TABLE ... DROP COLUMN` fail under the shipped defaults
(`hoodie.avro.schema.validate=false`,
`hoodie.datasource.write.schema.allow.auto.evolution.column.drop=false`):
```
org.apache.hudi.exception.MissingSchemaFieldException: Schema validation
failed due to missing field. Fields missing from incoming schema: {price}
```
Both statements succeed once
`hoodie.datasource.write.schema.allow.auto.evolution.column.drop=true` is set.
**To reproduce** (Spark SQL, COW or MOR):
```sql
set hoodie.schema.on.read.enable=true;
create table t (id int, name string, price double, ts long) using hudi
tblproperties (primaryKey = 'id', preCombineField = 'ts') location
'/tmp/t';
insert into t values (1, 'a1', 10.0, 1000);
alter table t rename column price to newprice; --
MissingSchemaFieldException
alter table t drop column price; --
MissingSchemaFieldException
```
**Cause:**
#13595 (`0fe119a0cf1e`, 2025-07-25) added a bare
`hoodieTable.validateSchema()` to `AlterTableCommand.commitWithSchema`
(`AlterTableCommand.scala:280`) so that columns backing a secondary index
cannot be evolved. `HoodieTable.validateSchema` only short-circuits when
`!shouldValidate && allowProjection`; with `allowProjection=false` it reaches
`HoodieSchemaCompatibility.checkSchemaCompatible`, whose first step is
`findMissingFields(tableSchema, writerSchema)` and which throws for any table
field absent from the new schema (`HoodieSchemaCompatibility.java:100-107`). A
rename or a drop always produces exactly that.
The same commit added `allow.auto.evolution.column.drop=true` to the four
pre-existing `TestSpark3DDL` rename/drop tests and to `TestTimeTravelTable`,
which is why CI did not flag the change in behaviour. Every rename/drop test in
the repo now carries that config; there is no test of the default path.
**Expected behavior:**
Schema-on-read rename/drop are explicit DDL, not an accidental writer-schema
projection, so they should not depend on `allow.auto.evolution.column.drop`.
The secondary-index guard from #13595 (`validateSecondaryIndexSchemaEvolution`)
is the part of `validateSchema` that should keep running on this path; the
missing-field check is the part that should not.
## Environment
Reproduced at `d291efccaad2`; the code path is unchanged on current master
(`18ae8c349058`). Spark 3.5 / Scala 2.12 profile, local filesystem. Found while
reviewing #19163, whose alter-table suite had to drop its rename/drop cases
because of this.
## Logs and Stack Trace
```
org.apache.hudi.exception.MissingSchemaFieldException: Schema validation
failed due to missing field. Fields missing from incoming schema: {price}
at
org.apache.hudi.common.schema.HoodieSchemaCompatibility.checkSchemaCompatible(HoodieSchemaCompatibility.java:104)
at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:936)
at
org.apache.spark.sql.hudi.command.AlterTableCommand$.commitWithSchema(AlterTableCommand.scala:280)
at
org.apache.spark.sql.hudi.command.AlterTableCommand.applyUpdateAction(AlterTableCommand.scala:180)
at
org.apache.spark.sql.hudi.command.AlterTableCommand.run(AlterTableCommand.scala:60)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
```
The `DROP COLUMN` stack is identical apart from `applyDeleteAction` in place
of `applyUpdateAction`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
