nsivabalan opened a new issue, #20371:
URL: https://github.com/apache/hudi/issues/20371
### Bug Description
`HoodieCatalog.alterTable(ident, changes*)` groups changes by class and
handles only `AddColumn`, `UpdateColumnType` and `UpdateColumnComment`:
1. Any `ColumnChange` subtype other than those two hits an inner `match`
with no default case and fails with `scala.MatchError`
(`HoodieCatalog.scala:238`): `DeleteColumn`, `RenameColumn`,
`UpdateColumnNullability`, `UpdateColumnPosition`, `UpdateColumnDefaultValue`.
A `MatchError` gives the caller no indication that the operation is unsupported.
2. `SetProperty` / `RemoveProperty` throw
`UnsupportedOperationException("not supported table change: …")` (`:255`) even
when the property value is unchanged. Spark's own `V2SessionCatalog` accepts
these; any DSv2 caller that syncs table properties through `alterTable` (Spark
4.1 Declarative Pipelines does this on **every** run for every table) fails on
the first re-run.
Additionally, `HoodieInternalV2Table.schema()`/`columns()` includes the five
`_hoodie_*` meta columns, so DSv2 callers that diff the user schema against
`Table.columns()` will issue `DeleteColumn` for them and hit (1).
Expected:
- A clear `UnsupportedOperationException` (or proper support) for unhandled
`ColumnChange`s instead of `MatchError`.
- `SetProperty`/`RemoveProperty` applied to the catalog table properties,
idempotently; reject only changes to Hudi table-identity properties
(`primaryKey`, `preCombineField`, `type`, key generator…) with an actionable
message.
- A decision on meta columns in `Table.columns()` (hide them for DSv2
callers, or treat `DeleteColumn` of `_hoodie_*` as a no-op).
Spark 4.1 also calls `createTable(Identifier, TableInfo)` → default
`createTable(Identifier, Column[], …)`, which `HoodieCatalog` does not override
(only the `StructType` overload is), so `DelegatingCatalogExtension` forwards
it to the session catalog and the table is registered with `provider=hudi` but
**no `hoodie.properties`**. It can be split into its own issue if preferred.
### Environment
Hudi 1.2.1 (`hudi-spark4.1-bundle_2.13`, Maven Central), Spark 4.1.1 (Spark
Connect server),
`spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog`,
`HoodieSparkSessionExtension`, local filesystem, macOS. Behaviour
cross-checked against current `master` source.
### Steps to reproduce
```scala
import org.apache.spark.sql.connector.catalog.{Identifier, TableChange}
spark.sql("CREATE TABLE default.t (id STRING, v INT) USING hudi
TBLPROPERTIES (primaryKey='id')")
val cat =
spark.sessionState.catalogManager.catalog("spark_catalog").asInstanceOf[org.apache.spark.sql.connector.catalog.TableCatalog]
val id = Identifier.of(Array("default"), "t")
cat.alterTable(id, TableChange.setProperty("comment", "x"))
// UnsupportedOperationException
cat.alterTable(id, TableChange.deleteColumn(Array("v"), false))
// scala.MatchError
cat.alterTable(id, TableChange.updateColumnNullability(Array("v"), true))
// scala.MatchError
```
### Logs and Stack Trace
```
java.lang.UnsupportedOperationException: not supported table change:
org.apache.spark.sql.connector.catalog.TableChange$SetProperty
at
org.apache.spark.sql.hudi.catalog.HoodieCatalog.$anonfun$alterTable$2(HoodieCatalog.scala:256)
scala.MatchError:
org.apache.spark.sql.connector.catalog.TableChange$DeleteColumn@1cdc90c (of
class org.apache.spark.sql.connector.catalog.TableChange$DeleteColumn)
at
org.apache.spark.sql.hudi.catalog.HoodieCatalog.$anonfun$alterTable$4(HoodieCatalog.scala:238)
at
org.apache.spark.sql.hudi.catalog.HoodieCatalog.alterTable(HoodieCatalog.scala:226)
```
Code pointer:
`hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala`
(`alterTable`, `createTable`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
