danhuawang opened a new issue, #11534: URL: https://github.com/apache/gravitino/issues/11534
### Version main branch ### Describe what's wrong When using the Gravitino Spark connector with a Glue catalog, `ALTER TABLE ADD COLUMNS` on an Iceberg table succeeds on the server side, but the subsequent `SELECT` that references the new column fails with `IllegalStateException: Couldn't find <column> in [<old columns>]`. The root cause is that `GravitinoGlueCatalog` maintains two separate catalogs: - `sparkCatalog` → `HiveTableCatalog` (for non-Iceberg tables) - `icebergGlueCatalog` → Iceberg `SparkCatalog` (for Iceberg tables, lazily created) `BaseCatalog.alterTable` only calls `sparkCatalog.invalidateTable(ident)`, which invalidates the HiveTableCatalog cache. The Iceberg `SparkCatalog`'s internal `CachingCatalog` (enabled by default) is never invalidated. On the next `loadSparkTable`, the Iceberg catalog returns a stale table with the pre-ALTER schema. `SparkIcebergTable.schema()` then reports the updated schema (from the fresh Gravitino table), but `newScanBuilder()` delegates to the stale Iceberg `SparkTable` with the old schema, causing Spark's `BindReferences` to fail. The same issue affects `dropTable`, `purgeTable`, `renameTable`, and `invalidateTable` for Iceberg tables in the Glue catalog. ### Error message and/or stacktrace ``` java.lang.IllegalStateException: Couldn't find age#240 in [id#238,name#239] at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:80) at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:73) at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:461) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:461) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:437) at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:405) at org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReference(BoundAttribute.scala:73) at org.apache.spark.sql.catalyst.expressions.BindReferences$.$anonfun$bindReferences$1(BoundAttribute.scala:94) at scala.collection.immutable.List.map(List.scala:297) at org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReferences(BoundAttribute.scala:94) at org.apache.spark.sql.execution.ProjectExec.doConsume(basicPhysicalOperators.scala:69) ``` ### How to reproduce 1. Create a Gravitino Glue catalog with Iceberg support via the Spark connector 2. Create an Iceberg table: `CREATE TABLE catalog.db.t (id INT, name STRING) USING iceberg` 3. Insert a row: `INSERT INTO catalog.db.t VALUES (1, 'Alice')` 4. Add a column: `ALTER TABLE catalog.db.t ADD COLUMNS (age INT)` 5. Query the new column: `SELECT id, name, age FROM catalog.db.t` 6. Step 5 fails with `IllegalStateException: Couldn't find age#<N> in [id#<N>,name#<N>]` This reproduces on the main branch with Spark 3.5 and Iceberg 1.11.0. ### Additional context The regular `GravitinoIcebergCatalog` does not have this bug because its `sparkCatalog` IS the Iceberg `SparkCatalog`, so `BaseCatalog.alterTable` → `sparkCatalog.invalidateTable(ident)` correctly evicts the cache. The Glue catalog is unique in that it routes Iceberg loads through a second, separate catalog instance that is never invalidated. Suggested fix: Override `invalidateTable` in `GravitinoGlueCatalog` to also call `icebergGlueCatalog.invalidateTable(ident)` when the table is an Iceberg table (or unconditionally, since invalidation on a non-existent entry is a no-op). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
