raviston opened a new pull request, #18989:
URL: https://github.com/apache/iceberg/pull/18989
Closes #17288
## Problem
With `spark_catalog` set to `SparkSessionCatalog` over a Hive catalog,
reading an Iceberg view that has column aliases fails:
```sql
CREATE VIEW v (c1, c2) AS VALUES (1, 2);
SELECT * FROM v;
-- [INCOMPATIBLE_VIEW_SCHEMA_CHANGE] The SQL query of view
`spark_catalog`.`default`.`v` has an incompatible
-- schema change and column c1 cannot be resolved. Expected 1 columns named
c1 but got [].
```
`WITH SCHEMA EVOLUTION` from the issue isn't required. The column aliases
are what trigger it.
The Hive catalog stores Iceberg views in the Hive Metastore as
`VIRTUAL_VIEW`s.
1. `SparkSessionCatalog.loadTable` fails to load the view as an Iceberg
table and falls back to the session catalog.
2. The session catalog returns the metastore entry as a Hive view
(`V1Table`).
3. Spark's `ResolveRelations` then resolves it as a v1 view before the
`ResolveViews` rule of the Iceberg extensions gets to it.
4. `SessionCatalog.fromCatalogTable` looks up each view column by name in
the output of the view's query. Without the `view.query.out.col.*` properties
that Spark writes for its own views, it looks for the view's column names
(`c1`). The query only outputs `col1` and `col2`, so the lookup fails.
Iceberg views are also missed by the rest of the Iceberg view handling in
this setup: `DESCRIBE`, `SHOW CREATE TABLE`, the schema binding mode, temp
object checks, and others.
## Fix
In `SparkSessionCatalog.loadTable`, the session catalog's result is not
returned when it is a view that the Iceberg view catalog knows (`viewExists`).
The original `NoSuchTableException` is rethrown instead, so `ResolveViews`
resolves the view as an Iceberg view, the same as for the other catalogs. The
extra `viewExists` call is only made when the session catalog returns a view.
Hive views that aren't Iceberg views are still resolved by Spark, because
`HiveCatalog.viewExists` is false for them.
## Testing
- New `TestSessionCatalogHiveViews` covers `SparkSessionCatalog` with
`type=hive`, which `TestViews` doesn't cover. It has three tests:
- column aliases over `VALUES`
- the exact SQL from the issue, with `WITH SCHEMA EVOLUTION`
- aliases with a column comment over a table
All three fail without the fix with the error above.
- Locally, I also ran `TestViews` with only the `SparkSessionCatalog` + Hive
configuration as its parameter. That run isn't part of this PR.
- Without the fix, 26 of its 86 tests fail.
- With the fix, 25 of those pass and no new failures appear.
- The remaining one, `renameViewToDifferentTargetCatalog`, fails with or
without the fix. `HiveCatalog.renameView` rejects the three-part target
identifier, which is a separate issue.
- On Spark 4.2, `TestSparkSessionCatalog`, `TestViews`,
`TestViewCatalogCache`, `TestReferencedByViewChain`, `TestTempViewRefresh` and
`TestSessionCatalogHiveViews` pass.
Spark 3.5, 4.0 and 4.1 have the same code. I'll backport in a follow-up once
this is reviewed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]