szehon-ho commented on PR #14984: URL: https://github.com/apache/iceberg/pull/14984#issuecomment-5199012029
Filed and fixed the Spark-side bug I mentioned in the dependency question in my review: [SPARK-58604](https://issues.apache.org/jira/browse/SPARK-58604) / apache/spark#57802. `CatalogV2Util.viewInfoBuilderFrom` copies every `View` field except the typed `viewDependencies`, so all three metadata-only ALTER VIEW execs (`SET`/`UNSET TBLPROPERTIES`, `WITH SCHEMA ...`) hand `replaceView` a payload with the dependency list cleared. Since dependencies are a first-class field rather than an encoded property, the catalog has no way to recover them. This PR isn't exposed to it, because the `IcebergAlterV2View*Exec` nodes commit through `updateProperties()` instead of Spark's rebuild path. It's relevant in two ways though: it matters if the open `spark.view-dependencies` question resolves toward persisting them, and it's one more reason the rebuild path isn't a safe thing to delegate to yet. Still to file upstream, and I'll link them here as they go up: - View DDL in `ResolveSessionCatalog` is routed by catalog name rather than by capability, so a `RelationCatalog` installed as `spark_catalog` never reaches the v2 view execs. This is what keeps `RewriteViewCommands` and `CheckViews` necessary. Same change should cover `ShowViewsExec` not being able to merge in temp views. - `ViewCatalog` has no partial-update operation, so every property change has to go through a full `replaceView`. That one needs a design discussion rather than a patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
