manuzhang opened a new issue, #18103: URL: https://github.com/apache/iceberg/issues/18103
### Apache Iceberg version main (development) ### Query engine Spark ### Please describe the bug 🐞 `TestViews.readFromViewReferencingTempFunction` failed on the Spark 4.2 extensions tests in a `main` push build: https://github.com/apache/iceberg/actions/runs/34809521107/job/103867881304 ``` TestViews > readFromViewReferencingTempFunction() > catalogName = spark_hive_with_views, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, default-namespace=default, cache-enabled=false} FAILED org.apache.spark.sql.catalyst.analysis.FunctionAlreadyExistsException: [ROUTINE_ALREADY_EXISTS] Cannot create the routine `test_avg919406` because a routine of that name already exists. Choose a different name, drop or replace the existing routine, or add the IF NOT EXISTS clause to tolerate a pre-existing routine. SQLSTATE: 42723 at app//org.apache.spark.sql.errors.QueryCompilationErrors$.functionAlreadyExistsError(QueryCompilationErrors.scala:1309) at app//org.apache.spark.sql.execution.command.CreateFunctionCommand.run(functions.scala:68) ... at app//org.apache.iceberg.spark.TestBase.sql(TestBase.java:131) at app//org.apache.iceberg.spark.extensions.TestViews.readFromViewReferencingTempFunction(TestViews.java:614) ``` #### Likely cause The temp function name is generated with `viewName("test_avg")`, which appends `new Random().nextInt(1000000)`. The temp function is never dropped. `TestViews` runs each test once per catalog parameter (`spark_with_views`, `spark_catalog`, `spark_hive_with_views`) against the same `SparkSession`. The temp functions created by the earlier parameter runs therefore stay registered. When the random suffix repeats, `CREATE TEMPORARY FUNCTION` fails. Spark 4.2's `CreateFunctionCommand` throws this error only when `SessionCatalog.isRegisteredFunction` finds the unqualified name in the session's temp function registry. It does not check the metastore, so the name must have been registered earlier in the same session. The collision is rare, which makes the test flaky. The same pattern (random suffix, no `DROP TEMPORARY FUNCTION`) exists in `readFromViewReferencingTempFunction`, `createViewReferencingTempFunction`, and `createViewReferencingQualifiedTempFunction`. It is present in the Spark 3.5, 4.0, 4.1 and 4.2 versions of `TestViews`. #### Possible fix - Drop the temp function in a `finally` block (`DROP TEMPORARY FUNCTION IF EXISTS ...`), as `createViewReferencingTemporaryVariable` already does for its temporary variable; and/or - generate collision-free names in `viewName()` (e.g. an incrementing counter or a UUID-based suffix) instead of `new Random().nextInt(1000000)`. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [x] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
