[
https://issues.apache.org/jira/browse/SPARK-52988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-52988:
----------------------------------
Parent: SPARK-51166
Issue Type: Sub-task (was: Bug)
> Race condition at the metastore function handling in SessionCatalog
> -------------------------------------------------------------------
>
> Key: SPARK-52988
> URL: https://issues.apache.org/jira/browse/SPARK-52988
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.1.0, 3.5.6, 4.0.0
> Reporter: Attila Zsolt Piros
> Assignee: Attila Zsolt Piros
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> Even when "CREATE FUNCTION IF NOT EXISTS" is used in parallel a run can fail
> with the following exception:
> {noformat}
> 2025-07-25 01:22:21,731 [AA-Rule-ThreadPoolExec-2] ERROR ***** - An error
> occured :
> org.apache.spark.sql.AnalysisException: Function default.SparkTestUDF already
> exists; line 1 pos 6734
> at
> org.apache.spark.sql.errors.QueryCompilationErrors$.functionAlreadyExistsError(QueryCompilationErrors.scala:654)
> at
> org.apache.spark.sql.catalyst.catalog.SessionCatalog.registerFunction(SessionCatalog.scala:1487)
> at
> org.apache.spark.sql.catalyst.catalog.SessionCatalog.resolvePersistentFunctionInternal(SessionCatalog.scala:1719)
> at
> org.apache.spark.sql.catalyst.catalog.SessionCatalog.resolvePersistentFunction(SessionCatalog.scala:1675)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveFunctions$$resolveV1Function(Analyzer.scala:2284)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$25$$anonfun$applyOrElse$103.$anonfun$applyOrElse$109(Analyzer.scala:2234)
> at scala.Option.getOrElse(Option.scala:189)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$25$$anonfun$applyOrElse$103.$anonfun$applyOrElse$108(Analyzer.scala:2231)
> at
> org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:60)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$25$$anonfun$applyOrElse$103.applyOrElse(Analyzer.scala:2231)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$25$$anonfun$applyOrElse$103.applyOrElse(Analyzer.scala:2205)
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
