[
https://issues.apache.org/jira/browse/SPARK-34804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17305065#comment-17305065
]
Apache Spark commented on SPARK-34804:
--------------------------------------
User 'sumeetmi2' has created a pull request for this issue:
https://github.com/apache/spark/pull/31903
> registerFunction shouldnt be logging warning message for same function being
> re-registered
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-34804
> URL: https://issues.apache.org/jira/browse/SPARK-34804
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.1
> Reporter: Sumeet Sharma
> Priority: Minor
>
> {code:java}
> test("function registry warning") {
> implicit val ss = spark
> import ss.implicits._
> val dd = Seq((1),(2)).toDF("a")
> val dd1 = udf((i: Int) => i*2)
> (1 to 4).foreach {
> _ =>
> dd.sparkSession.udf.register("function", dd1)
> Thread.sleep(1000)
> }
> dd.withColumn("aa", expr("function(a)")).show(10)
> }{code}
> logs:
> 21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession;
> some spark core configurations may not take effect.
> 21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession;
> the static sql configurations will not take effect.
> 21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing
> SparkSession; some spark core configurations may not take effect.
> 21/03/19 22:39:43 WARN SimpleFunctionRegistry : The function function
> replaced a previously registered function.
> 21/03/19 22:39:44 WARN SimpleFunctionRegistry : The function function
> replaced a previously registered function.
> 21/03/19 22:39:45 WARN SimpleFunctionRegistry : The function function
> replaced a previously registered function.
> +----+--+
> |a|aa|
> +----+--+
> |1|2|
> |2|4|
> +----+--+
> Basically in the FunctionRegistry implementation
> {code:java}
> override def registerFunction(
> name: FunctionIdentifier,
> info: ExpressionInfo,
> builder: FunctionBuilder): Unit = synchronized {
> val normalizedName = normalizeFuncName(name)
> val newFunction = (info, builder)
> functionBuilders.put(normalizedName, newFunction) match {
> case Some(previousFunction) if previousFunction != newFunction =>
> logWarning(s"The function $normalizedName replaced a previously registered
> function.")
> case _ =>
> }
> }{code}
> The *previousFunction != newFunction* equality comparison is incorrect
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
