[
https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maryann Xue updated SPARK-25690:
--------------------------------
Description:
This was fixed in SPARK-24891 and was then broken by SPARK-25044.
The unit test in {{AnalysisSuite}} added in SPARK-24891 should have failed but
didn't because it wasn't properly updated after the {{ScalaUDF}} constructor
signature change. In the meantime, the other two end-to-end tests added in
SPARK-24891 were shadowed by SPARK-24865.
So the unit test mentioned above, if updated properly, can reproduce this issue:
{code:java}
test("SPARK-24891 Fix HandleNullInputsForUDF rule") {
val a = testRelation.output(0)
val func = (x: Int, y: Int) => x + y
val udf1 = ScalaUDF(func, IntegerType, a :: a :: Nil, nullableTypes = false
:: false :: Nil)
val udf2 = ScalaUDF(func, IntegerType, a :: udf1 :: Nil, nullableTypes =
false :: false :: Nil)
val plan = Project(Alias(udf2, "")() :: Nil, testRelation)
comparePlans(plan.analyze, plan.analyze.analyze)
}{code}
was:
This was fixed in SPARK-24891 and was then broken by SPARK-25044.
The tests added in SPARK-24891 were not good enough and the expected failures
were shadowed by SPARK-24865. For more details, please refer to SPARK-25650.
Code changes and tests in
[https://github.com/apache/spark/pull/22060/files#diff-f70523b948b7af21abddfa3ab7e1d7d6R72]
can help reproduce the issue.
> Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied
> infinitely
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-25690
> URL: https://issues.apache.org/jira/browse/SPARK-25690
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
> Affects Versions: 2.4.0
> Reporter: Maryann Xue
> Priority: Major
>
> This was fixed in SPARK-24891 and was then broken by SPARK-25044.
> The unit test in {{AnalysisSuite}} added in SPARK-24891 should have failed
> but didn't because it wasn't properly updated after the {{ScalaUDF}}
> constructor signature change. In the meantime, the other two end-to-end tests
> added in SPARK-24891 were shadowed by SPARK-24865.
> So the unit test mentioned above, if updated properly, can reproduce this
> issue:
> {code:java}
> test("SPARK-24891 Fix HandleNullInputsForUDF rule") {
> val a = testRelation.output(0)
> val func = (x: Int, y: Int) => x + y
> val udf1 = ScalaUDF(func, IntegerType, a :: a :: Nil, nullableTypes = false
> :: false :: Nil)
> val udf2 = ScalaUDF(func, IntegerType, a :: udf1 :: Nil, nullableTypes =
> false :: false :: Nil)
> val plan = Project(Alias(udf2, "")() :: Nil, testRelation)
> comparePlans(plan.analyze, plan.analyze.analyze)
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
