[
https://issues.apache.org/jira/browse/SPARK-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lev Katzav updated SPARK-25368:
-------------------------------
Description:
there is a breaking change in spark 2.3 (I checked on 2.3.1 and 2.3.2-rc5)
the following code recreates the problem
(it's a bit convoluted examples, I tried to simplify it as much as possible
from my code)
{code:java}
import org.apache.spark.sql.{DataFrame, SQLContext}
import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._
import spark.implicits._
case class Data(a: Option[Int],b: String,c: Option[String],d: String)
val df1 = spark.createDataFrame(Seq(
Data(Some(1), "1", None, "1"),
Data(None, "2", Some("2"), "2")
))
val df2 = df1
.where( $"a".isNotNull)
.withColumn("e", lit(null).cast("string"))
val columns = df2.columns.map(c => col(c))
val df3 = df1
.select(
$"c",
$"b" as "e"
)
.withColumn("a", lit(null).cast("int"))
.withColumn("b", lit(null).cast("string"))
.withColumn("d", lit(null).cast("string"))
.select(columns :_*)
val df4 =
df2.union(df3)
.withColumn("e", last(col("e"), ignoreNulls =
true).over(Window.partitionBy($"c").orderBy($"d")))
.filter($"a".isNotNull)
df4.show
{code}
notice that the last statement in for df4 is to filter rows where a is null
in spark 2.2.1, the above code prints:
{code:java}
+---+---+----+---+---+
| a| b| c| d| e|
+---+---+----+---+---+
| 1| 1|null| 1| 1|
+---+---+----+---+---+
{code}
in spark 2.3.x, it prints:
{code:java}
+----+----+----+----+---+
| a| b| c| d| e|
+----+----+----+----+---+
|null|null|null|null| 1|
| 1| 1|null| 1| 1|
|null|null| 2|null| 2|
+----+----+----+----+---+
{code}
the column a still contains null values
{code:java}
== Parsed Logical Plan == 'Filter isnotnull('a) +- AnalysisBarrier +- Project
[a#0, b#1, c#2, d#3, e#104] +- Project [a#0, b#1, c#2, d#3, e#69, e#104, e#104]
+- Window [last(e#69, true) windowspecdefinition(c#2, d#3 ASC NULLS FIRST,
specifiedwindowframe(RangeFrame, unboundedpreceding$(), currentrow$())) AS
e#104], [c#2], [d#3 ASC NULLS FIRST] +- Project [a#0, b#1, c#2, d#3, e#69] +-
Union :- Project [a#0, b#1, c#2, d#3, cast(null as string) AS e#69] : +- Filter
isnotnull(a#0) : +- LocalRelation [a#0, b#1, c#2, d#3] +- Project [a#78, b#82,
c#2, d#87, e#75] +- Project [c#2, e#75, a#78, b#82, cast(null as string) AS
d#87] +- Project [c#2, e#75, a#78, cast(null as string) AS b#82] +- Project
[c#2, e#75, cast(null as int) AS a#78] +- Project [c#2, b#1 AS e#75] +-
LocalRelation [a#0, b#1, c#2, d#3]
{code}
was:
there is a breaking change in spark 2.3 (I checked on 2.3.1 and 2.3.2-rc5)
the following code recreates the problem
(it's a bit convoluted examples, I tried to simplify it as much as possible
from my code)
{code:java}
import org.apache.spark.sql.{DataFrame, SQLContext}
import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._
import spark.implicits._
case class Data(a: Option[Int],b: String,c: Option[String],d: String)
val df1 = spark.createDataFrame(Seq(
Data(Some(1), "1", None, "1"),
Data(None, "2", Some("2"), "2")
))
val df2 = df1
.where( $"a".isNotNull)
.withColumn("e", lit(null).cast("string"))
val columns = df2.columns.map(c => col(c))
val df3 = df1
.select(
$"c",
$"b" as "e"
)
.withColumn("a", lit(null).cast("int"))
.withColumn("b", lit(null).cast("string"))
.withColumn("d", lit(null).cast("string"))
.select(columns :_*)
val df4 =
df2.union(df3)
.withColumn("e", last(col("e"), ignoreNulls =
true).over(Window.partitionBy($"c").orderBy($"d")))
.filter($"a".isNotNull)
df4.show
{code}
notice that the last statement in for df4 is to filter rows where a is null
in spark 2.2.1, the above code prints:
{code:java}
+---+---+----+---+---+
| a| b| c| d| e|
+---+---+----+---+---+
| 1| 1|null| 1| 1|
+---+---+----+---+---+
{code}
in spark 2.3.x, it prints:
{code:java}
+----+----+----+----+---+
| a| b| c| d| e|
+----+----+----+----+---+
|null|null|null|null| 1|
| 1| 1|null| 1| 1|
|null|null| 2|null| 2|
+----+----+----+----+---+
{code}
the column a still contains null values
> Incorrect predicate pushdown returns in incorrect result
> --------------------------------------------------------
>
> Key: SPARK-25368
> URL: https://issues.apache.org/jira/browse/SPARK-25368
> Project: Spark
> Issue Type: Bug
> Components: Optimizer, SQL
> Affects Versions: 2.3.1, 2.3.2
> Reporter: Lev Katzav
> Priority: Blocker
>
> there is a breaking change in spark 2.3 (I checked on 2.3.1 and 2.3.2-rc5)
> the following code recreates the problem
> (it's a bit convoluted examples, I tried to simplify it as much as possible
> from my code)
> {code:java}
> import org.apache.spark.sql.{DataFrame, SQLContext}
> import org.apache.spark.sql.expressions.Window
> import org.apache.spark.sql.functions._
> import spark.implicits._
> case class Data(a: Option[Int],b: String,c: Option[String],d: String)
> val df1 = spark.createDataFrame(Seq(
> Data(Some(1), "1", None, "1"),
> Data(None, "2", Some("2"), "2")
> ))
> val df2 = df1
> .where( $"a".isNotNull)
> .withColumn("e", lit(null).cast("string"))
> val columns = df2.columns.map(c => col(c))
> val df3 = df1
> .select(
> $"c",
> $"b" as "e"
> )
> .withColumn("a", lit(null).cast("int"))
> .withColumn("b", lit(null).cast("string"))
> .withColumn("d", lit(null).cast("string"))
> .select(columns :_*)
> val df4 =
> df2.union(df3)
> .withColumn("e", last(col("e"), ignoreNulls =
> true).over(Window.partitionBy($"c").orderBy($"d")))
> .filter($"a".isNotNull)
> df4.show
> {code}
>
> notice that the last statement in for df4 is to filter rows where a is null
> in spark 2.2.1, the above code prints:
> {code:java}
> +---+---+----+---+---+
> | a| b| c| d| e|
> +---+---+----+---+---+
> | 1| 1|null| 1| 1|
> +---+---+----+---+---+
> {code}
> in spark 2.3.x, it prints:
> {code:java}
> +----+----+----+----+---+
> | a| b| c| d| e|
> +----+----+----+----+---+
> |null|null|null|null| 1|
> | 1| 1|null| 1| 1|
> |null|null| 2|null| 2|
> +----+----+----+----+---+
> {code}
> the column a still contains null values
> {code:java}
> == Parsed Logical Plan == 'Filter isnotnull('a) +- AnalysisBarrier +- Project
> [a#0, b#1, c#2, d#3, e#104] +- Project [a#0, b#1, c#2, d#3, e#69, e#104,
> e#104] +- Window [last(e#69, true) windowspecdefinition(c#2, d#3 ASC NULLS
> FIRST, specifiedwindowframe(RangeFrame, unboundedpreceding$(),
> currentrow$())) AS e#104], [c#2], [d#3 ASC NULLS FIRST] +- Project [a#0, b#1,
> c#2, d#3, e#69] +- Union :- Project [a#0, b#1, c#2, d#3, cast(null as string)
> AS e#69] : +- Filter isnotnull(a#0) : +- LocalRelation [a#0, b#1, c#2, d#3]
> +- Project [a#78, b#82, c#2, d#87, e#75] +- Project [c#2, e#75, a#78, b#82,
> cast(null as string) AS d#87] +- Project [c#2, e#75, a#78, cast(null as
> string) AS b#82] +- Project [c#2, e#75, cast(null as int) AS a#78] +- Project
> [c#2, b#1 AS e#75] +- LocalRelation [a#0, b#1, c#2, d#3]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
