Ivan Tsukanov created SPARK-28742:

             Summary: StackOverflowError when using otherwise(col()) in a loop
                 Key: SPARK-28742
                 URL: https://issues.apache.org/jira/browse/SPARK-28742
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.4.3, 2.4.0
            Reporter: Ivan Tsukanov

The following code

val rdd = sparkContext.makeRDD(Seq(Row("1")))
val schema = StructType(Seq(
  StructField("c1", StringType)

val df = sparkSession.createDataFrame(rdd, schema)
val column = when(col("c1").isin("1"), "1").otherwise(col("c1"))

(1 to 9).foldLeft(df) { case (acc, _) =>
  val res = acc.withColumn("c1", column)
falls with

   at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:395)

Probably, the problem is spark generates unexplainable big Physical Plan -




This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to