The version our job originally ran was spark 3.3.1 and Apache Iceberg to
1.2.0， But since we upgraded to spark3.4.1 and Apache Iceberg to 1.3.1,
jobs started to fail frequently， We tried to upgrade only iceberg without
upgrading spark, and the job did not report an error.
Detailed description：
When we execute this function writing data to the iceberg table：
> def appendToIcebergTable(targetTable: String, df: DataFrame): Unit = {
> _logger.warn(s"Append data to $targetTable")
>
> val (targetCols, sourceCols) = matchDFSchemaWithTargetTable(targetTable, df)
> df.createOrReplaceTempView("_temp")
> spark.sql(s"""
> INSERT INTO $targetTable ($targetCols) SELECT $sourceCols FROM _temp
> """)
> _logger.warn(s"Done append data to $targetTable")
> getIcebergLastAppendCountVerbose(targetTable)
> }
>
>
The error is reported as follows:
Caused by: java.lang.AssertionError: assertion failed at
> scala.Predef$.assert(Predef.scala:208) at
> org.apache.spark.sql.execution.ColumnarToRowExec.<init>(Columnar.scala:72) ...
> 191 more
Read the source code and find that the error is reported here：
case class ColumnarToRowExec(child: SparkPlan) extends
ColumnarToRowTransition with CodegenSupport {
> // supportsColumnar requires to be only called on driver side, see also
> SPARK-37779.
> *assert(Utils.isInRunningSparkTask || child.supportsColumnar)*
>
> override def output: Seq[Attribute] = child.output
>
> override def outputPartitioning: Partitioning = child.outputPartitioning
>
> override def outputOrdering: Seq[SortOrder] = child.outputOrdering
>
>
But we can't find the root cause，So seek help from the community ，If more
log information is required, please let me know.
thanks
