[
https://issues.apache.org/jira/browse/HUDI-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17408707#comment-17408707
]
ASF GitHub Bot commented on HUDI-1869:
--------------------------------------
melin removed a comment on pull request #3580:
URL: https://github.com/apache/hudi/pull/3580#issuecomment-911319873
spark 3.1.2
```
spark-sql> select * from test_hudi_table;
162943 [Executor task launch worker for task 0.0 in stage 32.0 (TID 1812)]
ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 32.0
(TID 1812)
java.lang.ClassCastException: scala.Some cannot be cast to
org.apache.spark.sql.catalyst.InternalRow
at
org.apache.spark.sql.hudi.HoodieMergeOnReadRDD$$anon$3.hasNext(HoodieMergeOnReadRDD.scala:240)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
at
org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:345)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
Do you test use this PR? I think this PR has solved this exception by
introduce the `HooodieAvroDeserializer` to match the `Option` result return by
spark 3.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Upgrading Spark3 To 3.1
> -----------------------
>
> Key: HUDI-1869
> URL: https://issues.apache.org/jira/browse/HUDI-1869
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Spark Integration
> Reporter: pengzhiwei
> Assignee: pengzhiwei
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> Spark 3.1 has changed some behavior of the internal class and interface for
> both spark-sql and spark-core module.
> Currently hudi can't compile success under the spark 3.1. We need support sql
> support for spark 3.1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
