[
https://issues.apache.org/jira/browse/HUDI-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shawn Chang updated HUDI-4467:
------------------------------
Description:
Currently some classes are copied from Spark32 module w/o/w only necessary
changes. we should port them from Spark 3.3 to use the latest implementation in
Spark
Classes copied:
Spark33NestedSchemaPruning
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
was:
Currently Spark33NestedSchemaPruning is copied from Spark32NestedSchemaPruning
with only necessary changes. The methods of this class are borrowed from Spark
and we should port this class from Spark 3.3 instead of copying.
> Port borrowed code from Spark 3.3
> ---------------------------------
>
> Key: HUDI-4467
> URL: https://issues.apache.org/jira/browse/HUDI-4467
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Shawn Chang
> Priority: Major
>
> Currently some classes are copied from Spark32 module w/o/w only necessary
> changes. we should port them from Spark 3.3 to use the latest implementation
> in Spark
>
> Classes copied:
> Spark33NestedSchemaPruning
> hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
> hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
> hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
