wangzheyuan opened a new issue, #6635:
URL: https://github.com/apache/hudi/issues/6635
**Describe the problem you faced**
Failed to build hudi 0.12.0 with spark 3.2.2
**To Reproduce**
Steps to reproduce the behavior:
```shell
mvn clean package -DskipTests -Dscala-2.12 -Dspark3.2 -Dspark32.version=3.2.2
```
**Environment Description**
* Hudi version : 0.12.0
* Spark version : 3.2.2
**Stacktrace**
```
[INFO] --- scala-maven-plugin:3.3.1:compile (scala-compile-first) @
hudi-spark3.2.x_2.12 ---
[INFO]
/root/hudi-release-0.12.0/hudi-spark-datasource/hudi-spark3.2.x/src/main/java:-1:
info: compiling
[INFO]
/root/hudi-release-0.12.0/hudi-spark-datasource/hudi-spark3.2.x/target/generated-sources/antlr4:-1:
info: compiling
[INFO]
/root/hudi-release-0.12.0/hudi-spark-datasource/hudi-spark3.2.x/src/main/scala:-1:
info: compiling
[INFO] Compiling 32 source files to
/root/hudi-release-0.12.0/hudi-spark-datasource/hudi-spark3.2.x/target/classes
at 1662601520623
[WARNING]
/root/hudi-release-0.12.0/hudi-spark-datasource/hudi-spark3.2.x/src/main/scala/org/apache/hudi/Spark32HoodieFileScanRDD.scala:21:
warning: imported `HoodieUnsafeRDD' is permanently hidden by definition of
trait HoodieUnsafeRDD in package hudi
[WARNING] import org.apache.hudi.HoodieUnsafeRDD
[WARNING] ^
[ERROR]
/root/hudi-release-0.12.0/hudi-spark-datasource/hudi-spark3.2.x/src/main/scala/org/apache/spark/sql/execution/datasources/Spark32NestedSchemaPruning.scala:92:
error: not enough arguments for method apply: (schema:
org.apache.spark.sql.types.StructType, output:
org.apache.spark.sql.catalyst.expressions.AttributeSet)org.apache.spark.sql.catalyst.expressions.ProjectionOverSchema
in object ProjectionOverSchema.
[ERROR] Unspecified value parameter output.
[ERROR] val projectionOverSchema =
ProjectionOverSchema(prunedDataSchema)
[ERROR] ^
[WARNING] one warning found
[ERROR] one error found
```
ProjectionOverSchema.scala:
```scala
//spark 3.2.1
case class ProjectionOverSchema(schema: StructType) {}
//spark 3.2.2 and spark 3.3.0
case class ProjectionOverSchema(schema: StructType, output: AttributeSet) {}
```
Spark32NestedSchemaPruning.scala:
```scala
if (countLeaves(dataSchema) > countLeaves(prunedDataSchema)) {
val prunedRelation = outputRelationBuilder(prunedDataSchema)
val projectionOverSchema = ProjectionOverSchema(prunedDataSchema)
Some(buildNewProjection(projects, normalizedProjects,
normalizedFilters,
prunedRelation, projectionOverSchema))
}
```
Spark33NestedSchemaPruning.scala:
```scala
if (countLeaves(dataSchema) > countLeaves(prunedDataSchema)) {
val prunedRelation = outputRelationBuilder(prunedDataSchema)
val projectionOverSchema =
ProjectionOverSchema(prunedDataSchema,AttributeSet(output))
Some(buildNewProjection(projects, normalizedProjects,
normalizedFilters,
prunedRelation, projectionOverSchema))
}
```
In spark 3.2.1, ProjectionOverSchema has a argument. But in spark 3.2.2 and
spark 3.3.0, ProjectionOverSchema has 2 arguments.
Spark32NestedSchemaPruning.scala is not compatible with spark 3.2.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
