yihua commented on code in PR #6151:
URL: https://github.com/apache/hudi/pull/6151#discussion_r928066027
##########
hudi-spark-datasource/hudi-spark/pom.xml:
##########
@@ -316,6 +332,12 @@
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
Review Comment:
Check whether this affects Spark bundle.
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestOrcBootstrap.java:
##########
@@ -168,11 +169,13 @@ public Schema generateNewDataSetAndReturnSchema(long
timestamp, int numRecords,
return AvroOrcUtils.createAvroSchemaWithDefaultValue(orcSchema,
"test_orc_record", null, true);
}
+ @Disabled("Disable due to hive's orc conflict.")
Review Comment:
Maybe we can add a `@Tag` like `Spark2_4only` for this class.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieIndexer.java:
##########
@@ -75,6 +76,7 @@
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
+@Disabled
Review Comment:
If this is due to multiple Spark context exception, we should use
`SparkClientFunctionalTestHarness` to rewrite this test and avoid initializing
the spark context again, to fix the tests.
##########
packaging/hudi-spark-bundle/pom.xml:
##########
@@ -95,6 +95,12 @@
<include>org.antlr:stringtemplate</include>
<include>org.apache.parquet:parquet-avro</include>
+
<include>com.fasterxml.jackson.core:jackson-annotations</include>
+ <include>com.fasterxml.jackson.core:jackson-core</include>
+
<include>com.fasterxml.jackson.core:jackson-databind</include>
+
<include>com.fasterxml.jackson.dataformat:jackson-dataformat-yaml</include>
+
<include>com.fasterxml.jackson.module:jackson-module-scala_${scala.binary.version}</include>
Review Comment:
Is this for fixing testing only? We should avoid introducing new changes
for production code and bundling. If really necessary, could you add these to
test scope only or integ-test-bundle?
##########
hudi-utilities/pom.xml:
##########
@@ -227,6 +227,10 @@
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ </exclusion>
Review Comment:
Could you clarify if this is needed? Any implication on Spark bundle (e.g.,
missing Hadoop-related classes)? Is this for test only?
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestOrcBootstrap.java:
##########
@@ -168,11 +169,13 @@ public Schema generateNewDataSetAndReturnSchema(long
timestamp, int numRecords,
return AvroOrcUtils.createAvroSchemaWithDefaultValue(orcSchema,
"test_orc_record", null, true);
}
+ @Disabled("Disable due to hive's orc conflict.")
Review Comment:
Could we re-enable these tests in Spark 2.4 in Github CI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
