xushiyan commented on code in PR #7327:
URL: https://github.com/apache/hudi/pull/7327#discussion_r1039292838
##########
hudi-utilities/pom.xml:
##########
@@ -263,12 +257,24 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-runtime</artifactId>
+ </exclusion>
+ </exclusions>
Review Comment:
this exclusion affects utilities bundle dependency as it is included. needs
to revisit the impact
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/io/storage/TestHoodieReaderWriterBase.java:
##########
@@ -258,10 +257,11 @@ private void verifyRecord(String schemaPath,
GenericRecord record, int index) {
if ("/exampleEvolvedSchemaColumnType.avsc".equals(schemaPath)) {
assertEquals(Integer.toString(index), record.get("number").toString());
} else if ("/exampleEvolvedSchemaDeleteColumn.avsc".equals(schemaPath)) {
- assertNull(record.get("number"));
+ assertFalse(record.hasField("number"));
} else {
assertEquals(index, record.get("number"));
}
- assertNull(record.get("added_field"));
+ // TODO temp disable
+ // assertNull(record.get("added_field"));
Review Comment:
need to revisit this test case
##########
hudi-integ-test/pom.xml:
##########
@@ -63,30 +69,22 @@
<dependency>
<groupId>org.apache.spark</groupId>
- <artifactId>spark-sql_${scala.binary.version}</artifactId>
+ <artifactId>spark-core_${scala.binary.version}</artifactId>
<exclusions>
<exclusion>
- <groupId>org.mortbay.jetty</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet.jsp</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet</groupId>
- <artifactId>*</artifactId>
Review Comment:
not seen these dep from spark-core or spark-sql
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestParquetColumnProjection.scala:
##########
@@ -229,10 +229,11 @@ class TestParquetColumnProjection extends
SparkClientFunctionalTestHarness with
// is invariant of the # of columns)
val fullColumnsReadStats: Array[(String, Long)] =
if (HoodieSparkUtils.isSpark3)
+ // TODO re-enable tests (these tests are very unstable currently)
Array(
- ("rider", 14167),
- ("rider,driver", 14167),
- ("rider,driver,tip_history", 14167))
+ ("rider", -1),
+ ("rider,driver", -1),
+ ("rider,driver,tip_history", -1))
Review Comment:
this test case wasn't run previously in GH actions CI under spark 3.2
profile because it was run under `unit-tests` profile while this is under
`functional-tests`. otherwise it would have failed before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
