alexeykudinkin commented on code in PR #7416:
URL: https://github.com/apache/hudi/pull/7416#discussion_r1044708335
##########
packaging/hudi-flink-bundle/pom.xml:
##########
@@ -106,6 +106,7 @@
<include>com.fasterxml.jackson.core:jackson-annotations</include>
<include>com.fasterxml.jackson.core:jackson-databind</include>
<include>com.fasterxml.jackson.core:jackson-core</include>
+
<include>com.fasterxml.jackson.datatype:jackson-datatype-jsr310</include>
Review Comment:
I think we'd need to add this one to Spark bundle as well
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestParquetColumnProjection.scala:
##########
@@ -229,10 +229,11 @@ class TestParquetColumnProjection extends
SparkClientFunctionalTestHarness with
// is invariant of the # of columns)
val fullColumnsReadStats: Array[(String, Long)] =
if (HoodieSparkUtils.isSpark3)
+ // TODO re-enable tests (these tests are very unstable currently)
Array(
- ("rider", 14167),
- ("rider,driver", 14167),
- ("rider,driver,tip_history", 14167))
+ ("rider", -1),
Review Comment:
What was the issue with this one?
##########
.github/workflows/bot.yml:
##########
@@ -73,19 +73,19 @@ jobs:
run: |
HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
./packaging/bundle-validation/ci_run.sh $HUDI_VERSION
- - name: Common Test
+ - name: UT - Common & Spark
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
FLINK_PROFILE: ${{ matrix.flinkProfile }}
if: ${{ !endsWith(env.SPARK_PROFILE, '2.4') }} # skip test spark 2.4
as it's covered by Azure CI
run:
- mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" '-Dtest=Test*' -pl hudi-common $MVN_ARGS
- - name: Spark SQL Test
+ mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-common,hudi-spark-datasource/hudi-spark $MVN_ARGS
+ - name: FT - Common & Spark
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
FLINK_PROFILE: ${{ matrix.flinkProfile }}
if: ${{ !endsWith(env.SPARK_PROFILE, '2.4') }} # skip test spark 2.4
as it's covered by Azure CI
run:
- mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" '-Dtest=Test*' -pl hudi-spark-datasource/hudi-spark $MVN_ARGS
+ mvn test -Pfunctional-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-common,hudi-spark-datasource/hudi-spark $MVN_ARGS
Review Comment:
Should we keep profiles for both unit and functional?
##########
packaging/hudi-flink-bundle/pom.xml:
##########
@@ -106,6 +106,7 @@
<include>com.fasterxml.jackson.core:jackson-annotations</include>
<include>com.fasterxml.jackson.core:jackson-databind</include>
<include>com.fasterxml.jackson.core:jackson-core</include>
+
<include>com.fasterxml.jackson.datatype:jackson-datatype-jsr310</include>
Review Comment:
Let's also enable Column Stats (write-path) in our Integration tests as well
to make sure we can capture these
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
