yihua commented on code in PR #12772:
URL: https://github.com/apache/hudi/pull/12772#discussion_r2076311555
##########
pom.xml:
##########
@@ -182,6 +185,8 @@
<apache-rat-plugin.version>0.16.1</apache-rat-plugin.version>
<scala-maven-plugin.version>3.3.1</scala-maven-plugin.version>
<scalatest.spark3.version>3.1.0</scalatest.spark3.version>
+ <!-- TODO: Need to upgrade to 3.2.18+ after removing deprecated FunSuite
-->
Review Comment:
Add JIRA ticket
##########
pom.xml:
##########
@@ -2620,6 +2631,67 @@
</activation>
</profile>
+ <profile>
+ <id>spark4</id>
Review Comment:
Make this `spark4.0`
##########
pom.xml:
##########
@@ -2398,7 +2408,8 @@
<bannedDependencies>
<excludes combine.children="append">
<exclude>*:*_2.11</exclude>
- <exclude>*:*_2.12</exclude>
+ <!-- flink-hadoop-compatibility_2.12 is necessary -->
+ <!--<exclude>*:*_2.12</exclude>-->
Review Comment:
Flink is not compilable on Scala 2.13 so there is no need to comment this
out.
##########
pom.xml:
##########
@@ -2620,6 +2631,67 @@
</activation>
</profile>
+ <profile>
+ <id>spark4</id>
+ <properties>
+ <spark4.version>${spark40.version}</spark4.version>
+ <spark.version>${spark4.version}</spark.version>
+ <sparkbundle.version>4</sparkbundle.version>
+ <scala13.version>2.13.14</scala13.version>
+ <scala.version>${scala13.version}</scala.version>
+ <scala.binary.version>2.13</scala.binary.version>
+ <hudi.spark.module>hudi-spark4.0.x</hudi.spark.module>
+ <!-- This glob has to include hudi-spark4-common -->
+ <hudi.spark.common.module>hudi-spark4-common</hudi.spark.common.module>
+ <scalatest.version>${scalatest.spark4.version}</scalatest.version>
+ <hadoop.version>3.4.0</hadoop.version>
+ <kafka.version>3.8.0</kafka.version>
+ <hive.storage.version>2.8.1</hive.storage.version>
+ <!-- NOTE: Some Hudi modules require standalone Parquet/Orc/etc
file-format dependency (hudi-hive-sync,
+ hudi-hadoop-mr, for ex). Since these Hudi modules might be
used from w/in the execution engine(s)
+ bringing these file-formats as dependencies as well, we
need to make sure that versions are
+ synchronized to avoid classpath ambiguity -->
+ <parquet.version>1.13.1</parquet.version>
+ <orc.spark.version>2.0.1</orc.spark.version>
+ <avro.version>1.11.3</avro.version>
Review Comment:
Are these aligned with dependency versions in Spark 4?
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJsonDFSSource.java:
##########
@@ -93,7 +94,17 @@ public void testCorruptedSourceFile() throws IOException {
}
protected void corruptFile(Path path) throws IOException {
- PrintStream os = new PrintStream(fs.appendFile(path).build());
+ PrintStream os;
+ try {
+ os = new PrintStream(fs.appendFile(path).build());
+ } catch (UnsupportedOperationException uoe) {
+ if (fs instanceof ProxyLocalFileSystem) {
Review Comment:
What is the cause for this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
