alexeykudinkin commented on code in PR #7702:
URL: https://github.com/apache/hudi/pull/7702#discussion_r1080866041
##########
pom.xml:
##########
@@ -440,34 +441,13 @@
<!-- common to all bundles -->
<artifactSet>
<includes>
- <!-- com.esotericsoftware:kryo-shaded -->
- <include>com.esotericsoftware:kryo-shaded</include>
- <include>com.esotericsoftware:minlog</include>
- <include>org.objenesis:objenesis</include>
<!-- org.apache.httpcomponents -->
<include>org.apache.httpcomponents:httpclient</include>
<include>org.apache.httpcomponents:httpcore</include>
<include>org.apache.httpcomponents:fluent-hc</include>
</includes>
</artifactSet>
<relocations>
- <!-- com.esotericsoftware:kryo-shaded -->
- <relocation>
- <pattern>com.esotericsoftware.kryo.</pattern>
-
<shadedPattern>org.apache.hudi.com.esotericsoftware.kryo.</shadedPattern>
- </relocation>
Review Comment:
We actually move it just to bundles that will have the Kryo included (Spark
and Flink won't have Kryo included)
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestUtils.java:
##########
@@ -92,10 +93,18 @@ public class HoodieClientTestUtils {
*/
public static SparkConf getSparkConfForTest(String appName) {
SparkConf sparkConf = new SparkConf().setAppName(appName)
- .set("spark.serializer",
"org.apache.spark.serializer.KryoSerializer").setMaster("local[4]")
+ .setMaster("local[4]")
+ .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .set("spark.kryo.registrator",
"org.apache.spark.HoodieSparkKryoRegistrar")
+ .set("spark.sql.extensions",
"org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
Review Comment:
This is exactly the method you're referring to (used in tests)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieCommonKryoRegistrar.java:
##########
@@ -81,7 +83,8 @@ public Class<?>[] registerClasses() {
HoodieRecordLocation.class,
HoodieRecordGlobalLocation.class
- };
+ })
+ .forEachOrdered(kryo::register);
Review Comment:
Agree in principle, but here we actually aligning it w/ an interface of
`KryoRegistrator`
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/HoodieSparkSqlTestBase.scala:
##########
@@ -51,13 +52,6 @@ class HoodieSparkSqlTestBase extends FunSuite with
BeforeAndAfterAll {
DateTimeZone.setDefault(DateTimeZone.UTC)
TimeZone.setDefault(DateTimeUtils.getTimeZone("UTC"))
protected lazy val spark: SparkSession = SparkSession.builder()
- .master("local[1]")
- .appName("hoodie sql test")
- .withExtensions(new HoodieSparkSessionExtension)
- .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
- .config("hoodie.insert.shuffle.parallelism", "4")
- .config("hoodie.upsert.shuffle.parallelism", "4")
- .config("hoodie.delete.shuffle.parallelism", "4")
Review Comment:
These are not removed -- they are replaced w/ options set in
`getSparkConfForTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
