codope commented on code in PR #5197:
URL: https://github.com/apache/hudi/pull/5197#discussion_r853695986
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/SerDeUtils.scala:
##########
@@ -30,7 +30,10 @@ object SerDeUtils {
private val kryoLocal = new ThreadLocal[Kryo] {
override protected def initialValue: Kryo = {
- val serializer = new KryoSerializer(new SparkConf(true))
+ val conf = new SparkConf(true)
+ conf.set("spark.kryo.referenceTracking", "true")
Review Comment:
Couple of concerns about setting this true for all:
1. Won't tracking references affect performance?
2. What about thrift server? In that case, default value is false. Setting
to true will affect default behaviour.
Would it be better to guard this by a config, or let users pass it as spark
config in the launch command? Users can enable/disable based on their
deployment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
