yihua commented on code in PR #7821:
URL: https://github.com/apache/hudi/pull/7821#discussion_r1093906025
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -328,7 +328,7 @@ case class MergeIntoHoodieTableCommand(mergeInto:
MergeIntoTable) extends Hoodie
}).toMap
// Serialize the Map[UpdateCondition, UpdateAssignments] to base64 string
val serializedUpdateConditionAndExpressions = Base64.getEncoder
- .encodeToString(SerDeUtils.toBytes(updateConditionToAssignments))
+ .encodeToString(Serializer.toBytes(updateConditionToAssignments))
Review Comment:
Does this work for all Spark versions?
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/payload/ExpressionPayload.scala:
##########
@@ -455,5 +456,50 @@ object ExpressionPayload {
field.schema, field.doc, field.defaultVal, field.order))
Schema.createRecord(a.getName, a.getDoc, a.getNamespace, a.isError,
mergedFields.asJava)
}
+
+
+ /**
+ * This object differs from Hudi's generic [[SerializationUtils]] in its
ability to serialize
+ * Spark's internal structures (various [[Expression]]s)
+ *
+ * For that purpose we re-use Spark's [[KryoSerializer]] instance sharing
configuration
+ * with enclosing [[SparkEnv]]. This is necessary to make sure that this
particular instance of Kryo
+ * user for serialization of Spark's internal structures (like
[[Expression]]s) is configured
+ * appropriately (class-loading, custom serializers, etc)
+ *
+ * TODO rebase on Spark's SerializerSupport
+ */
+ private[hudi] object Serializer {
+
Review Comment:
Have you tested this on all Spark versions (Spark 2.4, 3.1, 3.2, 3.3) in
cluster environment (multiple nodes)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
