danny0405 commented on code in PR #12417:
URL: https://github.com/apache/hudi/pull/12417#discussion_r1868761782
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java:
##########
@@ -279,15 +279,15 @@ public static HoodieRecord
createHoodieRecord(GenericRecord gr, HoodieKey hKey,
/**
* Drop records already present in the dataset.
*
- * @param jssc JavaSparkContext
+ * @param engineContext the spark engine context
* @param incomingHoodieRecords HoodieRecords to deduplicate
* @param writeConfig HoodieWriteConfig
*/
@SuppressWarnings("unchecked")
- public static JavaRDD<HoodieRecord> dropDuplicates(JavaSparkContext jssc,
JavaRDD<HoodieRecord> incomingHoodieRecords,
+ public static JavaRDD<HoodieRecord> dropDuplicates(HoodieSparkEngineContext
engineContext, JavaRDD<HoodieRecord> incomingHoodieRecords,
HoodieWriteConfig writeConfig) {
try {
- SparkRDDReadClient client = new SparkRDDReadClient<>(new
HoodieSparkEngineContext(jssc), writeConfig);
+ SparkRDDReadClient client = new SparkRDDReadClient<>(engineContext,
writeConfig);
Review Comment:
what's the gains?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
