[
https://issues.apache.org/jira/browse/HUDI-2117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17376987#comment-17376987
]
ASF GitHub Bot commented on HUDI-2117:
--------------------------------------
vinothchandar commented on a change in pull request #3207:
URL: https://github.com/apache/hudi/pull/3207#discussion_r665836241
##########
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -229,22 +229,6 @@ object HoodieSparkSqlWriter {
writeResult, parameters, writeClient, tableConfig, jsc,
TableInstantInfo(basePath, instantTime, commitActionType, operation))
- def unpersistRdd(rdd: RDD[_]): Unit = {
- if (sparkContext.getPersistentRDDs.contains(rdd.id)) {
- try {
- rdd.unpersist()
- } catch {
- case t: Exception => log.warn("Got excepting trying to unpersist
rdd", t)
- }
- }
- val parentRdds = rdd.dependencies.map(_.rdd)
Review comment:
is it okay to not chase down the dependencies like this? I think so.
want to confirm.
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1037,6 +1038,10 @@ protected void
setWriteSchemaForDeletes(HoodieTableMetaClient metaClient) {
}
}
+ protected void unpersistRDD() {
Review comment:
Can we just do this in `close()` instead of introducing a new api here?
regardless, we need to call this something spark agnostic since its in
client-common.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Unpersist the input rdd after the commit is completed
> -----------------------------------------------------
>
> Key: HUDI-2117
> URL: https://issues.apache.org/jira/browse/HUDI-2117
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: XiaoyuGeng
> Assignee: XiaoyuGeng
> Priority: Minor
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
