[
https://issues.apache.org/jira/browse/HUDI-2117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378853#comment-17378853
]
ASF GitHub Bot commented on HUDI-2117:
--------------------------------------
minihippo commented on a change in pull request #3207:
URL: https://github.com/apache/hudi/pull/3207#discussion_r667581108
##########
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -229,22 +229,6 @@ object HoodieSparkSqlWriter {
writeResult, parameters, writeClient, tableConfig, jsc,
TableInstantInfo(basePath, instantTime, commitActionType, operation))
- def unpersistRdd(rdd: RDD[_]): Unit = {
- if (sparkContext.getPersistentRDDs.contains(rdd.id)) {
- try {
- rdd.unpersist()
- } catch {
- case t: Exception => log.warn("Got excepting trying to unpersist
rdd", t)
- }
- }
- val parentRdds = rdd.dependencies.map(_.rdd)
Review comment:
Yes. Because all persisted rdd are stored in the map, the rdd dependency
which is cached is in the map too, and the map can be acessed by
`SparkContext.getPersistentRDDs()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Unpersist the input rdd after the commit is completed
> -----------------------------------------------------
>
> Key: HUDI-2117
> URL: https://issues.apache.org/jira/browse/HUDI-2117
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: XiaoyuGeng
> Assignee: XiaoyuGeng
> Priority: Minor
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
