Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14333
Yeah, I think you're right, because the unpersisted RDD can still be
recomputed but not a destroyed Broadcast. Hm, then isn't this also true of
`bcSyn0Global`?
I suppose I think we should prefer to keep it simple and correct first, and
only introduce complexity to optimize while preserving correctness. If some of
the current unpersist calls can't be safely changed to destroy, maybe best to
leave them rather than find a way to destroy them, if we don't know that it's a
problem.
I think we still have an RDD-related problem here in that the intermediate
RDDs aren't unpersisted, and all of them remain persisted after the loop. Kind
of a separate issue, I suppose.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
