Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/5475#issuecomment-92000380
Thanks for commenting.
For the first issue, I have registered the accumulators of table scan
`InMemoryColumnarTableScan`. If users explicitly uncache the table, these
accumulators can be released as I tested. If the table is not explicitly
uncached, these accumulators will be leaked as you said. Using a configuration
to initialize them sounds good. I will try it.
For the second one, I have tried doing this without a global hash map.
However, because the cache manager will call `InMemoryRelation.withOutput` to
create a new `InMemoryRelation` each time replacing the logical plan with
cached version. The accumulators will be created in these `InMemoryRelation`s.
But the cache manager only keeps the original `InMemoryRelation` and we uncache
based on it. So I add this global hash map to deal with this problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
