Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/791#issuecomment-43736593
- With the latest commit, the issue with dropping flag is gone - which is
great.
- There is a change of behavior w.r.t earlier code.
Whether the earlier code was the way it was intentionally or accidentally,
I am not sure - will let @mateiz or others comment.
Essentially there are a few things here :
a) What happens if existing block is re-added. Looks like this was probably
handled earlier also ?
I went up the call tree a bit, and did not look like this was prevented :
but maybe I missed it. Any comments @mateiz ?
b) What happens if same block is added in parallel by two threads.
If this was supported usecase, then the current PR breaks this - it is
possible for first thread to add it, and second to evict it from memory in case
it was not possible to host both two copies in memory (according to the free
space computed).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
