Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10748#issuecomment-171737226
Hmm, it looks like two tests are failing:
```
[info] - compute without caching when no partitions fit in memory ***
FAILED *** (3 seconds, 905 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage
0.0 (TID 6, localhost): org.apache.spark.storage.BlockException: Block manager
failed to return cached value for rdd_0_0!
```
and
```
[info] - compute when only some partitions fit in memory *** FAILED *** (3
seconds, 893 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 5 in stage 0.0 failed 4 times, most recent failure: Lost task 5.3 in stage
0.0 (TID 15, localhost): org.apache.spark.storage.BlockException: Block manager
failed to return cached value for rdd_0_5!
[info] at
org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:86)
```
My hunch is that the spill-to-disk fallback path works slightly differently
when called in `BlockManager.putIterator -> MemoryStore.putIterator` vs. the
old code which interacted directly with the memory manager and other block
manager components.
I'll take a look to see if I can spot what's going on. I think that the
control flow of the disk fallback path could be a bit better documented, so
I'll see about adding some comments to the existing code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
