Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/1578#discussion_r15426163
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
@@ -200,14 +200,21 @@ object BlockFetcherIterator {
// these all at once because they will just memory-map some files,
so they won't consume
// any memory that might exceed our maxBytesInFlight
for (id <- localBlocksToFetch) {
- getLocalFromDisk(id, serializer) match {
- case Some(iter) => {
- // Pass 0 as size since it's not in flight
- results.put(new FetchResult(id, 0, () => iter))
- logDebug("Got local block " + id)
+ try{
+ getLocalFromDisk(id, serializer) match {
+ case Some(iter) => {
+ // Pass 0 as size since it's not in flight
+ results.put(new FetchResult(id, 0, () => iter))
+ logDebug("Got local block " + id)
+ }
+ case None => {
+ throw new BlockException(id, "Could not get block " + id + "
from local machine")
+ }
}
- case None => {
- throw new BlockException(id, "Could not get block " + id + "
from local machine")
+ } catch {
+ case e: Exception => {
+ logError(s"Error occurred while fetch local block $id", e)
+ results.put(new FetchResult(id, -1, null))
}
--- End diff --
Why do we throw an exception above and then immediately catch it, instead
of doing results.put above? Is there any other kind of error that can happen
beyond getLocalFromDisk returning None?
Also, the current code seems to forget the exception: it just puts in a
failed result. Is this intentional, i.e. will get a FetchFailedException later?
It seems we should return from this method ASAP if there's a problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
