[jira] [Updated] (MAPREDUCE-6639) Process hangs in LocatedFileStatusFetcher if FileSystem.get throws
[ https://issues.apache.org/jira/browse/MAPREDUCE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-6639: -- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) I concur that it is a bug and that the patch fixes it. Although there's no test for (it'd be possible, but tricky), the IDE highlights that the field {{unknownError}} is only set in the conditional clause. That is: the code which sets the field can only be reached if the field is set. Accordingly, the field can never be set. This patch fixes that. > Process hangs in LocatedFileStatusFetcher if FileSystem.get throws > -- > > Key: MAPREDUCE-6639 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6639 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Ryan Blue >Assignee: Ryan Blue > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6639.2.patch > > > ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable > thread functions, > [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], > doesn't catch exceptions (the callbacks do). When an exception is thrown, > the thread exists and doesn't signal the error to the calling thread, which > continues waiting to be signaled. This can happen when a FS implementation > cannot be found. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6639) Process hangs in LocatedFileStatusFetcher if FileSystem.get throws
[ https://issues.apache.org/jira/browse/MAPREDUCE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated MAPREDUCE-6639: - Description: ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable thread functions, [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], doesn't catch exceptions (the callbacks do). When an exception is thrown, the thread exists and doesn't signal the error to the calling thread, which continues waiting to be signaled. This can happen when a FS implementation cannot be found. (was: ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable thread functions, [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], doesn't catch exceptions (the callbacks do). When an exception is thrown, the thread exists and doesn't signal the error to the calling thread, which continues waiting to be signaled. This can happen when a FS implementation cannot be found. The solution is to use a try-catch around the implementation of {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when any exception is caught.) > Process hangs in LocatedFileStatusFetcher if FileSystem.get throws > -- > > Key: MAPREDUCE-6639 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6639 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Ryan Blue >Assignee: Ryan Blue > Attachments: MAPREDUCE-6639.2.patch > > > ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable > thread functions, > [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], > doesn't catch exceptions (the callbacks do). When an exception is thrown, > the thread exists and doesn't signal the error to the calling thread, which > continues waiting to be signaled. This can happen when a FS implementation > cannot be found. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6639) Process hangs in LocatedFileStatusFetcher if FileSystem.get throws
[ https://issues.apache.org/jira/browse/MAPREDUCE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated MAPREDUCE-6639: - Attachment: MAPREDUCE-6639.2.patch Adding a better patch. The problem was in the error handling. The calling thread waits for the operation to succeed or until signaled that unknownError is set. When an error is passed to signal that thread, there's a check for whether another error has already been set, but the check is wrong. It only sets the error if another error is already set, rather than if no other error has been set. The result is that if there is an error, the caller is never signaled and waits indefinitely for all of the tasks to complete successfully. > Process hangs in LocatedFileStatusFetcher if FileSystem.get throws > -- > > Key: MAPREDUCE-6639 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6639 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Ryan Blue >Assignee: Ryan Blue > Attachments: MAPREDUCE-6639.2.patch > > > ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable > thread functions, > [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], > doesn't catch exceptions (the callbacks do). When an exception is thrown, > the thread exists and doesn't signal the error to the calling thread, which > continues waiting to be signaled. This can happen when a FS implementation > cannot be found. > The solution is to use a try-catch around the implementation of > {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when > any exception is caught. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6639) Process hangs in LocatedFileStatusFetcher if FileSystem.get throws
[ https://issues.apache.org/jira/browse/MAPREDUCE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated MAPREDUCE-6639: - Attachment: (was: MAPREDUCE-6639.1.patch) > Process hangs in LocatedFileStatusFetcher if FileSystem.get throws > -- > > Key: MAPREDUCE-6639 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6639 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Ryan Blue >Assignee: Ryan Blue > > ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable > thread functions, > [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], > doesn't catch exceptions (the callbacks do). When an exception is thrown, > the thread exists and doesn't signal the error to the calling thread, which > continues waiting to be signaled. This can happen when a FS implementation > cannot be found. > The solution is to use a try-catch around the implementation of > {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when > any exception is caught. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6639) Process hangs in LocatedFileStatusFetcher if FileSystem.get throws
[ https://issues.apache.org/jira/browse/MAPREDUCE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated MAPREDUCE-6639: - Description: ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable thread functions, [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], doesn't catch exceptions (the callbacks do). When an exception is thrown, the thread exists and doesn't signal the error to the calling thread, which continues waiting to be signaled. This can happen when a FS implementation cannot be found. The solution is to use a try-catch around the implementation of {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when any exception is caught. was: ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable thread functions, [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], doesn't catch exceptions (the others do). When an exception is thrown, the thread exists and doesn't signal the error to the calling thread, which continues waiting to be signaled. This can happen when a FS implementation cannot be found. The solution is to use a try-catch around the implementation of {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when any exception is caught. > Process hangs in LocatedFileStatusFetcher if FileSystem.get throws > -- > > Key: MAPREDUCE-6639 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6639 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Ryan Blue >Assignee: Ryan Blue > Attachments: MAPREDUCE-6639.1.patch > > > ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable > thread functions, > [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], > doesn't catch exceptions (the callbacks do). When an exception is thrown, > the thread exists and doesn't signal the error to the calling thread, which > continues waiting to be signaled. This can happen when a FS implementation > cannot be found. > The solution is to use a try-catch around the implementation of > {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when > any exception is caught. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6639) Process hangs in LocatedFileStatusFetcher if FileSystem.get throws
[ https://issues.apache.org/jira/browse/MAPREDUCE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated MAPREDUCE-6639: - Attachment: MAPREDUCE-6639.1.patch Attaching a fix. This is slightly different from what I suggested above. Because a result needs to be returned for the future, I've added an unknownError field to the results that the futures check. If there is an error, the futures call registerError. > Process hangs in LocatedFileStatusFetcher if FileSystem.get throws > -- > > Key: MAPREDUCE-6639 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6639 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Ryan Blue >Assignee: Ryan Blue > Attachments: MAPREDUCE-6639.1.patch > > > ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable > thread functions, > [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], > doesn't catch exceptions (the others do). When an exception is thrown, the > thread exists and doesn't signal the error to the calling thread, which > continues waiting to be signaled. This can happen when a FS implementation > cannot be found. > The solution is to use a try-catch around the implementation of > {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when > any exception is caught. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6639) Process hangs in LocatedFileStatusFetcher if FileSystem.get throws
[ https://issues.apache.org/jira/browse/MAPREDUCE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated MAPREDUCE-6639: - Assignee: Ryan Blue Affects Version/s: 2.7.2 Status: Patch Available (was: Open) > Process hangs in LocatedFileStatusFetcher if FileSystem.get throws > -- > > Key: MAPREDUCE-6639 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6639 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Ryan Blue >Assignee: Ryan Blue > > ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable > thread functions, > [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], > doesn't catch exceptions (the others do). When an exception is thrown, the > thread exists and doesn't signal the error to the calling thread, which > continues waiting to be signaled. This can happen when a FS implementation > cannot be found. > The solution is to use a try-catch around the implementation of > {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when > any exception is caught. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6639) Process hangs in LocatedFileStatusFetcher if FileSystem.get throws
[ https://issues.apache.org/jira/browse/MAPREDUCE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated MAPREDUCE-6639: - Description: ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable thread functions, [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], doesn't catch exceptions (the others do). When an exception is thrown, the thread exists and doesn't signal the error to the calling thread, which continues waiting to be signaled. This can happen when a FS implementation cannot be found. The solution is to use a try-catch around the implementation of {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when any exception is caught. was: ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable thread functions, [ProcessInitialInputPathCallable|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], doesn't catch exceptions (the others do). When an exception is thrown, the thread exists and doesn't signal the error to the calling thread, which continues waiting to be signaled. This can happen when a FS implementation cannot be found. The solution is to use a try-catch around the implementation of {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when any exception is caught. > Process hangs in LocatedFileStatusFetcher if FileSystem.get throws > -- > > Key: MAPREDUCE-6639 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6639 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Ryan Blue > > ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable > thread functions, > [{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306], > doesn't catch exceptions (the others do). When an exception is thrown, the > thread exists and doesn't signal the error to the calling thread, which > continues waiting to be signaled. This can happen when a FS implementation > cannot be found. > The solution is to use a try-catch around the implementation of > {{ProcessInitialInputPathCallable#call}} that calls {{registerError}} when > any exception is caught. -- This message was sent by Atlassian JIRA (v6.3.4#6332)