[jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1650#comment-1650
]
Gera Shegalov commented on MAPREDUCE-7108:
--
[~giovanni.fumarola] the change in {{slurp}} does not introduce any {{catch}}.
The patch is replacing try finally with the syntactic sugar version of the
same.
> TestFileOutputCommitter fails on Windows
>
>
> Key: MAPREDUCE-7108
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1
>Reporter: Zuoming Zhang
>Priority: Minor
> Labels: Windows
> Fix For: 3.1.0, 2.9.1
>
> Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt,
> MAPREDUCE-7108-TestFileOutputCommitter-Report.001.txt,
> MAPREDUCE-7108.000.patch, MAPREDUCE-7108.001.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TestFileOutputCommitter.java fails on Windows.
> Reason:
> This is because in testMapFileOutputCommitterInternal, after executing
> MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly
> closed. So other apis that attempt to delete the existing file fails due to
> the file handler already being used.
>
> Solution:
> Close all readers after call MapFileOutputFormat.getReaders(outDir, conf);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506661#comment-16506661 ] Giovanni Matteo Fumarola commented on MAPREDUCE-7108: - Thanks [~zuzhan] for the patch. The patch looks good to me. [~jira.shegalov], the current code can trigger future failures in Windows. The instruction new FileInputStream(f); should be within the try-catch. > TestFileOutputCommitter fails on Windows > > > Key: MAPREDUCE-7108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, > MAPREDUCE-7108-TestFileOutputCommitter-Report.001.txt, > MAPREDUCE-7108.000.patch, MAPREDUCE-7108.001.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > TestFileOutputCommitter.java fails on Windows. > Reason: > This is because in testMapFileOutputCommitterInternal, after executing > MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly > closed. So other apis that attempt to delete the existing file fails due to > the file handler already being used. > > Solution: > Close all readers after call MapFileOutputFormat.getReaders(outDir, conf); -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506652#comment-16506652
]
Gera Shegalov commented on MAPREDUCE-7108:
--
LGTM pending Jenkins although I'd rather not do the unrelated and semantically
neutral change in {{slurp}}
> TestFileOutputCommitter fails on Windows
>
>
> Key: MAPREDUCE-7108
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1
>Reporter: Zuoming Zhang
>Priority: Minor
> Labels: Windows
> Fix For: 3.1.0, 2.9.1
>
> Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt,
> MAPREDUCE-7108-TestFileOutputCommitter-Report.001.txt,
> MAPREDUCE-7108.000.patch, MAPREDUCE-7108.001.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TestFileOutputCommitter.java fails on Windows.
> Reason:
> This is because in testMapFileOutputCommitterInternal, after executing
> MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly
> closed. So other apis that attempt to delete the existing file fails due to
> the file handler already being used.
>
> Solution:
> Close all readers after call MapFileOutputFormat.getReaders(outDir, conf);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zuoming Zhang updated MAPREDUCE-7108: - Attachment: MAPREDUCE-7108-TestFileOutputCommitter-Report.001.txt > TestFileOutputCommitter fails on Windows > > > Key: MAPREDUCE-7108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, > MAPREDUCE-7108-TestFileOutputCommitter-Report.001.txt, > MAPREDUCE-7108.000.patch, MAPREDUCE-7108.001.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > TestFileOutputCommitter.java fails on Windows. > Reason: > This is because in testMapFileOutputCommitterInternal, after executing > MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly > closed. So other apis that attempt to delete the existing file fails due to > the file handler already being used. > > Solution: > Close all readers after call MapFileOutputFormat.getReaders(outDir, conf); -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506599#comment-16506599 ] Zuoming Zhang commented on MAPREDUCE-7108: -- Thanks [~jira.shegalov] & [~giovanni.fumarola], uploaded a new patch. > TestFileOutputCommitter fails on Windows > > > Key: MAPREDUCE-7108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, > MAPREDUCE-7108-TestFileOutputCommitter-Report.001.txt, > MAPREDUCE-7108.000.patch, MAPREDUCE-7108.001.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > TestFileOutputCommitter.java fails on Windows. > Reason: > This is because in testMapFileOutputCommitterInternal, after executing > MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly > closed. So other apis that attempt to delete the existing file fails due to > the file handler already being used. > > Solution: > Close all readers after call MapFileOutputFormat.getReaders(outDir, conf); -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zuoming Zhang updated MAPREDUCE-7108: - Attachment: MAPREDUCE-7108.001.patch > TestFileOutputCommitter fails on Windows > > > Key: MAPREDUCE-7108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, > MAPREDUCE-7108.000.patch, MAPREDUCE-7108.001.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > TestFileOutputCommitter.java fails on Windows. > Reason: > This is because in testMapFileOutputCommitterInternal, after executing > MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly > closed. So other apis that attempt to delete the existing file fails due to > the file handler already being used. > > Solution: > Close all readers after call MapFileOutputFormat.getReaders(outDir, conf); -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zuoming Zhang updated MAPREDUCE-7108: - Attachment: (was: MAPREDUCE-7108.001.patch) > TestFileOutputCommitter fails on Windows > > > Key: MAPREDUCE-7108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, > MAPREDUCE-7108.000.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > TestFileOutputCommitter.java fails on Windows. > Reason: > This is because in testMapFileOutputCommitterInternal, after executing > MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly > closed. So other apis that attempt to delete the existing file fails due to > the file handler already being used. > > Solution: > Close all readers after call MapFileOutputFormat.getReaders(outDir, conf); -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zuoming Zhang updated MAPREDUCE-7108: - Attachment: MAPREDUCE-7108.001.patch > TestFileOutputCommitter fails on Windows > > > Key: MAPREDUCE-7108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, > MAPREDUCE-7108.000.patch, MAPREDUCE-7108.001.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > TestFileOutputCommitter.java fails on Windows. > Reason: > This is because in testMapFileOutputCommitterInternal, after executing > MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly > closed. So other apis that attempt to delete the existing file fails due to > the file handler already being used. > > Solution: > Close all readers after call MapFileOutputFormat.getReaders(outDir, conf); -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5817) Mappers get rescheduled on node transition even after all reducers are completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506427#comment-16506427 ] Gera Shegalov commented on MAPREDUCE-5817: -- [~kartik.bhatia], this Jira is done. You can file and work on the next iteration separately. It will require tracking shuffle completions in the task attempt state machine. > Mappers get rescheduled on node transition even after all reducers are > completed > > > Key: MAPREDUCE-5817 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5817 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.3.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Major > Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: MAPREDUCE-5817.001.patch, MAPREDUCE-5817.002.patch, > mapreduce-5817.patch > > > We're seeing a behavior where a job runs long after all reducers were already > finished. We found that the job was rescheduling and running a number of > mappers beyond the point of reducer completion. In one situation, the job ran > for some 9 more hours after all reducers completed! > This happens because whenever a node transition (to an unusable state) comes > into the app master, it just reschedules all mappers that already ran on the > node in all cases. > Therefore, if any node transition has a potential to extend the job period. > Once this window opens, another node transition can prolong it, and this can > happen indefinitely in theory. > If there is some instability in the pool (unhealthy, etc.) for a duration, > then any big job is severely vulnerable to this problem. > If all reducers have been completed, JobImpl.actOnUnusableNode() should not > reschedule mapper tasks. If all reducers are completed, the mapper outputs > are no longer needed, and there is no need to reschedule mapper tasks as they > would not be consumed anyway. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506348#comment-16506348
]
Gera Shegalov edited comment on MAPREDUCE-7108 at 6/8/18 6:12 PM:
--
[~zuzhan] you can use one of the older methods {{cleanup(Log log,
java.io.Closeable... closeables)}} with log passed as null. Looks like
cleanupWithLogger can be used in 2.9.1
was (Author: jira.shegalov):
[~zuzhan] you can use one of the older methods {{cleanup(Log log,
java.io.Closeable... closeables)}} with log passed as null
> TestFileOutputCommitter fails on Windows
>
>
> Key: MAPREDUCE-7108
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1
>Reporter: Zuoming Zhang
>Priority: Minor
> Labels: Windows
> Fix For: 3.1.0, 2.9.1
>
> Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt,
> MAPREDUCE-7108.000.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TestFileOutputCommitter.java fails on Windows.
> Reason:
> This is because in testMapFileOutputCommitterInternal, after executing
> MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly
> closed. So other apis that attempt to delete the existing file fails due to
> the file handler already being used.
>
> Solution:
> Close all readers after call MapFileOutputFormat.getReaders(outDir, conf);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506348#comment-16506348
]
Gera Shegalov commented on MAPREDUCE-7108:
--
[~zuzhan] you can use one of the older methods {{cleanup(Log log,
java.io.Closeable... closeables)}} with log passed as null
> TestFileOutputCommitter fails on Windows
>
>
> Key: MAPREDUCE-7108
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1
>Reporter: Zuoming Zhang
>Priority: Minor
> Labels: Windows
> Fix For: 3.1.0, 2.9.1
>
> Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt,
> MAPREDUCE-7108.000.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TestFileOutputCommitter.java fails on Windows.
> Reason:
> This is because in testMapFileOutputCommitterInternal, after executing
> MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly
> closed. So other apis that attempt to delete the existing file fails due to
> the file handler already being used.
>
> Solution:
> Close all readers after call MapFileOutputFormat.getReaders(outDir, conf);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506340#comment-16506340 ] Zuoming Zhang commented on MAPREDUCE-7108: -- Thanks [~jira.shegalov] for the comment. However, I didn't see a function named as IOUtils.closeStreams in 2.9.1, is it a newly added function in Hadoop 3.1.0? > TestFileOutputCommitter fails on Windows > > > Key: MAPREDUCE-7108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, > MAPREDUCE-7108.000.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > TestFileOutputCommitter.java fails on Windows. > Reason: > This is because in testMapFileOutputCommitterInternal, after executing > MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly > closed. So other apis that attempt to delete the existing file fails due to > the file handler already being used. > > Solution: > Close all readers after call MapFileOutputFormat.getReaders(outDir, conf); -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506322#comment-16506322
]
Gera Shegalov edited comment on MAPREDUCE-7108 at 6/8/18 5:55 PM:
--
Thanks for the patch.
Instead of the manual a loop of closing Readers you can use
{{org.apache.hadoop.io.IOUtils#closeStreams}}}.
Since you are already changing this block, I think you can remove the catch
block
{code:java}
} catch (Exception e) {
fail("Fail to read from MapFileOutputFormat: " + e);
}{code}
because it does not look useful. And validation should be moved inside try as
well whereas fullyDelete be in finally. Would this make sense?
{code:java}
// Ensure getReaders call works and also ignores
// hidden filenames (_ or . prefixes)
MapFile.Reader[] readers = {};
try {
readers = getReaders(outDir, conf);
// validate output
validateMapFileOutputContent(FileSystem.get(job.getConfiguration()), outDir);
} finally {
IOUtils.closeStreams(r);
FileUtil.fullyDelete(new File(outDir.toString()));
} {code}
was (Author: jira.shegalov):
Thanks for the patch.
Instead of the manual a loop of closing Readers you can use
{{org.apache.hadoop.io.IOUtils#closeStreams}}}.
Since you are already changing this block, I think you can remove the catch
block
{code:java}
} catch (Exception e) {
fail("Fail to read from MapFileOutputFormat: " + e);
}{code}
because it does not look useful. And validation should be moved inside try as
well whereas fullyDelete be in finally. Would this make sense?
{code:java}
// Ensure getReaders call works and also ignores
// hidden filenames (_ or . prefixes)
MapFile.Reader[] readers = {};
try {
readers = getReaders(outDir, conf);
// validate output
validateMapFileOutputContent(FileSystem.get(job.getConfiguration()), outDir);
} finally {
IOUtils.closeStreams(r);
FileUtil.fullyDelete(new File(outDir.toString()));
}{code}
> TestFileOutputCommitter fails on Windows
>
>
> Key: MAPREDUCE-7108
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1
>Reporter: Zuoming Zhang
>Priority: Minor
> Labels: Windows
> Fix For: 3.1.0, 2.9.1
>
> Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt,
> MAPREDUCE-7108.000.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TestFileOutputCommitter.java fails on Windows.
> Reason:
> This is because in testMapFileOutputCommitterInternal, after executing
> MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly
> closed. So other apis that attempt to delete the existing file fails due to
> the file handler already being used.
>
> Solution:
> Close all readers after call MapFileOutputFormat.getReaders(outDir, conf);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506324#comment-16506324 ] Zuoming Zhang commented on MAPREDUCE-7108: -- Hi [~giovanni.fumarola], do you mean the "slurp" function instead of "testConcurrentCommitTaskWithSubDirV2"? > TestFileOutputCommitter fails on Windows > > > Key: MAPREDUCE-7108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, > MAPREDUCE-7108.000.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > TestFileOutputCommitter.java fails on Windows. > Reason: > This is because in testMapFileOutputCommitterInternal, after executing > MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly > closed. So other apis that attempt to delete the existing file fails due to > the file handler already being used. > > Solution: > Close all readers after call MapFileOutputFormat.getReaders(outDir, conf); -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506322#comment-16506322
]
Gera Shegalov commented on MAPREDUCE-7108:
--
Thanks for the patch.
Instead of the manual a loop of closing Readers you can use
{{org.apache.hadoop.io.IOUtils#closeStreams}}}.
Since you are already changing this block, I think you can remove the catch
block
{code:java}
} catch (Exception e) {
fail("Fail to read from MapFileOutputFormat: " + e);
}{code}
because it does not look useful. And validation should be moved inside try as
well whereas fullyDelete be in finally. Would this make sense?
{code:java}
// Ensure getReaders call works and also ignores
// hidden filenames (_ or . prefixes)
MapFile.Reader[] readers = {};
try {
readers = getReaders(outDir, conf);
// validate output
validateMapFileOutputContent(FileSystem.get(job.getConfiguration()), outDir);
} finally {
IOUtils.closeStreams(r);
FileUtil.fullyDelete(new File(outDir.toString()));
}{code}
> TestFileOutputCommitter fails on Windows
>
>
> Key: MAPREDUCE-7108
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1
>Reporter: Zuoming Zhang
>Priority: Minor
> Labels: Windows
> Fix For: 3.1.0, 2.9.1
>
> Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt,
> MAPREDUCE-7108.000.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TestFileOutputCommitter.java fails on Windows.
> Reason:
> This is because in testMapFileOutputCommitterInternal, after executing
> MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly
> closed. So other apis that attempt to delete the existing file fails due to
> the file handler already being used.
>
> Solution:
> Close all readers after call MapFileOutputFormat.getReaders(outDir, conf);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506292#comment-16506292
]
Giovanni Matteo Fumarola commented on MAPREDUCE-7108:
-
Thanks [~zuzhan] for the patch.
Do you mind change testConcurrentCommitTaskWithSubDirV2 in this way:
{code:java}
try (FileInputStream in = new FileInputStream(f)) {}{code}
instead of
{code:java}
FileInputStream in = new FileInputStream(f);
try {
} finally {
in.close();
}{code}
> TestFileOutputCommitter fails on Windows
>
>
> Key: MAPREDUCE-7108
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1
>Reporter: Zuoming Zhang
>Priority: Minor
> Labels: Windows
> Fix For: 3.1.0, 2.9.1
>
> Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt,
> MAPREDUCE-7108.000.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TestFileOutputCommitter.java fails on Windows.
> Reason:
> This is because in testMapFileOutputCommitterInternal, after executing
> MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly
> closed. So other apis that attempt to delete the existing file fails due to
> the file handler already being used.
>
> Solution:
> Close all readers after call MapFileOutputFormat.getReaders(outDir, conf);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506283#comment-16506283 ] Zuoming Zhang commented on MAPREDUCE-7108: -- [~elgoiri] test report attached. > TestFileOutputCommitter fails on Windows > > > Key: MAPREDUCE-7108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, > MAPREDUCE-7108.000.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > TestFileOutputCommitter.java fails on Windows. > Reason: > This is because in testMapFileOutputCommitterInternal, after executing > MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly > closed. So other apis that attempt to delete the existing file fails due to > the file handler already being used. > > Solution: > Close all readers after call MapFileOutputFormat.getReaders(outDir, conf); -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7108) TestFileOutputCommitter fails on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zuoming Zhang updated MAPREDUCE-7108: - Attachment: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt > TestFileOutputCommitter fails on Windows > > > Key: MAPREDUCE-7108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7108 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: MAPREDUCE-7108-TestFileOutputCommitter-Report.000.txt, > MAPREDUCE-7108.000.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > TestFileOutputCommitter.java fails on Windows. > Reason: > This is because in testMapFileOutputCommitterInternal, after executing > MapFileOutputFormat.getReaders(outDir, conf), the readers are not properly > closed. So other apis that attempt to delete the existing file fails due to > the file handler already being used. > > Solution: > Close all readers after call MapFileOutputFormat.getReaders(outDir, conf); -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
