[jira] [Commented] (MAPREDUCE-1125) SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
[
https://issues.apache.org/jira/browse/MAPREDUCE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993272#comment-14993272
]
Hudson commented on MAPREDUCE-1125:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #576 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/576/])
MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with (aw: rev
19a0c2660cce0dccf968376bed249de94b58b8a2)
* hadoop-tools/hadoop-pipes/src/main/native/utils/api/hadoop/SerialUtils.hh
* hadoop-mapreduce-project/CHANGES.txt
* hadoop-tools/hadoop-pipes/src/main/native/utils/impl/SerialUtils.cc
> SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
> ---
>
> Key: MAPREDUCE-1125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.21.0
>Reporter: Simone Leo
>Assignee: Simone Leo
> Labels: BB2015-05-TBR
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-1125-2.patch, MAPREDUCE-1125-3.patch,
> MAPREDUCE-1125.004.patch
>
>
> {noformat}
> *** SerialUtils.hh ***
> float deserializeFloat(InStream& stream);
> *** SerialUtils.cc ***
> void deserializeFloat(float& t, InStream& stream)
> {
> char buf[sizeof(float)];
> stream.read(buf, sizeof(float));
> XDR xdrs;
> xdrmem_create(&xdrs, buf, sizeof(float), XDR_DECODE);
> xdr_float(&xdrs, &t);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993271#comment-14993271 ] Hudson commented on MAPREDUCE-3914: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #576 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/576/]) MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes (aw: rev fc7cd46faf5cafc73d4316a21d59cb2b83a6b48f) * hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc * hadoop-mapreduce-project/CHANGES.txt > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl >Assignee: Joe Mudd > Fix For: 3.0.0 > > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1125) SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
[
https://issues.apache.org/jira/browse/MAPREDUCE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993080#comment-14993080
]
Hudson commented on MAPREDUCE-1125:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2574 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2574/])
MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with (aw: rev
19a0c2660cce0dccf968376bed249de94b58b8a2)
* hadoop-mapreduce-project/CHANGES.txt
* hadoop-tools/hadoop-pipes/src/main/native/utils/api/hadoop/SerialUtils.hh
* hadoop-tools/hadoop-pipes/src/main/native/utils/impl/SerialUtils.cc
> SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
> ---
>
> Key: MAPREDUCE-1125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.21.0
>Reporter: Simone Leo
>Assignee: Simone Leo
> Labels: BB2015-05-TBR
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-1125-2.patch, MAPREDUCE-1125-3.patch,
> MAPREDUCE-1125.004.patch
>
>
> {noformat}
> *** SerialUtils.hh ***
> float deserializeFloat(InStream& stream);
> *** SerialUtils.cc ***
> void deserializeFloat(float& t, InStream& stream)
> {
> char buf[sizeof(float)];
> stream.read(buf, sizeof(float));
> XDR xdrs;
> xdrmem_create(&xdrs, buf, sizeof(float), XDR_DECODE);
> xdr_float(&xdrs, &t);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993079#comment-14993079 ] Hudson commented on MAPREDUCE-3914: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2574 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2574/]) MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes (aw: rev fc7cd46faf5cafc73d4316a21d59cb2b83a6b48f) * hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc * hadoop-mapreduce-project/CHANGES.txt > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl >Assignee: Joe Mudd > Fix For: 3.0.0 > > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993070#comment-14993070 ] Hudson commented on MAPREDUCE-3914: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #1368 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1368/]) MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes (aw: rev fc7cd46faf5cafc73d4316a21d59cb2b83a6b48f) * hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc * hadoop-mapreduce-project/CHANGES.txt > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl >Assignee: Joe Mudd > Fix For: 3.0.0 > > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1125) SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
[
https://issues.apache.org/jira/browse/MAPREDUCE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993071#comment-14993071
]
Hudson commented on MAPREDUCE-1125:
---
SUCCESS: Integrated in Hadoop-Yarn-trunk #1368 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1368/])
MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with (aw: rev
19a0c2660cce0dccf968376bed249de94b58b8a2)
* hadoop-tools/hadoop-pipes/src/main/native/utils/impl/SerialUtils.cc
* hadoop-mapreduce-project/CHANGES.txt
* hadoop-tools/hadoop-pipes/src/main/native/utils/api/hadoop/SerialUtils.hh
> SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
> ---
>
> Key: MAPREDUCE-1125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.21.0
>Reporter: Simone Leo
>Assignee: Simone Leo
> Labels: BB2015-05-TBR
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-1125-2.patch, MAPREDUCE-1125-3.patch,
> MAPREDUCE-1125.004.patch
>
>
> {noformat}
> *** SerialUtils.hh ***
> float deserializeFloat(InStream& stream);
> *** SerialUtils.cc ***
> void deserializeFloat(float& t, InStream& stream)
> {
> char buf[sizeof(float)];
> stream.read(buf, sizeof(float));
> XDR xdrs;
> xdrmem_create(&xdrs, buf, sizeof(float), XDR_DECODE);
> xdr_float(&xdrs, &t);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: (was: MAPREDUCE-6096-v6.patch) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: zhangyubiao > Labels: BB2015-05-TBR, easyfix, patch > Attachments: MAPREDUCE-6096-v8.patch, > job_1446203652278_66705-1446308686422-dd_edw-insert+overwrite+table+bkactiv...dp%3D%27ACTIVE%27%28Stage-1446308802181-233-0-SUCCEEDED-bdp_jdw_corejob.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: (was: MAPREDUCE-6096-v7.patch) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: zhangyubiao > Labels: BB2015-05-TBR, easyfix, patch > Attachments: MAPREDUCE-6096-v8.patch, > job_1446203652278_66705-1446308686422-dd_edw-insert+overwrite+table+bkactiv...dp%3D%27ACTIVE%27%28Stage-1446308802181-233-0-SUCCEEDED-bdp_jdw_corejob.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: (was: MAPREDUCE-6096-v3.patch) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: zhangyubiao > Labels: BB2015-05-TBR, easyfix, patch > Attachments: MAPREDUCE-6096-v8.patch, > job_1446203652278_66705-1446308686422-dd_edw-insert+overwrite+table+bkactiv...dp%3D%27ACTIVE%27%28Stage-1446308802181-233-0-SUCCEEDED-bdp_jdw_corejob.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: (was: MAPREDUCE-6096-v5.patch) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: zhangyubiao > Labels: BB2015-05-TBR, easyfix, patch > Attachments: MAPREDUCE-6096-v8.patch, > job_1446203652278_66705-1446308686422-dd_edw-insert+overwrite+table+bkactiv...dp%3D%27ACTIVE%27%28Stage-1446308802181-233-0-SUCCEEDED-bdp_jdw_corejob.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: (was: MAPREDUCE-6096-v4.patch) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: zhangyubiao > Labels: BB2015-05-TBR, easyfix, patch > Attachments: MAPREDUCE-6096-v3.patch, MAPREDUCE-6096-v5.patch, > MAPREDUCE-6096-v6.patch, MAPREDUCE-6096-v7.patch, MAPREDUCE-6096-v8.patch, > job_1446203652278_66705-1446308686422-dd_edw-insert+overwrite+table+bkactiv...dp%3D%27ACTIVE%27%28Stage-1446308802181-233-0-SUCCEEDED-bdp_jdw_corejob.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: (was: job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: zhangyubiao > Labels: BB2015-05-TBR, easyfix, patch > Attachments: MAPREDUCE-6096-v3.patch, MAPREDUCE-6096-v5.patch, > MAPREDUCE-6096-v6.patch, MAPREDUCE-6096-v7.patch, MAPREDUCE-6096-v8.patch, > job_1446203652278_66705-1446308686422-dd_edw-insert+overwrite+table+bkactiv...dp%3D%27ACTIVE%27%28Stage-1446308802181-233-0-SUCCEEDED-bdp_jdw_corejob.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: (was: MAPREDUCE-6096-v2.patch) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: zhangyubiao > Labels: BB2015-05-TBR, easyfix, patch > Attachments: MAPREDUCE-6096-v3.patch, MAPREDUCE-6096-v5.patch, > MAPREDUCE-6096-v6.patch, MAPREDUCE-6096-v7.patch, MAPREDUCE-6096-v8.patch, > job_1446203652278_66705-1446308686422-dd_edw-insert+overwrite+table+bkactiv...dp%3D%27ACTIVE%27%28Stage-1446308802181-233-0-SUCCEEDED-bdp_jdw_corejob.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: (was: MAPREDUCE-6096.patch) > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: zhangyubiao > Labels: BB2015-05-TBR, easyfix, patch > Attachments: MAPREDUCE-6096-v3.patch, MAPREDUCE-6096-v5.patch, > MAPREDUCE-6096-v6.patch, MAPREDUCE-6096-v7.patch, MAPREDUCE-6096-v8.patch, > job_1446203652278_66705-1446308686422-dd_edw-insert+overwrite+table+bkactiv...dp%3D%27ACTIVE%27%28Stage-1446308802181-233-0-SUCCEEDED-bdp_jdw_corejob.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1125) SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
[
https://issues.apache.org/jira/browse/MAPREDUCE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993003#comment-14993003
]
Hudson commented on MAPREDUCE-1125:
---
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #644 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/644/])
MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with (aw: rev
19a0c2660cce0dccf968376bed249de94b58b8a2)
* hadoop-tools/hadoop-pipes/src/main/native/utils/api/hadoop/SerialUtils.hh
* hadoop-mapreduce-project/CHANGES.txt
* hadoop-tools/hadoop-pipes/src/main/native/utils/impl/SerialUtils.cc
> SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
> ---
>
> Key: MAPREDUCE-1125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.21.0
>Reporter: Simone Leo
>Assignee: Simone Leo
> Labels: BB2015-05-TBR
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-1125-2.patch, MAPREDUCE-1125-3.patch,
> MAPREDUCE-1125.004.patch
>
>
> {noformat}
> *** SerialUtils.hh ***
> float deserializeFloat(InStream& stream);
> *** SerialUtils.cc ***
> void deserializeFloat(float& t, InStream& stream)
> {
> char buf[sizeof(float)];
> stream.read(buf, sizeof(float));
> XDR xdrs;
> xdrmem_create(&xdrs, buf, sizeof(float), XDR_DECODE);
> xdr_float(&xdrs, &t);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993002#comment-14993002 ] Hudson commented on MAPREDUCE-3914: --- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #644 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/644/]) MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes (aw: rev fc7cd46faf5cafc73d4316a21d59cb2b83a6b48f) * hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc * hadoop-mapreduce-project/CHANGES.txt > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl >Assignee: Joe Mudd > Fix For: 3.0.0 > > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[
https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992995#comment-14992995
]
Hadoop QA commented on MAPREDUCE-5860:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 7m 0s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-06 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12662748/MAPREDUCE-5860.patch |
| JIRA Issue | MAPREDUCE-5860 |
| Optional Tests | asflicense cc unit javac compile |
| uname | Linux eab572b3b0f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
|
| git revision | trunk / bff7c90 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6127/testReport/ |
| modules | C: hadoop-tools/hadoop-pipes U: hadoop-tools/hadoop-pipes |
| Max memory used | 229MB |
| Powered by | Apache Yetus http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6127/console |
This message was automatically generated.
> Hadoop pipes Combiner is closed before all of its reduce calls
> --
>
> Key: MAPREDUCE-5860
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.23.0
> Environment: 0.23.0 on 64 bit linux
>Reporter: Joe Mudd
>
[jira] [Commented] (MAPREDUCE-1125) SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
[
https://issues.apache.org/jira/browse/MAPREDUCE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992984#comment-14992984
]
Hudson commented on MAPREDUCE-1125:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #634 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/634/])
MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with (aw: rev
19a0c2660cce0dccf968376bed249de94b58b8a2)
* hadoop-mapreduce-project/CHANGES.txt
* hadoop-tools/hadoop-pipes/src/main/native/utils/api/hadoop/SerialUtils.hh
* hadoop-tools/hadoop-pipes/src/main/native/utils/impl/SerialUtils.cc
> SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
> ---
>
> Key: MAPREDUCE-1125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.21.0
>Reporter: Simone Leo
>Assignee: Simone Leo
> Labels: BB2015-05-TBR
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-1125-2.patch, MAPREDUCE-1125-3.patch,
> MAPREDUCE-1125.004.patch
>
>
> {noformat}
> *** SerialUtils.hh ***
> float deserializeFloat(InStream& stream);
> *** SerialUtils.cc ***
> void deserializeFloat(float& t, InStream& stream)
> {
> char buf[sizeof(float)];
> stream.read(buf, sizeof(float));
> XDR xdrs;
> xdrmem_create(&xdrs, buf, sizeof(float), XDR_DECODE);
> xdr_float(&xdrs, &t);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992983#comment-14992983 ] Hudson commented on MAPREDUCE-3914: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #634 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/634/]) MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes (aw: rev fc7cd46faf5cafc73d4316a21d59cb2b83a6b48f) * hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc * hadoop-mapreduce-project/CHANGES.txt > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl >Assignee: Joe Mudd > Fix For: 3.0.0 > > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[
https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992974#comment-14992974
]
Hadoop QA commented on MAPREDUCE-1362:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 38s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 36s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 47s
{color} | {color:red} hadoop-mapreduce-client in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 47s {color}
| {color:red} hadoop-mapreduce-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 38s
{color} | {color:red} hadoop-mapreduce-client in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 38s {color}
| {color:red} hadoop-mapreduce-client in the patch failed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 26s
{color} | {color:red} Patch generated 104 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 18, now 122).
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 114 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 22s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 55s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 19s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 8s {color} |
{color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 20s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 22s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.mapreduce.
[jira] [Commented] (MAPREDUCE-6539) Make hadoop-tools/hadoop-pipes Native code -Wall-clean
[
https://issues.apache.org/jira/browse/MAPREDUCE-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992972#comment-14992972
]
Hadoop QA commented on MAPREDUCE-6539:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 5s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 5m 41s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-06 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12743072/HADOOP-12114.002.patch
|
| JIRA Issue | MAPREDUCE-6539 |
| Optional Tests | asflicense cc unit javac compile |
| uname | Linux f5509b54239f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
|
| git revision | trunk / 19a0c26 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6126/testReport/ |
| modules | C: hadoop-tools/hadoop-pipes U: hadoop-tools/hadoop-pipes |
| Max memory used | 226MB |
| Powered by | Apache Yetus http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6126/console |
This message was automatically generated.
> Make hadoop-tools/hadoop-pipes Native code -Wall-clean
> --
>
> Key: MAPREDUCE-6539
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6539
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: pipes
>Affects Versions: 2.7.0
>Reporter: Alan Burlison
>Assignee: Alan Burlison
> Attachment
[jira] [Commented] (MAPREDUCE-6536) hadoop-pipes doesn't use maven properties for openssl
[
https://issues.apache.org/jira/browse/MAPREDUCE-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992968#comment-14992968
]
Hadoop QA commented on MAPREDUCE-6536:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 8m 53s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-06 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12770221/HADOOP-12518.03.patch
|
| JIRA Issue | MAPREDUCE-6536 |
| Optional Tests | asflicense javac javadoc mvninstall unit xml compile
cc |
| uname | Linux b500ef9386b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
|
| git revision | trunk / 19a0c26 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-B
[jira] [Commented] (MAPREDUCE-6537) hadoop pipes examples aren't in the mvn package tar ball
[
https://issues.apache.org/jira/browse/MAPREDUCE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992945#comment-14992945
]
Hadoop QA commented on MAPREDUCE-6537:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 9s
{color} | {color:green} hadoop-assemblies in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 10s
{color} | {color:green} hadoop-assemblies in the patch passed with JDK
v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 5m 49s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.0 Server=1.7.0
Image:test-patch-base-hadoop-date2015-11-06 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12763984/HADOOP-12381.00.patch
|
| JIRA Issue | MAPREDUCE-6537 |
| Optional Tests | asflicense javac javadoc mvninstall unit xml |
| uname | Linux a0b74760c7fd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
|
| git revision | trunk / 19a0c26 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_66
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6123/testReport/ |
| modules | C: hadoop-assemblies U: hadoop-assemblies |
| Max memory used | 228MB |
| Powered by | Apache Yetus http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6123/console |
This message was automatically generated.
> hadoop pipes examples aren't in the mvn package tar ball
>
>
> Key: MAPREDUCE-6537
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6537
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Kai Sasaki
>Priority: Blocker
> Attachments: HADOOP-12381.00.patch
>
>
> Hadoop pipes examples are built but never packaged.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-4090) Branch 1 pipes doesn't work on MR2 clusters
[
https://issues.apache.org/jira/browse/MAPREDUCE-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved MAPREDUCE-4090.
-
Resolution: Won't Fix
It is expected to require recompilation if not outright code changes on major
version upgrades. This is documented in the Hadoop Compatibility Guide.
Closing as won't fix.
> Branch 1 pipes doesn't work on MR2 clusters
> ---
>
> Key: MAPREDUCE-4090
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4090
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2, pipes
>Affects Versions: 0.23.1, 1.0.1
>Reporter: Ahmed Radwan
>
> If I compile pipes examples on branch 1:
> {code}
> ant -Dcompile.c++=yes examples
> {code}
> And then try to run it on an MR2 cluster, the pipes job hangs forever at map
> 0% reduce 0%, I can see in the maps stderr:
> {code}
> Hadoop Pipes Exception: Illegal text protocol command
> {code}
> The issue here is that if users have older pipes job, they won't be able to
> run it on MR2 (without recompilation). Is this expected or there is something
> to be fixed so jobs can be used interchangeably? Or should we document it as
> an incompatibility?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4090) Branch 1 pipes doesn't work on MR2 clusters
[
https://issues.apache.org/jira/browse/MAPREDUCE-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-4090:
Component/s: pipes
> Branch 1 pipes doesn't work on MR2 clusters
> ---
>
> Key: MAPREDUCE-4090
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4090
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2, pipes
>Affects Versions: 0.23.1, 1.0.1
>Reporter: Ahmed Radwan
>
> If I compile pipes examples on branch 1:
> {code}
> ant -Dcompile.c++=yes examples
> {code}
> And then try to run it on an MR2 cluster, the pipes job hangs forever at map
> 0% reduce 0%, I can see in the maps stderr:
> {code}
> Hadoop Pipes Exception: Illegal text protocol command
> {code}
> The issue here is that if users have older pipes job, they won't be able to
> run it on MR2 (without recompilation). Is this expected or there is something
> to be fixed so jobs can be used interchangeably? Or should we document it as
> an incompatibility?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992912#comment-14992912 ] Hudson commented on MAPREDUCE-3914: --- FAILURE: Integrated in Hadoop-trunk-Commit #8764 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8764/]) MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes (aw: rev fc7cd46faf5cafc73d4316a21d59cb2b83a6b48f) * hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc * hadoop-mapreduce-project/CHANGES.txt > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl >Assignee: Joe Mudd > Fix For: 3.0.0 > > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-2446) HCE 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-2446. - Resolution: Duplicate This work has effectively been done in MAPREDUCE-2841. Closing as a dupe. > HCE 2.0 > --- > > Key: MAPREDUCE-2446 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2446 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/streaming, pipes, task >Reporter: Dong Yang > > Enhancing MapReduce by Task-level Optimization. Except for yielding speedups > of up to 130% on original Streaming Program, Hce 2.0 provides some more > flexible programming interfaces including c++, java, python, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1125) SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
[
https://issues.apache.org/jira/browse/MAPREDUCE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992913#comment-14992913
]
Hudson commented on MAPREDUCE-1125:
---
FAILURE: Integrated in Hadoop-trunk-Commit #8764 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8764/])
MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with (aw: rev
19a0c2660cce0dccf968376bed249de94b58b8a2)
* hadoop-tools/hadoop-pipes/src/main/native/utils/impl/SerialUtils.cc
* hadoop-tools/hadoop-pipes/src/main/native/utils/api/hadoop/SerialUtils.hh
* hadoop-mapreduce-project/CHANGES.txt
> SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
> ---
>
> Key: MAPREDUCE-1125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.21.0
>Reporter: Simone Leo
>Assignee: Simone Leo
> Labels: BB2015-05-TBR
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-1125-2.patch, MAPREDUCE-1125-3.patch,
> MAPREDUCE-1125.004.patch
>
>
> {noformat}
> *** SerialUtils.hh ***
> float deserializeFloat(InStream& stream);
> *** SerialUtils.cc ***
> void deserializeFloat(float& t, InStream& stream)
> {
> char buf[sizeof(float)];
> stream.read(buf, sizeof(float));
> XDR xdrs;
> xdrmem_create(&xdrs, buf, sizeof(float), XDR_DECODE);
> xdr_float(&xdrs, &t);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5860) Hadoop pipes Combiner is closed before all of its reduce calls
[
https://issues.apache.org/jira/browse/MAPREDUCE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992914#comment-14992914
]
Hadoop QA commented on MAPREDUCE-5860:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 5s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 5m 35s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-06 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12662748/MAPREDUCE-5860.patch |
| JIRA Issue | MAPREDUCE-5860 |
| Optional Tests | asflicense cc unit javac compile |
| uname | Linux 0fb51396181e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
|
| git revision | trunk / 19a0c26 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6122/testReport/ |
| modules | C: hadoop-tools/hadoop-pipes U: hadoop-tools/hadoop-pipes |
| Max memory used | 227MB |
| Powered by | Apache Yetus http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6122/console |
This message was automatically generated.
> Hadoop pipes Combiner is closed before all of its reduce calls
> --
>
> Key: MAPREDUCE-5860
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5860
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.23.0
> Environment: 0.23.0 on 64 bit linux
>Reporter: Joe Mudd
>
[jira] [Resolved] (MAPREDUCE-1270) Hadoop C++ Extention
[ https://issues.apache.org/jira/browse/MAPREDUCE-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-1270. - Resolution: Duplicate Tags: (was: PIPES C++ ) Given the last message, closing this as a dupe. > Hadoop C++ Extention > > > Key: MAPREDUCE-1270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1270 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 0.20.1 > Environment: hadoop linux >Reporter: Wang Shouyan > Attachments: HADOOP-HCE-1.0.0.patch, HCE InstallMenu.pdf, HCE > Performance Report.pdf, HCE Tutorial.pdf, Overall Design of Hadoop C++ > Extension.doc > > > Hadoop C++ extension is an internal project in baidu, We start it for these > reasons: >1 To provide C++ API. We mostly use Streaming before, and we also try to > use PIPES, but we do not find PIPES is more efficient than Streaming. So we > think a new C++ extention is needed for us. >2 Even using PIPES or Streaming, it is hard to control memory of hadoop > map/reduce Child JVM. >3 It costs so much to read/write/sort TB/PB data by Java. When using > PIPES or Streaming, pipe or socket is not efficient to carry so huge data. >What we want to do: >1 We do not use map/reduce Child JVM to do any data processing, which just > prepares environment, starts C++ mapper, tells mapper which split it should > deal with, and reads report from mapper until that finished. The mapper will > read record, ivoke user defined map, to do partition, write spill, combine > and merge into file.out. We think these operations can be done by C++ code. >2 Reducer is similar to mapper, it was started after sort finished, it > read from sorted files, ivoke user difined reduce, and write to user defined > record writer. >3 We also intend to rewrite shuffle and sort with C++, for efficience and > memory control. >at first, 1 and 2, then 3. >What's the difference with PIPES: >1 Yes, We will reuse most PIPES code. >2 And, We should do it more completely, nothing changed in scheduling and > management, but everything in execution. > *UPDATE:* > Now you can get a test version of HCE from this link > http://docs.google.com/leaf?id=0B5xhnqH1558YZjcxZmI0NzEtODczMy00NmZiLWFkNjAtZGM1MjZkMmNkNWFk&hl=zh_CN&pli=1 > This is a full package with all hadoop source code. > Following document "HCE InstallMenu.pdf" in attachment, you will build and > deploy it in your cluster. > Attachment "HCE Tutorial.pdf" will lead you to write the first HCE program > and give other specifications of the interface. > Attachment "HCE Performance Report.pdf" gives a performance report of HCE > compared to Java MapRed and Pipes. > Any comments are welcomed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6539) Make hadoop-tools/hadoop-pipes Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer moved HADOOP-12114 to MAPREDUCE-6539: -- Affects Version/s: (was: 2.7.0) 2.7.0 Component/s: (was: native) pipes Key: MAPREDUCE-6539 (was: HADOOP-12114) Project: Hadoop Map/Reduce (was: Hadoop Common) > Make hadoop-tools/hadoop-pipes Native code -Wall-clean > -- > > Key: MAPREDUCE-6539 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6539 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 2.7.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Attachments: HADOOP-12114.001.patch, HADOOP-12114.002.patch > > > As we specify -Wall as a default compilation flag, it would be helpful if the > Native code was -Wall-clean -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6539) Make hadoop-tools/hadoop-pipes Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6539: Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-6390 > Make hadoop-tools/hadoop-pipes Native code -Wall-clean > -- > > Key: MAPREDUCE-6539 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6539 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: pipes >Affects Versions: 2.7.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Attachments: HADOOP-12114.001.patch, HADOOP-12114.002.patch > > > As we specify -Wall as a default compilation flag, it would be helpful if the > Native code was -Wall-clean -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6538) Deprecate hadoop-pipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer moved HADOOP-12547 to MAPREDUCE-6538: -- Target Version/s: 3.0.0 (was: 2.8.0) Issue Type: Wish (was: Improvement) Key: MAPREDUCE-6538 (was: HADOOP-12547) Project: Hadoop Map/Reduce (was: Hadoop Common) > Deprecate hadoop-pipes > -- > > Key: MAPREDUCE-6538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6538 > Project: Hadoop Map/Reduce > Issue Type: Wish > Components: pipes >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > Development appears to have stopped on hadoop-pipes upstream for the last few > years, aside from very basic maintenance. Hadoop streaming seems to be a > better alternative, since it supports more programming languages and is > better implemented. > There were no responses to a message on the mailing list asking for users of > Hadoop pipes... and in my experience, I have never seen anyone use this. We > should remove it to reduce our maintenance burden and build times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6538) Deprecate hadoop-pipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6538: Component/s: pipes > Deprecate hadoop-pipes > -- > > Key: MAPREDUCE-6538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6538 > Project: Hadoop Map/Reduce > Issue Type: Wish > Components: pipes >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > Development appears to have stopped on hadoop-pipes upstream for the last few > years, aside from very basic maintenance. Hadoop streaming seems to be a > better alternative, since it supports more programming languages and is > better implemented. > There were no responses to a message on the mailing list asking for users of > Hadoop pipes... and in my experience, I have never seen anyone use this. We > should remove it to reduce our maintenance burden and build times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6537) hadoop pipes examples aren't in the mvn package tar ball
[ https://issues.apache.org/jira/browse/MAPREDUCE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer moved HADOOP-12381 to MAPREDUCE-6537: -- Affects Version/s: (was: 3.0.0) 3.0.0 Target Version/s: 3.0.0 (was: 3.0.0) Component/s: (was: tools/pipes) pipes Key: MAPREDUCE-6537 (was: HADOOP-12381) Project: Hadoop Map/Reduce (was: Hadoop Common) > hadoop pipes examples aren't in the mvn package tar ball > > > Key: MAPREDUCE-6537 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6537 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Kai Sasaki >Priority: Blocker > Attachments: HADOOP-12381.00.patch > > > Hadoop pipes examples are built but never packaged. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6536) hadoop-pipes doesn't use maven properties for openssl
[ https://issues.apache.org/jira/browse/MAPREDUCE-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer moved HADOOP-12518 to MAPREDUCE-6536: -- Affects Version/s: (was: 3.0.0) 3.0.0 Component/s: (was: tools/pipes) pipes Key: MAPREDUCE-6536 (was: HADOOP-12518) Project: Hadoop Map/Reduce (was: Hadoop Common) > hadoop-pipes doesn't use maven properties for openssl > - > > Key: MAPREDUCE-6536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6536 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 3.0.0 > Environment: OS X >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-12518.00.patch, HADOOP-12518.01.patch, > HADOOP-12518.02.patch, HADOOP-12518.03.patch > > > hadoop-common has some maven properties that are used to define where OpenSSL > lives. hadoop-pipes should also use them so we can enable automated testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-4769) Pipes build problem with recent OpenSSL libs
[
https://issues.apache.org/jira/browse/MAPREDUCE-4769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved MAPREDUCE-4769.
-
Resolution: Duplicate
> Pipes build problem with recent OpenSSL libs
>
>
> Key: MAPREDUCE-4769
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4769
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 3.0.0, 2.0.3-alpha
> Environment: CentOS 6.2 x86_64
>Reporter: Andrew Purtell
>
> Seems to be a problem with CMake not figuring that the linker needs -lcrypto
> too with recent OpenSSL. Observed on two CentOS 6 build servers occurring
> after 'yum update' pulled down an openssl-devel update.
> {noformat}
> [exec] Linking CXX executable examples/pipes-sort
> [exec] /usr/bin/cmake -E cmake_link_script
> CMakeFiles/pipes-sort.dir/li
> [exec] /usr/bin/c++-g -Wall -O2 -D_REENTRANT
> -D_FILE_OFFSET_BITS=64
> [exec] /usr/bin/ld: libhadooppipes.a(HadoopPipes.cc.o): undefined
> refer
> [exec] /usr/bin/ld: note: 'BIO_ctrl' is defined in DSO
> /lib64/libcrypto
> [exec] /lib64/libcrypto.so.10: could not read symbols: Invalid
> operatio
> [exec] collect2: ld returned 1 exit status
> [exec] make[3]: *** [examples/pipes-sort] Error 1
> [exec] make[2]: *** [CMakeFiles/pipes-sort.dir/all] Error 2
> [exec] make[1]: *** [all] Error 2
> {noformat}
> This works around the problem:
> {noformat}
> diff --git hadoop-tools/hadoop-pipes/src/CMakeLists.txt hadoop-tools/hadoop-
> index a1ee97d..29cfba7 100644
> --- hadoop-tools/hadoop-pipes/src/CMakeLists.txt
> +++ hadoop-tools/hadoop-pipes/src/CMakeLists.txt
> @@ -71,5 +71,6 @@ add_library(hadooppipes STATIC
> )
> target_link_libraries(hadooppipes
> ${OPENSSL_LIBRARIES}
> +crypto
> pthread
> )
> {noformat}
> Builds with -Pnative won't complete without this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-384) Exception thrown from pipes Map task is not handled properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-384. Resolution: Unresolved Closing this for now. If this issue re-surfaces, please open a new JIRA. > Exception thrown from pipes Map task is not handled properly > > > Key: MAPREDUCE-384 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-384 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Reporter: Amareshwari Sriramadasu > > In Pipes environment, when map task throws an exception it is not killed > immediately. But the tracker is waiting for 603 seconds (10 mins) for the > report and then killing the task. > When I threw exception from word count sample program the maptask exits after > 603 seconds showing the following on console: > 07/09/14 11:09:26 INFO mapred.JobClient: Task Id : > task_200709141017_0002_m_01_1, Status : FAILED > task_200709141017_0002_m_01_1: terminate called after throwing an > instance of 'std::exception' > task_200709141017_0002_m_01_1: what(): St9exception > And the Job UI shows: > Task task_200709141017_0002_m_01_1 failed to report status for 603 > seconds. Killing! > Thus, each map task is taking 10 mins for exiting and is tried 4 times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-1362: Labels: (was: BB2015-05-TBR) > Pipes should be ported to the new mapreduce API > --- > > Key: MAPREDUCE-1362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1362 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: pipes >Reporter: Bassam Tabbara > Attachments: MAPREDUCE-1362-trunk.patch, MAPREDUCE-1362.patch, > MAPREDUCE-1362.patch > > > Pipes is still currently using the old mapred API. This prevents us from > using pipes with HBase's TableInputFormat, HRegionPartitioner, etc. > Here is a rough proposal for how to accomplish this: > * Add a new package org.apache.hadoop.mapreduce.pipes that uses the new > mapred API. > * the new pipes package will run side by side with the old one. old one > should get deprecated at some point. > * the wire protocol used between PipesMapper and PipesReducer and C++ > programs must not change. > * bin/hadoop should support both pipes (old api) and pipes2 (new api) > Does this sound reasonable? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1362) Pipes should be ported to the new mapreduce API
[
https://issues.apache.org/jira/browse/MAPREDUCE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992839#comment-14992839
]
Hadoop QA commented on MAPREDUCE-1362:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 17s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-mapreduce-client in the patch failed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 28s {color}
| {color:red} hadoop-mapreduce-client in the patch failed with JDK v1.8.0_60.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 31s
{color} | {color:red} hadoop-mapreduce-client in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 31s {color}
| {color:red} hadoop-mapreduce-client in the patch failed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s
{color} | {color:red} Patch generated 104 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 18, now 122).
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 114 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 16s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 32s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 16s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 52s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 18s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 20s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
hadoop.mapreduce
[jira] [Updated] (MAPREDUCE-1125) SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
[
https://issues.apache.org/jira/browse/MAPREDUCE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-1125:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
+1 committed
Thanks!
> SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
> ---
>
> Key: MAPREDUCE-1125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.21.0
>Reporter: Simone Leo
>Assignee: Simone Leo
> Labels: BB2015-05-TBR
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-1125-2.patch, MAPREDUCE-1125-3.patch,
> MAPREDUCE-1125.004.patch
>
>
> {noformat}
> *** SerialUtils.hh ***
> float deserializeFloat(InStream& stream);
> *** SerialUtils.cc ***
> void deserializeFloat(float& t, InStream& stream)
> {
> char buf[sizeof(float)];
> stream.read(buf, sizeof(float));
> XDR xdrs;
> xdrmem_create(&xdrs, buf, sizeof(float), XDR_DECODE);
> xdr_float(&xdrs, &t);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3914: Labels: (was: BB2015-05-TBR) > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl >Assignee: Joe Mudd > Fix For: 3.0.0 > > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3914: Resolution: Fixed Fix Version/s: 3.0.0 Target Version/s: (was: ) Status: Resolved (was: Patch Available) +1 committed. Thanks! > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl >Assignee: Joe Mudd > Fix For: 3.0.0 > > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3914: Assignee: Joe Mudd (was: Charles Earl) > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl >Assignee: Joe Mudd > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1125) SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
[
https://issues.apache.org/jira/browse/MAPREDUCE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992817#comment-14992817
]
Hadoop QA commented on MAPREDUCE-1125:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 5s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 5m 31s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-06 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12741423/MAPREDUCE-1125.004.patch
|
| JIRA Issue | MAPREDUCE-1125 |
| Optional Tests | asflicense cc unit javac compile |
| uname | Linux 4773eb2d375b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
|
| git revision | trunk / 286cc64 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6121/testReport/ |
| modules | C: hadoop-tools/hadoop-pipes U: hadoop-tools/hadoop-pipes |
| Max memory used | 228MB |
| Powered by | Apache Yetus http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6121/console |
This message was automatically generated.
> SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
> ---
>
> Key: MAPREDUCE-1125
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1125
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.21.0
>Reporter: Simone Leo
>Assignee: Simone Leo
>
[jira] [Updated] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3914: Assignee: Charles Earl > Mismatched free() / delete / delete [] in HadoopPipes > - > > Key: MAPREDUCE-3914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 0.20.205.0, 0.23.0, 1.0.0 > Environment: Based upon map reduce pipes task executed on Ubuntu 11.10 >Reporter: Charles Earl >Assignee: Charles Earl > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-3914-branch-0.23.patch, > MAPREDUCE-3914-branch-1.0.patch, MAPREDUCE-3914.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When running valgrind on a simple MapReduce pipes job, valgrind identifies a > mismatched new / delete: > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328A5: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1171) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c5b540 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E5D: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1121) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== > ==20394== Mismatched free() / delete / delete [] > ==20394==at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387) > ==20394==by 0x4328AF: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1172) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > ==20394== Address 0x9c7b580 is 0 bytes inside a block of size 131,072 alloc'd > ==20394==at 0x4C2864B: operator new[](unsigned long) > (vg_replace_malloc.c:305) > ==20394==by 0x431E6A: HadoopPipes::runTask(HadoopPipes::Factory const&) > (HadoopPipes.cc:1122) > ==20394==by 0x424C33: main (ProcessRow.cpp:118) > The new [] calls in Lines 1121 and 1122 of HadoopPipes.cc: > bufin = new char[bufsize]; > bufout = new char[bufsize]; > should have matching delete [] calls but are instead bracketed my delete on > lines 1171 and 1172: > delete bufin; > delete bufout; > So these should be replaced by delete[] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3914) Mismatched free() / delete / delete [] in HadoopPipes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992805#comment-14992805
]
Hadoop QA commented on MAPREDUCE-3914:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 5s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hadoop-pipes in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 5m 56s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-06 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12652401/MAPREDUCE-3914.patch |
| JIRA Issue | MAPREDUCE-3914 |
| Optional Tests | asflicense cc unit javac compile |
| uname | Linux 120d39c7f1be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
|
| git revision | trunk / 286cc64 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6119/testReport/ |
| modules | C: hadoop-tools/hadoop-pipes U: hadoop-tools/hadoop-pipes |
| Max memory used | 226MB |
| Powered by | Apache Yetus http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6119/console |
This message was automatically generated.
> Mismatched free() / delete / delete [] in HadoopPipes
> -
>
> Key: MAPREDUCE-3914
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3914
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 0.20.205.0, 0.23.0, 1.0.0
> Environment: Based upon map reduce pipes task executed on Ubuntu 11.10
>
[jira] [Commented] (MAPREDUCE-6535) TaskID default constructor results in NPE on toString()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992181#comment-14992181
]
Rakesh commented on MAPREDUCE-6535:
---
I agree with above solution to make it default to TaskType.REDUCE as we found
this issue in reduce phase while creating custom output format to create
different directories based on the key.
Thanks for prompt response on this. Please let me know if I can provide a fix
on this.
> TaskID default constructor results in NPE on toString()
> ---
>
> Key: MAPREDUCE-6535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.6.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>
> This code will reproduce the issue:
> {code}
> new TaskAttemptID().toString();
> {code}
> The issue is that the default constructor leaves the type {{null}}. The
> {{get()}} in {{CharTaskTypesMaps.getRepresentingCharacter()}} then throws an
> NPE on the null type key.
> The simplest solution would be to only call the {{get()}} on line 288 of
> {{TaskID.java}} if {{type}} is not {{null}} and return some other literal
> otherwise. Since no part of the code is tripping on the NPE, what we choose
> for the literal shouldn't matter. How about "x"?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5485) Allow repeating job commit by extending OutputCommitter API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992102#comment-14992102 ] Junping Du commented on MAPREDUCE-5485: --- Thanks Bikas for review and comments! bq. If the commit actually failed then there does not seem any reason to assume that retrying it will succeed. IMO if the commit reports a failure then AM should fail. Similarly, if a commit failure file exists (from a previous AM version) then the new version of the AM should respect that and fail since the commit has been reported to be failed. There are still reasons that related to AM specific, i.e. previous AM cannot connect to FS (FS or other CloudFS), committer mis-behavior because of getting loaded incorrect (due to classpath or other defect), etc. I think it make sense to do the best effort to retry the commit failure (like other reason to cause AM failure) given the commit is repeatable and all tasks are done successfully. bq. Javadoc could be improved. Inline Yes. I will. bq. num-retries instead of retries? Also, if its num-retries then default should be 0. If its num-attempts then default should be 1. Ok. I will update to attempts. default to be 1 means no retry to keep consistent with previous behavior. bq. Retry count checking code in the catch block subsumes the check retry count check in the while block? I don't think so. Can you take a look at it again? bq. The previous operation could delete the path after the if check has succeeded. So we probably also need to catch FileNotFoundException exception class here and ignore it if repeatableCommit is true. That's good point. Will fix it. bq. Do testcases need an @Test annotation? No. The test class extends TestCase, so all method start with "test" will be executed automatically. bq. firstTimeFail is probably a more clear name for what its doing - failing on the first attempt. Would be good to have a test that version 2 and retry = 1 will fail also. Testcases missing for specific changes in FileOutputCommitter for create/delete operation changes? Sounds good. Will fix/add later. > Allow repeating job commit by extending OutputCommitter API > --- > > Key: MAPREDUCE-5485 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5485 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.1.0-beta >Reporter: Nemon Lou >Assignee: Junping Du >Priority: Critical > Attachments: MAPREDUCE-5485-demo-2.patch, MAPREDUCE-5485-demo.patch, > MAPREDUCE-5485-v1.patch > > > There are chances MRAppMaster crush during job committing,or NodeManager > restart cause the committing AM exit due to container expire.In these cases > ,the job will fail. > However,some jobs can redo commit so failing the job becomes unnecessary. > Let clients tell AM to allow redo commit or not is a better choice. > This idea comes from Jason Lowe's comments in MAPREDUCE-4819 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991825#comment-14991825
]
Sunil G commented on MAPREDUCE-5870:
Test case of {{TestUberAM}} is passing locally and also its fine for JDK 1.7.
But Jenkins reports failure for JDK 1.8. I will check jenkins again to see the
results.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991564#comment-14991564
]
Hadoop QA commented on MAPREDUCE-5870:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 51s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 57s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 52s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s
{color} | {color:red} Patch generated 9 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 381, now 384).
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 7s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 47s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 4s {color} |
{color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 121m 0s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 19s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 12s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 19s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.7.0_79.
[jira] [Commented] (MAPREDUCE-5485) Allow repeating job commit by extending OutputCommitter API
[
https://issues.apache.org/jira/browse/MAPREDUCE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991486#comment-14991486
]
Bikas Saha commented on MAPREDUCE-5485:
---
I dont think AM should reboot to retry after commit failure. Commit repeatable
implies that a commit can be retried if it was stopped while in progress. If
the commit actually failed then there does not seem any reason to assume that
retrying it will succeed. IMO if the commit reports a failure then AM should
fail. Similarly, if a commit failure file exists (from a previous AM version)
then the new version of the AM should respect that and fail since the commit
has been reported to be failed.
So, IMO, we are looking to change the code here. Thoughts?
{code} //The commit is still pending, commit error
shutDownMessage =
"Job commit from a prior MRAppMaster attempt is " +
"potentially in progress. Preventing multiple commit executions";
forcedState = JobStateInternal.ERROR;
{code}
Javadoc could be improved. Inline
{code}
/**
+ * Is repeatable job commit supported so that job commit can be repeated
again
+ * from previous failures. Returns true if an in-progress job commit
can be retried. If the MR AM is re-run then it will check this value to
determine if it can retry an in-progress commit that was started by a previous
version. Note that in rare scenarios, the previous AM version might still be
running at that time, due to system anomalies. Hence if this method returns
true then the retry commit operation should be able to run concurrently with
the previous operation.
+ *
+ * If repeatable job commit is supported, job restart can tolerant previous
+ * failures during job is committing.
+ *
+ * By default, it is not supported. Extended classes (like:
+ * FileOutputCommitter) should explicitly override it if provide support. <<
if they support it
+ *
+ * @param jobContext
+ * Context of the job whose output is being written.
+ * @return true repeatable job commit is supported,
+ * false otherwise
+ */
{code}
num-retries instead of retries? Also, if its num-retries then default should be
0. If its num-attempts then default should be 1.
{code}+ public static final String FILEOUTPUTCOMMITTER_FAILURE_RETRIES =
+ "mapreduce.fileoutputcommitter.failures.retry";{code}
Retry count checking code in the catch block subsumes the check retry count
check in the while block?
{code}+while (jobCommitNotFinished && (retries++ < retriesOnFailure)) {
...
+ } catch (Exception e) {
+if (retries >= retriesOnFailure) {{code}
The previous operation could delete the path after the if check has succeeded.
So we probably also need to catch FileNotFoundException exception class here
and ignore it if repeatableCommit is true.
{code}+ // if job allow repeatable commit and pendingJobAttemptsPath could
be
+ // deleted by previous AM, we do nothing here.
+ if (isCommitJobRepeatable(context) &&
!fs.exists(pendingJobAttemptsPath)) {
+return;
+ }{code}
Do testcases need an @Test annotation?
{code}+ public void testCommitterWithFailureV1() throws Exception {
+testCommitterWithFailureInternal(1);{code}
firstTimeFail is probably a more clear name for what its doing - failing on the
first attempt.
{code}+boolean firstTimeCommit = true;{code}
Would be good to have a test that version 2 and retry = 1 will fail also.
Testcases missing for specific changes in FileOutputCommitter for create/delete
operation changes?
> Allow repeating job commit by extending OutputCommitter API
> ---
>
> Key: MAPREDUCE-5485
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5485
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 2.1.0-beta
>Reporter: Nemon Lou
>Assignee: Junping Du
>Priority: Critical
> Attachments: MAPREDUCE-5485-demo-2.patch, MAPREDUCE-5485-demo.patch,
> MAPREDUCE-5485-v1.patch
>
>
> There are chances MRAppMaster crush during job committing,or NodeManager
> restart cause the committing AM exit due to container expire.In these cases
> ,the job will fail.
> However,some jobs can redo commit so failing the job becomes unnecessary.
> Let clients tell AM to allow redo commit or not is a better choice.
> This idea comes from Jason Lowe's comments in MAPREDUCE-4819
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
