[jira] [Updated] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated MAPREDUCE-6068:
-
Attachment: MAPREDUCE-6068.003.patch
Thanks for the review and nice catch [~djp] ! Update the patch.
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Labels: BB2015-05-RFC
> Attachments: MAPREDUCE-6068.002.patch, MAPREDUCE-6068.003.patch,
> MAPREDUCE-6068.v1.patch
>
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088723#comment-15088723
]
Binglin Chang commented on MAPREDUCE-6068:
--
Looked findbug warnings and test errors, all unrelated to this code.
[~djp] Could you help review this?
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Labels: BB2015-05-RFC
> Attachments: MAPREDUCE-6068.002.patch, MAPREDUCE-6068.v1.patch
>
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15084984#comment-15084984
]
Binglin Chang commented on MAPREDUCE-6068:
--
It's a very simple one line fix, not sure a regression test is needed.
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Labels: BB2015-05-RFC
> Attachments: MAPREDUCE-6068.002.patch, MAPREDUCE-6068.v1.patch
>
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6417) MapReduceClient's primitives.h is toxic and should be extirpated
[
https://issues.apache.org/jira/browse/MAPREDUCE-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055444#comment-15055444
]
Binglin Chang commented on MAPREDUCE-6417:
--
I think it's probably OK to remove them, most of those method are added because
of old compilers(gcc3) or glibc, which have inefficient memcpy & memcmp, like
which we used in 2011 in our production environment, I recall it's the main
reason adding them.
At least on macosx clang and gcc4.4+, I see memcpy are pretty fast.
So if we are drop old compilers/oses support and need to support sparc/arm, use
system library is better.
> MapReduceClient's primitives.h is toxic and should be extirpated
>
>
> Key: MAPREDUCE-6417
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6417
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: client
>Affects Versions: 3.0.0
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Blocker
> Attachments: MAPREDUCE-6417.001.patch
>
>
> MapReduceClient's primitives.h attempts to provide optimised versions of
> standard library memory copy and comparison functions. It has been the
> subject of several portability-related bugs:
> * HADOOP-11505 hadoop-mapreduce-client-nativetask uses bswap where be32toh is
> needed, doesn't work on non-x86
> * HADOOP-11665 Provide and unify cross platform byteorder support in native
> code
> * MAPREDUCE-6397 MAPREDUCE makes many endian-dependent assumptions
> * HADOOP-11484 hadoop-mapreduce-client-nativetask fails to build on ARM
> AARCH64 due to x86 asm statements
> At present it only works on x86 and ARM64 as it lacks definitions for bswap
> and bswap64 for any platforms other than those.
> However it has even more serious problems on non-x86 architectures, for
> example on SPARC simple_memcpy simply doesn't work at all:
> {code}
> $ cat bang.cc
> #include
> #define SIMPLE_MEMCPY
> #include "primitives.h"
> int main(int argc, char **argv)
> {
> char b1[9];
> char b2[9];
> simple_memcpy(b2, b1, sizeof(b1));
> }
> $ gcc -o bang bang.cc && ./bang
> Bus Error (core dumped)
> {code}
> That's because simple_memcpy does pointer fiddling that results in misaligned
> accesses, which are illegal on SPARC.
> fmemcmp is also broken. Even if a definition of bswap is provided, on
> big-endian architectures the result is simply wrong because of its
> unconditional use of bswap:
> {code}
> $ cat thud.cc
> #include
> #include
> #include "primitives.h"
> int main(int argc, char **argv)
> {
> char a[] = { 0,1,2,0 };
> char b[] = { 0,2,1,0 };
> printf("%lld %d\n", fmemcmp(a, b, sizeof(a), memcmp(a, b, sizeof(a;
> }
> $ g++ -o thud thud.cc && ./thud
> 65280 -1
> {code}
> And in addition fmemcmp suffers from the same misalignment issues as
> simple_memcpy and coredumps on SPARC when asked to compare odd-sized buffers.
> primitives.h provides the following functions:
> * bswap - used in 12 files in MRC but as HADOOP-11505 points out, mostly
> incorrectly as it takes no account of platform endianness
> * bswap64 - used in 4 files in MRC, same comments as per bswap apply
> * simple_memcpy - used in 3 files in MRC, should be replaced with the
> standard memcpy
> * fmemcmp - used in 1 file, should be replaced with the standard memcmp
> * fmemeq - used in 1 file, should be replaced with the standard memcmp
> * frmemeq - not used at all, should just be removed
> *Summary*: primitives.h should simply be deleted and replaced with the
> standard memory copy & compare functions, or with thin wrappers around them
> where the APIs are different.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740031#comment-14740031 ] Binglin Chang commented on MAPREDUCE-6241: -- The code is basically copied from https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c which checks for __GNUC__ > Native compilation fails for Checksum.cc due to an incompatibility of > assembler register constraint for PowerPC > > > Key: MAPREDUCE-6241 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0, 2.6.0 > Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux > gcc (Debian 4.9.1-19) > protobuf 2.6.1 > OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2) > OpenJDK Zero VM (build 24.65-b04, interpreted mode) > source was cloned (and updated) from Apache-Hadoop's git repository >Reporter: Stephan Drescher >Assignee: Binglin Chang > Labels: BB2015-05-TBR, features > Attachments: MAPREDUCE-6241.001.patch, MAPREDUCE-6241.002.patch, > MAPREDUCE-6241.003.patch > > > Issue when using assembler code for performance optimization on the powerpc > platform (compiled for 32bit) > mvn compile -Pnative -DskipTests > [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY > -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native > -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux > -isystem > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include > -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > [exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target > 'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed > [exec] make[2]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] CMakeFiles/Makefile2:95: recipe for target > 'CMakeFiles/nativetask.dir/all' failed > [exec] make[1]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] Makefile:76: recipe for target 'all' failed > [exec] > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc: > In function ‘void NativeTask::init_cpu_support_flag()’: > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc:611:14: > error: impossible register constraint in ‘asm’ > --> > "popl %%ebx" : "=a" (eax), [ebx] "=r"(ebx), "=c"(ecx), "=d"(edx) : "a" > (eax_in) : "cc"); > <-- -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6346) mapred.nativetask.kvtest.KVTest crashes on PPC64LE
[
https://issues.apache.org/jira/browse/MAPREDUCE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560437#comment-14560437
]
Binglin Chang commented on MAPREDUCE-6346:
--
Yes, that should work
> mapred.nativetask.kvtest.KVTest crashes on PPC64LE
> --
>
> Key: MAPREDUCE-6346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6346
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
> Environment: RHEL 7.1 - PPC64 LE - OpenJDK
> rhel-2.5.5.1.ael7b_1-ppc64le u79-b14
>Reporter: Tony Reix
> Attachments: TR
>
>
> Test org.apache.hadoop.mapred.nativetask.kvtest.KVTest (and 5 or 6 other
> tests) crashes on PPC64LE .
>
> 15/04/28 10:46:06 INFO Mid-spill: { id: 4, collect: 245 ms, in-memory sort:
> 32 ms, in-memory records: 48202, merge&spill: 80 ms, uncompressed size:
> 5031451, real size: 3739319 path:
> /tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local408221154_0008/attempt_local408221154_0008_m_00_0/output/spill4.out
> }
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x3fff6c7d8e50, pid=945, tid=70366264881616
> #
> # JRE version: OpenJDK Runtime Environment (7.0_79-b14) (build
> 1.7.0_79-mockbuild_2015_04_10_10_48-b00)
> # Java VM: OpenJDK 64-Bit Server VM (24.79-b02 mixed mode linux-ppc64
> compressed oops)
> # Derivative: IcedTea 2.5.5
> # Distribution: Built on Red Hat Enterprise Linux Server release 7.1 (Maipo)
> (Fri Apr 10 10:48:01 EDT 2015)
> # Problematic frame:
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
> #
> # Core dump written. Default location:
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/core
> or core.945
> #
> # An error report file with more information is saved as:
> # /tmp/jvm-945/hs_error.log
> #
> # If you would like to submit a bug report, please include
> # instructions on how to reproduce the bug and visit:
> # http://icedtea.classpath.org/bugzilla
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> /bin/sh: line 1: 945 Aborted (core dumped)
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/jre/bin/java
> -Xmx4096m -XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefirebooter9078773752877532263.jar
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire4138802116387705281tmp
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire_01525011254551870798tmp
> /tmp/jvm-945/hs_error.log :
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6346) mapred.nativetask.kvtest.KVTest crashes on PPC64LE
[
https://issues.apache.org/jira/browse/MAPREDUCE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559195#comment-14559195
]
Binglin Chang commented on MAPREDUCE-6346:
--
See the TR file you uploaded:
{noformat}
15/05/22 08:05:33 INFO Native Total MemoryBlockPool: num_partitions 1,
min_block_size 1280K, max_block_size 4096K, capacity 5M
15/05/22 08:05:34 INFO Mid-spill: { id: 0, collect: 1013 ms, in-memory sort: 36
ms, in-memory records: 262144, merge&spill: 80 ms, uncompressed size: 3670018,
real size: 3523082 path:
/tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local739893329_0006/attempt_local739893329_0006_m_00_0/output/spill0.out
}
15/05/22 08:05:35 INFO Mid-spill: { id: 1, collect: 975 ms, in-memory sort: 36
ms, in-memory records: 262144, merge&spill: 80 ms, uncompressed size: 3670018,
real size: 3523317 path:
/tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local739893329_0006/attempt_local739893329_0006_m_00_0/output/spill1.out
}
15/05/22 08:05:36 INFO Mid-spill: { id: 2, collect: 978 ms, in-memory sort: 36
ms, in-memory records: 262144, merge&spill: 80 ms, uncompressed size: 3670018,
real size: 3523469 path:
/tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local739893329_0006/attempt_local739893329_0006_m_00_0/output/spill2.out
}
MapOutputCollector::finalSpill 0
MapOutputCollector::finalSpill 1 IFileWriter::create:filepath
:/tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local739893329_0006/attempt_local739893329_0006_m_00_0/output/file.out
{noformat}
the files spill0.out, spill1.out, spill2.out are input files for
merge(previously written by , so could you stop the process when merge starts,
and compare those files with x86_64 env, to see if they are the same?
> mapred.nativetask.kvtest.KVTest crashes on PPC64LE
> --
>
> Key: MAPREDUCE-6346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6346
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
> Environment: RHEL 7.1 - PPC64 LE - OpenJDK
> rhel-2.5.5.1.ael7b_1-ppc64le u79-b14
>Reporter: Tony Reix
> Attachments: TR
>
>
> Test org.apache.hadoop.mapred.nativetask.kvtest.KVTest (and 5 or 6 other
> tests) crashes on PPC64LE .
>
> 15/04/28 10:46:06 INFO Mid-spill: { id: 4, collect: 245 ms, in-memory sort:
> 32 ms, in-memory records: 48202, merge&spill: 80 ms, uncompressed size:
> 5031451, real size: 3739319 path:
> /tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local408221154_0008/attempt_local408221154_0008_m_00_0/output/spill4.out
> }
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x3fff6c7d8e50, pid=945, tid=70366264881616
> #
> # JRE version: OpenJDK Runtime Environment (7.0_79-b14) (build
> 1.7.0_79-mockbuild_2015_04_10_10_48-b00)
> # Java VM: OpenJDK 64-Bit Server VM (24.79-b02 mixed mode linux-ppc64
> compressed oops)
> # Derivative: IcedTea 2.5.5
> # Distribution: Built on Red Hat Enterprise Linux Server release 7.1 (Maipo)
> (Fri Apr 10 10:48:01 EDT 2015)
> # Problematic frame:
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
> #
> # Core dump written. Default location:
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/core
> or core.945
> #
> # An error report file with more information is saved as:
> # /tmp/jvm-945/hs_error.log
> #
> # If you would like to submit a bug report, please include
> # instructions on how to reproduce the bug and visit:
> # http://icedtea.classpath.org/bugzilla
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> /bin/sh: line 1: 945 Aborted (core dumped)
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/jre/bin/java
> -Xmx4096m -XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefirebooter9078773752877532263.jar
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire4138802116387705281tmp
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire_01525011254551870798tmp
> /tmp/jvm-945/hs_error.log :
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char con
[jira] [Commented] (MAPREDUCE-6346) mapred.nativetask.kvtest.KVTest crashes on PPC64LE
[
https://issues.apache.org/jira/browse/MAPREDUCE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14557945#comment-14557945
]
Binglin Chang commented on MAPREDUCE-6346:
--
Sorry for the late reply, from the bug descriptions, I suspect this is
bigendian/littleendian or unaligned memory access issue.
Not sure how PPC64LE handles unaligned memory access. I don't have a PPC64LE
env,
looks like the crash occours when doing final merge & spill, basically mapper
write mid-spill files, and read them to do merge at last, so could you diff the
generatered mid-spill files to see if they are the same? If not, trace the
IFile writing code?
> mapred.nativetask.kvtest.KVTest crashes on PPC64LE
> --
>
> Key: MAPREDUCE-6346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6346
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
> Environment: RHEL 7.1 - PPC64 LE - OpenJDK
> rhel-2.5.5.1.ael7b_1-ppc64le u79-b14
>Reporter: Tony Reix
> Attachments: TR
>
>
> Test org.apache.hadoop.mapred.nativetask.kvtest.KVTest (and 5 or 6 other
> tests) crashes on PPC64LE .
>
> 15/04/28 10:46:06 INFO Mid-spill: { id: 4, collect: 245 ms, in-memory sort:
> 32 ms, in-memory records: 48202, merge&spill: 80 ms, uncompressed size:
> 5031451, real size: 3739319 path:
> /tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local408221154_0008/attempt_local408221154_0008_m_00_0/output/spill4.out
> }
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x3fff6c7d8e50, pid=945, tid=70366264881616
> #
> # JRE version: OpenJDK Runtime Environment (7.0_79-b14) (build
> 1.7.0_79-mockbuild_2015_04_10_10_48-b00)
> # Java VM: OpenJDK 64-Bit Server VM (24.79-b02 mixed mode linux-ppc64
> compressed oops)
> # Derivative: IcedTea 2.5.5
> # Distribution: Built on Red Hat Enterprise Linux Server release 7.1 (Maipo)
> (Fri Apr 10 10:48:01 EDT 2015)
> # Problematic frame:
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
> #
> # Core dump written. Default location:
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/core
> or core.945
> #
> # An error report file with more information is saved as:
> # /tmp/jvm-945/hs_error.log
> #
> # If you would like to submit a bug report, please include
> # instructions on how to reproduce the bug and visit:
> # http://icedtea.classpath.org/bugzilla
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> /bin/sh: line 1: 945 Aborted (core dumped)
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/jre/bin/java
> -Xmx4096m -XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefirebooter9078773752877532263.jar
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire4138802116387705281tmp
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire_01525011254551870798tmp
> /tmp/jvm-945/hs_error.log :
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6241: - Attachment: MAPREDUCE-6241.001.patch This patch makes Checksum.cc more compatible for compiling, just like code in hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c > Native compilation fails for Checksum.cc due to an incompatibility of > assembler register constraint for PowerPC > > > Key: MAPREDUCE-6241 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 > Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux > gcc (Debian 4.9.1-19) > protobuf 2.6.1 > OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2) > OpenJDK Zero VM (build 24.65-b04, interpreted mode) > source was cloned (and updated) from Apache-Hadoop's git repository >Reporter: Stephan Drescher >Assignee: Binglin Chang >Priority: Minor > Labels: features > Attachments: MAPREDUCE-6241.001.patch > > > Issue when using assembler code for performance optimization on the powerpc > platform (compiled for 32bit) > mvn compile -Pnative -DskipTests > [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY > -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native > -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux > -isystem > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include > -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > [exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target > 'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed > [exec] make[2]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] CMakeFiles/Makefile2:95: recipe for target > 'CMakeFiles/nativetask.dir/all' failed > [exec] make[1]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] Makefile:76: recipe for target 'all' failed > [exec] > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc: > In function ‘void NativeTask::init_cpu_support_flag()’: > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc:611:14: > error: impossible register constraint in ‘asm’ > --> > "popl %%ebx" : "=a" (eax), [ebx] "=r"(ebx), "=c"(ecx), "=d"(edx) : "a" > (eax_in) : "cc"); > <-- -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6241: - Assignee: Binglin Chang Status: Patch Available (was: Open) > Native compilation fails for Checksum.cc due to an incompatibility of > assembler register constraint for PowerPC > > > Key: MAPREDUCE-6241 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 > Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux > gcc (Debian 4.9.1-19) > protobuf 2.6.1 > OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2) > OpenJDK Zero VM (build 24.65-b04, interpreted mode) > source was cloned (and updated) from Apache-Hadoop's git repository >Reporter: Stephan Drescher >Assignee: Binglin Chang >Priority: Minor > Labels: features > > Issue when using assembler code for performance optimization on the powerpc > platform (compiled for 32bit) > mvn compile -Pnative -DskipTests > [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY > -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native > -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux > -isystem > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include > -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > [exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target > 'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed > [exec] make[2]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] CMakeFiles/Makefile2:95: recipe for target > 'CMakeFiles/nativetask.dir/all' failed > [exec] make[1]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] Makefile:76: recipe for target 'all' failed > [exec] > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc: > In function ‘void NativeTask::init_cpu_support_flag()’: > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc:611:14: > error: impossible register constraint in ‘asm’ > --> > "popl %%ebx" : "=a" (eax), [ebx] "=r"(ebx), "=c"(ecx), "=d"(edx) : "a" > (eax_in) : "cc"); > <-- -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang moved HADOOP-11537 to MAPREDUCE-6241: --- Component/s: (was: build) build Target Version/s: 3.0.0 (was: 3.0.0) Affects Version/s: (was: 3.0.0) 3.0.0 Key: MAPREDUCE-6241 (was: HADOOP-11537) Project: Hadoop Map/Reduce (was: Hadoop Common) > Native compilation fails for Checksum.cc due to an incompatibility of > assembler register constraint for PowerPC > > > Key: MAPREDUCE-6241 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 > Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux > gcc (Debian 4.9.1-19) > protobuf 2.6.1 > OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2) > OpenJDK Zero VM (build 24.65-b04, interpreted mode) > source was cloned (and updated) from Apache-Hadoop's git repository >Reporter: Stephan Drescher >Priority: Minor > Labels: features > > Issue when using assembler code for performance optimization on the powerpc > platform (compiled for 32bit) > mvn compile -Pnative -DskipTests > [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY > -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native > -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux > -isystem > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include > -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > [exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target > 'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed > [exec] make[2]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] CMakeFiles/Makefile2:95: recipe for target > 'CMakeFiles/nativetask.dir/all' failed > [exec] make[1]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] Makefile:76: recipe for target 'all' failed > [exec] > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc: > In function ‘void NativeTask::init_cpu_support_flag()’: > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc:611:14: > error: impossible register constraint in ‘asm’ > --> > "popl %%ebx" : "=a" (eax), [ebx] "=r"(ebx), "=c"(ecx), "=d"(edx) : "a" > (eax_in) : "cc"); > <-- -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated MAPREDUCE-6068:
-
Attachment: MAPREDUCE-6068.002.patch
Rebase to trunk
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-6068.002.patch, MAPREDUCE-6068.v1.patch
>
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated MAPREDUCE-6068:
-
Target Version/s: 2.7.0
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-6068.002.patch, MAPREDUCE-6068.v1.patch
>
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6115) TestPipeApplication#testSubmitter fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated MAPREDUCE-6115:
-
Attachment: MAPREDUCE-6115.v1.patch
Simple fix, looks like it's caused by MAPREDUCE-5945
> TestPipeApplication#testSubmitter fails in trunk
>
>
> Key: MAPREDUCE-6115
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6115
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Binglin Chang
>Priority: Minor
> Attachments: MAPREDUCE-6115.v1.patch
>
>
> This can be reproduced locally:
> {code}
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.89 sec <<<
> FAILURE! - in org.apache.hadoop.mapred.pipes.TestPipeApplication
> testSubmitter(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time
> elapsed: 0.061 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:310)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6115) TestPipeApplication#testSubmitter fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang reassigned MAPREDUCE-6115:
Assignee: Binglin Chang
> TestPipeApplication#testSubmitter fails in trunk
>
>
> Key: MAPREDUCE-6115
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6115
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Binglin Chang
>Priority: Minor
>
> This can be reproduced locally:
> {code}
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.89 sec <<<
> FAILURE! - in org.apache.hadoop.mapred.pipes.TestPipeApplication
> testSubmitter(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time
> elapsed: 0.061 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:310)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6115) TestPipeApplication#testSubmitter fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated MAPREDUCE-6115:
-
Status: Patch Available (was: Open)
> TestPipeApplication#testSubmitter fails in trunk
>
>
> Key: MAPREDUCE-6115
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6115
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Binglin Chang
>Priority: Minor
>
> This can be reproduced locally:
> {code}
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.89 sec <<<
> FAILURE! - in org.apache.hadoop.mapred.pipes.TestPipeApplication
> testSubmitter(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time
> elapsed: 0.061 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:310)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6078) native-task: fix gtest build on macosx
[ https://issues.apache.org/jira/browse/MAPREDUCE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6078: - Attachment: MAPREDUCE-6078.v2.patch Right, should upgrade to gtest1.7 instead, update the patch, manually tested on macoxs, now can build successfully. > native-task: fix gtest build on macosx > -- > > Key: MAPREDUCE-6078 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6078 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Trivial > Attachments: MAPREDUCE-6078.v1.patch, MAPREDUCE-6078.v2.patch > > > Try compile the HEAD code in macos but failed, looks like MAPREDUCE-5977 > separate gtest compile from nttest in order to surpress compile warnings, but > it forget to add addition compile flags added to nttest is also required for > gtest build, this patch fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated MAPREDUCE-6068:
-
Attachment: MAPREDUCE-6068.v1.patch
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-6068.v1.patch
>
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated MAPREDUCE-6068:
-
Status: Patch Available (was: Open)
Add some trace log:
{code}
2014-09-11 17:49:38,282 WARN [pool-65-thread-1] mapred.Merger
(Merger.java:next(566)) - totalBytesProcessed: 12076353 progPerByte:
8.294227E-8 result: 1.0016402
2014-09-11 17:49:38,282 WARN [pool-65-thread-1] util.Progress
(Progress.java:set(180)) - Illegal progress value found, progress is larger
than 1. Progress will be changed to 1
{code}
Merger use estimated progress calculation, when data size is large, it is
possible that the estimated value is a little above 1.0.
The patch makes sure Merger will never set progress larger than 1.0.
More thoughts(not related to this jira) on this part of code, the current code
set progress 1 or 2 times for each k/v pairs, Progress.set is synchronized, and
the calculation and code in Progress.set involve a lot floating point
operations, this part of code may impact performance a little, though I have
not tested.
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang reassigned MAPREDUCE-6068:
Assignee: Binglin Chang
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6078) native-task: fix gtest build on macosx
[ https://issues.apache.org/jira/browse/MAPREDUCE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128250#comment-14128250 ] Binglin Chang commented on MAPREDUCE-6078: -- Looks like cmake 2.6(which is in currently build requirement) does require this? The doc is a little different, I guess I could verify this in a cmake2.6 env(need to get a old centos env..) But I think to careful is good. http://www.cmake.org/cmake/help/cmake2.6docs.html#command:if Evaluates the given expression. If the result is true, the commands in the THEN section are invoked. Otherwise, the commands in the else section are invoked. The elseif and else sections are optional. You may have multiple elseif clauses. **Note that the same expression must be given to if, and endif.** Long expressions can be used and the order or precedence is that the EXISTS, COMMAND, and DEFINED operators will be evaluated first. Then any EQUAL, LESS, GREATER, STRLESS, STRGREATER, STREQUAL, MATCHES will be evaluated. Then NOT operators and finally AND, OR operators will be evaluated. Possible expressions are: > native-task: fix gtest build on macosx > -- > > Key: MAPREDUCE-6078 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6078 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Trivial > Attachments: MAPREDUCE-6078.v1.patch > > > Try compile the HEAD code in macos but failed, looks like MAPREDUCE-5977 > separate gtest compile from nttest in order to surpress compile warnings, but > it forget to add addition compile flags added to nttest is also required for > gtest build, this patch fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6078) native-task: fix gtest build on macosx
[ https://issues.apache.org/jira/browse/MAPREDUCE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127974#comment-14127974 ] Binglin Chang commented on MAPREDUCE-6078: -- What do you mean? I guess thats the weird cmake syntax. > native-task: fix gtest build on macosx > -- > > Key: MAPREDUCE-6078 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6078 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Trivial > Attachments: MAPREDUCE-6078.v1.patch > > > Try compile the HEAD code in macos but failed, looks like MAPREDUCE-5977 > separate gtest compile from nttest in order to surpress compile warnings, but > it forget to add addition compile flags added to nttest is also required for > gtest build, this patch fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6078) native-task: fix gtest build on macosx
[ https://issues.apache.org/jira/browse/MAPREDUCE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6078: - Attachment: MAPREDUCE-6078.v1.patch Should found this issue earlier, we can resolve it later after the the merge is done. > native-task: fix gtest build on macosx > -- > > Key: MAPREDUCE-6078 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6078 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Trivial > Attachments: MAPREDUCE-6078.v1.patch > > > Try compile the HEAD code in macos but failed, looks like MAPREDUCE-5977 > separate gtest compile from nttest in order to surpress compile warnings, but > it forget to add addition compile flags added to nttest is also required for > gtest build, this patch fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6078) native-task: fix gtest build on macosx
Binglin Chang created MAPREDUCE-6078: Summary: native-task: fix gtest build on macosx Key: MAPREDUCE-6078 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6078 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Trivial Try compile the HEAD code in macos but failed, looks like MAPREDUCE-5977 separate gtest compile from nttest in order to surpress compile warnings, but it forget to add addition compile flags added to nttest is also required for gtest build, this patch fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6067) native-task: fix some counter issues
[ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122503#comment-14122503 ] Binglin Chang commented on MAPREDUCE-6067: -- I have committed this. > native-task: fix some counter issues > > > Key: MAPREDUCE-6067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-6067.v1.patch, MAPREDUCE-6067.v2.patch, > MAPREDUCE-6067.v3.patch, MAPREDUCE-6067.v4.patch, MAPREDUCE-6067.v5.patch, > native-counters.html, trunk-counters.html > > > After running a terasort, I see the spilled records counter at 5028651606, > which is about half what I expected to see. Using the non-native collector I > see the expected count of 100. It seems the correct number of records > were indeed spilled, because the job's output record count is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6067) native-task: fix some counter issues
[ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6067: - Summary: native-task: fix some counter issues (was: native-task: spilled records counter is incorrect) > native-task: fix some counter issues > > > Key: MAPREDUCE-6067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-6067.v1.patch, MAPREDUCE-6067.v2.patch, > MAPREDUCE-6067.v3.patch, MAPREDUCE-6067.v4.patch, MAPREDUCE-6067.v5.patch, > native-counters.html, trunk-counters.html > > > After running a terasort, I see the spilled records counter at 5028651606, > which is about half what I expected to see. Using the non-native collector I > see the expected count of 100. It seems the correct number of records > were indeed spilled, because the job's output record count is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122496#comment-14122496 ] Binglin Chang commented on MAPREDUCE-6067: -- Thanks for the review and tests, guys. Will commit soon. > native-task: spilled records counter is incorrect > - > > Key: MAPREDUCE-6067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-6067.v1.patch, MAPREDUCE-6067.v2.patch, > MAPREDUCE-6067.v3.patch, MAPREDUCE-6067.v4.patch, MAPREDUCE-6067.v5.patch, > native-counters.html, trunk-counters.html > > > After running a terasort, I see the spilled records counter at 5028651606, > which is about half what I expected to see. Using the non-native collector I > see the expected count of 100. It seems the correct number of records > were indeed spilled, because the job's output record count is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6067: - Attachment: MAPREDUCE-6067.v5.patch Attach new patch, changes: 1. move all added counters to MapOutputCollector 2. remove RECUDE_INPUT_RECORDS verify when doing combine test 3. inline runNativeTest and runNormalTest 4. add mapoutput file path to log 5. mid-spill sometimes is actually final-spill, make log changes to address this > native-task: spilled records counter is incorrect > - > > Key: MAPREDUCE-6067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-6067.v1.patch, MAPREDUCE-6067.v2.patch, > MAPREDUCE-6067.v3.patch, MAPREDUCE-6067.v4.patch, MAPREDUCE-6067.v5.patch, > native-counters.html, trunk-counters.html > > > After running a terasort, I see the spilled records counter at 5028651606, > which is about half what I expected to see. Using the non-native collector I > see the expected count of 100. It seems the correct number of records > were indeed spilled, because the job's output record count is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
[
https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122351#comment-14122351
]
Binglin Chang commented on MAPREDUCE-6067:
--
bq. I did saw some seeming trivival minor changes, like one or two lines of
change, will impact the CPU cache efficiency.
Never mind, I already move those counters to mapoutputcollector
bq. think operation team will want this.
In java, output file path is not printed also, but I think I can add them back
to mid-spill and final-spill log
bq. merge log, spill log, I do think we should keep them
I just move those info to mid-spill and final-spill log in a more formatted
way, not really deleting them
bq. I think more changes are better than this. Some test maven plugin allows
tests to be runned in parallel, sharing mutable stuff in Test Cases is wrong.
Sure, will make the change
bq. Yes, but you only have a declaration, but never use it?
Here, the variable name is different
{code}
+
+ _outputBytes->increase(realOutputSize);
{code}
> native-task: spilled records counter is incorrect
> -
>
> Key: MAPREDUCE-6067
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-6067.v1.patch, MAPREDUCE-6067.v2.patch,
> MAPREDUCE-6067.v3.patch, MAPREDUCE-6067.v4.patch, native-counters.html,
> trunk-counters.html
>
>
> After running a terasort, I see the spilled records counter at 5028651606,
> which is about half what I expected to see. Using the non-native collector I
> see the expected count of 100. It seems the correct number of records
> were indeed spilled, because the job's output record count is correct.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
[
https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122316#comment-14122316
]
Binglin Chang commented on MAPREDUCE-6067:
--
Thanks for the comments Manu and Sean.
bq. Since the new added line lies in the critical path of performance. May be
it is risky to change here?
1. The added code just increase 2 counters, the performance impact should be
negligible, and we need a way to get the counter number right? java side also
increase counters for every kv pair.
bq. For these two, they are not inited in the constructor.
2. I will move all the added counters to MapOutputCollector, and added them to
constructor
bq. Log is removed due to it is too noisy? The log was added after real pain
and practices in troubleshootings
3. In common practice, log added when troubleshooting bug should be remove
after the bug is found and fixed. Too bad we don't have debug log level. I
removed 2 logs(spill file path and buffer is full), spill file path is useful
only for debugging only, and buffer is full is very common situation, and is
always implied by later mid-spill log, don't see why it is useful anymore when
the bug is already gone.
bq. Can we make normalJob and nativeJob local var instead of field member?
Since it is a test file, Test case should share nothing except immutable
things defined in test setup.
4. Simple use local var doesn't work, if we want to eliminate field member, we
need a way to get both outputpath and job from sub-methods(runNativeTest,
runNormalTest), perhaps just inline them into test method, this is lot change
compare to current approach, if you think it's OK, I will make more aggressive
changes.
bq. By deleting the cleanup code, have you confirmed that it will leak any
garbage file on local disk?
5 . I see you already add cleanUp method to remove root dir, so the old
cleanup code is removed
bq. It is confusing by looking at this line of change.
6. It's confusing, again see comment 4
bq. Maybe we can add some message in assert
7. When assertion fails, the failed line(number?) is showed, which already have
Counter name info, I though that should be enough. OK, will add some message
bq. I cannot find you use this counter anywhere.
Here, it's in c++ code
{code}
@@ -157,6 +158,8 @@ void MapOutputCollector::configure(Config * config) {
Counter * spilledRecord =
NativeObjectFactory::GetCounter(TaskCounters::TASK_COUNTER_GROUP,
TaskCounters::SPILLED_RECORDS);
+ Counter * materializedBytes =
NativeObjectFactory::GetCounter(TaskCounters::TASK_COUNTER_GROUP,
+ TaskCounters::MAP_OUTPUT_MATERIALIZED_BYTES);
ICombineRunner * combiner = NULL;
if (NULL != config->get(NATIVE_COMBINER)
{code}
> native-task: spilled records counter is incorrect
> -
>
> Key: MAPREDUCE-6067
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Attachments: MAPREDUCE-6067.v1.patch, MAPREDUCE-6067.v2.patch,
> MAPREDUCE-6067.v3.patch, MAPREDUCE-6067.v4.patch, native-counters.html,
> trunk-counters.html
>
>
> After running a terasort, I see the spilled records counter at 5028651606,
> which is about half what I expected to see. Using the non-native collector I
> see the expected count of 100. It seems the correct number of records
> were indeed spilled, because the job's output record count is correct.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6067: - Attachment: MAPREDUCE-6067.v4.patch Fix a minor bug in MAP_OUTPUT_BYTES counter calculation. > native-task: spilled records counter is incorrect > - > > Key: MAPREDUCE-6067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-6067.v1.patch, MAPREDUCE-6067.v2.patch, > MAPREDUCE-6067.v3.patch, MAPREDUCE-6067.v4.patch, native-counters.html, > trunk-counters.html > > > After running a terasort, I see the spilled records counter at 5028651606, > which is about half what I expected to see. Using the non-native collector I > see the expected count of 100. It seems the correct number of records > were indeed spilled, because the job's output record count is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6067: - Attachment: MAPREDUCE-6067.v3.patch The earlier patch can't apply, rebase to lastest brach HEAD > native-task: spilled records counter is incorrect > - > > Key: MAPREDUCE-6067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-6067.v1.patch, MAPREDUCE-6067.v2.patch, > MAPREDUCE-6067.v3.patch, native-counters.html, trunk-counters.html > > > After running a terasort, I see the spilled records counter at 5028651606, > which is about half what I expected to see. Using the non-native collector I > see the expected count of 100. It seems the correct number of records > were indeed spilled, because the job's output record count is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6067: - Attachment: MAPREDUCE-6067.v2.patch Update the patch, changes: 1. add counter support for MAP_OUTPUT_RECORDS, MAP_OUTPUT_BYTES, MAP_OUTPUT_MATERIALIZED_BYTES 2. add counter verification(only verify MAP_OUTPUT_RECORDS and REDUCE_INPUT_GROUPS and REDUCE_INPUT_RECORDS), due to different serialization methods MAP_OUTPUT_BYTES is not the same so it is not verified, due to map output record order may not the same, and compression, MAP_OUTPUT_MATERIALIZED_BYTES also may not be the same. 3. update to TaskCounter from old API's Task$Counter 4. remove some of the unused counters 5. clean up some log > native-task: spilled records counter is incorrect > - > > Key: MAPREDUCE-6067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-6067.v1.patch, MAPREDUCE-6067.v2.patch, > native-counters.html, trunk-counters.html > > > After running a terasort, I see the spilled records counter at 5028651606, > which is about half what I expected to see. Using the non-native collector I > see the expected count of 100. It seems the correct number of records > were indeed spilled, because the job's output record count is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6058) native-task: KVTest and LargeKVTest should check mr job is sucessful
[ https://issues.apache.org/jira/browse/MAPREDUCE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120897#comment-14120897 ] Binglin Chang commented on MAPREDUCE-6058: -- Yes, the test requires compression codec to run, which requires libhadoop.so. If the whole project is not compiled with Pnative, I thinks those tests should be skipped. > native-task: KVTest and LargeKVTest should check mr job is sucessful > > > Key: MAPREDUCE-6058 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6058 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-6058.v1.patch, MAPREDUCE-6058.v2.patch, > MAPREDUCE-6058.v3.patch > > > When running KVTest and LargeKVTest, if the job failed for some reason(lack > libhadoop.so etc), both native and normal job failed, and both compare empty > output directory, so the test passes without noticing failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6067: - Attachment: MAPREDUCE-6067.v1.patch The patch should fix spill record error, along within the patch, remove some redundant code in StatusReportChecker.java > native-task: spilled records counter is incorrect > - > > Key: MAPREDUCE-6067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-6067.v1.patch, native-counters.html, > trunk-counters.html > > > After running a terasort, I see the spilled records counter at 5028651606, > which is about half what I expected to see. Using the non-native collector I > see the expected count of 100. It seems the correct number of records > were indeed spilled, because the job's output record count is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang reassigned MAPREDUCE-6067: Assignee: Binglin Chang > native-task: spilled records counter is incorrect > - > > Key: MAPREDUCE-6067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: native-counters.html, trunk-counters.html > > > After running a terasort, I see the spilled records counter at 5028651606, > which is about half what I expected to see. Using the non-native collector I > see the expected count of 100. It seems the correct number of records > were indeed spilled, because the job's output record count is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119375#comment-14119375 ] Binglin Chang commented on MAPREDUCE-6067: -- Thanks for the report Todd. If remember that I have done terasort long time ago, and the counter was accurate, will look into it. > native-task: spilled records counter is incorrect > - > > Key: MAPREDUCE-6067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon > Attachments: native-counters.html, trunk-counters.html > > > After running a terasort, I see the spilled records counter at 5028651606, > which is about half what I expected to see. Using the non-native collector I > see the expected count of 100. It seems the correct number of records > were indeed spilled, because the job's output record count is correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6065) native-task: warnings about illegal Progress values
[ https://issues.apache.org/jira/browse/MAPREDUCE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119219#comment-14119219 ] Binglin Chang commented on MAPREDUCE-6065: -- Hi I looked into the log, looks like both normal and nativetask jobs all print excessive logs, so it may not be a bug introduced by nativetask. > native-task: warnings about illegal Progress values > --- > > Key: MAPREDUCE-6065 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6065 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Manu Zhang > > In running terasort tests, I see a few warnings like this: > 2014-09-02 18:50:34,623 WARN [main] org.apache.hadoop.util.Progress: Illegal > progress value found, progress is larger than 1. Progress will be changed to 1 > It sounds like we're improperly calculating task progress somewhere. We > should fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6056) nativetask: move system test working dir to target dir and cleanup test config xml files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang resolved MAPREDUCE-6056. -- Resolution: Fixed Committed, thanks Manu. > nativetask: move system test working dir to target dir and cleanup test > config xml files > > > Key: MAPREDUCE-6056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6056 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Manu Zhang >Priority: Minor > Attachments: mapreduce-6056-v2.txt, mapreduce-6056-v3.txt, > mapreduce-6056-v4.txt, mapreduce-6056.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6056) nativetask: move system test working dir to target dir and cleanup test config xml files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118018#comment-14118018 ] Binglin Chang commented on MAPREDUCE-6056: -- Patch lgtm, +1, will commit soon. > nativetask: move system test working dir to target dir and cleanup test > config xml files > > > Key: MAPREDUCE-6056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6056 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Manu Zhang >Priority: Minor > Attachments: mapreduce-6056-v2.txt, mapreduce-6056-v3.txt, > mapreduce-6056-v4.txt, mapreduce-6056.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6058) native-task: KVTest and LargeKVTest should check mr job is sucessful
[ https://issues.apache.org/jira/browse/MAPREDUCE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang resolved MAPREDUCE-6058. -- Resolution: Fixed I have committed this, thanks Sean and Todd. > native-task: KVTest and LargeKVTest should check mr job is sucessful > > > Key: MAPREDUCE-6058 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6058 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-6058.v1.patch, MAPREDUCE-6058.v2.patch, > MAPREDUCE-6058.v3.patch > > > When running KVTest and LargeKVTest, if the job failed for some reason(lack > libhadoop.so etc), both native and normal job failed, and both compare empty > output directory, so the test passes without noticing failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6058) native-task: KVTest and LargeKVTest should check mr job is sucessful
[ https://issues.apache.org/jira/browse/MAPREDUCE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6058: - Attachment: MAPREDUCE-6058.v3.patch Thanks for the review and comments Sean, update the patch addressing the comments. > native-task: KVTest and LargeKVTest should check mr job is sucessful > > > Key: MAPREDUCE-6058 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6058 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-6058.v1.patch, MAPREDUCE-6058.v2.patch, > MAPREDUCE-6058.v3.patch > > > When running KVTest and LargeKVTest, if the job failed for some reason(lack > libhadoop.so etc), both native and normal job failed, and both compare empty > output directory, so the test passes without noticing failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6056) nativetask: move system test working dir to target dir and cleanup test config xml files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117830#comment-14117830 ] Binglin Chang commented on MAPREDUCE-6056: -- Sure, prefer not catch exceptions too > nativetask: move system test working dir to target dir and cleanup test > config xml files > > > Key: MAPREDUCE-6056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6056 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Manu Zhang >Priority: Minor > Attachments: mapreduce-6056-v2.txt, mapreduce-6056-v3.txt, > mapreduce-6056.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6056) nativetask: move system test working dir to target dir and cleanup test config xml files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117333#comment-14117333 ] Binglin Chang commented on MAPREDUCE-6056: -- Hi Manu, I see in the patch you ignore all the exceptions, and just log them, why? If exception occurs, it should just fail the test. > nativetask: move system test working dir to target dir and cleanup test > config xml files > > > Key: MAPREDUCE-6056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6056 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Manu Zhang >Priority: Minor > Attachments: mapreduce-6056-v2.txt, mapreduce-6056-v3.txt, > mapreduce-6056.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5977) Fix or suppress native-task gcc warnings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115166#comment-14115166 ] Binglin Chang commented on MAPREDUCE-5977: -- decster:~/projects/hadoop-trunk> git la 2014-08-27 bfd1d75 (Todd Lipcon): MAPREDUCE-6054. native-task: Speed up tests. Contributed by Todd Lipcon. 2014-08-27 fad4524 (Todd Lipcon): MAPREDUCE-5977. Fix or suppress native-task gcc warnings. Contributed by Manu Zhang. > Fix or suppress native-task gcc warnings > > > Key: MAPREDUCE-5977 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5977 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Manu Zhang > Attachments: gcc_compile.log, mapreduce-5977-v2.txt, > mapreduce-5977-v3.txt, mapreduce-5977.txt > > > Currently, building the native task code on gcc 4.8 has a fair number of > warnings. We should fix or suppress them so that new warnings are easier to > see. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5977) Fix or suppress native-task gcc warnings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115162#comment-14115162 ] Binglin Chang commented on MAPREDUCE-5977: -- Haodop has moved to git, see https://wiki.apache.org/hadoop/HowToCommitWithGit > Fix or suppress native-task gcc warnings > > > Key: MAPREDUCE-5977 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5977 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Manu Zhang > Attachments: gcc_compile.log, mapreduce-5977-v2.txt, > mapreduce-5977-v3.txt, mapreduce-5977.txt > > > Currently, building the native task code on gcc 4.8 has a fair number of > warnings. We should fix or suppress them so that new warnings are easier to > see. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6056) nativetask: move system test working dir to target dir and cleanup test config xml files
[
https://issues.apache.org/jira/browse/MAPREDUCE-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115051#comment-14115051
]
Binglin Chang commented on MAPREDUCE-6056:
--
Thanks for the patch Manu, some comments:
1. most test use System.getProperty("test.build.data", "target/test/data") to
locate test work dir, better to follow that rule rather than hard code into code
2. those xml config files require apache license header
> nativetask: move system test working dir to target dir and cleanup test
> config xml files
>
>
> Key: MAPREDUCE-6056
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6056
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Binglin Chang
>Assignee: Manu Zhang
>Priority: Minor
> Attachments: mapreduce-6056-v2.txt, mapreduce-6056.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6056) nativetask: move system test working dir to target dir and cleanup test config xml files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114897#comment-14114897 ] Binglin Chang commented on MAPREDUCE-6056: -- Hi, the patch cannot apply, please check again? > nativetask: move system test working dir to target dir and cleanup test > config xml files > > > Key: MAPREDUCE-6056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6056 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Manu Zhang >Priority: Minor > Attachments: mapreduce-6056.txt > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6058) native-task: KVTest and LargeKVTest should check mr job is sucessful
[ https://issues.apache.org/jira/browse/MAPREDUCE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6058: - Attachment: MAPREDUCE-6058.v2.patch Changes: 1. add assumeTrue to skip tests when native library is not available 2. add more run job checks 3. since test can be automatically skipped, remove test configs in pom.xml > native-task: KVTest and LargeKVTest should check mr job is sucessful > > > Key: MAPREDUCE-6058 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6058 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-6058.v1.patch, MAPREDUCE-6058.v2.patch > > > When running KVTest and LargeKVTest, if the job failed for some reason(lack > libhadoop.so etc), both native and normal job failed, and both compare empty > output directory, so the test passes without noticing failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6058) native-task: KVTest and LargeKVTest should check mr job is sucessful
[ https://issues.apache.org/jira/browse/MAPREDUCE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114797#comment-14114797 ] Binglin Chang commented on MAPREDUCE-6058: -- If we are testing without -Pnative in pom.xml, those tests are skipped, I encounter the error mainly because I run test inside hadoop-mapreduce-client-nativetask(on mac mvn test -Pnative will fail), anyway I think it's good to add assumeTrue, will update the patch soon. > native-task: KVTest and LargeKVTest should check mr job is sucessful > > > Key: MAPREDUCE-6058 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6058 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-6058.v1.patch > > > When running KVTest and LargeKVTest, if the job failed for some reason(lack > libhadoop.so etc), both native and normal job failed, and both compare empty > output directory, so the test passes without noticing failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6058) native-task: KVTest and LargeKVTest should check mr job is sucessful
[ https://issues.apache.org/jira/browse/MAPREDUCE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6058: - Attachment: MAPREDUCE-6058.v1.patch > native-task: KVTest and LargeKVTest should check mr job is sucessful > > > Key: MAPREDUCE-6058 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6058 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-6058.v1.patch > > > When running KVTest and LargeKVTest, if the job failed for some reason(lack > libhadoop.so etc), both native and normal job failed, and both compare empty > output directory, so the test passes without noticing failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-6058) native-task: KVTest and LargeKVTest should check mr job is sucessful
Binglin Chang created MAPREDUCE-6058: Summary: native-task: KVTest and LargeKVTest should check mr job is sucessful Key: MAPREDUCE-6058 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6058 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor When running KVTest and LargeKVTest, if the job failed for some reason(lack libhadoop.so etc), both native and normal job failed, and both compare empty output directory, so the test passes without noticing failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6056) nativetask: move system test working dir to target dir and cleanup test config xml files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113522#comment-14113522 ] Binglin Chang commented on MAPREDUCE-6056: -- OK > nativetask: move system test working dir to target dir and cleanup test > config xml files > > > Key: MAPREDUCE-6056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6056 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Manu Zhang >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6056) nativetask: move system test working dir to target dir and cleanup test config xml files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113419#comment-14113419 ] Binglin Chang commented on MAPREDUCE-6056: -- Most config's are input/output locations which is not useful, key/value type list is fixed, so most of them can be set just in code, right? Most hadoop tests do this. > nativetask: move system test working dir to target dir and cleanup test > config xml files > > > Key: MAPREDUCE-6056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6056 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Manu Zhang >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-6056) nativetask: move system test working dir to target dir and cleanup test config xml files
Binglin Chang created MAPREDUCE-6056: Summary: nativetask: move system test working dir to target dir and cleanup test config xml files Key: MAPREDUCE-6056 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6056 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Binglin Chang Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6054) native-task: speed up test runs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112030#comment-14112030
]
Binglin Chang commented on MAPREDUCE-6054:
--
Patch lgtm, +1
When I run the tests, there are some other issues, will file other jiras for
those
1. I first compile without -Pnative, java mr jobs all failed, but the test
succeeded, have not look into it yet;
2. I compile with -Pnative, java mr jobs succeed, but the log file contains
excessive logs like following, looks like Progress.java:set is called to often
some where
3. test work dir is not in target and not cleaned up after test finishes.
4. test uses lots of xml files as config, in other tests configs are just set
in code.
{noformat}
2014-08-27 16:15:33,130 WARN [pool-65-thread-1] util.Progress
(Progress.java:set(180)) - Illegal progress value found, progress is larger
than 1. Progress will be changed to 1
2014-08-27 16:15:33,130 WARN [pool-65-thread-1] util.Progress
(Progress.java:set(180)) - Illegal progress value found, progress is larger
than 1. Progress will be changed to 1
2014-08-27 16:15:33,130 WARN [pool-65-thread-1] util.Progress
(Progress.java:set(180)) - Illegal progress value found, progress is larger
than 1. Progress will be changed to 1
2014-08-27 16:15:33,130 WARN [pool-65-thread-1] util.Progress
(Progress.java:set(180)) - Illegal progress value found, progress is larger
than 1. Progress will be changed to 1
{noformat}
> native-task: speed up test runs
> ---
>
> Key: MAPREDUCE-6054
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6054
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Attachments: mapreduce-6054.txt
>
>
> Currently the KVTest compatibility test takes so long on my machine that it
> regularly times out maven. We should speed it up.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5992) native-task test logs should not write to console
[ https://issues.apache.org/jira/browse/MAPREDUCE-5992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14108836#comment-14108836 ] Binglin Chang commented on MAPREDUCE-5992: -- Nativetask uses a FILE * LOG_DEVICE as log destination(default stderr), a simple hack is to open a pipe, make write side to LOG_DEVICE, and read side in a java thread log each line in async way, make sure java side can write all logs to the disk before exit > native-task test logs should not write to console > - > > Key: MAPREDUCE-5992 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5992 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon > > Most of our unit tests are configured with a log4j.properties test resource > so they don't spout a bunch of output to the console. We need to do the same > for native-task. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14098364#comment-14098364 ] Binglin Chang commented on MAPREDUCE-6025: -- Patch lgtm, +1 > native-task: fix native library distribution > > > Key: MAPREDUCE-6025 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Manu Zhang >Assignee: Manu Zhang > Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, > mapreduce-6025-v4.txt, mapreduce-6025.txt > > > currently running "mvn install -Pdist" fails and nativetask native library is > not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092417#comment-14092417 ] Binglin Chang commented on MAPREDUCE-6025: -- Hi Sean, as nativetask is part of mapreduce-client and mapreduce-client distribution need to include nativetask libraries, some config requires to be included in mapreduce-client pom(whether it is written explicitly like this patch or inherent from parent pom) > native-task: fix native library distribution > > > Key: MAPREDUCE-6025 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Manu Zhang >Assignee: Manu Zhang > Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, > mapreduce-6025.txt > > > currently running "mvn install -Pdist" fails and nativetask native library is > not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092379#comment-14092379 ] Binglin Chang commented on MAPREDUCE-6025: -- Hi Manu, I see you patch, please add use *cp -R* rather than cp to preserve symlinks > native-task: fix native library distribution > > > Key: MAPREDUCE-6025 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Manu Zhang >Assignee: Manu Zhang > Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, > mapreduce-6025.txt > > > currently running "mvn install -Pdist" fails and nativetask native library is > not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090526#comment-14090526 ] Binglin Chang commented on MAPREDUCE-6025: -- Hi Manu, patch looks good, comments: 1. use space rather than tabs 2. I can see you reuse some code in hadoop-project-dist, I think *cp -R* is much simpler than *tar xxx*, look at hadoop-dist/pom.xml, which already uses cp -r > native-task: fix native library distribution > > > Key: MAPREDUCE-6025 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Manu Zhang >Assignee: Manu Zhang > Attachments: mapreduce-6025-v2.txt, mapreduce-6025.txt > > > currently running "mvn install -Pdist" fails and nativetask native library is > not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[
https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090342#comment-14090342
]
Binglin Chang commented on MAPREDUCE-6025:
--
I am not sure this can work, the current hadoop-dist script only copy
sub-module jars, not native library paths, it does not have
target/hadoop-mapreduce-client-nativetask-${project.version} at all.
Add some script into hadoop-mapreduce-project/pom.xml dist profile like
hadoop-common/hadoop-hdfs(they inherent dist profile from
hadoop-project-dist/pom.xml) would work.
> native-task: fix native library distribution
>
>
> Key: MAPREDUCE-6025
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Attachments: mapreduce-6025.txt
>
>
> currently running "mvn install -Pdist" fails and nativetask native library is
> not distributed to hadoop tar
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090267#comment-14090267 ] Binglin Chang commented on MAPREDUCE-6025: -- bq. adding a dist profile in nativetask which will do so Adding a dist profile in hadoop-mapreduce-project/pom.xml? > native-task: fix native library distribution > > > Key: MAPREDUCE-6025 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Manu Zhang >Assignee: Manu Zhang > Attachments: mapreduce-6025.txt > > > currently running "mvn install -Pdist" fails and nativetask native library is > not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[
https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090260#comment-14090260
]
Binglin Chang commented on MAPREDUCE-6025:
--
bq. Hi Binglin, jars are copied over to
"hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/" as other sub
projects under hadoop-mapreduce-client.
I see, hadoop-mapreduce-dist.xml do this automatically to all sub-modules,
still I think it's better to make native libs first copied in hadoop-mapreduce
project, then dist script well copy them to root project automatically, like
other projects(hadoop-common, hadoop-hdfs).
And the current patch will fail when build without native profile *mvn package
-Pdist -DskipTests* (cause native llib path does not exist)
{noformat}
[exec] $ cp -r
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT/bin
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT/etc
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT/libexec
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT/sbin
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT/share
.
[exec] $ cp -r
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/target/usr/local/lib/lib*
./lib/native/
[exec]
[exec] Failed!
{noformat}
> native-task: fix native library distribution
>
>
> Key: MAPREDUCE-6025
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Attachments: mapreduce-6025.txt
>
>
> currently running "mvn install -Pdist" fails and nativetask native library is
> not distributed to hadoop tar
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089027#comment-14089027 ] Binglin Chang commented on MAPREDUCE-6025: -- Hi Manu, the patch only copies native libs but not jars. I am thinking another way of doing this, follow hadoop-assemblies rules, you can see the contents in hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml > native-task: fix native library distribution > > > Key: MAPREDUCE-6025 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Manu Zhang >Assignee: Manu Zhang > Attachments: mapreduce-6025.txt > > > currently running "mvn install -Pdist" fails and nativetask native library is > not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6006) native-task: add native tests to maven and fix bug in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6006: - Attachment: MAPREDUCE-6006.v3.patch update the patch, forget to pass skipTests env to nttest. > native-task: add native tests to maven and fix bug in pom.xml > - > > Key: MAPREDUCE-6006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6006 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-6006.v1.patch, MAPREDUCE-6006.v2.patch, > MAPREDUCE-6006.v3.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6006) native-task: add native tests to maven and fix bug in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6006: - Attachment: MAPREDUCE-6006.v2.patch Chages: Revert some changes done by MAPREDUCE-5991. To [~mauzhang]: generally the tests(LargeKVTest, KVTest) takes too much memory and time to run, in my default env it produces OOM, (those tests are important, but we may need to make it smaller when run as unit test), and the test output are written to project sources directory rather than target directory, which pollutes project structure. > native-task: add native tests to maven and fix bug in pom.xml > - > > Key: MAPREDUCE-6006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6006 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-6006.v1.patch, MAPREDUCE-6006.v2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6006) native-task: add native tests to maven and fix bug in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6006: - Attachment: MAPREDUCE-6006.v1.patch Attach first version of the patch, changes: 1. bugfix use hadoop-mapreduce-client-common normal jar rather than test-jar 2. invoke nttest in maven native build 3. remove copy test.sh in pom.xml, use cmake configure_file instead, which can pass libjvm.so path to test script. Note that the run *mvn test -Pnative* still fail(or OOM) cause some java tests are still failing, I skip all java tests by running: *mvn test -Pnative -Dtest=ddd* Manually tested on macos and ubuntu, native tests run successfully. > native-task: add native tests to maven and fix bug in pom.xml > - > > Key: MAPREDUCE-6006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6006 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-6006.v1.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5984) native-task: reuse lz4 sources in hadoop-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087308#comment-14087308 ] Binglin Chang commented on MAPREDUCE-5984: -- Thanks Sean and Todd, I have committed this. > native-task: reuse lz4 sources in hadoop-common > --- > > Key: MAPREDUCE-5984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-5984.v1.patch, MAPREDUCE-5984.v2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5984) native-task: reuse lz4 sources in hadoop-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang resolved MAPREDUCE-5984. -- Resolution: Fixed > native-task: reuse lz4 sources in hadoop-common > --- > > Key: MAPREDUCE-5984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-5984.v1.patch, MAPREDUCE-5984.v2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5984) native-task: reuse lz4 sources in hadoop-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-5984: - Summary: native-task: reuse lz4 sources in hadoop-common (was: native-task: upgrade lz4 to lastest version) > native-task: reuse lz4 sources in hadoop-common > --- > > Key: MAPREDUCE-5984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-5984.v1.patch, MAPREDUCE-5984.v2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5987) native-task: Unit test TestGlibCBug fails on ubuntu
[ https://issues.apache.org/jira/browse/MAPREDUCE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087270#comment-14087270 ] Binglin Chang commented on MAPREDUCE-5987: -- At least on my ubuntu env, the bug doesn't show up. ] Sean can you give more comments? > native-task: Unit test TestGlibCBug fails on ubuntu > --- > > Key: MAPREDUCE-5987 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5987 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Sean Zhong >Assignee: Sean Zhong >Priority: Minor > > On ubuntu12, glibc: 2.15-0ubuntu10.3, UT TestGlibCBug fails > [ RUN ] IFile.TestGlibCBug > 14/07/21 15:55:30 INFO TestGlibCBug ./testData/testGlibCBugSpill.out > /home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc:186: > Failure > Value of: realKey > Actual: 1127504685 > Expected: expect[index] > Which is: 4102672832 > [ FAILED ] IFile.TestGlibCBug (0 ms) > [--] 2 tests from IFile (240 ms total) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang resolved MAPREDUCE-6005. -- Resolution: Fixed > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch, > MAPREDUCE-6005.v3.patch, MAPREDUCE-6005.v4.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084351#comment-14084351 ] Binglin Chang commented on MAPREDUCE-6005: -- I have committed this, thanks Sean. > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch, > MAPREDUCE-6005.v3.patch, MAPREDUCE-6005.v4.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5991) native-task should not run unit tests if native profile is not enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14082069#comment-14082069 ] Binglin Chang commented on MAPREDUCE-5991: -- Thanks for the notice Todd, will remember to do that in later jiras. > native-task should not run unit tests if native profile is not enabled > -- > > Key: MAPREDUCE-5991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5991 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-5991.v1.patch > > > Currently, running "mvn test" without the 'native' profile enabled causes all > of the native-task tests to fail. In order to integrate to trunk, we need to > fix this - either using JUnit "Assume" commands in each test that depends on > native code, or disabling the tests from the pom unless -Pnative is specified -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6005: - Attachment: MAPREDUCE-6005.v4.patch Thanks for the comments Sean. I change toString(double) and toString(float) to use Format, which is safe. For other fixed int types, buffer size 32 should be sufficient. > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch, > MAPREDUCE-6005.v3.patch, MAPREDUCE-6005.v4.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6005: - Attachment: MAPREDUCE-6005.v3.patch > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch, > MAPREDUCE-6005.v3.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080524#comment-14080524 ] Binglin Chang commented on MAPREDUCE-6005: -- Hi Sean, bq. About the leak, at src / main / native / src / lib / Merge.cc, there is a similar mem leak, you only fixed the leak in PartitionBucketIterator.cc In merger, all MergeEntryPtr is owned by Merger::_entries, and is deleted in ~Merger at end, so it doesn't require additional care. bq. maybe better use snsprintf(buf_ptr, "%02X", ...) Sure, I will update the patch > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080476#comment-14080476 ] Binglin Chang commented on MAPREDUCE-6005: -- Thanks for the review Sean. bq. I believe we can rename StringUtil::ToString(const void * v, uint32_t len) to StringUtil::md5BinaryToString(const void * v, uint32_t len), and also make ToHex(uint8_t v) private or inlined to md5BinaryToString. I make the function name from ToString to ToHexString, hex means hex format binary string, which is md5's format, this format is not restricted to md5 only, so I think ToHexString is more suitable. About ToHex, using static prefix makes this method private, thats why I add static. bq. There is another leak at Merge What do you mean? I already add delete in the code you specified. Do we need to adding comments only? > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6005: - Attachment: MAPREDUCE-6005.v2.patch > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch, MAPREDUCE-6005.v2.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077684#comment-14077684 ] Binglin Chang commented on MAPREDUCE-6005: -- update patch, changes: 1. fix more valgrind errors in nttest, run nttest now only show errors in libz and libjvm 2. remove some excessive log in test 3. remove java filesystem and related factory methods (getDefaultFsUri, FileSystem::get) 4. FileInputStream/FileOutputStream use open instead of fopen, this is mostly an optimization. 5. add unittest for CounterSet 6. add unittest for ToHexString 7. fix memory leak in PartitionBucket > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-6006) native-task: add native tests to maven and fix bug in pom.xml
Binglin Chang created MAPREDUCE-6006: Summary: native-task: add native tests to maven and fix bug in pom.xml Key: MAPREDUCE-6006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6006 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5991) native-task should not run unit tests if native profile is not enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074187#comment-14074187 ] Binglin Chang commented on MAPREDUCE-5991: -- [~mauzhang], thanks for the comments, you are right. I should use hadoop-mapreduce-client-common normal jar rather than test-jar. I will add the fix into another jira. > native-task should not run unit tests if native profile is not enabled > -- > > Key: MAPREDUCE-5991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5991 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-5991.v1.patch > > > Currently, running "mvn test" without the 'native' profile enabled causes all > of the native-task tests to fail. In order to integrate to trunk, we need to > fix this - either using JUnit "Assume" commands in each test that depends on > native code, or disabling the tests from the pom unless -Pnative is specified -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5987) native-task: Unit test TestGlibCBug fails on ubuntu
[ https://issues.apache.org/jira/browse/MAPREDUCE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074179#comment-14074179 ] Binglin Chang commented on MAPREDUCE-5987: -- bq. memcpy seems to perform better memmov Yes, so we should use memcpy when we can make sure src & dest are not overlap, but in fillGet memmove should be used, whether or not GlibCBug exists. > native-task: Unit test TestGlibCBug fails on ubuntu > --- > > Key: MAPREDUCE-5987 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5987 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Sean Zhong >Assignee: Sean Zhong >Priority: Minor > > On ubuntu12, glibc: 2.15-0ubuntu10.3, UT TestGlibCBug fails > [ RUN ] IFile.TestGlibCBug > 14/07/21 15:55:30 INFO TestGlibCBug ./testData/testGlibCBugSpill.out > /home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc:186: > Failure > Value of: realKey > Actual: 1127504685 > Expected: expect[index] > Which is: 4102672832 > [ FAILED ] IFile.TestGlibCBug (0 ms) > [--] 2 tests from IFile (240 ms total) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5984) native-task: upgrade lz4 to lastest version
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074160#comment-14074160 ] Binglin Chang commented on MAPREDUCE-5984: -- Hi [~cnauroth], the patch reuses lz4 code files in hadoop-common, could you help review the patch? > native-task: upgrade lz4 to lastest version > --- > > Key: MAPREDUCE-5984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-5984.v1.patch, MAPREDUCE-5984.v2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5987) native-task: Unit test TestGlibCBug fails on ubuntu
[ https://issues.apache.org/jira/browse/MAPREDUCE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074157#comment-14074157 ] Binglin Chang commented on MAPREDUCE-5987: -- Hi Sean, I use valgrind to check TestGlibCBug, and find some bugs, one of them is causing TestGlibCBug to fail on ubuntu, create d MAPREDUCE-6005 for this. About TestGlibCBug, I suspect the real cause is bug in ReadBuffer::fillGet, in which I used memcpy rather than memmove to move a range of bytes from buffer end to buffer beginning, the two ranges may overlap, according to the man page of memcpy: bq. If dst and src overlap, behavior is undefined. Applications in which dst and src might overlap should use memmove(3) instead. So the bytes in the buffer may be corrupted. I think that is why it the test fails, it is not related to some glibc bug. > native-task: Unit test TestGlibCBug fails on ubuntu > --- > > Key: MAPREDUCE-5987 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5987 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Sean Zhong >Assignee: Sean Zhong >Priority: Minor > > On ubuntu12, glibc: 2.15-0ubuntu10.3, UT TestGlibCBug fails > [ RUN ] IFile.TestGlibCBug > 14/07/21 15:55:30 INFO TestGlibCBug ./testData/testGlibCBugSpill.out > /home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc:186: > Failure > Value of: realKey > Actual: 1127504685 > Expected: expect[index] > Which is: 4102672832 > [ FAILED ] IFile.TestGlibCBug (0 ms) > [--] 2 tests from IFile (240 ms total) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6005) native-task: fix some valgrind errors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6005: - Attachment: MAPREDUCE-6005.v1.patch Changes: 1. Bugfix: should use memmove instead of memcpy in ReadBuffer::fillGet, this probably cause TestGlibCBug fails on ubuntu, after the patch, TestGlibCBug no longer fails. 2. use open instead of fopen in FileInputStream/FileOutputStream 3. Bugfix: set need to use a comparator 4. Bugfix: StringUtil::ToString(const void * v, uint32_t len) 5. Bugfix: object not free in TestIFile.cc 6. Bugfix: object not free in TestMain.cc > native-task: fix some valgrind errors > -- > > Key: MAPREDUCE-6005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-6005.v1.patch > > > Running test with valgrind shows there are some bugs, this jira try to fix > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-6005) native-task: fix some valgrind errors
Binglin Chang created MAPREDUCE-6005: Summary: native-task: fix some valgrind errors Key: MAPREDUCE-6005 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6005 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Binglin Chang Assignee: Binglin Chang Running test with valgrind shows there are some bugs, this jira try to fix them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5991) native-task should not run unit tests if native profile is not enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang resolved MAPREDUCE-5991. -- Resolution: Fixed Committed to branch. Thanks Todd. > native-task should not run unit tests if native profile is not enabled > -- > > Key: MAPREDUCE-5991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5991 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-5991.v1.patch > > > Currently, running "mvn test" without the 'native' profile enabled causes all > of the native-task tests to fail. In order to integrate to trunk, we need to > fix this - either using JUnit "Assume" commands in each test that depends on > native code, or disabling the tests from the pom unless -Pnative is specified -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5987) native-task: Unit test TestGlibCBug fails on ubuntu
[ https://issues.apache.org/jira/browse/MAPREDUCE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073046#comment-14073046 ] Binglin Chang commented on MAPREDUCE-5987: -- Hi Sean, I don't know why the steps in your comments is related to the test code? The test code just read from a file sequentially and check the data is not corrupted. > native-task: Unit test TestGlibCBug fails on ubuntu > --- > > Key: MAPREDUCE-5987 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5987 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Sean Zhong >Assignee: Sean Zhong >Priority: Minor > > On ubuntu12, glibc: 2.15-0ubuntu10.3, UT TestGlibCBug fails > [ RUN ] IFile.TestGlibCBug > 14/07/21 15:55:30 INFO TestGlibCBug ./testData/testGlibCBugSpill.out > /home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc:186: > Failure > Value of: realKey > Actual: 1127504685 > Expected: expect[index] > Which is: 4102672832 > [ FAILED ] IFile.TestGlibCBug (0 ms) > [--] 2 tests from IFile (240 ms total) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5991) native-task should not run unit tests if native profile is not enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-5991: - Attachment: MAPREDUCE-5991.v1.patch Changes: 1. add hadoop-mapreduce-client-common test jar, so to remove additionalClasspathElements 2. remove system tests in default profile, add them in native profile now just run "mvn test" succeed. note that "mvn test -Pnative" is still failing > native-task should not run unit tests if native profile is not enabled > -- > > Key: MAPREDUCE-5991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5991 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > Attachments: MAPREDUCE-5991.v1.patch > > > Currently, running "mvn test" without the 'native' profile enabled causes all > of the native-task tests to fail. In order to integrate to trunk, we need to > fix this - either using JUnit "Assume" commands in each test that depends on > native code, or disabling the tests from the pom unless -Pnative is specified -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (MAPREDUCE-5991) native-task should not run unit tests if native profile is not enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang reassigned MAPREDUCE-5991: Assignee: Binglin Chang > native-task should not run unit tests if native profile is not enabled > -- > > Key: MAPREDUCE-5991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5991 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Binglin Chang > > Currently, running "mvn test" without the 'native' profile enabled causes all > of the native-task tests to fail. In order to integrate to trunk, we need to > fix this - either using JUnit "Assume" commands in each test that depends on > native code, or disabling the tests from the pom unless -Pnative is specified -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5996) native-task: Rename system tests into standard directory layout
[ https://issues.apache.org/jira/browse/MAPREDUCE-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072827#comment-14072827 ] Binglin Chang commented on MAPREDUCE-5996: -- Oh I see this patch depends on MAPREDUCE-5994, please ignore my comments before. lgtm +1 > native-task: Rename system tests into standard directory layout > --- > > Key: MAPREDUCE-5996 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5996 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: mapreduce-5996.txt > > > Currently there are a number of tests in src/java/system. This confuses IDEs > which think that the package should then be system.org.apache.hadoop instead > of just org.apache.hadoop. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5991) native-task should not run unit tests if native profile is not enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072723#comment-14072723
]
Binglin Chang commented on MAPREDUCE-5991:
--
Hi, Todd, the patch does not apply, mind take a look?
{code}
error: patch failed:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/system/function/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java:1
error:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/system/function/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java:
patch does not apply
error: patch failed:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/system/function/org/apache/hadoop/mapred/nativetask/testutil/BytesFactory.java:1
error:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/system/function/org/apache/hadoop/mapred/nativetask/testutil/BytesFactory.java:
patch does not apply
{code}
> native-task should not run unit tests if native profile is not enabled
> --
>
> Key: MAPREDUCE-5991
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5991
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
>Reporter: Todd Lipcon
>
> Currently, running "mvn test" without the 'native' profile enabled causes all
> of the native-task tests to fail. In order to integrate to trunk, we need to
> fix this - either using JUnit "Assume" commands in each test that depends on
> native code, or disabling the tests from the pom unless -Pnative is specified
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5996) native-task: Rename system tests into standard directory layout
[ https://issues.apache.org/jira/browse/MAPREDUCE-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072724#comment-14072724 ] Binglin Chang commented on MAPREDUCE-5996: -- Hi, Todd, the patch does not apply, mind take a look? error: patch failed: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/system/function/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java:1 error: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/system/function/org/apache/hadoop/mapred/nativetask/kvtest/KVJob.java: patch does not apply error: patch failed: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/system/function/org/apache/hadoop/mapred/nativetask/testutil/BytesFactory.java:1 error: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/java/system/function/org/apache/hadoop/mapred/nativetask/testutil/BytesFactory.java: patch does not apply > native-task: Rename system tests into standard directory layout > --- > > Key: MAPREDUCE-5996 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5996 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: mapreduce-5996.txt > > > Currently there are a number of tests in src/java/system. This confuses IDEs > which think that the package should then be system.org.apache.hadoop instead > of just org.apache.hadoop. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5991) native-task should not run unit tests if native profile is not enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14071392#comment-14071392 ] Binglin Chang commented on MAPREDUCE-5991: -- Also some tests in a non-standard directory: src/test/java/system, and a data file is also included in this directory, which is not used. > native-task should not run unit tests if native profile is not enabled > -- > > Key: MAPREDUCE-5991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5991 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Todd Lipcon > > Currently, running "mvn test" without the 'native' profile enabled causes all > of the native-task tests to fail. In order to integrate to trunk, we need to > fix this - either using JUnit "Assume" commands in each test that depends on > native code, or disabling the tests from the pom unless -Pnative is specified -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5984) native-task: upgrade lz4 to lastest version
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-5984: - Attachment: MAPREDUCE-5984.v2.patch update patch, changes: 1. copy lz4.h/lz4.c/lz4_encoder.h from hadoop-common before compile 2. include lz4.h in source file instead of using extern declaration 3. fix a typo in TestIFile.cc > native-task: upgrade lz4 to lastest version > --- > > Key: MAPREDUCE-5984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-5984.v1.patch, MAPREDUCE-5984.v2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5984) native-task: upgrade lz4 to lastest version
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-5984: - Status: Open (was: Patch Available) > native-task: upgrade lz4 to lastest version > --- > > Key: MAPREDUCE-5984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-5984.v1.patch, MAPREDUCE-5984.v2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-2841) Task level native optimization
[ https://issues.apache.org/jira/browse/MAPREDUCE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069945#comment-14069945 ] Binglin Chang commented on MAPREDUCE-2841: -- Hi Sean, the test succeed on macosx, but failed on ubuntu12, I update the test a little in MAPREDUCE-5985. > Task level native optimization > -- > > Key: MAPREDUCE-2841 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2841 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task > Environment: x86-64 Linux/Unix >Reporter: Binglin Chang >Assignee: Sean Zhong > Attachments: DESIGN.html, MAPREDUCE-2841.v1.patch, > MAPREDUCE-2841.v2.patch, dualpivot-0.patch, dualpivotv20-0.patch, > fb-shuffle.patch, hadoop-3.0-mapreduce-2841-2014-7-17.patch > > > I'm recently working on native optimization for MapTask based on JNI. > The basic idea is that, add a NativeMapOutputCollector to handle k/v pairs > emitted by mapper, therefore sort, spill, IFile serialization can all be done > in native code, preliminary test(on Xeon E5410, jdk6u24) showed promising > results: > 1. Sort is about 3x-10x as fast as java(only binary string compare is > supported) > 2. IFile serialization speed is about 3x of java, about 500MB/s, if hardware > CRC32C is used, things can get much faster(1G/ > 3. Merge code is not completed yet, so the test use enough io.sort.mb to > prevent mid-spill > This leads to a total speed up of 2x~3x for the whole MapTask, if > IdentityMapper(mapper does nothing) is used > There are limitations of course, currently only Text and BytesWritable is > supported, and I have not think through many things right now, such as how to > support map side combine. I had some discussion with somebody familiar with > hive, it seems that these limitations won't be much problem for Hive to > benefit from those optimizations, at least. Advices or discussions about > improving compatibility are most welcome:) > Currently NativeMapOutputCollector has a static method called canEnable(), > which checks if key/value type, comparator type, combiner are all compatible, > then MapTask can choose to enable NativeMapOutputCollector. > This is only a preliminary test, more work need to be done. I expect better > final results, and I believe similar optimization can be adopt to reduce task > and shuffle too. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5984) native-task: upgrade lz4 to lastest version
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069939#comment-14069939 ] Binglin Chang commented on MAPREDUCE-5984: -- bq. but I'm wondering if it's possible to reuse the lz4 source files that are already checked in for hadoop-common Sure, I will update the patch to copy lz4 files to building path. And we can upgrading the version in hadoop-common in trunk. > native-task: upgrade lz4 to lastest version > --- > > Key: MAPREDUCE-5984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: MAPREDUCE-5984.v1.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
