[jira] [Updated] (HDFS-11841) libc.so.6 detected double free on TestDFSIO write when erasure coding and enabled Intel ISA-L
[
https://issues.apache.org/jira/browse/HDFS-11841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liaoyuxiangqin updated HDFS-11841:
--
Description:
when i execute hadoop jar
hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar TestDFSIO
-write -nrFiles 1 -size 1MB on above environment, glibc detected double
free or corruption, detail information as follows:
{pane}
File System Counters
FILE: Number of bytes read=489391103089
FILE: Number of bytes written=42020064724
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5751218285
HDFS: Number of bytes written=52456018688978
HDFS: Number of read operations=150165017
HDFS: Number of large read operations=0
HDFS: Number of write operations=150065006
Map-Reduce Framework
Map input records=1
Map output records=5
Map output bytes=707116
Map output materialized bytes=867116
Input split bytes=1288890
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=867116
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=34575
Total committed heap usage (bytes)=46386259689472
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1148890
File Output Format Counters
Bytes Written=88
glibc detected /home/hadoop/jdk1.8.0_111/bin/java: double free or corruption
(out): 0x7fcd84a6c6c0
=== Backtrace: =
/lib64/libc.so.6[0x3d1fe724a6]
/lib64/libc.so.6(cfree+0x6c)[0x3d1fe771bc]
[0x7fcdb138e8e6]
=== Memory map:
0040-00401000 r-xp 08:01 3720754
/home/hadoop/jdk1.8.0_111/bin/java
0060-00601000 rw-p 08:01 3720754
/home/hadoop/jdk1.8.0_111/bin/java
0204f000-0460 rw-p 00:00 0 [heap]
64940-6f210 rw-p 00:00 0
6f210-74318 ---p 00:00 0
74318-7c000 rw-p 00:00 0
7c000-7c03c rw-p 00:00 0
7c03c-8 ---p 00:00 0
3d1fa0-3d1fa21000 r-xp 08:01 8126508
/lib64/ld-2.14.1.so
3d1fc2-3d1fc21000 r--p 0002 08:01 8126508
/lib64/ld-2.14.1.so
3d1fc21000-3d1fc23000 rw-p 00021000 08:01 8126508
/lib64/ld-2.14.1.so
3d1fe0-3d1ff83000 r-xp 08:01 8126562
/lib64/libc-2.14.1.so
3d1ff83000-3d20182000 ---p 00183000 08:01 8126562
/lib64/libc-2.14.1.so
/home/hadoop/jdk1.8.0_111/jre/lib/jsse.jar
7fcd9260a000-7fcd9260f000 r--s 00044000 08:01 3852117
/home/gy/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.0.0-alpha3-SNAPSHOT.jar
7fcd9260f000-7fcd92612000 r--s 0001f000 08:01 3852191
/home/gy/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-3.0.0-alpha3-SNAPSHOT.jarAborted
{panel}
was:
when i execute hadoop jar
hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar TestDFSIO
-write -nrFiles 1 -size 1MB on above environment, glibc detected double
free or corruption, detail information as follows:
{pane}
File System Counters
FILE: Number of bytes read=489391103089
FILE: Number of bytes written=42020064724
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5751218285
HDFS: Number of bytes written=52456018688978
HDFS: Number of read operations=150165017
HDFS: Number of large read operations=0
HDFS: Number of write operations=150065006
Map-Reduce Framework
Map input records=1
Map output records=5
Map output bytes=707116
Map output materialized bytes=867116
Input split bytes=1288890
Combine input records=0
Combine output records=0
Reduce
[jira] [Updated] (HDFS-11841) libc.so.6 detected double free on TestDFSIO write when erasure coding and enabled Intel ISA-L
[
https://issues.apache.org/jira/browse/HDFS-11841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liaoyuxiangqin updated HDFS-11841:
--
Component/s: native
> libc.so.6 detected double free on TestDFSIO write when erasure coding and
> enabled Intel ISA-L
> --
>
> Key: HDFS-11841
> URL: https://issues.apache.org/jira/browse/HDFS-11841
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, native
> Environment: cluster: 3 nodes
> os:(Red Hat 2.6.33.20, Red Hat 3.10.0-514.6.1.el7.x86_64,
> Ubuntu4.4.0-31-generic)
> hadoop version: hadoop-3.0.0-alpha3
> erasure coding: XOR-2-1-64k and enabled Intel ISA-L
> TestDFSIO: hadoop jar
> hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar TestDFSIO
> -write -nrFiles 1 -size 1MB
>Reporter: liaoyuxiangqin
>Priority: Critical
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> when i execute hadoop jar
> hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar TestDFSIO
> -write -nrFiles 1 -size 1MB on above environment, glibc detected double
> free or corruption, detail information as follows:
> {pane}
> File System Counters
> FILE: Number of bytes read=489391103089
> FILE: Number of bytes written=42020064724
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=5751218285
> HDFS: Number of bytes written=52456018688978
> HDFS: Number of read operations=150165017
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=150065006
> Map-Reduce Framework
> Map input records=1
> Map output records=5
> Map output bytes=707116
> Map output materialized bytes=867116
> Input split bytes=1288890
> Combine input records=0
> Combine output records=0
> Reduce input groups=5
> Reduce shuffle bytes=867116
> Reduce input records=5
> Reduce output records=5
> Spilled Records=10
> Shuffled Maps =1
> Failed Shuffles=0
> Merged Map outputs=1
> GC time elapsed (ms)=34575
> Total committed heap usage (bytes)=46386259689472
> Shuffle Errors
> BAD_ID=0
> CONNECTION=0
> IO_ERROR=0
> WRONG_LENGTH=0
> WRONG_MAP=0
> WRONG_REDUCE=0
> File Input Format Counters
> Bytes Read=1148890
> File Output Format Counters
> Bytes Written=88
> glibc detected /home/hadoop/jdk1.8.0_111/bin/java: double free or
> corruption (out): 0x7fcd84a6c6c0
> === Backtrace: =
> /lib64/libc.so.6[0x3d1fe724a6]
> /lib64/libc.so.6(cfree+0x6c)[0x3d1fe771bc]
> [0x7fcdb138e8e6]
> === Memory map:
> 0040-00401000 r-xp 08:01 3720754
> /home/hadoop/jdk1.8.0_111/bin/java
> 0060-00601000 rw-p 08:01 3720754
> /home/hadoop/jdk1.8.0_111/bin/java
> 0204f000-0460 rw-p 00:00 0
> [heap]
> 64940-6f210 rw-p 00:00 0
> 6f210-74318 ---p 00:00 0
> 74318-7c000 rw-p 00:00 0
> 7c000-7c03c rw-p 00:00 0
> 7c03c-8 ---p 00:00 0
> 3d1fa0-3d1fa21000 r-xp 08:01 8126508
> /lib64/ld-2.14.1.so
> 3d1fc2-3d1fc21000 r--p 0002 08:01 8126508
> /lib64/ld-2.14.1.so
> 3d1fc21000-3d1fc23000 rw-p 00021000 08:01 8126508
> /lib64/ld-2.14.1.so
> 3d1fe0-3d1ff83000 r-xp 08:01 8126562
> /lib64/libc-2.14.1.so
> 3d1ff83000-3d20182000 ---p 00183000 08:01 8126562
> /lib64/libc-2.14.1.so
> 3d20182000-3d20186000 r--p 00182000 08:01 8126562
> /lib64/libc-2.14.1.so
> 3d20186000-3d20187000 rw-p 00186000 08:01 8126562
> /lib64/libc-2.14.1.so
> 3d20187000-3d2018c000 rw-p 00:00 0
> 3d2020-3d20282000 r-xp 08:01 8126567
> /lib64/libm-2.14.1.so
> 3d20282000-3d20481000 ---p 00082000 08:01 8126567
> /lib64/libm-2.14.1.so
> 3d20481000-3d20482000 r--p 00081000 08:01 8126567
> /lib64/libm-2.14.1.so
> 3d20482000-3d20483000 rw-p 00082000 08:01 8126567
> /lib64/libm-2.14.1.so
>
[jira] [Updated] (HDFS-11841) libc.so.6 detected double free on TestDFSIO write when erasure coding and enabled Intel ISA-L
[
https://issues.apache.org/jira/browse/HDFS-11841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liaoyuxiangqin updated HDFS-11841:
--
Description:
when i execute hadoop jar
hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar TestDFSIO
-write -nrFiles 1 -size 1MB on above environment, glibc detected double
free or corruption, detail information as follows:
{pane}
File System Counters
FILE: Number of bytes read=489391103089
FILE: Number of bytes written=42020064724
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5751218285
HDFS: Number of bytes written=52456018688978
HDFS: Number of read operations=150165017
HDFS: Number of large read operations=0
HDFS: Number of write operations=150065006
Map-Reduce Framework
Map input records=1
Map output records=5
Map output bytes=707116
Map output materialized bytes=867116
Input split bytes=1288890
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=867116
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=34575
Total committed heap usage (bytes)=46386259689472
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1148890
File Output Format Counters
Bytes Written=88
glibc detected /home/hadoop/jdk1.8.0_111/bin/java: double free or corruption
(out): 0x7fcd84a6c6c0
=== Backtrace: =
/lib64/libc.so.6[0x3d1fe724a6]
/lib64/libc.so.6(cfree+0x6c)[0x3d1fe771bc]
[0x7fcdb138e8e6]
=== Memory map:
0040-00401000 r-xp 08:01 3720754
/home/hadoop/jdk1.8.0_111/bin/java
0060-00601000 rw-p 08:01 3720754
/home/hadoop/jdk1.8.0_111/bin/java
0204f000-0460 rw-p 00:00 0 [heap]
64940-6f210 rw-p 00:00 0
6f210-74318 ---p 00:00 0
74318-7c000 rw-p 00:00 0
7c000-7c03c rw-p 00:00 0
7c03c-8 ---p 00:00 0
3d1fa0-3d1fa21000 r-xp 08:01 8126508
/lib64/ld-2.14.1.so
3d1fc2-3d1fc21000 r--p 0002 08:01 8126508
/lib64/ld-2.14.1.so
3d1fc21000-3d1fc23000 rw-p 00021000 08:01 8126508
/lib64/ld-2.14.1.so
3d1fe0-3d1ff83000 r-xp 08:01 8126562
/lib64/libc-2.14.1.so
3d1ff83000-3d20182000 ---p 00183000 08:01 8126562
/lib64/libc-2.14.1.so
3d20182000-3d20186000 r--p 00182000 08:01 8126562
/lib64/libc-2.14.1.so
3d20186000-3d20187000 rw-p 00186000 08:01 8126562
/lib64/libc-2.14.1.so
3d20187000-3d2018c000 rw-p 00:00 0
3d2020-3d20282000 r-xp 08:01 8126567
/lib64/libm-2.14.1.so
3d20282000-3d20481000 ---p 00082000 08:01 8126567
/lib64/libm-2.14.1.so
3d20481000-3d20482000 r--p 00081000 08:01 8126567
/lib64/libm-2.14.1.so
3d20482000-3d20483000 rw-p 00082000 08:01 8126567
/lib64/libm-2.14.1.so
3d20a0-3d20a02000 r-xp 08:01 8126764
/lib64/libdl-2.14.1.so
3d20a02000-3d20c02000 ---p 2000 08:01 8126764
/lib64/libdl-2.14.1.so
3d20c02000-3d20c03000 r--p 2000 08:01 8126764
/lib64/libdl-2.14.1.so
3d20c03000-3d20c04000 rw-p 3000 08:01 8126764
/lib64/libdl-2.14.1.so
3d2120-3d21218000 r-xp 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21218000-3d21417000 ---p 00018000 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21417000-3d21418000 r--p 00017000 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21418000-3d21419000 rw-p 00018000 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21419000-3d2141d000 rw-p 00:00 0
3d21a0-3d21a0a000 r-xp 08:01 8126578
/lib64/librt-2.14.1.so
3d21a0a000-3d21c09000 ---p a000 08:01 8126578
[jira] [Updated] (HDFS-11841) libc.so.6 detected double free on TestDFSIO write when erasure coding and enabled Intel ISA-L
[
https://issues.apache.org/jira/browse/HDFS-11841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liaoyuxiangqin updated HDFS-11841:
--
Description:
when i execute hadoop jar
hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar TestDFSIO
-write -nrFiles 1 -size 1MB on above environment, glibc detected double
free or corruption, detail information as follows:
{pane}
File System Counters
FILE: Number of bytes read=489391103089
FILE: Number of bytes written=42020064724
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5751218285
HDFS: Number of bytes written=52456018688978
HDFS: Number of read operations=150165017
HDFS: Number of large read operations=0
HDFS: Number of write operations=150065006
Map-Reduce Framework
Map input records=1
Map output records=5
Map output bytes=707116
Map output materialized bytes=867116
Input split bytes=1288890
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=867116
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=34575
Total committed heap usage (bytes)=46386259689472
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1148890
File Output Format Counters
Bytes Written=88
*** glibc detected *** /home/hadoop/jdk1.8.0_111/bin/java: double free or
corruption (out): 0x7fcd84a6c6c0 ***
=== Backtrace: =
/lib64/libc.so.6[0x3d1fe724a6]
/lib64/libc.so.6(cfree+0x6c)[0x3d1fe771bc]
[0x7fcdb138e8e6]
=== Memory map:
0040-00401000 r-xp 08:01 3720754
/home/hadoop/jdk1.8.0_111/bin/java
0060-00601000 rw-p 08:01 3720754
/home/hadoop/jdk1.8.0_111/bin/java
0204f000-0460 rw-p 00:00 0 [heap]
64940-6f210 rw-p 00:00 0
6f210-74318 ---p 00:00 0
74318-7c000 rw-p 00:00 0
7c000-7c03c rw-p 00:00 0
7c03c-8 ---p 00:00 0
3d1fa0-3d1fa21000 r-xp 08:01 8126508
/lib64/ld-2.14.1.so
3d1fc2-3d1fc21000 r--p 0002 08:01 8126508
/lib64/ld-2.14.1.so
3d1fc21000-3d1fc23000 rw-p 00021000 08:01 8126508
/lib64/ld-2.14.1.so
3d1fe0-3d1ff83000 r-xp 08:01 8126562
/lib64/libc-2.14.1.so
3d1ff83000-3d20182000 ---p 00183000 08:01 8126562
/lib64/libc-2.14.1.so
3d20182000-3d20186000 r--p 00182000 08:01 8126562
/lib64/libc-2.14.1.so
3d20186000-3d20187000 rw-p 00186000 08:01 8126562
/lib64/libc-2.14.1.so
3d20187000-3d2018c000 rw-p 00:00 0
3d2020-3d20282000 r-xp 08:01 8126567
/lib64/libm-2.14.1.so
3d20282000-3d20481000 ---p 00082000 08:01 8126567
/lib64/libm-2.14.1.so
3d20481000-3d20482000 r--p 00081000 08:01 8126567
/lib64/libm-2.14.1.so
3d20482000-3d20483000 rw-p 00082000 08:01 8126567
/lib64/libm-2.14.1.so
3d20a0-3d20a02000 r-xp 08:01 8126764
/lib64/libdl-2.14.1.so
3d20a02000-3d20c02000 ---p 2000 08:01 8126764
/lib64/libdl-2.14.1.so
3d20c02000-3d20c03000 r--p 2000 08:01 8126764
/lib64/libdl-2.14.1.so
3d20c03000-3d20c04000 rw-p 3000 08:01 8126764
/lib64/libdl-2.14.1.so
3d2120-3d21218000 r-xp 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21218000-3d21417000 ---p 00018000 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21417000-3d21418000 r--p 00017000 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21418000-3d21419000 rw-p 00018000 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21419000-3d2141d000 rw-p 00:00 0
3d21a0-3d21a0a000 r-xp 08:01 8126578
/lib64/librt-2.14.1.so
3d21a0a000-3d21c09000 ---p a000 08:01 8126578
[jira] [Updated] (HDFS-11841) libc.so.6 detected double free on TestDFSIO write when erasure coding and enabled Intel ISA-L
[
https://issues.apache.org/jira/browse/HDFS-11841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liaoyuxiangqin updated HDFS-11841:
--
Description:
when i execute hadoop jar
hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar TestDFSIO
-write -nrFiles 1 -size 1MB on above environment, glibc detected double
free or corruption, detail information as follows:
{panel:title=My Title|borderStyle=dashed}
File System Counters
FILE: Number of bytes read=489391103089
FILE: Number of bytes written=42020064724
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5751218285
HDFS: Number of bytes written=52456018688978
HDFS: Number of read operations=150165017
HDFS: Number of large read operations=0
HDFS: Number of write operations=150065006
Map-Reduce Framework
Map input records=1
Map output records=5
Map output bytes=707116
Map output materialized bytes=867116
Input split bytes=1288890
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=867116
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=34575
Total committed heap usage (bytes)=46386259689472
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1148890
File Output Format Counters
Bytes Written=88
*** glibc detected *** /home/hadoop/jdk1.8.0_111/bin/java: double free or
corruption (out): 0x7fcd84a6c6c0 ***
=== Backtrace: =
/lib64/libc.so.6[0x3d1fe724a6]
/lib64/libc.so.6(cfree+0x6c)[0x3d1fe771bc]
[0x7fcdb138e8e6]
=== Memory map:
0040-00401000 r-xp 08:01 3720754
/home/hadoop/jdk1.8.0_111/bin/java
0060-00601000 rw-p 08:01 3720754
/home/hadoop/jdk1.8.0_111/bin/java
0204f000-0460 rw-p 00:00 0 [heap]
64940-6f210 rw-p 00:00 0
6f210-74318 ---p 00:00 0
74318-7c000 rw-p 00:00 0
7c000-7c03c rw-p 00:00 0
7c03c-8 ---p 00:00 0
3d1fa0-3d1fa21000 r-xp 08:01 8126508
/lib64/ld-2.14.1.so
3d1fc2-3d1fc21000 r--p 0002 08:01 8126508
/lib64/ld-2.14.1.so
3d1fc21000-3d1fc23000 rw-p 00021000 08:01 8126508
/lib64/ld-2.14.1.so
3d1fe0-3d1ff83000 r-xp 08:01 8126562
/lib64/libc-2.14.1.so
3d1ff83000-3d20182000 ---p 00183000 08:01 8126562
/lib64/libc-2.14.1.so
3d20182000-3d20186000 r--p 00182000 08:01 8126562
/lib64/libc-2.14.1.so
3d20186000-3d20187000 rw-p 00186000 08:01 8126562
/lib64/libc-2.14.1.so
3d20187000-3d2018c000 rw-p 00:00 0
3d2020-3d20282000 r-xp 08:01 8126567
/lib64/libm-2.14.1.so
3d20282000-3d20481000 ---p 00082000 08:01 8126567
/lib64/libm-2.14.1.so
3d20481000-3d20482000 r--p 00081000 08:01 8126567
/lib64/libm-2.14.1.so
3d20482000-3d20483000 rw-p 00082000 08:01 8126567
/lib64/libm-2.14.1.so
3d20a0-3d20a02000 r-xp 08:01 8126764
/lib64/libdl-2.14.1.so
3d20a02000-3d20c02000 ---p 2000 08:01 8126764
/lib64/libdl-2.14.1.so
3d20c02000-3d20c03000 r--p 2000 08:01 8126764
/lib64/libdl-2.14.1.so
3d20c03000-3d20c04000 rw-p 3000 08:01 8126764
/lib64/libdl-2.14.1.so
3d2120-3d21218000 r-xp 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21218000-3d21417000 ---p 00018000 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21417000-3d21418000 r--p 00017000 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21418000-3d21419000 rw-p 00018000 08:01 8126576
/lib64/libpthread-2.14.1.so
3d21419000-3d2141d000 rw-p 00:00 0
3d21a0-3d21a0a000 r-xp 08:01 8126578
/lib64/librt-2.14.1.so
3d21a0a000-3d21c09000
