[jira] [Created] (HBASE-16015) Usability - VerifyReplication performance is too slow
KarthikP created HBASE-16015:
Summary: Usability - VerifyReplication performance is too slow
Key: HBASE-16015
URL: https://issues.apache.org/jira/browse/HBASE-16015
Project: HBase
Issue Type: Improvement
Components: Usability
Reporter: KarthikP
Priority: Critical
I see VerifyReplication is too slow in Geo replication cluster, then I dig into
the code where default Input scanner caching set as 1 for target cluster
request.
This value should be optimal or could be exposed in usage command.
-Dhbase.mapreduce.scan.cachedrows=100
{code:title=TableInputFormat.java|borderStyle=solid}
public static final String SCAN_CACHEDROWS = "hbase.mapreduce.scan.cachedrows";
{code}
{code:title=VerifyReplication.java|borderStyle=solid}
Configuration conf = context.getConfiguration();
final Scan scan = new Scan();
scan.setCaching(conf.getInt(TableInputFormat.SCAN_CACHEDROWS, 1));
{code}
If agree, then I will add this line into printUsage method as shown below,
{code:title=VerifyReplication.java|borderStyle=solid}
System.err.println("For performance consider the following option, Input
scanner caching for source to target cluster request\n"
+ "-Dhbase.mapreduce.scan.cachedrows=100");
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15309) GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
[
https://issues.apache.org/jira/browse/HBASE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
karthikp updated HBASE-15309:
-
Description:
I'd memory leak issue in regionserver process, VM and RSS memory continiously
increasing 64MB.
I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
I've huge write load and frequent minor compaction, We've used GZip hfile
compression.
Max java regionserver heap size is 32GB.
{noformat}
top - 14:28:30 up 201 days, 21:06, 3 users, load average: 5.67, 3.72, 3.31
Tasks: 803 total, 1 running, 802 sleeping, 0 stopped, 0 zombie
Cpu(s): 8.3%us, 2.1%sy, 0.0%ni, 85.9%id, 3.5%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 65932340k total, 63961912k used, 1970428k free, 2394528k buffers
Swap: 29659132k total,63532k used, 29595600k free, 1095268k cached
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
57335 hbase 20 0 46.4g 44g 9296 S 98.2 70.9 13319:10 java
{noformat}
{noformat}
[hbase@xx-hslave ~]$ pmap -x 57335 | sort -k 3 -nr | more
total kB48695984 46765792 46756512
7ff31246 33171072 33169464 33169464 rwx--[ anon ]
4010a000 1448552 1448552 1448552 rwx--[ anon ]
7ff2d181 612120 603124 603124 rwx--[ anon ]
7ff2fadff000 383364 383364 383364 rwx--[ anon ]
7ff0e800 131072 131072 131072 rwx--[ anon ]
7ff21800 131048 131048 131048 rwx--[ anon ]
7ff12800 131068 131048 131048 rwx--[ anon ]
7ff23000 131044 131044 131044 rwx--[ anon ]
7ff0 131036 131036 131036 rwx--[ anon ]
7fefe000 131060 131036 131036 rwx--[ anon ]
7ff23c00 65536 65536 65536 rwx--[ anon ]
7ff0a400 65536 65536 65536 rwx--[ anon ]
7ff05400 65536 65536 65536 rwx--[ anon ]
7ff01c00 65536 65536 65536 rwx--[ anon ]
7fefb400 65536 65536 65536 rwx--[ anon ]
7ff22c00 65532 65532 65532 rwx--[ anon ]
7ff11000 65532 65532 65532 rwx--[ anon ]
7ff10c00 65532 65532 65532 rwx--[ anon ]
7ff0b800 65532 65532 65532 rwx--[ anon ]
7ff09c00 65532 65532 65532 rwx--[ anon ]
7feff800 65532 65532 65532 rwx--[ anon ]
7ff25000 65528 65528 65528 rwx--[ anon ]
--More--
{noformat}
{noformat}
$pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print $i}'
| wc -l
146
{noformat}
Regionserver process has many anon pages, size varying from 64000 to 65536 KB
that shown above.
was:
I'd memory leak issue in regionserver process, VM and RSS memory continiously
increasing 64MB.
I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
I've huge write load and frequent minor compaction, We've used GZip hfile
compression.
Max java regionserver heap size is 32GB.
{noformat}
top - 14:28:30 up 201 days, 21:06, 3 users, load average: 5.67, 3.72, 3.31
Tasks: 803 total, 1 running, 802 sleeping, 0 stopped, 0 zombie
Cpu(s): 8.3%us, 2.1%sy, 0.0%ni, 85.9%id, 3.5%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 65932340k total, 63961912k used, 1970428k free, 2394528k buffers
Swap: 29659132k total,63532k used, 29595600k free, 1095268k cached
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
57335 hbase 20 0 46.4g 44g 9296 S 98.2 70.9 13319:10 java
{noformat}
{noformat}
[hbase@xx-hslave ~]$ pmap -x 57335 | sort -k 3 -nr | more
total kB48695984 46765792 46756512
7ff31246 33171072 33169464 33169464 rwx--[ anon ]
4010a000 1448552 1448552 1448552 rwx--[ anon ]
7ff2d181 612120 603124 603124 rwx--[ anon ]
7ff2fadff000 383364 383364 383364 rwx--[ anon ]
7ff0e800 131072 131072 131072 rwx--[ anon ]
7ff21800 131048 131048 131048 rwx--[ anon ]
7ff12800 131068 131048 131048 rwx--[ anon ]
7ff23000 131044 131044 131044 rwx--[ anon ]
7ff0 131036 131036 131036 rwx--[ anon ]
7fefe000 131060 131036 131036 rwx--[ anon ]
7ff23c00 65536 65536 65536 rwx--[ anon ]
7ff0a400 65536 65536 65536 rwx--[ anon ]
7ff05400 65536 65536 65536 rwx--[ anon ]
7ff01c00 65536 65536 65536 rwx--[ anon ]
7fefb400 65536 65536 65536 rwx--[ anon ]
7ff22c00 65532 65532 65532 rwx--[ anon ]
7ff11000 65532 65532 65532 rwx--[ anon ]
7ff10c00 65532 65532 65532 rwx--[ anon ]
7ff0b800 65532 65532 65532 rwx--[ anon ]
7ff09c00 65532 65532 65532 rwx--[ anon ]
7feff800 65532 65532 65532 rwx--[ anon ]
7ff25000 65528 65528 65528 rwx--[ anon ]
--More--
{noformat}
{noformat}
$pmap -x 57335 | awk '{print
[jira] [Updated] (HBASE-15309) GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
[
https://issues.apache.org/jira/browse/HBASE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
karthikp updated HBASE-15309:
-
Priority: Blocker (was: Critical)
> GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
>
>
> Key: HBASE-15309
> URL: https://issues.apache.org/jira/browse/HBASE-15309
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.6
>Reporter: karthikp
>Assignee: Todd Lipcon
>Priority: Blocker
>
> I'd memory leak issue in regionserver process, VM and RSS memory continiously
> increasing 64MB.
> I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
> I've huge write load and frequent minor compaction, We've used GZip hfile
> compression.
> Max java regionserver heap size is 32GB.
> {noformat}
> top - 14:28:30 up 201 days, 21:06, 3 users, load average: 5.67, 3.72, 3.31
> Tasks: 803 total, 1 running, 802 sleeping, 0 stopped, 0 zombie
> Cpu(s): 8.3%us, 2.1%sy, 0.0%ni, 85.9%id, 3.5%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 65932340k total, 63961912k used, 1970428k free, 2394528k buffers
> Swap: 29659132k total,63532k used, 29595600k free, 1095268k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 57335 hbase 20 0 46.4g 44g 9296 S 98.2 70.9 13319:10 java
> {noformat}
> {noformat}
> [hbase@xx-hslave ~]$ pmap -x 57335 | sort -k 3 -nr | more
> total kB48695984 46765792 46756512
> 7ff31246 33171072 33169464 33169464 rwx--[ anon ]
> 4010a000 1448552 1448552 1448552 rwx--[ anon ]
> 7ff2d181 612120 603124 603124 rwx--[ anon ]
> 7ff2fadff000 383364 383364 383364 rwx--[ anon ]
> 7ff0e800 131072 131072 131072 rwx--[ anon ]
> 7ff21800 131048 131048 131048 rwx--[ anon ]
> 7ff12800 131068 131048 131048 rwx--[ anon ]
> 7ff23000 131044 131044 131044 rwx--[ anon ]
> 7ff0 131036 131036 131036 rwx--[ anon ]
> 7fefe000 131060 131036 131036 rwx--[ anon ]
> 7ff23c00 65536 65536 65536 rwx--[ anon ]
> 7ff0a400 65536 65536 65536 rwx--[ anon ]
> 7ff05400 65536 65536 65536 rwx--[ anon ]
> 7ff01c00 65536 65536 65536 rwx--[ anon ]
> 7fefb400 65536 65536 65536 rwx--[ anon ]
> 7ff22c00 65532 65532 65532 rwx--[ anon ]
> 7ff11000 65532 65532 65532 rwx--[ anon ]
> 7ff10c00 65532 65532 65532 rwx--[ anon ]
> 7ff0b800 65532 65532 65532 rwx--[ anon ]
> 7ff09c00 65532 65532 65532 rwx--[ anon ]
> 7feff800 65532 65532 65532 rwx--[ anon ]
> 7ff25000 65528 65528 65528 rwx--[ anon ]
> --More--
> {noformat}
> {noformat}
> $pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print
> $i}' | wc -l
> 146
> {noformat}
> Will MALLOC_AREANA have exactly 65536KB?
> Regionserver process has many anon pages, size varying from 64000 to 65536 KB
> that shown above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15309) GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
[
https://issues.apache.org/jira/browse/HBASE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
karthikp updated HBASE-15309:
-
Assignee: Todd Lipcon
> GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
>
>
> Key: HBASE-15309
> URL: https://issues.apache.org/jira/browse/HBASE-15309
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.6
>Reporter: karthikp
>Assignee: Todd Lipcon
>Priority: Critical
>
> I'd memory leak issue in regionserver process, VM and RSS memory continiously
> increasing 64MB.
> I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
> I've huge write load and frequent minor compaction, We've used GZip hfile
> compression.
> Max java regionserver heap size is 32GB.
> {noformat}
> top - 14:28:30 up 201 days, 21:06, 3 users, load average: 5.67, 3.72, 3.31
> Tasks: 803 total, 1 running, 802 sleeping, 0 stopped, 0 zombie
> Cpu(s): 8.3%us, 2.1%sy, 0.0%ni, 85.9%id, 3.5%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 65932340k total, 63961912k used, 1970428k free, 2394528k buffers
> Swap: 29659132k total,63532k used, 29595600k free, 1095268k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 57335 hbase 20 0 46.4g 44g 9296 S 98.2 70.9 13319:10 java
> {noformat}
> {noformat}
> [hbase@xx-hslave ~]$ pmap -x 57335 | sort -k 3 -nr | more
> total kB48695984 46765792 46756512
> 7ff31246 33171072 33169464 33169464 rwx--[ anon ]
> 4010a000 1448552 1448552 1448552 rwx--[ anon ]
> 7ff2d181 612120 603124 603124 rwx--[ anon ]
> 7ff2fadff000 383364 383364 383364 rwx--[ anon ]
> 7ff0e800 131072 131072 131072 rwx--[ anon ]
> 7ff21800 131048 131048 131048 rwx--[ anon ]
> 7ff12800 131068 131048 131048 rwx--[ anon ]
> 7ff23000 131044 131044 131044 rwx--[ anon ]
> 7ff0 131036 131036 131036 rwx--[ anon ]
> 7fefe000 131060 131036 131036 rwx--[ anon ]
> 7ff23c00 65536 65536 65536 rwx--[ anon ]
> 7ff0a400 65536 65536 65536 rwx--[ anon ]
> 7ff05400 65536 65536 65536 rwx--[ anon ]
> 7ff01c00 65536 65536 65536 rwx--[ anon ]
> 7fefb400 65536 65536 65536 rwx--[ anon ]
> 7ff22c00 65532 65532 65532 rwx--[ anon ]
> 7ff11000 65532 65532 65532 rwx--[ anon ]
> 7ff10c00 65532 65532 65532 rwx--[ anon ]
> 7ff0b800 65532 65532 65532 rwx--[ anon ]
> 7ff09c00 65532 65532 65532 rwx--[ anon ]
> 7feff800 65532 65532 65532 rwx--[ anon ]
> 7ff25000 65528 65528 65528 rwx--[ anon ]
> --More--
> {noformat}
> {noformat}
> $pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print
> $i}' | wc -l
> 146
> {noformat}
> Will MALLOC_AREANA have exactly 65536KB?
> Regionserver process has many anon pages, size varying from 64000 to 65536 KB
> that shown above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15309) GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
[
https://issues.apache.org/jira/browse/HBASE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
karthikp updated HBASE-15309:
-
Assignee: Ted Yu
> GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
>
>
> Key: HBASE-15309
> URL: https://issues.apache.org/jira/browse/HBASE-15309
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.6
>Reporter: karthikp
>Assignee: Ted Yu
>Priority: Critical
>
> I'd memory leak issue in regionserver process, VM and RSS memory continiously
> increasing 64MB.
> I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
> I've huge write load and frequent minor compaction, We've used GZip hfile
> compression.
> Max java regionserver heap size is 32GB.
> {noformat}
> top - 14:28:30 up 201 days, 21:06, 3 users, load average: 5.67, 3.72, 3.31
> Tasks: 803 total, 1 running, 802 sleeping, 0 stopped, 0 zombie
> Cpu(s): 8.3%us, 2.1%sy, 0.0%ni, 85.9%id, 3.5%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 65932340k total, 63961912k used, 1970428k free, 2394528k buffers
> Swap: 29659132k total,63532k used, 29595600k free, 1095268k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 57335 hbase 20 0 46.4g 44g 9296 S 98.2 70.9 13319:10 java
> {noformat}
> {noformat}
> [hbase@xx-hslave ~]$ pmap -x 57335 | sort -k 3 -nr | more
> total kB48695984 46765792 46756512
> 7ff31246 33171072 33169464 33169464 rwx--[ anon ]
> 4010a000 1448552 1448552 1448552 rwx--[ anon ]
> 7ff2d181 612120 603124 603124 rwx--[ anon ]
> 7ff2fadff000 383364 383364 383364 rwx--[ anon ]
> 7ff0e800 131072 131072 131072 rwx--[ anon ]
> 7ff21800 131048 131048 131048 rwx--[ anon ]
> 7ff12800 131068 131048 131048 rwx--[ anon ]
> 7ff23000 131044 131044 131044 rwx--[ anon ]
> 7ff0 131036 131036 131036 rwx--[ anon ]
> 7fefe000 131060 131036 131036 rwx--[ anon ]
> 7ff23c00 65536 65536 65536 rwx--[ anon ]
> 7ff0a400 65536 65536 65536 rwx--[ anon ]
> 7ff05400 65536 65536 65536 rwx--[ anon ]
> 7ff01c00 65536 65536 65536 rwx--[ anon ]
> 7fefb400 65536 65536 65536 rwx--[ anon ]
> 7ff22c00 65532 65532 65532 rwx--[ anon ]
> 7ff11000 65532 65532 65532 rwx--[ anon ]
> 7ff10c00 65532 65532 65532 rwx--[ anon ]
> 7ff0b800 65532 65532 65532 rwx--[ anon ]
> 7ff09c00 65532 65532 65532 rwx--[ anon ]
> 7feff800 65532 65532 65532 rwx--[ anon ]
> 7ff25000 65528 65528 65528 rwx--[ anon ]
> --More--
> {noformat}
> {noformat}
> $pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print
> $i}' | wc -l
> 146
> {noformat}
> Will MALLOC_AREANA have exactly 65536KB?
> Regionserver process has many anon pages, size varying from 64000 to 65536 KB
> that shown above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15309) GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
[
https://issues.apache.org/jira/browse/HBASE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
karthikp updated HBASE-15309:
-
Assignee: (was: Ted Yu)
> GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
>
>
> Key: HBASE-15309
> URL: https://issues.apache.org/jira/browse/HBASE-15309
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.6
>Reporter: karthikp
>Priority: Critical
>
> I'd memory leak issue in regionserver process, VM and RSS memory continiously
> increasing 64MB.
> I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
> I've huge write load and frequent minor compaction, We've used GZip hfile
> compression.
> Max java regionserver heap size is 32GB.
> {noformat}
> top - 14:28:30 up 201 days, 21:06, 3 users, load average: 5.67, 3.72, 3.31
> Tasks: 803 total, 1 running, 802 sleeping, 0 stopped, 0 zombie
> Cpu(s): 8.3%us, 2.1%sy, 0.0%ni, 85.9%id, 3.5%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 65932340k total, 63961912k used, 1970428k free, 2394528k buffers
> Swap: 29659132k total,63532k used, 29595600k free, 1095268k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 57335 hbase 20 0 46.4g 44g 9296 S 98.2 70.9 13319:10 java
> {noformat}
> {noformat}
> [hbase@xx-hslave ~]$ pmap -x 57335 | sort -k 3 -nr | more
> total kB48695984 46765792 46756512
> 7ff31246 33171072 33169464 33169464 rwx--[ anon ]
> 4010a000 1448552 1448552 1448552 rwx--[ anon ]
> 7ff2d181 612120 603124 603124 rwx--[ anon ]
> 7ff2fadff000 383364 383364 383364 rwx--[ anon ]
> 7ff0e800 131072 131072 131072 rwx--[ anon ]
> 7ff21800 131048 131048 131048 rwx--[ anon ]
> 7ff12800 131068 131048 131048 rwx--[ anon ]
> 7ff23000 131044 131044 131044 rwx--[ anon ]
> 7ff0 131036 131036 131036 rwx--[ anon ]
> 7fefe000 131060 131036 131036 rwx--[ anon ]
> 7ff23c00 65536 65536 65536 rwx--[ anon ]
> 7ff0a400 65536 65536 65536 rwx--[ anon ]
> 7ff05400 65536 65536 65536 rwx--[ anon ]
> 7ff01c00 65536 65536 65536 rwx--[ anon ]
> 7fefb400 65536 65536 65536 rwx--[ anon ]
> 7ff22c00 65532 65532 65532 rwx--[ anon ]
> 7ff11000 65532 65532 65532 rwx--[ anon ]
> 7ff10c00 65532 65532 65532 rwx--[ anon ]
> 7ff0b800 65532 65532 65532 rwx--[ anon ]
> 7ff09c00 65532 65532 65532 rwx--[ anon ]
> 7feff800 65532 65532 65532 rwx--[ anon ]
> 7ff25000 65528 65528 65528 rwx--[ anon ]
> --More--
> {noformat}
> {noformat}
> $pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print
> $i}' | wc -l
> 146
> {noformat}
> Will MALLOC_AREANA have exactly 65536KB?
> Regionserver process has many anon pages, size varying from 64000 to 65536 KB
> that shown above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15309) GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
karthikp created HBASE-15309:
Summary: GZIP HFile Compression & Decompression Memory Leak,
Similar Issue HBASE-5516
Key: HBASE-15309
URL: https://issues.apache.org/jira/browse/HBASE-15309
Project: HBase
Issue Type: Bug
Components: HFile
Affects Versions: 0.94.6
Reporter: karthikp
Priority: Critical
I'd memory leak issue in regionserver process, VM and RSS memory continiously
increasing 64MB.
I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
I've huge write load and frequent minor compaction, We've used GZip hfile
compression.
Max java regionserver heap size is 32GB.
{noformat}
top - 14:28:30 up 201 days, 21:06, 3 users, load average: 5.67, 3.72, 3.31
Tasks: 803 total, 1 running, 802 sleeping, 0 stopped, 0 zombie
Cpu(s): 8.3%us, 2.1%sy, 0.0%ni, 85.9%id, 3.5%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 65932340k total, 63961912k used, 1970428k free, 2394528k buffers
Swap: 29659132k total,63532k used, 29595600k free, 1095268k cached
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
57335 hbase 20 0 46.4g 44g 9296 S 98.2 70.9 13319:10 java
{noformat}
{noformat}
[hbase@xx-hslave ~]$ pmap -x 57335 | sort -k 3 -nr | more
total kB48695984 46765792 46756512
7ff31246 33171072 33169464 33169464 rwx--[ anon ]
4010a000 1448552 1448552 1448552 rwx--[ anon ]
7ff2d181 612120 603124 603124 rwx--[ anon ]
7ff2fadff000 383364 383364 383364 rwx--[ anon ]
7ff0e800 131072 131072 131072 rwx--[ anon ]
7ff21800 131048 131048 131048 rwx--[ anon ]
7ff12800 131068 131048 131048 rwx--[ anon ]
7ff23000 131044 131044 131044 rwx--[ anon ]
7ff0 131036 131036 131036 rwx--[ anon ]
7fefe000 131060 131036 131036 rwx--[ anon ]
7ff23c00 65536 65536 65536 rwx--[ anon ]
7ff0a400 65536 65536 65536 rwx--[ anon ]
7ff05400 65536 65536 65536 rwx--[ anon ]
7ff01c00 65536 65536 65536 rwx--[ anon ]
7fefb400 65536 65536 65536 rwx--[ anon ]
7ff22c00 65532 65532 65532 rwx--[ anon ]
7ff11000 65532 65532 65532 rwx--[ anon ]
7ff10c00 65532 65532 65532 rwx--[ anon ]
7ff0b800 65532 65532 65532 rwx--[ anon ]
7ff09c00 65532 65532 65532 rwx--[ anon ]
7feff800 65532 65532 65532 rwx--[ anon ]
7ff25000 65528 65528 65528 rwx--[ anon ]
--More--
{noformat}
{noformat}
$pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print $i}'
| wc -l
146
{noformat}
Will MALLOC_AREANA have exactly 65536KB?
Regionserver process has many anon pages, size varying from 64000 to 65536 KB
that shown above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5516) GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90.
[
https://issues.apache.org/jira/browse/HBASE-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156704#comment-15156704
]
karthikp commented on HBASE-5516:
-
[~ram_krish] [~zhi...@ebaysf.com]
I'd similar memory leak issue in regionserver process, VM and RSS memory
continiously increasing 64MB.
I'm using CDH4-hbase 0.94.6 , Above fix are avialable the same and
MALLOC_ARENA_MAX=4 was set in hbase-env.sh
I've huge write load and frequent minor compaction, We've used GZip hfile
compression.
Max java regionserver heap size is 32GB.
{noformat}
top - 14:28:30 up 201 days, 21:06, 3 users, load average: 5.67, 3.72, 3.31
Tasks: 803 total, 1 running, 802 sleeping, 0 stopped, 0 zombie
Cpu(s): 8.3%us, 2.1%sy, 0.0%ni, 85.9%id, 3.5%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 65932340k total, 63961912k used, 1970428k free, 2394528k buffers
Swap: 29659132k total,63532k used, 29595600k free, 1095268k cached
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
57335 hbase 20 0 46.4g 44g 9296 S 98.2 70.9 13319:10 java
{noformat}
{noformat}
[hbase@pb1hsl5-hslave ~]$ pmap -x 57335 | sort -k 3 -nr | more
total kB48695984 46765792 46756512
7ff31246 33171072 33169464 33169464 rwx--[ anon ]
4010a000 1448552 1448552 1448552 rwx--[ anon ]
7ff2d181 612120 603124 603124 rwx--[ anon ]
7ff2fadff000 383364 383364 383364 rwx--[ anon ]
7ff0e800 131072 131072 131072 rwx--[ anon ]
7ff21800 131048 131048 131048 rwx--[ anon ]
7ff12800 131068 131048 131048 rwx--[ anon ]
7ff23000 131044 131044 131044 rwx--[ anon ]
7ff0 131036 131036 131036 rwx--[ anon ]
7fefe000 131060 131036 131036 rwx--[ anon ]
7ff23c00 65536 65536 65536 rwx--[ anon ]
7ff0a400 65536 65536 65536 rwx--[ anon ]
7ff05400 65536 65536 65536 rwx--[ anon ]
7ff01c00 65536 65536 65536 rwx--[ anon ]
7fefb400 65536 65536 65536 rwx--[ anon ]
7ff22c00 65532 65532 65532 rwx--[ anon ]
7ff11000 65532 65532 65532 rwx--[ anon ]
7ff10c00 65532 65532 65532 rwx--[ anon ]
7ff0b800 65532 65532 65532 rwx--[ anon ]
7ff09c00 65532 65532 65532 rwx--[ anon ]
7feff800 65532 65532 65532 rwx--[ anon ]
7ff25000 65528 65528 65528 rwx--[ anon ]
--More--
{noformat}
{noformat}
$pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print $i}'
| wc -l
146
{noformat}
Will MALLOC_AREANA create exactly 65536KB?
Regionserver process has many anon pages, size varying from 64000 to 65536 KB
that shown above.
Appreciate any help you can provide!!
> GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for
> 0.90.
>
>
> Key: HBASE-5516
> URL: https://issues.apache.org/jira/browse/HBASE-5516
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.5
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.7
>
> Attachments: HBASE-5516_2_0.90.patch, HBASE-5516_3_0.90.patch
>
>
> Usage of GZip is leading to resident memory leak in 0.90.
> We need to have something similar to HBASE-5387 in 0.90.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
