Carol Pearson created TRAFODION-2606:
----------------------------------------
Summary: Regionserver log on CDH 5.7 swamped with
global.memstore.upperLimit deprecated messages when Trafodion under high
transactional load
Key: TRAFODION-2606
URL: https://issues.apache.org/jira/browse/TRAFODION-2606
Project: Apache Trafodion
Issue Type: Bug
Components: dtm
Affects Versions: 2.2-incubating
Environment: Testing trafodion on AWS 4 node d2.2xlarge cluster with
Trafodion 2.2 daily build, using CDH 5.7. Ran a version of the OrderEntry
benchmark to produce workload
Reporter: Carol Pearson
Was running a basic Trafodion checkout to validate a recently installed
instance, and saw huge growth in the regionserver's log file. In under 5
hours, I was close to rolling a log, and after 10 hours, I was 122MB into the
second log. On a different cluster, it was days before the log rolled. So I
investigated further.
At the top, I found the usual startup stuff, and messages as I loaded my test
tables,
Then came pages and pages and pages and pages of errors about a deprecated call:
2017-05-03 19:52:18,858 WARN
org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil:
hbase.regionserver.global.memstore.upperLimit is deprecated by
hbase.regionserver.global.memstore.size
How many in this 5 hour old instance?
grep " hbase.regionserver.global.memstore.upperLimit is deprecated by
hbase.regionserver.global.memstore.size"
hbase-cmf-trafhbase-REGIONSERVER-glacier01.cluster.local.log.out | wc -l
1084218
[trafodion@glacier01 hbase]$
In other words, a lot of extra energy is spent logging in the hbase
regionserver. Over a million of them.
The loads to set up my tables are not transactional, so I didn't see the same
level of message generation.
To reproduce the problem, run a transactional workload on HBase 1.2/CDH 5.7. I
use a version of the order entry benchmark.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
