[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107757#comment-15107757 ] Hudson commented on HBASE-14058: SUCCESS: Integrated in HBase-1.2-IT #401 (See [https://builds.apache.org/job/HBase-1.2-IT/401/]) HBASE-14058 Stabilizing default heap memory tuner (eclark: rev e738e69f8cc59581a454207483aca42e7f314396) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java > Stabilizing default heap memory tuner > - > > Key: HBASE-14058 > URL: https://issues.apache.org/jira/browse/HBASE-14058 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Abhilash >Assignee: Abhilash > Fix For: 2.0.0, 1.3.0 > > Attachments: 0001-Stabilizing-default-heap-memory-tuner.patch, > HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, > before_modifications.png > > > The memory tuner works well in general cases but when we have a work load > that is both read heavy as well as write heavy the tuner does too many > tuning. We should try to control the number of tuner operation and stabilize > it. The main problem was that the tuner thinks it is in steady state even if > it sees just one neutral tuner period thus does too many tuning operations > and too many reverts that too with large step sizes(step size was set to > maximum even after one neutral period). So to stop this I have thought of > these steps: > 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically > ~62% periods will lie outside this range, which means 62% of the data points > are considered either high or low which is too much. Use μ + δ*0.8 and μ - > δ*0.8 instead. On expectations it will decrease number of tuner operations > per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values > will be considered to be high and 31% will be considered to be low (2*0.31 * > 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% > will be high(2*0.22*0.22 ~ 0.10). > 2) Defining proper steady state by looking at past few periods(it is equal to > hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last > tuner operation. We say tuner is in steady state when last few tuner periods > were NEUTRAL. We keep decreasing step size unless it is extremely low. Then > leave system in that state for some time. > 3) Rather then decreasing step size only while reverting, decrease the > magnitude of step size whenever we are trying to revert tuning done in last > few periods(sum the changes of last few periods and compare to current step) > rather than just looking at last period. When its magnitude gets too low then > make tuner steps NEUTRAL(no operation). This will cause step size to > continuously decrease unless we reach steady state. After that tuning process > will restart (tuner step size rests again when we reach steady state). > 4) The tuning done in last few periods will be decaying sum of past tuner > steps with sign. This parameter will be positive for increase in memstore and > negative for increase in block cache. Rather than using arithmetic mean we > use this to give more priority to recent tuner steps. > Please see the attachments. One represents the size of memstore(green) and > size of block cache(blue) adjusted by tuner without these modification and > other with the above modifications. The x-axis is time axis and y-axis is the > fraction of heap memory available to memstore and block cache at that time(it > always sums up to 80%). I configured min/max ranges for both components to > 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and > 0.7). In both cases the tuner tries to distribute memory by giving ~15% to > memstore and ~65% to block cache. But the modified one does it much more > smoothly. > I got these results from YCSB test. The test was doing approximately 5000 > inserts and 500 reads per second (for one region server). The results can be > further fine tuned and number of tuner operation can be reduced with these > changes in configuration. > For more fine tuning: > a) lower max step size (suggested = 4%) > b) lower min step size ( default if also fine ) > To further decrease frequency of tuning operations: > c) increase the number of lookup periods ( in the tests it was just 10, > default is 60 ) > d) increase tuner period ( in the tests it was just 20 secs, default is > 60secs) > I used smaller tuner period/ number of look up periods to get more data > points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107903#comment-15107903 ] Hudson commented on HBASE-14058: FAILURE: Integrated in HBase-1.2 #512 (See [https://builds.apache.org/job/HBase-1.2/512/]) HBASE-14058 Stabilizing default heap memory tuner (eclark: rev e738e69f8cc59581a454207483aca42e7f314396) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java > Stabilizing default heap memory tuner > - > > Key: HBASE-14058 > URL: https://issues.apache.org/jira/browse/HBASE-14058 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Abhilash >Assignee: Abhilash > Fix For: 2.0.0, 1.3.0 > > Attachments: 0001-Stabilizing-default-heap-memory-tuner.patch, > HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, > before_modifications.png > > > The memory tuner works well in general cases but when we have a work load > that is both read heavy as well as write heavy the tuner does too many > tuning. We should try to control the number of tuner operation and stabilize > it. The main problem was that the tuner thinks it is in steady state even if > it sees just one neutral tuner period thus does too many tuning operations > and too many reverts that too with large step sizes(step size was set to > maximum even after one neutral period). So to stop this I have thought of > these steps: > 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically > ~62% periods will lie outside this range, which means 62% of the data points > are considered either high or low which is too much. Use μ + δ*0.8 and μ - > δ*0.8 instead. On expectations it will decrease number of tuner operations > per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values > will be considered to be high and 31% will be considered to be low (2*0.31 * > 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% > will be high(2*0.22*0.22 ~ 0.10). > 2) Defining proper steady state by looking at past few periods(it is equal to > hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last > tuner operation. We say tuner is in steady state when last few tuner periods > were NEUTRAL. We keep decreasing step size unless it is extremely low. Then > leave system in that state for some time. > 3) Rather then decreasing step size only while reverting, decrease the > magnitude of step size whenever we are trying to revert tuning done in last > few periods(sum the changes of last few periods and compare to current step) > rather than just looking at last period. When its magnitude gets too low then > make tuner steps NEUTRAL(no operation). This will cause step size to > continuously decrease unless we reach steady state. After that tuning process > will restart (tuner step size rests again when we reach steady state). > 4) The tuning done in last few periods will be decaying sum of past tuner > steps with sign. This parameter will be positive for increase in memstore and > negative for increase in block cache. Rather than using arithmetic mean we > use this to give more priority to recent tuner steps. > Please see the attachments. One represents the size of memstore(green) and > size of block cache(blue) adjusted by tuner without these modification and > other with the above modifications. The x-axis is time axis and y-axis is the > fraction of heap memory available to memstore and block cache at that time(it > always sums up to 80%). I configured min/max ranges for both components to > 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and > 0.7). In both cases the tuner tries to distribute memory by giving ~15% to > memstore and ~65% to block cache. But the modified one does it much more > smoothly. > I got these results from YCSB test. The test was doing approximately 5000 > inserts and 500 reads per second (for one region server). The results can be > further fine tuned and number of tuner operation can be reduced with these > changes in configuration. > For more fine tuning: > a) lower max step size (suggested = 4%) > b) lower min step size ( default if also fine ) > To further decrease frequency of tuning operations: > c) increase the number of lookup periods ( in the tests it was just 10, > default is 60 ) > d) increase tuner period ( in the tests it was just 20 secs, default is > 60secs) > I used smaller tuner period/ number of look up periods to get more data > points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14637726#comment-14637726 ] Hudson commented on HBASE-14058: SUCCESS: Integrated in HBase-1.3 #70 (See [https://builds.apache.org/job/HBase-1.3/70/]) HBASE-14058 Stabilizing default heap memory tuner (eclark: rev 89ecb93c95b63159cca427100b6003802870f1d8) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Fix For: 2.0.0, 1.3.0 Attachments: 0001-Stabilizing-default-heap-memory-tuner.patch, HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of look up periods to get more data points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14637685#comment-14637685 ] Hudson commented on HBASE-14058: FAILURE: Integrated in HBase-TRUNK #6671 (See [https://builds.apache.org/job/HBase-TRUNK/6671/]) HBASE-14058 Stabilizing default heap memory tuner (eclark: rev 20739542fdca185eb857813bf269d6262c11b652) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Fix For: 2.0.0, 1.3.0 Attachments: 0001-Stabilizing-default-heap-memory-tuner.patch, HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of look up periods to get more data points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14637723#comment-14637723 ] Hudson commented on HBASE-14058: SUCCESS: Integrated in HBase-1.3-IT #55 (See [https://builds.apache.org/job/HBase-1.3-IT/55/]) HBASE-14058 Stabilizing default heap memory tuner (eclark: rev 89ecb93c95b63159cca427100b6003802870f1d8) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Fix For: 2.0.0, 1.3.0 Attachments: 0001-Stabilizing-default-heap-memory-tuner.patch, HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of look up periods to get more data points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[
https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14637640#comment-14637640
]
Hadoop QA commented on HBASE-14058:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12746603/0001-Stabilizing-default-heap-memory-tuner.patch
against master branch at commit 5ec5552be0534dbf4b07ef6607741ae6f9ab0495.
ATTACHMENT ID: 12746603
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14862//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14862//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14862//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14862//console
This message is automatically generated.
Stabilizing default heap memory tuner
-
Key: HBASE-14058
URL: https://issues.apache.org/jira/browse/HBASE-14058
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 2.0.0, 1.2.0, 1.3.0
Reporter: Abhilash
Assignee: Abhilash
Fix For: 2.0.0, 1.3.0
Attachments: 0001-Stabilizing-default-heap-memory-tuner.patch,
HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png,
before_modifications.png
The memory tuner works well in general cases but when we have a work load
that is both read heavy as well as write heavy the tuner does too many
tuning. We should try to control the number of tuner operation and stabilize
it. The main problem was that the tuner thinks it is in steady state even if
it sees just one neutral tuner period thus does too many tuning operations
and too many reverts that too with large step sizes(step size was set to
maximum even after one neutral period). So to stop this I have thought of
these steps:
1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically
~62% periods will lie outside this range, which means 62% of the data points
are considered either high or low which is too much. Use μ + δ*0.8 and μ -
δ*0.8 instead. On expectations it will decrease number of tuner operations
per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values
will be considered to be high and 31% will be considered to be low (2*0.31 *
0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22%
will be high(2*0.22*0.22 ~ 0.10).
2) Defining proper steady state by looking at past few periods(it is equal to
hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last
tuner operation. We say tuner is in steady state when last few tuner periods
were NEUTRAL. We keep decreasing step size unless it is extremely low. Then
leave system in that state for some time.
3) Rather then decreasing step size only while reverting, decrease the
magnitude of step size whenever we are trying to revert tuning done in last
few periods(sum the changes of last few periods and compare to current step)
rather than just looking at last period. When its magnitude gets too low then
make tuner steps NEUTRAL(no operation). This will cause step size to
continuously decrease unless we reach
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630305#comment-14630305 ] Elliott Clark commented on HBASE-14058: --- +1 from me. I'll commit unless someone has another comment Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Attachments: HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of look up periods to get more data points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626032#comment-14626032 ] Anoop Sam John commented on HBASE-14058: The eviction usage related discussion let us have in another Jira. Let us get this improvement in first. Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Attachments: HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of look up periods to get more data points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626568#comment-14626568 ] Abhilash commented on HBASE-14058: -- Then lets get this patch in as there as no other reviews for this patch. Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Attachments: HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of look up periods to get more data points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626773#comment-14626773 ] Lars Hofhansl commented on HBASE-14058: --- bq. IMO it wont be a good idea to use it for stat calculation as it will tamper the actual trend and stats. Hmm... We want to capture the trend and not the current state. I.e. if we trend towards write load for a while then we tune for writeloads, if we trend towards readloads we tune for that, but only after a bit. Maybe I'm missing something. Lemme actually look at the patch. Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Attachments: HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of look up periods to get more data points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[
https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14624518#comment-14624518
]
Hadoop QA commented on HBASE-14058:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12744997/HBASE-14058-v1.patch
against master branch at commit 5e708746b8d301c2fb22a85b8756129147012374.
ATTACHMENT ID: 12744997
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14754//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14754//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14754//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14754//console
This message is automatically generated.
Stabilizing default heap memory tuner
-
Key: HBASE-14058
URL: https://issues.apache.org/jira/browse/HBASE-14058
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 2.0.0, 1.2.0, 1.3.0
Reporter: Abhilash
Assignee: Abhilash
Attachments: HBASE-14058-v1.patch, HBASE-14058.patch,
after_modifications.png, before_modifications.png
The memory tuner works well in general cases but when we have a work load
that is both read heavy as well as write heavy the tuner does too many
tuning. We should try to control the number of tuner operation and stabilize
it. The main problem was that the tuner thinks it is in steady state even if
it sees just one neutral tuner period thus does too many tuning operations
and too many reverts that too with large step sizes(step size was set to
maximum even after one neutral period). So to stop this I have thought of
these steps:
1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically
~62% periods will lie outside this range, which means 62% of the data points
are considered either high or low which is too much. Use μ + δ*0.8 and μ -
δ*0.8 instead. On expectations it will decrease number of tuner operations
per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values
will be considered to be high and 31% will be considered to be low (2*0.31 *
0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22%
will be high(2*0.22*0.22 ~ 0.10).
2) Defining proper steady state by looking at past few periods(it is equal to
hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last
tuner operation. We say tuner is in steady state when last few tuner periods
were NEUTRAL. We keep decreasing step size unless it is extremely low. Then
leave system in that state for some time.
3) Rather then decreasing step size only while reverting, decrease the
magnitude of step size whenever we are trying to revert tuning done in last
few periods(sum the changes of last few periods and compare to current step)
rather than just looking at last period. When its magnitude gets too low then
make tuner steps NEUTRAL(no operation). This will cause step size to
continuously decrease unless we reach steady state. After that tuning process
will restart (tuner step size rests again when we reach steady
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[
https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14624695#comment-14624695
]
Ted Yu commented on HBASE-14058:
{code}
Fetching the console output from the URL
Printing hanging tests
Printing Failing tests
Failing test : org.apache.hadoop.hbase.master.TestDistributedLogSplitting
Failing test : org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2
{code}
The failed tests were not related to the patch.
Stabilizing default heap memory tuner
-
Key: HBASE-14058
URL: https://issues.apache.org/jira/browse/HBASE-14058
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 2.0.0, 1.2.0, 1.3.0
Reporter: Abhilash
Assignee: Abhilash
Attachments: HBASE-14058-v1.patch, HBASE-14058.patch,
after_modifications.png, before_modifications.png
The memory tuner works well in general cases but when we have a work load
that is both read heavy as well as write heavy the tuner does too many
tuning. We should try to control the number of tuner operation and stabilize
it. The main problem was that the tuner thinks it is in steady state even if
it sees just one neutral tuner period thus does too many tuning operations
and too many reverts that too with large step sizes(step size was set to
maximum even after one neutral period). So to stop this I have thought of
these steps:
1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically
~62% periods will lie outside this range, which means 62% of the data points
are considered either high or low which is too much. Use μ + δ*0.8 and μ -
δ*0.8 instead. On expectations it will decrease number of tuner operations
per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values
will be considered to be high and 31% will be considered to be low (2*0.31 *
0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22%
will be high(2*0.22*0.22 ~ 0.10).
2) Defining proper steady state by looking at past few periods(it is equal to
hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last
tuner operation. We say tuner is in steady state when last few tuner periods
were NEUTRAL. We keep decreasing step size unless it is extremely low. Then
leave system in that state for some time.
3) Rather then decreasing step size only while reverting, decrease the
magnitude of step size whenever we are trying to revert tuning done in last
few periods(sum the changes of last few periods and compare to current step)
rather than just looking at last period. When its magnitude gets too low then
make tuner steps NEUTRAL(no operation). This will cause step size to
continuously decrease unless we reach steady state. After that tuning process
will restart (tuner step size rests again when we reach steady state).
4) The tuning done in last few periods will be decaying sum of past tuner
steps with sign. This parameter will be positive for increase in memstore and
negative for increase in block cache. Rather than using arithmetic mean we
use this to give more priority to recent tuner steps.
Please see the attachments. One represents the size of memstore(green) and
size of block cache(blue) adjusted by tuner without these modification and
other with the above modifications. The x-axis is time axis and y-axis is the
fraction of heap memory available to memstore and block cache at that time(it
always sums up to 80%). I configured min/max ranges for both components to
0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and
0.7). In both cases the tuner tries to distribute memory by giving ~15% to
memstore and ~65% to block cache. But the modified one does it much more
smoothly.
I got these results from YCSB test. The test was doing approximately 5000
inserts and 500 reads per second (for one region server). The results can be
further fine tuned and number of tuner operation can be reduced with these
changes in configuration.
For more fine tuning:
a) lower max step size (suggested = 4%)
b) lower min step size ( default if also fine )
To further decrease frequency of tuning operations:
c) increase the number of lookup periods ( in the tests it was just 10,
default is 60 )
d) increase tuner period ( in the tests it was just 20 secs, default is
60secs)
I used smaller tuner period/ number of look up periods to get more data
points.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14624759#comment-14624759 ] Lars Hofhansl commented on HBASE-14058: --- Similar to the design suggested in the patch, we could also have a moving average of the metrics we're tracking to make tuning decisions. So heavy read and write loads will ramp up slowly (as far as the tuner is concerned) and short spikes will not cause any tuner actions. So each time we calculate the metric we calculate it to metric = oldValue * (1-gamma) + newValue * gamma. Then we can tune gamma (high to make quick tuning decision, low to make them smooth). We can also make the tuner step size proportional to the change of the metric. (Sorry haven't looked at the patch, yet) Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Attachments: HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of look up periods to get more data points. -- This message was
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625006#comment-14625006 ] Abhilash commented on HBASE-14058: -- Thanks for the suggestions. We already maintain moving average and deviation of metrics over past periods(number of past periods to consider is configurable). Then compare mean and deviation to current metric value to decide tuning. Did you mean to use above value instead of current metric value while comparing or use it for that stats calculation itself ? IMO it wont be a good idea to use it for stat calculation as it will tamper the actual trend and stats. It will be very interesting/challenging to try to resize step size based on variation. But I feel it may lead to the same problem of over tuning. Instead of doing this resize every time. How about we do it only when we reset our step size ? Instead of resetting step size to maximum we can set it to some mediocre value depending on current variation and tuner will try to settle down things in following periods and then go to steady state. Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Attachments: HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[
https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623985#comment-14623985
]
Ted Yu commented on HBASE-14058:
Went through the patch:
{code}
+ * parameter idecayingAvgTunerStepSize/i which is sum of past tuner steps
with
{code}
Since the variable is sum, you can drop the 'Avg' in variable name.
{code}
+ public static final float DEFAULT_MIN_STEP_VALUE = 0.00125f; // 0.5%
{code}
Make the comment match actual percentage.
{code}
+// are likely to occur with probability 56%, 22%, 22% respectively.
Hence with there is at
{code}
Please correct the syntax for 'Hence with there is at'
Stabilizing default heap memory tuner
-
Key: HBASE-14058
URL: https://issues.apache.org/jira/browse/HBASE-14058
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 2.0.0, 1.2.0, 1.3.0
Reporter: Abhilash
Assignee: Abhilash
Attachments: HBASE-14058.patch, after_modifications.png,
before_modifications.png
The memory tuner works well in general cases but when we have a work load
that is both read heavy as well as write heavy the tuner does too many
tuning. We should try to control the number of tuner operation and stabilize
it. The main problem was that the tuner thinks it is in steady state even if
it sees just one neutral tuner period thus does too many tuning operations
and too many reverts that too with large step sizes(step size was set to
maximum even after one neutral period). So to stop this I have thought of
these steps:
1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically
~62% periods will lie outside this range, which means 62% of the data points
are considered either high or low which is too much. Use μ + δ*0.8 and μ -
δ*0.8 instead. On expectations it will decrease number of tuner operations
per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values
will be considered to be high and 31% will be considered to be low (2*0.31 *
0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22%
will be high(2*0.22*0.22 ~ 0.10).
2) Defining proper steady state by looking at past few periods(it is equal to
hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last
tuner operation. We say tuner is in steady state when last few tuner periods
were NEUTRAL. We keep decreasing step size unless it is extremely low. Then
leave system in that state for some time.
3) Rather then decreasing step size only while reverting, decrease the
magnitude of step size whenever we are trying to revert tuning done in last
few periods(sum the changes of last few periods and compare to current step)
rather than just looking at last period. When its magnitude gets too low then
make tuner steps NEUTRAL(no operation). This will cause step size to
continuously decrease unless we reach steady state. After that tuning process
will restart (tuner step size rests again when we reach steady state).
4) The tuning done in last few periods will be decaying sum of past tuner
steps with sign. This parameter will be positive for increase in memstore and
negative for increase in block cache. Rather than using arithmetic mean we
use this to give more priority to recent tuner steps.
Please see the attachments. One represents the size of memstore(green) and
size of block cache(blue) adjusted by tuner without these modification and
other with the above modifications. The x-axis is time axis and y-axis is the
fraction of heap memory available to memstore and block cache at that time(it
always sums up to 80%). I configured min/max ranges for both components to
0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and
0.7). In both cases the tuner tries to distribute memory by giving ~15% to
memstore and ~65% to block cache. But the modified one does it much more
smoothly.
I got these results from YCSB test. The test was doing approximately 5000
inserts and 500 reads per second (for one region server). The results can be
further fine tuned and number of tuner operation can be reduced with these
changes in configuration.
For more fine tuning:
a) lower max step size (suggested = 4%)
b) lower min step size ( default if also fine )
To further decrease frequency of tuning operations:
c) increase the number of lookup periods ( in the tests it was just 10,
default is 60 )
d) increase tuner period ( in the tests it was just 20 secs, default is
60secs)
I used smaller tuner period/ number of look up periods to get more data
points.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14624029#comment-14624029 ] Abhilash commented on HBASE-14058: -- Thanks a lot for the reviews Ted. I think idecayingTunerStepSizeSum/i will be correct name for it. I saw that percentage mistake yesterday, I was just waiting for further reviews. Will commit soon if there are no further reviews. Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Attachments: HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of look up periods to get more data points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623304#comment-14623304 ] Anoop Sam John commented on HBASE-14058: I will have to check the patch and description. But just one point to note. The evict count we make use in the tune decisions. The eviction count can be some time larger due to write also. In a write and read heavy system, the flushes are happening constantly and so compaction. After the compaction all the blocks from old files will get evicted. This also getting added to the evict count stat now! The concurrent reads would have made the flushed files into BC. If cache of write is ON that also will make it into BC. May be we have to check wrt this area also.. There should be separate tracking for evictions due to compaction (Say because of HFile removal) Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Attachments: HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623670#comment-14623670 ] Abhilash commented on HBASE-14058: -- Yes you are absolutely right. In read/write heavy scenario evictions due to compaction is a problem. We should try to distinguish between normal evictions and eviction caused due to compactions. But we are also using cache misses(for operations that were set to be cached while read) which gives more clear indication if we need to increase size of BC or not. It is more helpful in both cases that you mentioned i.e ignore evictions caused by compaction and caching on writes. Should we completely remove use of evictions form tuner decisions and just use cache misses ? Stabilizing default heap memory tuner - Key: HBASE-14058 URL: https://issues.apache.org/jira/browse/HBASE-14058 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.0, 1.2.0, 1.3.0 Reporter: Abhilash Assignee: Abhilash Attachments: HBASE-14058.patch, after_modifications.png, before_modifications.png The memory tuner works well in general cases but when we have a work load that is both read heavy as well as write heavy the tuner does too many tuning. We should try to control the number of tuner operation and stabilize it. The main problem was that the tuner thinks it is in steady state even if it sees just one neutral tuner period thus does too many tuning operations and too many reverts that too with large step sizes(step size was set to maximum even after one neutral period). So to stop this I have thought of these steps: 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically ~62% periods will lie outside this range, which means 62% of the data points are considered either high or low which is too much. Use μ + δ*0.8 and μ - δ*0.8 instead. On expectations it will decrease number of tuner operations per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values will be considered to be high and 31% will be considered to be low (2*0.31 * 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% will be high(2*0.22*0.22 ~ 0.10). 2) Defining proper steady state by looking at past few periods(it is equal to hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last tuner operation. We say tuner is in steady state when last few tuner periods were NEUTRAL. We keep decreasing step size unless it is extremely low. Then leave system in that state for some time. 3) Rather then decreasing step size only while reverting, decrease the magnitude of step size whenever we are trying to revert tuning done in last few periods(sum the changes of last few periods and compare to current step) rather than just looking at last period. When its magnitude gets too low then make tuner steps NEUTRAL(no operation). This will cause step size to continuously decrease unless we reach steady state. After that tuning process will restart (tuner step size rests again when we reach steady state). 4) The tuning done in last few periods will be decaying sum of past tuner steps with sign. This parameter will be positive for increase in memstore and negative for increase in block cache. Rather than using arithmetic mean we use this to give more priority to recent tuner steps. Please see the attachments. One represents the size of memstore(green) and size of block cache(blue) adjusted by tuner without these modification and other with the above modifications. The x-axis is time axis and y-axis is the fraction of heap memory available to memstore and block cache at that time(it always sums up to 80%). I configured min/max ranges for both components to 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and 0.7). In both cases the tuner tries to distribute memory by giving ~15% to memstore and ~65% to block cache. But the modified one does it much more smoothly. I got these results from YCSB test. The test was doing approximately 5000 inserts and 500 reads per second (for one region server). The results can be further fine tuned and number of tuner operation can be reduced with these changes in configuration. For more fine tuning: a) lower max step size (suggested = 4%) b) lower min step size ( default if also fine ) To further decrease frequency of tuning operations: c) increase the number of lookup periods ( in the tests it was just 10, default is 60 ) d) increase tuner period ( in the tests it was just 20 secs, default is 60secs) I used smaller tuner period/ number of look up periods to get more data points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[
https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623220#comment-14623220
]
Hadoop QA commented on HBASE-14058:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12744836/HBASE-14058.patch
against master branch at commit 5e708746b8d301c2fb22a85b8756129147012374.
ATTACHMENT ID: 12744836
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14748//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14748//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14748//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14748//console
This message is automatically generated.
Stabilizing default heap memory tuner
-
Key: HBASE-14058
URL: https://issues.apache.org/jira/browse/HBASE-14058
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 2.0.0, 1.2.0, 1.3.0
Reporter: Abhilash
Assignee: Abhilash
Attachments: HBASE-14058.patch, after_modifications.png,
before_modifications.png
The memory tuner works well in general cases but when we have a work load
that is both read heavy as well as write heavy the tuner does too many
tuning. We should try to control the number of tuner operation and stabilize
it. The main problem was that the tuner thinks it is in steady state even if
it sees just one neutral tuner period thus does too many tuning operations
and too many reverts that too with large step sizes(step size was set to
maximum even after one neutral period). So to stop this I have thought of
these steps:
1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically
~62% periods will lie outside this range, which means 62% of the data points
are considered either high or low which is too much. Use μ + δ*0.8 and μ -
δ*0.8 instead. On expectations it will decrease number of tuner operations
per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values
will be considered to be high and 31% will be considered to be low (2*0.31 *
0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22%
will be high(2*0.22*0.22 ~ 0.10).
2) Defining proper steady state by looking at past few periods(it is equal to
hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last
tuner operation. We say tuner is in steady state when last few tuner periods
were NEUTRAL. We keep decreasing step size unless it is extremely low. Then
leave system in that state for some time.
3) Rather then decreasing step size only while reverting, decrease the
magnitude of step size whenever we are trying to revert tuning done in last
few periods(sum the changes of last few periods and compare to current step)
rather than just looking at last period. When its magnitude gets too low then
make tuner steps NEUTRAL(no operation). This will cause step size to
continuously decrease unless we reach steady state. After that tuning process
will restart (tuner step size rests again when we reach steady state).
4) The tuning done in last
