[jira] [Updated] (HBASE-21000) Default limits for PressureAwareCompactionThroughputController are too low

[Biju Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Biju Nair updated HBASE-21000:
--
Component/s: Compaction

> Default limits for PressureAwareCompactionThroughputController are too low
> --
>
> Key: HBASE-21000
> URL: https://issues.apache.org/jira/browse/HBASE-21000
> Project: HBase
>  Issue Type: Improvement
>  Components: Compaction
>Affects Versions: 1.5.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: HBASE-21000-branch-1.patch, HBASE-21000.patch
>
>
> In PressureAwareCompactionThroughputController:
> {code:java}
> /**   
>   
> 
>  * A throughput controller which uses the follow schema to limit throughput   
>   
> 
>  *
>   
> 
>  * If compaction pressure is greater than 1.0, no limitation.
>   
> 
>  * In off peak hours, use a fixed throughput limitation   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_OFFPEAK}  
>   
> 
>  * In normal hours, the max throughput is tuned between   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND} and   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND}, using the 
> formula lower + 
>   
>  * (higher - lower) * compactionPressure, where compactionPressure is 
> in range [0.0, 1.0]  
>
>  *   
>   
> 
>  */
> {code}
> The lower and upper bounds are 10MB/sec and 20MB/sec, respectively:
> {code:java}
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   "hbase.hstore.compaction.throughput.higher.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   20L * 1024 * 1024;
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   "hbase.hstore.compaction.throughput.lower.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   10L * 1024 * 1024;
> {code}
> (In contrast, in PressureAwareFlushThroughputController the lower and upper 
> bounds are 10x of those limits, at 100MB/sec and 200MB/sec, respectively.)
> In fairly light load scenarios we see compaction quickly falls behind and 
> write clients are backed off or failing due to RegionTooBusy exceptions. 
> Although compaction throughput becomes unbounded after the store reaches the 
> blocking file count, in the lead up to this the default settings do not 
> provide enough bandwidth to stave off blocking. The defaults should be 
> increased. 
> I'm not sure what new good defaults make sense. We could start by doubling 
> them to 20MB/sec and 40MB/sec respectively. Might need to be doubled again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21000) Default limits for PressureAwareCompactionThroughputController are too low

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-21000:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Default limits for PressureAwareCompactionThroughputController are too low
> --
>
> Key: HBASE-21000
> URL: https://issues.apache.org/jira/browse/HBASE-21000
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.5.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: HBASE-21000-branch-1.patch, HBASE-21000.patch
>
>
> In PressureAwareCompactionThroughputController:
> {code:java}
> /**   
>   
> 
>  * A throughput controller which uses the follow schema to limit throughput   
>   
> 
>  *
>   
> 
>  * If compaction pressure is greater than 1.0, no limitation.
>   
> 
>  * In off peak hours, use a fixed throughput limitation   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_OFFPEAK}  
>   
> 
>  * In normal hours, the max throughput is tuned between   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND} and   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND}, using the 
> formula lower + 
>   
>  * (higher - lower) * compactionPressure, where compactionPressure is 
> in range [0.0, 1.0]  
>
>  *   
>   
> 
>  */
> {code}
> The lower and upper bounds are 10MB/sec and 20MB/sec, respectively:
> {code:java}
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   "hbase.hstore.compaction.throughput.higher.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   20L * 1024 * 1024;
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   "hbase.hstore.compaction.throughput.lower.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   10L * 1024 * 1024;
> {code}
> (In contrast, in PressureAwareFlushThroughputController the lower and upper 
> bounds are 10x of those limits, at 100MB/sec and 200MB/sec, respectively.)
> In fairly light load scenarios we see compaction quickly falls behind and 
> write clients are backed off or failing due to RegionTooBusy exceptions. 
> Although compaction throughput becomes unbounded after the store reaches the 
> blocking file count, in the lead up to this the default settings do not 
> provide enough bandwidth to stave off blocking. The defaults should be 
> increased. 
> I'm not sure what new good defaults make sense. We could start by doubling 
> them to 20MB/sec and 40MB/sec respectively. Might need to be doubled again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21000) Default limits for PressureAwareCompactionThroughputController are too low

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-21000:
---
Fix Version/s: (was: 2.2.0)
   (was: 3.0.0)

> Default limits for PressureAwareCompactionThroughputController are too low
> --
>
> Key: HBASE-21000
> URL: https://issues.apache.org/jira/browse/HBASE-21000
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.5.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: HBASE-21000-branch-1.patch, HBASE-21000.patch
>
>
> In PressureAwareCompactionThroughputController:
> {code:java}
> /**   
>   
> 
>  * A throughput controller which uses the follow schema to limit throughput   
>   
> 
>  *
>   
> 
>  * If compaction pressure is greater than 1.0, no limitation.
>   
> 
>  * In off peak hours, use a fixed throughput limitation   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_OFFPEAK}  
>   
> 
>  * In normal hours, the max throughput is tuned between   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND} and   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND}, using the 
> formula lower + 
>   
>  * (higher - lower) * compactionPressure, where compactionPressure is 
> in range [0.0, 1.0]  
>
>  *   
>   
> 
>  */
> {code}
> The lower and upper bounds are 10MB/sec and 20MB/sec, respectively:
> {code:java}
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   "hbase.hstore.compaction.throughput.higher.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   20L * 1024 * 1024;
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   "hbase.hstore.compaction.throughput.lower.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   10L * 1024 * 1024;
> {code}
> (In contrast, in PressureAwareFlushThroughputController the lower and upper 
> bounds are 10x of those limits, at 100MB/sec and 200MB/sec, respectively.)
> In fairly light load scenarios we see compaction quickly falls behind and 
> write clients are backed off or failing due to RegionTooBusy exceptions. 
> Although compaction throughput becomes unbounded after the store reaches the 
> blocking file count, in the lead up to this the default settings do not 
> provide enough bandwidth to stave off blocking. The defaults should be 
> increased. 
> I'm not sure what new good defaults make sense. We could start by doubling 
> them to 20MB/sec and 40MB/sec respectively. Might need to be doubled again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21000) Default limits for PressureAwareCompactionThroughputController are too low

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-21000:
---
Attachment: HBASE-21000-branch-1.patch

> Default limits for PressureAwareCompactionThroughputController are too low
> --
>
> Key: HBASE-21000
> URL: https://issues.apache.org/jira/browse/HBASE-21000
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.5.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21000-branch-1.patch, HBASE-21000.patch
>
>
> In PressureAwareCompactionThroughputController:
> {code:java}
> /**   
>   
> 
>  * A throughput controller which uses the follow schema to limit throughput   
>   
> 
>  *
>   
> 
>  * If compaction pressure is greater than 1.0, no limitation.
>   
> 
>  * In off peak hours, use a fixed throughput limitation   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_OFFPEAK}  
>   
> 
>  * In normal hours, the max throughput is tuned between   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND} and   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND}, using the 
> formula lower + 
>   
>  * (higher - lower) * compactionPressure, where compactionPressure is 
> in range [0.0, 1.0]  
>
>  *   
>   
> 
>  */
> {code}
> The lower and upper bounds are 10MB/sec and 20MB/sec, respectively:
> {code:java}
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   "hbase.hstore.compaction.throughput.higher.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   20L * 1024 * 1024;
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   "hbase.hstore.compaction.throughput.lower.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   10L * 1024 * 1024;
> {code}
> (In contrast, in PressureAwareFlushThroughputController the lower and upper 
> bounds are 10x of those limits, at 100MB/sec and 200MB/sec, respectively.)
> In fairly light load scenarios we see compaction quickly falls behind and 
> write clients are backed off or failing due to RegionTooBusy exceptions. 
> Although compaction throughput becomes unbounded after the store reaches the 
> blocking file count, in the lead up to this the default settings do not 
> provide enough bandwidth to stave off blocking. The defaults should be 
> increased. 
> I'm not sure what new good defaults make sense. We could start by doubling 
> them to 20MB/sec and 40MB/sec respectively. Might need to be doubled again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21000) Default limits for PressureAwareCompactionThroughputController are too low

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-21000:
---
Status: Patch Available  (was: Open)

> Default limits for PressureAwareCompactionThroughputController are too low
> --
>
> Key: HBASE-21000
> URL: https://issues.apache.org/jira/browse/HBASE-21000
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.5.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21000.patch
>
>
> In PressureAwareCompactionThroughputController:
> {code:java}
> /**   
>   
> 
>  * A throughput controller which uses the follow schema to limit throughput   
>   
> 
>  *
>   
> 
>  * If compaction pressure is greater than 1.0, no limitation.
>   
> 
>  * In off peak hours, use a fixed throughput limitation   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_OFFPEAK}  
>   
> 
>  * In normal hours, the max throughput is tuned between   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND} and   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND}, using the 
> formula lower + 
>   
>  * (higher - lower) * compactionPressure, where compactionPressure is 
> in range [0.0, 1.0]  
>
>  *   
>   
> 
>  */
> {code}
> The lower and upper bounds are 10MB/sec and 20MB/sec, respectively:
> {code:java}
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   "hbase.hstore.compaction.throughput.higher.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   20L * 1024 * 1024;
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   "hbase.hstore.compaction.throughput.lower.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   10L * 1024 * 1024;
> {code}
> (In contrast, in PressureAwareFlushThroughputController the lower and upper 
> bounds are 10x of those limits, at 100MB/sec and 200MB/sec, respectively.)
> In fairly light load scenarios we see compaction quickly falls behind and 
> write clients are backed off or failing due to RegionTooBusy exceptions. 
> Although compaction throughput becomes unbounded after the store reaches the 
> blocking file count, in the lead up to this the default settings do not 
> provide enough bandwidth to stave off blocking. The defaults should be 
> increased. 
> I'm not sure what new good defaults make sense. We could start by doubling 
> them to 20MB/sec and 40MB/sec respectively. Might need to be doubled again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21000) Default limits for PressureAwareCompactionThroughputController are too low

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-21000:
---
Attachment: HBASE-21000.patch

> Default limits for PressureAwareCompactionThroughputController are too low
> --
>
> Key: HBASE-21000
> URL: https://issues.apache.org/jira/browse/HBASE-21000
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.5.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21000.patch
>
>
> In PressureAwareCompactionThroughputController:
> {code:java}
> /**   
>   
> 
>  * A throughput controller which uses the follow schema to limit throughput   
>   
> 
>  *
>   
> 
>  * If compaction pressure is greater than 1.0, no limitation.
>   
> 
>  * In off peak hours, use a fixed throughput limitation   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_OFFPEAK}  
>   
> 
>  * In normal hours, the max throughput is tuned between   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND} and   
>   
> 
>  * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND}, using the 
> formula lower + 
>   
>  * (higher - lower) * compactionPressure, where compactionPressure is 
> in range [0.0, 1.0]  
>
>  *   
>   
> 
>  */
> {code}
> The lower and upper bounds are 10MB/sec and 20MB/sec, respectively:
> {code:java}
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   "hbase.hstore.compaction.throughput.higher.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
>   20L * 1024 * 1024;
>   public static final String 
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   "hbase.hstore.compaction.throughput.lower.bound";
>   private static final long 
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
>   10L * 1024 * 1024;
> {code}
> (In contrast, in PressureAwareFlushThroughputController the lower and upper 
> bounds are 10x of those limits, at 100MB/sec and 200MB/sec, respectively.)
> In fairly light load scenarios we see compaction quickly falls behind and 
> write clients are backed off or failing due to RegionTooBusy exceptions. 
> Although compaction throughput becomes unbounded after the store reaches the 
> blocking file count, in the lead up to this the default settings do not 
> provide enough bandwidth to stave off blocking. The defaults should be 
> increased. 
> I'm not sure what new good defaults make sense. We could start by doubling 
> them to 20MB/sec and 40MB/sec respectively. Might need to be doubled again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)