[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for your review, Ted. > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch, HBASE-18133.002.patch, > HBASE-18133.003.patch, HBASE-18133.004.patch, HBASE-18133.005.patch, > HBASE-18133.006.patch, HBASE-18133.007.patch, HBASE-18133.008.patch, > HBASE-18133.009.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Attachment: HBASE-18133.009.patch > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch, HBASE-18133.002.patch, > HBASE-18133.003.patch, HBASE-18133.004.patch, HBASE-18133.005.patch, > HBASE-18133.006.patch, HBASE-18133.007.patch, HBASE-18133.008.patch, > HBASE-18133.009.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Attachment: HBASE-18133.008.patch > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch, HBASE-18133.002.patch, > HBASE-18133.003.patch, HBASE-18133.004.patch, HBASE-18133.005.patch, > HBASE-18133.006.patch, HBASE-18133.007.patch, HBASE-18133.008.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Attachment: HBASE-18133.007.patch > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch, HBASE-18133.002.patch, > HBASE-18133.003.patch, HBASE-18133.004.patch, HBASE-18133.005.patch, > HBASE-18133.006.patch, HBASE-18133.007.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Attachment: HBASE-18133.006.patch > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch, HBASE-18133.002.patch, > HBASE-18133.003.patch, HBASE-18133.004.patch, HBASE-18133.005.patch, > HBASE-18133.006.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Attachment: HBASE-18133.005.patch > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch, HBASE-18133.002.patch, > HBASE-18133.003.patch, HBASE-18133.004.patch, HBASE-18133.005.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Attachment: HBASE-18133.004.patch > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch, HBASE-18133.002.patch, > HBASE-18133.003.patch, HBASE-18133.004.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Attachment: HBASE-18133.003.patch > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch, HBASE-18133.002.patch, > HBASE-18133.003.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Attachment: HBASE-18133.002.patch .002 is a rebase onto master after the space quota including hbase snapshots work. Will likely throw it up on RB. > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch, HBASE-18133.002.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Attachment: HBASE-18133.001.patch .001 which was a nice little change in the end. Still requires HBASE-17752 and HBASE-17840 to land. > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Status: Patch Available (was: Open) > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18133) Low-latency space quota size reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18133: --- Fix Version/s: 3.0.0 > Low-latency space quota size reports > > > Key: HBASE-18133 > URL: https://issues.apache.org/jira/browse/HBASE-18133 > Project: HBase > Issue Type: Improvement >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 3.0.0 > > Attachments: HBASE-18133.001.patch > > > Presently space quota enforcement relies on RegionServers sending reports to > the master about each Region that they host. This is done by periodically, > reading the cached size of each HFile in each Region (which was ultimately > computed from HDFS). > This means that the Master is unaware of Region size growth until the the > next time this chore in a RegionServer fires which is a fair amount of > latency (a few minutes, by default). Operations like flushes, compactions, > and bulk-loads are delayed even though the RegionServer is running those > operations locally. > Instead, we can create an API which these operations could invoke that would > automatically update the size of the Region being operated on. For example, a > successful flush can report that the size of a Region increased by the size > of the flush. A compaction can subtract the size of the input files of the > compaction and add in the size of the resulting file. > This de-couples the computation of a Region's size from sending the Region > sizes to the Master, allowing us to send reports more frequently, increasing > the responsiveness of the cluster to size changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
