[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[ https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17854: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Pushed into master branch and closing issue. Thanks all for review. > Use StealJobQueue in HFileCleaner after HBASE-17215 > --- > > Key: HBASE-17854 > URL: https://issues.apache.org/jira/browse/HBASE-17854 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Yu Li >Assignee: Yu Li > Fix For: 2.0.0 > > Attachments: HBASE-17854.patch, HBASE-17854.v2.patch, > HBASE-17854.v3.patch, HBASE-17854.v4.patch, HBASE-17854.v5.patch, > HBASE-17854.v5.patch > > > In HBASE-17215 we use specific threads for deleting large/small (archived) > hfiles, and will improve it from below aspects in this JIRA: > 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs > from small queue, based on the experience that in real world there'll be much > more small hfiles > 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete > from the larger file in the queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[ https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17854: -- Attachment: HBASE-17854.v5.patch > Use StealJobQueue in HFileCleaner after HBASE-17215 > --- > > Key: HBASE-17854 > URL: https://issues.apache.org/jira/browse/HBASE-17854 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17854.patch, HBASE-17854.v2.patch, > HBASE-17854.v3.patch, HBASE-17854.v4.patch, HBASE-17854.v5.patch, > HBASE-17854.v5.patch > > > In HBASE-17215 we use specific threads for deleting large/small (archived) > hfiles, and will improve it from below aspects in this JIRA: > 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs > from small queue, based on the experience that in real world there'll be much > more small hfiles > 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete > from the larger file in the queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[ https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17854: -- Attachment: HBASE-17854.v5.patch Fix the awkward error introduced by v4 patch that caused UT failure... > Use StealJobQueue in HFileCleaner after HBASE-17215 > --- > > Key: HBASE-17854 > URL: https://issues.apache.org/jira/browse/HBASE-17854 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17854.patch, HBASE-17854.v2.patch, > HBASE-17854.v3.patch, HBASE-17854.v4.patch, HBASE-17854.v5.patch > > > In HBASE-17215 we use specific threads for deleting large/small (archived) > hfiles, and will improve it from below aspects in this JIRA: > 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs > from small queue, based on the experience that in real world there'll be much > more small hfiles > 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete > from the larger file in the queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[ https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17854: -- Attachment: HBASE-17854.v4.patch Add configuration for setting initial size of large/small queue, default to 10240 > Use StealJobQueue in HFileCleaner after HBASE-17215 > --- > > Key: HBASE-17854 > URL: https://issues.apache.org/jira/browse/HBASE-17854 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17854.patch, HBASE-17854.v2.patch, > HBASE-17854.v3.patch, HBASE-17854.v4.patch > > > In HBASE-17215 we use specific threads for deleting large/small (archived) > hfiles, and will improve it from below aspects in this JIRA: > 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs > from small queue, based on the experience that in real world there'll be much > more small hfiles > 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete > from the larger file in the queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[ https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17854: -- Attachment: HBASE-17854.v3.patch > Use StealJobQueue in HFileCleaner after HBASE-17215 > --- > > Key: HBASE-17854 > URL: https://issues.apache.org/jira/browse/HBASE-17854 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17854.patch, HBASE-17854.v2.patch, > HBASE-17854.v3.patch > > > In HBASE-17215 we use specific threads for deleting large/small (archived) > hfiles, and will improve it from below aspects in this JIRA: > 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs > from small queue, based on the experience that in real world there'll be much > more small hfiles > 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete > from the larger file in the queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[ https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17854: -- Attachment: HBASE-17854.v2.patch New patch resolves fingbugs issue. > Use StealJobQueue in HFileCleaner after HBASE-17215 > --- > > Key: HBASE-17854 > URL: https://issues.apache.org/jira/browse/HBASE-17854 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17854.patch, HBASE-17854.v2.patch > > > In HBASE-17215 we use specific threads for deleting large/small (archived) > hfiles, and will improve it from below aspects in this JIRA: > 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs > from small queue, based on the experience that in real world there'll be much > more small hfiles > 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete > from the larger file in the queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[ https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17854: -- Status: Patch Available (was: Open) Back to this one since HBASE-17215 is closed. Submit for HadoopQA to check. > Use StealJobQueue in HFileCleaner after HBASE-17215 > --- > > Key: HBASE-17854 > URL: https://issues.apache.org/jira/browse/HBASE-17854 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17854.patch > > > In HBASE-17215 we use specific threads for deleting large/small (archived) > hfiles, and will improve it from below aspects in this JIRA: > 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs > from small queue, based on the experience that in real world there'll be much > more small hfiles > 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete > from the larger file in the queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[ https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17854: -- Affects Version/s: (was: 2.0) 2.0.0 > Use StealJobQueue in HFileCleaner after HBASE-17215 > --- > > Key: HBASE-17854 > URL: https://issues.apache.org/jira/browse/HBASE-17854 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17854.patch > > > In HBASE-17215 we use specific threads for deleting large/small (archived) > hfiles, and will improve it from below aspects in this JIRA: > 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs > from small queue, based on the experience that in real world there'll be much > more small hfiles > 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete > from the larger file in the queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[ https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17854: -- Attachment: HBASE-17854.patch Uploading the initial patch, which is based on HBASE-17215 changes. Notice that after changing to {{StealJobQueue}}, the queue size is unbounded, and if there're already too many archived hfiles left for cleaning, there's potential risk of HMaster OOM. Searched for some implementation of bounded PriorityBlockingQueue but seems not that easy, see this [link|http://stackoverflow.com/questions/2341615/bounded-priorityblockingqueue] on stackoverflow. > Use StealJobQueue in HFileCleaner after HBASE-17215 > --- > > Key: HBASE-17854 > URL: https://issues.apache.org/jira/browse/HBASE-17854 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17854.patch > > > In HBASE-17215 we use specific threads for deleting large/small (archived) > hfiles, and will improve it from below aspects in this JIRA: > 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs > from small queue, based on the experience that in real world there'll be much > more small hfiles > 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete > from the larger file in the queues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)