[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215

2017-04-05 Thread Yu Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17854:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
   Status: Resolved  (was: Patch Available)

Pushed into master branch and closing issue. Thanks all for review.

> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch, 
> HBASE-17854.v3.patch, HBASE-17854.v4.patch, HBASE-17854.v5.patch, 
> HBASE-17854.v5.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived) 
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs 
> from small queue, based on the experience that in real world there'll be much 
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete 
> from the larger file in the queues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215

2017-04-05 Thread Yu Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17854:
--
Attachment: HBASE-17854.v5.patch

> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch, 
> HBASE-17854.v3.patch, HBASE-17854.v4.patch, HBASE-17854.v5.patch, 
> HBASE-17854.v5.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived) 
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs 
> from small queue, based on the experience that in real world there'll be much 
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete 
> from the larger file in the queues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215

2017-04-04 Thread Yu Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17854:
--
Attachment: HBASE-17854.v5.patch

Fix the awkward error introduced by v4 patch that caused UT failure...

> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch, 
> HBASE-17854.v3.patch, HBASE-17854.v4.patch, HBASE-17854.v5.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived) 
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs 
> from small queue, based on the experience that in real world there'll be much 
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete 
> from the larger file in the queues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215

2017-04-04 Thread Yu Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17854:
--
Attachment: HBASE-17854.v4.patch

Add configuration for setting initial size of large/small queue, default to 
10240

> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch, 
> HBASE-17854.v3.patch, HBASE-17854.v4.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived) 
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs 
> from small queue, based on the experience that in real world there'll be much 
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete 
> from the larger file in the queues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215

2017-04-03 Thread Yu Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17854:
--
Attachment: HBASE-17854.v3.patch

> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch, 
> HBASE-17854.v3.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived) 
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs 
> from small queue, based on the experience that in real world there'll be much 
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete 
> from the larger file in the queues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215

2017-04-01 Thread Yu Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17854:
--
Attachment: HBASE-17854.v2.patch

New patch resolves fingbugs issue.

> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived) 
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs 
> from small queue, based on the experience that in real world there'll be much 
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete 
> from the larger file in the queues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215

2017-03-31 Thread Yu Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17854:
--
Status: Patch Available  (was: Open)

Back to this one since HBASE-17215 is closed. Submit for HadoopQA to check.

> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived) 
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs 
> from small queue, based on the experience that in real world there'll be much 
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete 
> from the larger file in the queues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215

2017-03-30 Thread Duo Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17854:
--
Affects Version/s: (was: 2.0)
   2.0.0

> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived) 
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs 
> from small queue, based on the experience that in real world there'll be much 
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete 
> from the larger file in the queues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215

2017-03-30 Thread Yu Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17854:
--
Attachment: HBASE-17854.patch

Uploading the initial patch, which is based on HBASE-17215 changes.

Notice that after changing to {{StealJobQueue}}, the queue size is unbounded, 
and if there're already too many archived hfiles left for cleaning, there's 
potential risk of HMaster OOM.

Searched for some implementation of bounded PriorityBlockingQueue but seems not 
that easy, see this 
[link|http://stackoverflow.com/questions/2341615/bounded-priorityblockingqueue] 
on stackoverflow.

> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived) 
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs 
> from small queue, based on the experience that in real world there'll be much 
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete 
> from the larger file in the queues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)