[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed again. So resolving. As per Hadoop QA all the failed tests passes now. Thanks for the reviews. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-17738_10.patch, HBASE-17738_11.patch, > HBASE-17738_2.patch, HBASE-17738_2.patch, HBASE-17738_3.patch, > HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738_9.patch, > HBASE-17738_log.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Patch Available (was: Open) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-17738_10.patch, HBASE-17738_11.patch, > HBASE-17738_2.patch, HBASE-17738_2.patch, HBASE-17738_3.patch, > HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738_9.patch, > HBASE-17738_log.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_11.patch REsubmitting the patch with a simple change to hbase-server so that we run tests from that module also. Then will commit this. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-17738_10.patch, HBASE-17738_11.patch, > HBASE-17738_2.patch, HBASE-17738_2.patch, HBASE-17738_3.patch, > HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738_9.patch, > HBASE-17738_log.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Open (was: Patch Available) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-17738_10.patch, HBASE-17738_2.patch, > HBASE-17738_2.patch, HBASE-17738_3.patch, HBASE-17738_4.patch, > HBASE-17738_5_withoutUnsafe.patch, HBASE-17738_6_withoutUnsafe.patch, > HBASE-17738_8.patch, HBASE-17738_9.patch, HBASE-17738_log.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_10.patch The Math.round() was causing the issue. In case we need to create 25 buffers and 16 threads are available, then perthreadCount by doing round off was creating 2 bytebuffers and the last threads had to create -5 (which is wrong) and also per thread it was creating more than what is required. So now we do a Math.floor() and round it off to int. Creating this case with a test case is difficult so I just reproduced the above case by making ByteBufferarray non final and mocking the available thread count. So this test will fail based on the number of available threads in the system and not on any other system if the resulting calculation yields better result. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-17738_10.patch, HBASE-17738_2.patch, > HBASE-17738_2.patch, HBASE-17738_3.patch, HBASE-17738_4.patch, > HBASE-17738_5_withoutUnsafe.patch, HBASE-17738_6_withoutUnsafe.patch, > HBASE-17738_8.patch, HBASE-17738_9.patch, HBASE-17738_log.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Patch Available (was: Open) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-17738_10.patch, HBASE-17738_2.patch, > HBASE-17738_2.patch, HBASE-17738_3.patch, HBASE-17738_4.patch, > HBASE-17738_5_withoutUnsafe.patch, HBASE-17738_6_withoutUnsafe.patch, > HBASE-17738_8.patch, HBASE-17738_9.patch, HBASE-17738_log.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Open (was: Patch Available) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738_9.patch, > HBASE-17738_log.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_log.patch Adding a LOG to see the number of threads. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738_9.patch, > HBASE-17738_log.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Patch Available (was: Reopened) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738_9.patch, > HBASE-17738_log.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Resolution: Fixed Fix Version/s: 2.0.0-alpha-2 Status: Resolved (was: Patch Available) Corrected the test case name on commit. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738_9.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_9.patch Patch to fix the findbugs warning. Made the inner class static private. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738_9.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Patch Available (was: Open) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738_9.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Open (was: Patch Available) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Open (was: Patch Available) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_8.patch Updated patch implementing the suggestion with test case. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Patch Available (was: Open) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738_8.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_6_withoutUnsafe.patch Patch that splits the buffer creation based on the number of threads and creates only those many callables as the number of threads. This one and the previous patch both takes 11 secs only. How ever before the patch it was taking 34secs. The previous patch was easy to read and this has certain index manipulation to be done. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738_6_withoutUnsafe.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Open (was: Patch Available) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Patch Available (was: Open) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_5_withoutUnsafe.patch This patch is similar to the first patch 'HBASE-17738.patch' without the usage of Unsafe APIs. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Open (was: Patch Available) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Patch Available (was: Open) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_4.patch Adds a check in the code to see if the capacity is greater than the maxDirectMemory. Bit.java is package-private so we cannot make use of it. Thread count min count 10 gives good performance and the max is now controlled by the num of max available processors. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Open (was: Patch Available) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Patch Available (was: Open) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_3.patch Updated patch fixing the findbug comment and the review comment. One problem here is that even if the size of bucket cache is bigger than the available RAM no OOME is thrown and the server just starts. So may be better way could be to find the offheap memory configured and see if the current bufferCount allocation would exceed the available offheap memory if so throw error, if not proceed with the allocation. That is not done in this patch and just asking for suggestions. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Open (was: Patch Available) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Status: Patch Available (was: Reopened) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_2.patch Resubmitting for QA. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738_2.patch This patch uses the logic of creating bytebuffers as done by snazy/ohc. It makes the RS start up even faster. The 150G BB itself takes 5 secs only but the catch is that this does not honour the --XXDirectMemorySize param meter as the allocation is just done assuming we have enough memory. This is what snazy/ohc also makes use of. So for 125G of offheap memory allocation BEfore patch - 40 sec With HBASE-17738.patch - 10 to 12 secs With HBASE-17738_2.patch -5 to 6 secs. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Resolution: Fixed Status: Resolved (was: Patch Available) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Fix Version/s: 2.0.0 Affects Version/s: 2.0.0 Status: Patch Available (was: Open) > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17738: --- Attachment: HBASE-17738.patch Patch that speeds up the start up process of bucket cache. 75G bucket cache just starts in 7 secs whereas previously it takes 24 secs. 100G bucket cache starts in 10 secs whereas it took 40secs previously. Added a TODO in the patch to see if we need to make things configurable regarding the thread pool that does this init. Will once try this FileMMap engine to see what happens if parallely we try to allocate mmaped BBs. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
