[jira] [Commented] (HBASE-22300) SLB doesn't perform well with increase in number of regions
[
https://issues.apache.org/jira/browse/HBASE-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363299#comment-17363299
]
Biju Nair commented on HBASE-22300:
---
Go for it [~dmanning]
> SLB doesn't perform well with increase in number of regions
> ---
>
> Key: HBASE-22300
> URL: https://issues.apache.org/jira/browse/HBASE-22300
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Biju Nair
>Priority: Major
> Labels: balancer
> Attachments: CostFromRegionLoadFunctionNew.rtf
>
>
> With increase in number of regions in a cluster the number of steps taken by
> balancer in 30 sec (default balancer runtime) reduces noticeably. The
> following is the number of steps taken with by balancer with region loads set
> and running it without the loads being set i.e. cost functions using region
> loads are not fully exercised.
> {noformat}
> Nodes regions Tables # of steps # of steps
> with RS Load With no load
> 5 50 5 20 20
> 100 2000 110 104707 100
>
> 100 1 40 19911 100
>
> 200 10 400 870 100
> {noformat}
> As one would expect the reduced number of steps also makes the balancer take
> long time to get to an optimal cost. Note that only 2 data points were used
> in the region load histogram while in practice 15 region load data points are
> remembered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22535) TestShellRSGroups fails when run on JDK11
[
https://issues.apache.org/jira/browse/HBASE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856001#comment-16856001
]
Biju Nair commented on HBASE-22535:
---
Thanks [~jatsakthi] for the information.
> TestShellRSGroups fails when run on JDK11
> -
>
> Key: HBASE-22535
> URL: https://issues.apache.org/jira/browse/HBASE-22535
> Project: HBase
> Issue Type: Bug
> Components: java, shell
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Labels: jdk11
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.2.1, 2.1.6
>
> Attachments: hbase-22535.master.001.patch
>
>
> When built on Java 8 and unit tests are run on Java 11, TestShellRSGroups
> fails. A sample stacktrace:
> {code}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 19.293 s <<<
> FAILURE! - in org.apache.hadoop.hbase.client.rsgroup.TestShellRSGroups
> testRunShellTests(org.apache.hadoop.hbase.client.rsgroup.TestShellRSGroups)
> Time elapsed: 2.879 s <<< ERROR!
> org.jruby.embed.EvalFailedException: (RuntimeError) Shell unit tests failed.
> Check output file for details.
> at
> org.apache.hadoop.hbase.client.rsgroup.TestShellRSGroups.testRunShellTests(TestShellRSGroups.java:102)
> Caused by: org.jruby.exceptions.RaiseException: (RuntimeError) Shell unit
> tests failed. Check output file for details.
> Error:
> ^[[48;5;16;38;5;226;1mtest_Test_bogus_arguments(Hbase::RSGroupShellTest)^[[0m:
> TypeError: illegal access on constructor for type 'RSGroupAdminClient':
> class org.jruby.javasupport.JavaConstructor cannot access a member of class
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminClient with modifiers "public"
> /Users/jatsakthi/dev/hbase/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb:30:in
> `setup'
> org/jruby/RubyArray.java:1735:in `each'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22538) Prevent graceful_stop.sh from shutting down RS too early before finishing unloading regions
[ https://issues.apache.org/jira/browse/HBASE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855566#comment-16855566 ] Biju Nair commented on HBASE-22538: --- [~Jeongdae Kim] Thanks for the detailed explanation which helped understand the issue and the fix :-). > Prevent graceful_stop.sh from shutting down RS too early before finishing > unloading regions > --- > > Key: HBASE-22538 > URL: https://issues.apache.org/jira/browse/HBASE-22538 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.4.9 >Reporter: Jeongdae Kim >Assignee: Jeongdae Kim >Priority: Minor > Attachments: HBASE-22538.branch-1.4.001.patch > > > We can stop or restart region servers gracefully using graceful_stop.sh > command > This command should guarantee that all regions are moved out before shutting > down a region server. > However, sometimes i saw many requests failed while restarting a region > server with this command in our production clusters(v1.2.5) > affected clients got many RegionServerStoppedExceptions and exhausted retry > count. > I found it took 0.03 sec to move a region, it’s too fast. and, > moving(unloading) regions in the region server wasn’t finished, even didn’t > closed yet when region server got shutdown signal. > Because a region server serving regions (didn't be closed) were stopped, > clients got many exception (RegionServerStoppedException) > But, region_mover should wait until a region is served by other region > server(meta changed) > https://github.com/apache/hbase/blob/branch-1.2/bin/region_mover.rb#L153 > I figured out why this early shutdown happened. > a) our clusters use upper case hostname > b) region server makes ServerName with lowercase hostname, and it will be > sent to the master > https://github.com/apache/hbase/blob/branch-1.2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L542 > c) when updating meta, server name will keep its own case > https://github.com/apache/hbase/blob/branch-1.2/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java#L1527 > d) region_mover.rb just compare b) and c), so it is always false > https://github.com/apache/hbase/blob/branch-1.2/bin/region_mover.rb#L91 > https://github.com/apache/hbase/blob/branch-1.2/bin/region_mover.rb#L52 > I think region_mover should compare server name between master and meta with > the same case(lower) > With patch, I confirmed region_mover waited until finishing moving all > regions, then triggered shutting down region sever. (also observed only > RegionMovedException before shutdown log, and no exception after starting > shutdown) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22535) TestShellRSGroups fails when run on JDK11
[
https://issues.apache.org/jira/browse/HBASE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1682#comment-1682
]
Biju Nair commented on HBASE-22535:
---
Hi [~jatsakthi], For your v11 tests which flavor of Java JDK are you using?
Oracle or OpenJDK?
> TestShellRSGroups fails when run on JDK11
> -
>
> Key: HBASE-22535
> URL: https://issues.apache.org/jira/browse/HBASE-22535
> Project: HBase
> Issue Type: Bug
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-22535.master.001.patch
>
>
> When built on Java 8 and unit tests are run on Java 11, TestShellRSGroups
> fails. A sample stacktrace:
> {code}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 19.293 s <<<
> FAILURE! - in org.apache.hadoop.hbase.client.rsgroup.TestShellRSGroups
> testRunShellTests(org.apache.hadoop.hbase.client.rsgroup.TestShellRSGroups)
> Time elapsed: 2.879 s <<< ERROR!
> org.jruby.embed.EvalFailedException: (RuntimeError) Shell unit tests failed.
> Check output file for details.
> at
> org.apache.hadoop.hbase.client.rsgroup.TestShellRSGroups.testRunShellTests(TestShellRSGroups.java:102)
> Caused by: org.jruby.exceptions.RaiseException: (RuntimeError) Shell unit
> tests failed. Check output file for details.
> Error:
> ^[[48;5;16;38;5;226;1mtest_Test_bogus_arguments(Hbase::RSGroupShellTest)^[[0m:
> TypeError: illegal access on constructor for type 'RSGroupAdminClient':
> class org.jruby.javasupport.JavaConstructor cannot access a member of class
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminClient with modifiers "public"
> /Users/jatsakthi/dev/hbase/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb:30:in
> `setup'
> org/jruby/RubyArray.java:1735:in `each'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22531) The HFileReaderImpl#shouldUseHeap return the incorrect true when disabled BlockCache
[
https://issues.apache.org/jira/browse/HBASE-22531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855545#comment-16855545
]
Biju Nair commented on HBASE-22531:
---
Thanks for the clarification [~openinx]. You want to change the status of the
ticket to {{patch available}} so that it can be verified and merged?
> The HFileReaderImpl#shouldUseHeap return the incorrect true when disabled
> BlockCache
> -
>
> Key: HBASE-22531
> URL: https://issues.apache.org/jira/browse/HBASE-22531
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-22531.HBASE-21879.v1.patch,
> async-prof-pid-13311-alloc-4.svg, async-prof-pid-8590-alloc-2.svg
>
>
> I'm having a benchmark with block cache disabled for HBASE-21879 branch.
> Just caurious about why still so many heap allocation in the heap allocation
> flame graph [async-prof-pid-13311-alloc-4.svg |
> https://issues.apache.org/jira/secure/attachment/12970648/async-prof-pid-13311-alloc-4.svg],
>actually, I've set the following config, which means all allocation should
> be offheap, while it's not:
> {code}
> # Disable the block cache
> hfile.block.cache.size=0
> hbase.ipc.server.reservoir.minimal.allocating.size=0 # Let all allocation
> from pooled allocator.
> {code}
> Checked the code, I found the problem here:
> {code}
> private boolean shouldUseHeap(BlockType expectedBlockType) {
> if (cacheConf.getBlockCache() == null) {
> return false;
> } else if (!cacheConf.isCombinedBlockCache()) {
> // Block to cache in LruBlockCache must be an heap one. So just
> allocate block memory from
> // heap for saving an extra off-heap to heap copying.
> return true;
> }
> return expectedBlockType != null && !expectedBlockType.isData();
> }
> {code}
> Say, the CacheConfig#getBlockCache will return a Optional,
> which is always non-null:
> {code}
> /**
>* Returns the block cache.
>*
>* @return the block cache, or null if caching is completely disabled
>*/
> public Optional getBlockCache() {
> return Optional.ofNullable(this.blockCache);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855106#comment-16855106
]
Biju Nair commented on HBASE-21415:
---
Sorry for the delay [~elserj]. The updates in the patch looks fine.
> Admin#snapshot method documentation is misleading.
> --
>
> Key: HBASE-21415
> URL: https://issues.apache.org/jira/browse/HBASE-21415
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Philippe Laflamme
>Assignee: Philippe Laflamme
>Priority: Trivial
> Attachments: HBASE-21415.002.patch, HBASE-21415.003.patch,
> HBASE-21415.patch, HBASE-21415.patch
>
>
> The documentation for the {{Admin#snapshot}} function is misleading in
> regards to snapshot concurrency.
> It currently states the following:
> {quote}Only a single snapshot should be taken at a time for an instance of
> HBase, or results may be undefined (you can tell multiple HBase clusters to
> snapshot at the same time, but only one at a time for a single cluster).
> {quote}
> This seems to indicate that it's dangerous to run snapshots concurrently when
> in fact the HBase master will simply run them sequentially (as per this Slack
> thread [https://apache-hbase.slack.com/archives/C13K8NVAM/p1540994948005900)]
> The suggested fix is to simply remove this entire sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22531) The HFileReaderImpl#shouldUseHeap return the incorrect true when disabled BlockCache
[
https://issues.apache.org/jira/browse/HBASE-22531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855067#comment-16855067
]
Biju Nair commented on HBASE-22531:
---
{quote}
After applied the patch.v1, seems all of the allocation happened on heap now
{quote}
You meant off heap [~openinx]?
> The HFileReaderImpl#shouldUseHeap return the incorrect true when disabled
> BlockCache
> -
>
> Key: HBASE-22531
> URL: https://issues.apache.org/jira/browse/HBASE-22531
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-22531.HBASE-21879.v1.patch,
> async-prof-pid-13311-alloc-4.svg, async-prof-pid-8590-alloc-2.svg
>
>
> I'm having a benchmark with block cache disabled for HBASE-21879 branch.
> Just caurious about why still so many heap allocation in the heap allocation
> flame graph [async-prof-pid-13311-alloc-4.svg |
> https://issues.apache.org/jira/secure/attachment/12970648/async-prof-pid-13311-alloc-4.svg],
>actually, I've set the following config, which means all allocation should
> be offheap, while it's not:
> {code}
> # Disable the block cache
> hfile.block.cache.size=0
> hbase.ipc.server.reservoir.minimal.allocating.size=0 # Let all allocation
> from pooled allocator.
> {code}
> Checked the code, I found the problem here:
> {code}
> private boolean shouldUseHeap(BlockType expectedBlockType) {
> if (cacheConf.getBlockCache() == null) {
> return false;
> } else if (!cacheConf.isCombinedBlockCache()) {
> // Block to cache in LruBlockCache must be an heap one. So just
> allocate block memory from
> // heap for saving an extra off-heap to heap copying.
> return true;
> }
> return expectedBlockType != null && !expectedBlockType.isData();
> }
> {code}
> Say, the CacheConfig#getBlockCache will return a Optional,
> which is always non-null:
> {code}
> /**
>* Returns the block cache.
>*
>* @return the block cache, or null if caching is completely disabled
>*/
> public Optional getBlockCache() {
> return Optional.ofNullable(this.blockCache);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-12169) Document IPC binding options
[ https://issues.apache.org/jira/browse/HBASE-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16854993#comment-16854993 ] Biju Nair commented on HBASE-12169: --- Attached an updated patch. FYI [~busbey]. > Document IPC binding options > > > Key: HBASE-12169 > URL: https://issues.apache.org/jira/browse/HBASE-12169 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 0.98.0, 0.94.7, 0.99.0 >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-12169-001.PATCH, HBASE-12169-002.PATCH, > HBASE-12169.patch > > > HBASE-8148 added options to change binding component services, but there > aren't any docs for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-12169) Document IPC binding options
[ https://issues.apache.org/jira/browse/HBASE-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-12169: -- Attachment: HBASE-12169-002.PATCH > Document IPC binding options > > > Key: HBASE-12169 > URL: https://issues.apache.org/jira/browse/HBASE-12169 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 0.98.0, 0.94.7, 0.99.0 >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-12169-001.PATCH, HBASE-12169-002.PATCH, > HBASE-12169.patch > > > HBASE-8148 added options to change binding component services, but there > aren't any docs for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22514) make RSGroupAdminClient public
[ https://issues.apache.org/jira/browse/HBASE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16854432#comment-16854432 ] Biju Nair commented on HBASE-22514: --- FYI. [~toffer], [~thiruvel] > make RSGroupAdminClient public > -- > > Key: HBASE-22514 > URL: https://issues.apache.org/jira/browse/HBASE-22514 > Project: HBase > Issue Type: Improvement >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Major > Attachments: HBASE-22514.master.001.patch, > image-2019-05-31-18-25-38-217.png > > > The class RSGroupAdminClient is not public > we need to use java api RSGroupAdminClient to manager RSG > so RSGroupAdminClient should be public > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22504) Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
[
https://issues.apache.org/jira/browse/HBASE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22504:
--
Component/s: BucketCache
> Optimize the MultiByteBuff#get(ByteBuffer, offset, len)
> ---
>
> Key: HBASE-22504
> URL: https://issues.apache.org/jira/browse/HBASE-22504
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-22504.HBASE-21879.v01.patch
>
>
> In HBASE-22483, we saw that the BucketCacheWriter thread was quite busy
> [^BucketCacheWriter-is-busy.png], the flame graph also indicated that the
> ByteBufferArray#internalTransfer cost ~6% CPU (see
> [async-prof-pid-25042-cpu-1.svg|https://issues.apache.org/jira/secure/attachment/12970294/async-prof-pid-25042-cpu-1.svg]).
> because we used the hbase.ipc.server.allocator.buffer.size=64KB, each
> HFileBlock will be backend by a MultiByteBuff: one 64KB offheap ByteBuffer
> and one small heap ByteBuffer.
> The path is depending on the MultiByteBuff#get(ByteBuffer, offset, len) now:
> {code:java}
> RAMQueueEntry#writeToCache
> |--> ByteBufferIOEngine#write
> |--> ByteBufferArray#internalTransfer
> |--> ByteBufferArray$WRITER
> |--> MultiByteBuff#get(ByteBuffer, offset, len)
> {code}
> While the MultiByteBuff#get impl is simple and crude now, can optimze this
> implementation:
> {code:java}
> @Override
> public void get(ByteBuffer out, int sourceOffset,
> int length) {
> checkRefCount();
> // Not used from real read path actually. So not going with
> // optimization
> for (int i = 0; i < length; ++i) {
> out.put(this.get(sourceOffset + i));
> }
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22483) It's better to use 65KB as the default buffer size in ByteBuffAllocator
[ https://issues.apache.org/jira/browse/HBASE-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-22483: -- Component/s: BucketCache > It's better to use 65KB as the default buffer size in ByteBuffAllocator > --- > > Key: HBASE-22483 > URL: https://issues.apache.org/jira/browse/HBASE-22483 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: 121240.stack, BucketCacheWriter-is-busy.png, > checksum-stacktrace.png, with-buffer-size-64KB.png, with-buffer-size-65KB.png > > > There're some reason why it's better to choose 65KB as the default buffer > size: > 1. Almost all of the data block have a block size: 64KB + delta, whose delta > is very small, depends on the size of lastKeyValue. If we use the default > hbase.ipc.server.allocator.buffer.size=64KB, then each block will be > allocated as a MultiByteBuff: one 64KB DirectByteBuffer and delta bytes > HeapByteBuffer, the HeapByteBuffer will increase the GC pressure. Ideally, we > should let the data block to be allocated as a SingleByteBuff, it has simpler > data structure, faster access speed, less heap usage... > 2. In my benchmark, I found some checksum stack traces . (see > [checksum-stacktrace.png > |https://issues.apache.org/jira/secure/attachment/12969905/checksum-stacktrace.png]) > > Since the block are MultiByteBuff, so we have to calculate the checksum by > an temp heap copying ( see HBASE-21917), while if we're a SingleByteBuff, we > can speed the checksum by calling the hadoop' checksum in native lib, it's > more faster. > 3. Seems the BucketCacheWriters were always busy because of the higher cost > of copying from MultiByteBuff to DirectByteBuffer. For SingleByteBuff, we > can just use the unsafe array copying while for MultiByteBuff we have to copy > byte one by one. > Anyway, I will give a benchmark for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22513) Admin#getQuota does not work correctly if exceedThrottleQuota is set
[
https://issues.apache.org/jira/browse/HBASE-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853984#comment-16853984
]
Biju Nair commented on HBASE-22513:
---
Interesting find :-).
> Admin#getQuota does not work correctly if exceedThrottleQuota is set
>
>
> Key: HBASE-22513
> URL: https://issues.apache.org/jira/browse/HBASE-22513
> Project: HBase
> Issue Type: Bug
> Components: Quotas
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
>
> Admin#getQuota get nothing if exceedThrottleQuota is set, because
> exceedThrottleQuota is a special row key in quota table and can not be parsed
> to a QuotaSettings.
> The shell command results are as follows:
> {code:java}
> hbase(main):018:0> list_quotas
> OWNER QUOTAS
> 0 row(s)
> Took 0.0342 seconds
> hbase(main):019:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> exceedThrottleQuota column=q:s, timestamp=1559199136449, value=\x00
> n.ang column=q:s, timestamp=1559122413584,
> value=PBUF\x12\x08*\x06\x08\x04\x10" \x02
> n.ns1 column=q:s, timestamp=1559203286943,
> value=PBUF\x12\x10\x1A\x06\x08\x04\x10\x05 \x02*\x06\x08\x04\x10\x05
> \x02\x1A\x0A\x08\x
> 80\x80\x80\x80\x80\xC0\x0C\x10\x03
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22513) Admin#getQuota does not work correctly if exceedThrottleQuota is set
[
https://issues.apache.org/jira/browse/HBASE-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22513:
--
Component/s: Quotas
> Admin#getQuota does not work correctly if exceedThrottleQuota is set
>
>
> Key: HBASE-22513
> URL: https://issues.apache.org/jira/browse/HBASE-22513
> Project: HBase
> Issue Type: Bug
> Components: Quotas
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
>
> Admin#getQuota get nothing if exceedThrottleQuota is set, because
> exceedThrottleQuota is a special row key in quota table and can not be parsed
> to a QuotaSettings.
> The shell command results are as follows:
> {code:java}
> hbase(main):018:0> list_quotas
> OWNER QUOTAS
> 0 row(s)
> Took 0.0342 seconds
> hbase(main):019:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> exceedThrottleQuota column=q:s, timestamp=1559199136449, value=\x00
> n.ang column=q:s, timestamp=1559122413584,
> value=PBUF\x12\x08*\x06\x08\x04\x10" \x02
> n.ns1 column=q:s, timestamp=1559203286943,
> value=PBUF\x12\x10\x1A\x06\x08\x04\x10\x05 \x02*\x06\x08\x04\x10\x05
> \x02\x1A\x0A\x08\x
> 80\x80\x80\x80\x80\xC0\x0C\x10\x03
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-12169) Document IPC binding options
[
https://issues.apache.org/jira/browse/HBASE-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853975#comment-16853975
]
Biju Nair commented on HBASE-12169:
---
Looking at the
[xslt|https://github.com/apache/hbase/blob/4ce6f486d063553a78ed1d60670e68564d61a483/src/main/xslt/configuration_to_asciidoc_chapter.xsl#L66],
only the values in {{hbase-default}} section gets updated with the data from
{{hbase-default.xml}} i.e. we can hardcode the default values for these two
properties in {{ops_mgmt}} section as done in {{connection timeouts}} section
which has the risk of document going out of sync with the default values if in
the future when there is any change. And the other option is to add these in
the {{hbase-default}} section which the initial patch did. Thoughts?
> Document IPC binding options
>
>
> Key: HBASE-12169
> URL: https://issues.apache.org/jira/browse/HBASE-12169
> Project: HBase
> Issue Type: Task
> Components: documentation
>Affects Versions: 0.98.0, 0.94.7, 0.99.0
>Reporter: Sean Busbey
>Assignee: Mike Drob
>Priority: Minor
> Attachments: HBASE-12169-001.PATCH, HBASE-12169.patch
>
>
> HBASE-8148 added options to change binding component services, but there
> aren't any docs for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-12169) Document IPC binding options
[
https://issues.apache.org/jira/browse/HBASE-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853971#comment-16853971
]
Biju Nair commented on HBASE-12169:
---
Assuming adding {{hbase.master.dns.interface}} and
{{hbase.regionserver.dns.interface}} properties with {{default}} values should
fix it. Let me update the patch.
> Document IPC binding options
>
>
> Key: HBASE-12169
> URL: https://issues.apache.org/jira/browse/HBASE-12169
> Project: HBase
> Issue Type: Task
> Components: documentation
>Affects Versions: 0.98.0, 0.94.7, 0.99.0
>Reporter: Sean Busbey
>Assignee: Mike Drob
>Priority: Minor
> Attachments: HBASE-12169-001.PATCH, HBASE-12169.patch
>
>
> HBASE-8148 added options to change binding component services, but there
> aren't any docs for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22415) HBOSS: Reduce log verbosity in ZKTreeLockManager when waiting on a parent/child node lock
[ https://issues.apache.org/jira/browse/HBASE-22415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853319#comment-16853319 ] Biju Nair commented on HBASE-22415: --- Changes looks fine to me [~busbey]. +1 (non-binding). Thanks for checking. > HBOSS: Reduce log verbosity in ZKTreeLockManager when waiting on a > parent/child node lock > - > > Key: HBASE-22415 > URL: https://issues.apache.org/jira/browse/HBASE-22415 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Labels: HBOSS > Fix For: hbase-filesystem-1.0.0-alpha1 > > Attachments: HBASE-22415.master.001.patch, > HBASE-22415.master.002.patch > > > On normal read/write workloads, it will be very often that requests will > reach an temporary "locked" node by some other requests, causing process logs > to flood with the messages been currently logged as WARN. To improve > supportability, am proposing a patch that reduces these logging to DEBUG. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22489) When regionServer stopping open region cause server aborting
[
https://issues.apache.org/jira/browse/HBASE-22489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22489:
--
Component/s: regionserver
> When regionServer stopping open region cause server aborting
>
>
> Key: HBASE-22489
> URL: https://issues.apache.org/jira/browse/HBASE-22489
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: binlijin
>Priority: Major
>
> When regionserver stopping, region open will ABORTING regionserver, need to
> take care it.
> {code}
> 2019-05-28 20:32:48,672 WARN
> [RS_OPEN_REGION-regionserver/hbaseperf-8:16020-15]
> handler.AssignRegionHandler: Fatal error occurred while opening region
> user,f768,1559033479968.6f1508a4480447698cf8b04cd850698f., aborting...
> org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server
> 10.11.45.52,16020,1559046489201 stopping
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1505)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2206)
> at
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:139)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-05-28 20:32:48,677 ERROR
> [RS_OPEN_REGION-regionserver/hbaseperf-8:16020-15]
> regionserver.HRegionServer: * ABORTING region server
> 10.11.45.52,16020,1559046489201: Failed to open region
> user,f768,1559033479968.6f1508a4480447698cf8b04cd850698f. and can not
> recover *
> org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server
> 10.11.45.52,16020,1559046489201 stopping
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1505)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2206)
> at
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:139)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-05-28 20:32:48,677 ERROR
> [RS_OPEN_REGION-regionserver/hbaseperf-8:16020-15]
> regionserver.HRegionServer: RegionServer abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22493) HBOSS: Document supported hadoop versions.
[ https://issues.apache.org/jira/browse/HBASE-22493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-22493: -- Component/s: documentation > HBOSS: Document supported hadoop versions. > -- > > Key: HBASE-22493 > URL: https://issues.apache.org/jira/browse/HBASE-22493 > Project: HBase > Issue Type: Task > Components: documentation, hboss >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Labels: HBOSS > Attachments: HBASE-22493.master.001 > > > This mentions supported and tested hadoop versions (2.9.2 and 3.2.0) to hboss > readme file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22493) HBOSS: Document supported hadoop versions.
[ https://issues.apache.org/jira/browse/HBASE-22493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-22493: -- Component/s: hboss > HBOSS: Document supported hadoop versions. > -- > > Key: HBASE-22493 > URL: https://issues.apache.org/jira/browse/HBASE-22493 > Project: HBase > Issue Type: Task > Components: hboss >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Labels: HBOSS > Attachments: HBASE-22493.master.001 > > > This mentions supported and tested hadoop versions (2.9.2 and 3.2.0) to hboss > readme file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22494) Use different wals for different write workload table
[ https://issues.apache.org/jira/browse/HBASE-22494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-22494: -- Component/s: wal > Use different wals for different write workload table > - > > Key: HBASE-22494 > URL: https://issues.apache.org/jira/browse/HBASE-22494 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: binlijin >Priority: Major > > I observe some cluster have many wal files because there are different tables > have different > write workloads, there share the same wal, and write heavy workloads table > make wal files increasing fast, but write light workloads table make wal > files archive slow... > There is a possible way is that make them in different namespace and use > multiwal(namespace Strategies) to make them use different wals, but it need > to design it at first before providing service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-12169) Document IPC binding options
[ https://issues.apache.org/jira/browse/HBASE-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850303#comment-16850303 ] Biju Nair commented on HBASE-12169: --- [~busbey], the new patch helps? > Document IPC binding options > > > Key: HBASE-12169 > URL: https://issues.apache.org/jira/browse/HBASE-12169 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 0.98.0, 0.94.7, 0.99.0 >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-12169-001.PATCH, HBASE-12169.patch > > > HBASE-8148 added options to change binding component services, but there > aren't any docs for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-12169) Document IPC binding options
[ https://issues.apache.org/jira/browse/HBASE-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-12169: -- Attachment: HBASE-12169-001.PATCH > Document IPC binding options > > > Key: HBASE-12169 > URL: https://issues.apache.org/jira/browse/HBASE-12169 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 0.98.0, 0.94.7, 0.99.0 >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-12169-001.PATCH, HBASE-12169.patch > > > HBASE-8148 added options to change binding component services, but there > aren't any docs for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15898) Document G1GC Recommendations
[ https://issues.apache.org/jira/browse/HBASE-15898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850292#comment-16850292 ] Biju Nair commented on HBASE-15898: --- Attached an updated patch which takes care of the previous comments. Hope this helps to get this information into the doc [~stack]. > Document G1GC Recommendations > - > > Key: HBASE-15898 > URL: https://issues.apache.org/jira/browse/HBASE-15898 > Project: HBase > Issue Type: Bug > Components: documentation, java >Affects Versions: 2.0.0 >Reporter: Misty Linville >Assignee: Misty Linville >Priority: Major > Attachments: HBASE-15898-001.PATCH, HBASE-15898.patch > > > Document G1GC recommendations for HBase -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-15898) Document G1GC Recommendations
[ https://issues.apache.org/jira/browse/HBASE-15898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-15898: -- Attachment: HBASE-15898-001.PATCH > Document G1GC Recommendations > - > > Key: HBASE-15898 > URL: https://issues.apache.org/jira/browse/HBASE-15898 > Project: HBase > Issue Type: Bug > Components: documentation, java >Affects Versions: 2.0.0 >Reporter: Misty Linville >Assignee: Misty Linville >Priority: Major > Attachments: HBASE-15898-001.PATCH, HBASE-15898.patch > > > Document G1GC recommendations for HBase -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-18280) Manual Array Copy Cleanup: Automated
[
https://issues.apache.org/jira/browse/HBASE-18280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850053#comment-16850053
]
Biju Nair edited comment on HBASE-18280 at 5/28/19 7:24 PM:
[~chia7712] Will {{System.arraycopy}} perform better than the manual looping?
If so can this be merged? There are some graphs on
[performance|https://stackoverflow.com/questions/18638743/is-it-better-to-use-system-arraycopy-than-a-for-loop-for-copying-arrays]
comparison between manual looping, Arrays.copy and System.arraycopy.
was (Author: gsbiju):
[~chia7712] Will {{System.arraycopy}} perform better than the manual looping?
If so can this be merged? There are some graphs on
[performance|https://stackoverflow.com/questions/18638743/is-it-better-to-use-system-arraycopy-than-a-for-loop-for-copying-arrays].
> Manual Array Copy Cleanup: Automated
>
>
> Key: HBASE-18280
> URL: https://issues.apache.org/jira/browse/HBASE-18280
> Project: HBase
> Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: 0001-HBASE-18280-Remove-Manual-Array-Copies.patch
>
>
> Remove Manual Array Copies: Style Issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18280) Manual Array Copy Cleanup: Automated
[
https://issues.apache.org/jira/browse/HBASE-18280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850053#comment-16850053
]
Biju Nair commented on HBASE-18280:
---
[~chia7712] Will {{System.arraycopy}} perform better than the manual looping?
If so can this be merged? There are some graphs on
[performance|https://stackoverflow.com/questions/18638743/is-it-better-to-use-system-arraycopy-than-a-for-loop-for-copying-arrays].
> Manual Array Copy Cleanup: Automated
>
>
> Key: HBASE-18280
> URL: https://issues.apache.org/jira/browse/HBASE-18280
> Project: HBase
> Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: 0001-HBASE-18280-Remove-Manual-Array-Copies.patch
>
>
> Remove Manual Array Copies: Style Issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17233) See if we should replace System.arrayCopy with Arrays.copyOfRange
[
https://issues.apache.org/jira/browse/HBASE-17233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850046#comment-16850046
]
Biju Nair commented on HBASE-17233:
---
So the conclusion is {{Arrays.copyOfRange}} is better performing than
{{System.arraycopy}}. Correct? By how much?
> See if we should replace System.arrayCopy with Arrays.copyOfRange
> -
>
> Key: HBASE-17233
> URL: https://issues.apache.org/jira/browse/HBASE-17233
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Major
>
> Just saw this interesting comment in PB code. Since we deal with byte[]
> extensively (when we are onheap) we do lot of copies too.
> {code}
> * One of the noticeable costs of copying a byte[] into a new array using
>* {@code System.arraycopy} is nullification of a new buffer before the
> copy. It has been shown
>* the Hotspot VM is capable to intrisicfy {@code Arrays.copyOfRange}
> operation to avoid this
>* expensive nullification and provide substantial performance gain.
> Unfortunately this does not
>* hold on Android runtimes and could make the copy slightly slower due to
> additional code in
>* the {@code Arrays.copyOfRange}.
> {code}
> So since we are hotspot VM we could see if the places we use System.arrayCopy
> can be replaced with Arrays.copyOfRange.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-12057) solaris -d64 jvm crash getLong workaround
[ https://issues.apache.org/jira/browse/HBASE-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-12057: -- Status: Open (was: Patch Available) There were no patches attached. > solaris -d64 jvm crash getLong workaround > - > > Key: HBASE-12057 > URL: https://issues.apache.org/jira/browse/HBASE-12057 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.98.6.1 > Environment: solaris -d64 >Reporter: Valera V. Kharseko >Priority: Major > Attachments: hs_err_pid18208.log > > > solaris 64 fix based on > https://issues.apache.org/jira/secure/attachment/12625582/solaris_unsafe_fix.patch > https://bugs.openjdk.java.net/browse/JDK-8021574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-11157) [hbck] NotServingRegionException: Received close for but we are not serving it
[
https://issues.apache.org/jira/browse/HBASE-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-11157:
--
Resolution: Invalid
Status: Resolved (was: Patch Available)
The code in master is changed and the patch is not applicable anymore. Closing
this as {{invalid}}.
> [hbck] NotServingRegionException: Received close for but we are
> not serving it
> ---
>
> Key: HBASE-11157
> URL: https://issues.apache.org/jira/browse/HBASE-11157
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 0.94.13, 0.94.19
>Reporter: dailidong
>Priority: Trivial
> Attachments: HBASE-11157-v2.patch, HBASE-11157.patch
>
>
> if hbck close a region then meet a NotServerRegionException,hbck will hang up
> . we will close the region on the regionserver, but this regionserver is not
> serving the region, so we should try catch this exception.
> Trying to fix unassigned region...
> Exception in thread "main" org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hbase.NotServingRegionException: Received close for
> regionName but we are not serving it
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3204)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3185)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1012)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
> at com.sun.proxy.$Proxy7.closeRegion(Unknown Source)
> at
> org.apache.hadoop.hbase.util.HBaseFsckRepair.closeRegionSilentlyAndWait(HBaseFsckRepair.java:150)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.closeRegion(HBaseFsck.java:1565)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1704)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1406)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:419)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:438)
> at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3670)
> at org.apache.hadoop.hbase.util.HBaseFsck.run(HBaseFsck.java:3489)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3483)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-10653) Incorrect table status in HBase shell Describe
[
https://issues.apache.org/jira/browse/HBASE-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-10653:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Checked the output from the current shell and the table state is displayed on a
separate line which was the origin of the issue. Closing the issue as fixed.
Please re-open if required.
{noformat}
hbase(main):002:0> describe 't1'
Table t1 is ENABLED
t1
COLUMN FAMILIES DESCRIPTION
{NAME => 'cf1', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false',
NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE',
CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING =
> 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0',
> BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false',
> CACHE_BLOOMS_ON_WRITE => 'false'
, PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
'true', BLOCKSIZE => '65536'}{noformat}
> Incorrect table status in HBase shell Describe
> --
>
> Key: HBASE-10653
> URL: https://issues.apache.org/jira/browse/HBASE-10653
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 0.94.17
>Reporter: Biju Nair
>Assignee: Rekha Joshi
>Priority: Major
> Labels: HbaseShell, describe
> Fix For: 0.98.0
>
> Attachments: HBASE-10653.1.patch, HBASE-10653.2.patch
>
>
> Describe output of table which is disabled shows as enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-10827) Making HBase use multiple ethernet cards will improve the performance
[
https://issues.apache.org/jira/browse/HBASE-10827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-10827:
--
Resolution: Incomplete
Status: Resolved (was: Patch Available)
Resolving the issue as {{incomplete}}. Please re-open this issue if required.
At this time {{bonding}} would be the option users can use to utilize multiple
network cards with HBase.
> Making HBase use multiple ethernet cards will improve the performance
> -
>
> Key: HBASE-10827
> URL: https://issues.apache.org/jira/browse/HBASE-10827
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.99.0
>Reporter: zhaojianbo
>Assignee: zhaojianbo
>Priority: Major
> Attachments: HBASE-10827-0.98-branch.patch
>
>
> In our online cluster, usually there are multiple ethernet cards in one
> machine, one for outer network, one for inner network. But the current
> version of HBase can not use all of them which waste the network bandwidth of
> one ethernet card. If we make HBase use multiple ethernet cards concurrently,
> the performance of HBase will be improved.
> So I did the work, and test a simple scenario:
> 8 clients scan the same region data from a different machine with two
> ethernet cards.(machine of regionserver also with two ethernet cards)
> The Environment is:
> * I start HBase cluster with a master, a regionserver, a zookeeper in a
> machine.
> * HDFS cluster with a Namenode, a datanode, a secondary namenode is also
> started in the same machine.
> * 8 client run on different machine.
> * all data local
> * 22GB data size
> I measure the performance before and after the optimization.
> The results are:
> ||client||time before optimization||time after optimization||
> | 8 | 1665.07s | 1242.45s |
> The patch is uploaded. What I did is the following:
> # create new RPC getAllServerAddress which obtain all the addresses of
> regionserver
> # client call the RPC to obtain the addresses, choose one of them randomly,
> validate the address and use the address as the regionLocation address
> # add a cache serverAddressMap to avoid redundant RPC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-9171) Canary monitor tool not loading hbase configuration
[ https://issues.apache.org/jira/browse/HBASE-9171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-9171: - Resolution: Fixed Status: Resolved (was: Patch Available) In the [current master|https://github.com/apache/hbase/blob/0797243365f53a80dcdbc1c1e5514e6b1ae15ee6/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java#L1629] this issue seems to be fixed. Closing. > Canary monitor tool not loading hbase configuration > --- > > Key: HBASE-9171 > URL: https://issues.apache.org/jira/browse/HBASE-9171 > Project: HBase > Issue Type: Bug >Affects Versions: 0.95.1 >Reporter: Xiong LIU >Priority: Minor > Labels: Canary > Attachments: HBASE-9171-trunk-v1.patch, HBASE-9171-v0.patch, > HBASE-9171-v1.patch > > > When running bin/hbase org.apache.hadoop.hbase.tool.Canary, Canary throws > exception that it cannot connect to zookeeper ensemble localhost:2181. The > zookeeper server address to be connected is wrong because on my cluster setup > the port number is 3181. > The configuration Canary loaded is hadoop's default > configuration(core-default.xml) rather than hbase's > configuration(hbase-site.xml and hbase-default.xml). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17808) FastPath for RWQueueRpcExecutor
[ https://issues.apache.org/jira/browse/HBASE-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847087#comment-16847087 ] Biju Nair commented on HBASE-17808: --- [~allan163], still pursuing this or can we close this ticket? > FastPath for RWQueueRpcExecutor > --- > > Key: HBASE-17808 > URL: https://issues.apache.org/jira/browse/HBASE-17808 > Project: HBase > Issue Type: Improvement > Components: rpc, Scheduler >Affects Versions: 2.0.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-17808.patch, HBASE-17808.v2.patch > > > FastPath for the FIFO rpcscheduler was introduced in HBASE-16023. But it is > not implemented for RW queues. In this issue, I use > FastPathBalancedQueueRpcExecutor in RW queues. So anyone who want to isolate > their read/write requests can also benefit from the fastpath. > I haven't test the performance yet. But since I haven't change any of the > core implemention of FastPathBalancedQueueRpcExecutor, it should have the > same performance in HBASE-16023. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-5198) PerformanceEvaluation --presplit option should not recreate TestTable if it exists
[ https://issues.apache.org/jira/browse/HBASE-5198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847086#comment-16847086 ] Biju Nair commented on HBASE-5198: -- Hi [~ndimiduk], FYI. Hope the new patch makes sense. > PerformanceEvaluation --presplit option should not recreate TestTable if it > exists > -- > > Key: HBASE-5198 > URL: https://issues.apache.org/jira/browse/HBASE-5198 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Sujee Maniyam >Assignee: Sujee Maniyam >Priority: Minor > Labels: benchmark > Attachments: HBASE-5198.PATCH, > PerformanceEvaluation_HBASE_5198_2.patch > > > --presplit option was added in > https://issues.apache.org/jira/browse/HBASE-4440 > It drops and re-creates TestTable if the table exists. > This behavior in-consistent with previous ones. If TestTable exists, it > shouldn't be re-created -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15898) Document G1GC Recommendations
[ https://issues.apache.org/jira/browse/HBASE-15898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847077#comment-16847077 ] Biju Nair commented on HBASE-15898: --- Since this information will help users can we take in the changes and have another ticket for the comments here which are trivial so someone else can work on it. [~stack], thoughts? > Document G1GC Recommendations > - > > Key: HBASE-15898 > URL: https://issues.apache.org/jira/browse/HBASE-15898 > Project: HBase > Issue Type: Bug > Components: documentation, java >Affects Versions: 2.0.0 >Reporter: Misty Linville >Assignee: Misty Linville >Priority: Major > Attachments: HBASE-15898.patch > > > Document G1GC recommendations for HBase -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847068#comment-16847068
]
Biju Nair commented on HBASE-21415:
---
Looks like the new patch takes into account the previous comments. Good to
commit [~elserj] ?
> Admin#snapshot method documentation is misleading.
> --
>
> Key: HBASE-21415
> URL: https://issues.apache.org/jira/browse/HBASE-21415
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Philippe Laflamme
>Assignee: Philippe Laflamme
>Priority: Trivial
> Attachments: HBASE-21415.patch, HBASE-21415.patch
>
>
> The documentation for the {{Admin#snapshot}} function is misleading in
> regards to snapshot concurrency.
> It currently states the following:
> {quote}Only a single snapshot should be taken at a time for an instance of
> HBase, or results may be undefined (you can tell multiple HBase clusters to
> snapshot at the same time, but only one at a time for a single cluster).
> {quote}
> This seems to indicate that it's dangerous to run snapshots concurrently when
> in fact the HBase master will simply run them sequentially (as per this Slack
> thread [https://apache-hbase.slack.com/archives/C13K8NVAM/p1540994948005900)]
> The suggested fix is to simply remove this entire sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-12169) Document IPC binding options
[ https://issues.apache.org/jira/browse/HBASE-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847066#comment-16847066 ] Biju Nair commented on HBASE-12169: --- Will be good to have this property documented for user visibility. The changes in the patch good [~busbey] ? > Document IPC binding options > > > Key: HBASE-12169 > URL: https://issues.apache.org/jira/browse/HBASE-12169 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 0.98.0, 0.94.7, 0.99.0 >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-12169.patch > > > HBASE-8148 added options to change binding component services, but there > aren't any docs for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16291) Explain the correct semantic of hbase.client.retries.number in doc
[
https://issues.apache.org/jira/browse/HBASE-16291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847058#comment-16847058
]
Biju Nair commented on HBASE-16291:
---
Since the current stable release is 1.x we want to add this change to the
document. [~apurtell] thoughts?
> Explain the correct semantic of hbase.client.retries.number in doc
> --
>
> Key: HBASE-16291
> URL: https://issues.apache.org/jira/browse/HBASE-16291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.5, 1.2.2, 0.98.20, 2.0.0
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Major
> Attachments: HBASE-16291-v1.patch
>
>
> In hbase-default.xml (and also in HBase Book), the description of
> hbase.client.retries.number is:
> {quote}
> Maximum retries. Used as maximum for all retryable operations such as the
> getting of a cell's value, starting a row update, etc. Retry interval is a
> rough function based on hbase.client.pause. At first we retry at this
> interval but then with backoff, we pretty quickly reach retrying every ten
> seconds. See HConstants#RETRY_BACKOFF for how the backup ramps up. Change
> this setting and hbase.client.pause to suit your workload.
> {quote}
> However, the semantic of this conf is confusing and different in various
> branches. After HBASE-14521, in master it means "the number of RETRIES", but
> in all the other branches it still means "the number of TRIES".
> So I think we need make the doc clear and tell users what it is in 0.98/1.x
> and what it will be from 2.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22453) A "NullPointerException" could be thrown; "tableDescriptor" is nullable
[
https://issues.apache.org/jira/browse/HBASE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846369#comment-16846369
]
Biju Nair commented on HBASE-22453:
---
[~q977734161], minor suggestion about the message in the exception for your
consideration
{noformat}
throw new IOException("Can't find " + tableName + "'s descriptor ");{noformat}
> A "NullPointerException" could be thrown; "tableDescriptor" is nullable
> ---
>
> Key: HBASE-22453
> URL: https://issues.apache.org/jira/browse/HBASE-22453
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.0.0, 2.1.2, 2.1.4
>Reporter: lixiaobao
>Assignee: lixiaobao
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-22453.patch
>
>
> In hbase-backup model the class
> "org.apache.hadoop.hbase.backup.util.RestoreTool"'s method
> incrementalRestoreTable(),A "NullPointerException" could be thrown;
> "tableDescriptor" is nullable here.
> {code:java}
> // adjust table schema
> for (int i = 0; i < tableNames.length; i++) {
> TableName tableName = tableNames[i];
> TableDescriptor tableDescriptor = getTableDescriptor(fileSys, tableName,
> incrBackupId);
> LOG.debug("Found descriptor " + tableDescriptor + " through " +
> incrBackupId);
> TableName newTableName = newTableNames[i];
> TableDescriptor newTableDescriptor = admin.getDescriptor(newTableName);
> List families =
> Arrays.asList(tableDescriptor.getColumnFamilies());
> List existingFamilies =
> Arrays.asList(newTableDescriptor.getColumnFamilies());
> TableDescriptorBuilder builder =
> TableDescriptorBuilder.newBuilder(newTableDescriptor);
> boolean schemaChangeNeeded = false;
> for (ColumnFamilyDescriptor family : families) {
> if (!existingFamilies.contains(family)) {
> builder.setColumnFamily(family);
> schemaChangeNeeded = true;
> }
> }
> for (ColumnFamilyDescriptor family : existingFamilies) {
> if (!families.contains(family)) {
> builder.removeColumnFamily(family.getName());
> schemaChangeNeeded = true;
> }
> }
> if (schemaChangeNeeded) {
> modifyTableSync(conn, builder.build());
> LOG.info("Changed " + newTableDescriptor.getTableName() + " to: " +
> newTableDescriptor);
> }
> }
> RestoreJob restoreService = BackupRestoreFactory.getRestoreJob(conf);
> restoreService.run(logDirs, tableNames, newTableNames, false);
> }
> //代码占位符
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17550) replication.sink.client.ops.timeout should have a documented default
[ https://issues.apache.org/jira/browse/HBASE-17550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846352#comment-16846352 ] Biju Nair commented on HBASE-17550: --- Thanks for the review [~ashu210890]. Attached an updated patch. > replication.sink.client.ops.timeout should have a documented default > > > Key: HBASE-17550 > URL: https://issues.apache.org/jira/browse/HBASE-17550 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Biju Nair >Priority: Minor > Attachments: HBASE-17550.PATCH, HBASE-17550.patch > > > The config is undocumented and is not referenced in replication logs either > when a timeout actually occurs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17550) replication.sink.client.ops.timeout should have a documented default
[ https://issues.apache.org/jira/browse/HBASE-17550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-17550: -- Attachment: HBASE-17550.PATCH > replication.sink.client.ops.timeout should have a documented default > > > Key: HBASE-17550 > URL: https://issues.apache.org/jira/browse/HBASE-17550 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Biju Nair >Priority: Minor > Attachments: HBASE-17550.PATCH, HBASE-17550.patch > > > The config is undocumented and is not referenced in replication logs either > when a timeout actually occurs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-10081) Since HBASE-7091, HBASE_OPTS cannot be set on the command line
[ https://issues.apache.org/jira/browse/HBASE-10081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-10081: -- Resolution: Fixed Status: Resolved (was: Patch Available) The proposed change is in current master. > Since HBASE-7091, HBASE_OPTS cannot be set on the command line > -- > > Key: HBASE-10081 > URL: https://issues.apache.org/jira/browse/HBASE-10081 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nicolas Liochon >Assignee: Liu Shaohui >Priority: Minor > Attachments: HBASE-10081-v1.diff > > > Discussed in HBASE-7091. > It's not critical, but a little bit surprising, as the comments in bin/hbase > doesn't say anything about this. If you create your own hbase-env then it's > not an issue... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-9996) move the log io operation out of updateLock scope
[
https://issues.apache.org/jira/browse/HBASE-9996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-9996:
-
Resolution: Invalid
Status: Resolved (was: Patch Available)
{{HLog.java}} not in current master.
> move the log io operation out of updateLock scope
> -
>
> Key: HBASE-9996
> URL: https://issues.apache.org/jira/browse/HBASE-9996
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 0.94.13
>Reporter: Liang Xie
>Assignee: Liang Xie
>Priority: Major
> Attachments: HBase-9996-0.94.txt
>
>
> I am working at another write outlier issue, and just saw this code pattern:
> {code}
> synchronized(updateLock) {
> ...
> LOG.info(xxx)
> ...
> }
> {code}
> let's move the log statement out of this sync block. seems it's ok in trunk
> code, just a 0.94 issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22280) Separate read/write handler for priority request(especially for meta).
[ https://issues.apache.org/jira/browse/HBASE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-22280: -- Component/s: Scheduler > Separate read/write handler for priority request(especially for meta). > -- > > Key: HBASE-22280 > URL: https://issues.apache.org/jira/browse/HBASE-22280 > Project: HBase > Issue Type: New Feature > Components: Scheduler >Reporter: binlijin >Assignee: binlijin >Priority: Major > Attachments: HBASE-22280.patch > > > Client may give too many read pressure on meta, so blocking master write meta > for region open. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-5198) PerformanceEvaluation --presplit option should not recreate TestTable if it exists
[ https://issues.apache.org/jira/browse/HBASE-5198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844293#comment-16844293 ] Biju Nair commented on HBASE-5198: -- Attached a new patch to drop table only when the requested perf test is not read. > PerformanceEvaluation --presplit option should not recreate TestTable if it > exists > -- > > Key: HBASE-5198 > URL: https://issues.apache.org/jira/browse/HBASE-5198 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Sujee Maniyam >Assignee: Sujee Maniyam >Priority: Minor > Labels: benchmark > Attachments: HBASE-5198.PATCH, > PerformanceEvaluation_HBASE_5198_2.patch > > > --presplit option was added in > https://issues.apache.org/jira/browse/HBASE-4440 > It drops and re-creates TestTable if the table exists. > This behavior in-consistent with previous ones. If TestTable exists, it > shouldn't be re-created -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-5198) PerformanceEvaluation --presplit option should not recreate TestTable if it exists
[ https://issues.apache.org/jira/browse/HBASE-5198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-5198: - Attachment: HBASE-5198.PATCH > PerformanceEvaluation --presplit option should not recreate TestTable if it > exists > -- > > Key: HBASE-5198 > URL: https://issues.apache.org/jira/browse/HBASE-5198 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Sujee Maniyam >Assignee: Sujee Maniyam >Priority: Minor > Labels: benchmark > Attachments: HBASE-5198.PATCH, > PerformanceEvaluation_HBASE_5198_2.patch > > > --presplit option was added in > https://issues.apache.org/jira/browse/HBASE-4440 > It drops and re-creates TestTable if the table exists. > This behavior in-consistent with previous ones. If TestTable exists, it > shouldn't be re-created -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-5198) PerformanceEvaluation --presplit option should not recreate TestTable if it exists
[ https://issues.apache.org/jira/browse/HBASE-5198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844280#comment-16844280 ] Biju Nair commented on HBASE-5198: -- Dropping a table for different pre-split when a read test is performed is an issue. New patch need to be created since the current one is pretty old for the code base. > PerformanceEvaluation --presplit option should not recreate TestTable if it > exists > -- > > Key: HBASE-5198 > URL: https://issues.apache.org/jira/browse/HBASE-5198 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Sujee Maniyam >Assignee: Sujee Maniyam >Priority: Minor > Labels: benchmark > Attachments: PerformanceEvaluation_HBASE_5198_2.patch > > > --presplit option was added in > https://issues.apache.org/jira/browse/HBASE-4440 > It drops and re-creates TestTable if the table exists. > This behavior in-consistent with previous ones. If TestTable exists, it > shouldn't be re-created -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17329) Improve message when CallQueueTooBigException
[ https://issues.apache.org/jira/browse/HBASE-17329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-17329: -- Component/s: Scheduler > Improve message when CallQueueTooBigException > - > > Key: HBASE-17329 > URL: https://issues.apache.org/jira/browse/HBASE-17329 > Project: HBase > Issue Type: Improvement > Components: Scheduler >Reporter: stack >Assignee: stack >Priority: Trivial > Attachments: HBASE-17329.master.001.patch > > > Print out queue item counts and queue sizes. While here correct queue name > spellings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22443) Add hbase-vote script details to documentation
[
https://issues.apache.org/jira/browse/HBASE-22443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844152#comment-16844152
]
Biju Nair commented on HBASE-22443:
---
Thanks [~apurtell]. Sorry about the typo.
> Add hbase-vote script details to documentation
> --
>
> Key: HBASE-22443
> URL: https://issues.apache.org/jira/browse/HBASE-22443
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Biju Nair
>Assignee: Biju Nair
>Priority: Minor
> Attachments: HBASE-22443.PATCH
>
>
> In HBASE-21963 [~taklwu] provided {{hbase-vote.sh}} which helps with
> verification of new release. Adding it to the documentation will help anyone
> who would like to understand the verification process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-19646) Add CRON To Major Compaction
[
https://issues.apache.org/jira/browse/HBASE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair resolved HBASE-19646.
---
Resolution: Invalid
> Add CRON To Major Compaction
>
>
> Key: HBASE-19646
> URL: https://issues.apache.org/jira/browse/HBASE-19646
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: David Mollitor
>Priority: Minor
>
> HBase provides _hbase.hregion.majorcompaction_
> {quote}
> Time between major compactions, expressed in milliseconds. Set to 0 to
> disable time-based automatic major compactions. User-requested and size-based
> major compactions will still run. This value is multiplied by
> hbase.hregion.majorcompaction.jitter to cause compaction to start at a
> somewhat-random time during a given window of time. The default value is 7
> days, expressed in milliseconds. If major compactions are causing disruption
> in your environment, you can configure them to run at off-peak times for your
> deployment, or disable time-based major compactions by setting this parameter
> to 0, and run major compactions in a cron job or by another external
> mechanism.
> {quote}
> Instead of this existing mechanism, that adds randomness into a production
> system (ugh), let's simply allow users to specify a cron string and replace
> this simple periodic (+jitter) scheduling mechanism. CRON is useful for
> systems that have known windows of time (i.e. weekend, nights) that are known
> to be good times for compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-9934) Mesh replication (a.k.a. multi master replication)
[ https://issues.apache.org/jira/browse/HBASE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair resolved HBASE-9934. -- Resolution: Later Resolving. Can be re-opened for "later" consideration. > Mesh replication (a.k.a. multi master replication) > -- > > Key: HBASE-9934 > URL: https://issues.apache.org/jira/browse/HBASE-9934 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Ishan Chhabra >Assignee: Ishan Chhabra >Priority: Minor > > This is to setup NxN replication. > See background discussion here: > http://mail-archives.apache.org/mod_mbox/hbase-user/201311.mbox/%3CCAOiuM-4UMmLA7UHMp4hhjpLWUrHDxg1t4tN4aWvnZUMcTxG%2BKQ%40mail.gmail.com%3E > We can add a new mode in replication to not forward edits from other > clusters. Not sure what should be done when some clusters are configured with > this setting and some aren't. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HBASE-22443) Add hbase-vote script details to documentation
[
https://issues.apache.org/jira/browse/HBASE-22443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-22443 started by Biju Nair.
-
> Add hbase-vote script details to documentation
> --
>
> Key: HBASE-22443
> URL: https://issues.apache.org/jira/browse/HBASE-22443
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Biju Nair
>Assignee: Biju Nair
>Priority: Minor
> Attachments: HBASE-22443.PATCH
>
>
> In HBASE-21963 [~taklwu] provided {{hbase-vote.sh}} which helps with
> verification of new release. Adding it to the documentation will help anyone
> who would like to understand the verification process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22443) Add hbase-vote script details to documentation
[
https://issues.apache.org/jira/browse/HBASE-22443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22443:
--
Attachment: HBASE-22443.PATCH
> Add hbase-vote script details to documentation
> --
>
> Key: HBASE-22443
> URL: https://issues.apache.org/jira/browse/HBASE-22443
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Biju Nair
>Assignee: Biju Nair
>Priority: Minor
> Attachments: HBASE-22443.PATCH
>
>
> In HBASE-21963 [~taklwu] provided {{hbase-vote.sh}} which helps with
> verification of new release. Adding it to the documentation will help anyone
> who would like to understand the verification process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22443) Add hbase-vote script details to documentation
[
https://issues.apache.org/jira/browse/HBASE-22443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22443:
--
Status: Patch Available (was: In Progress)
Attached a document patch.
> Add hbase-vote script details to documentation
> --
>
> Key: HBASE-22443
> URL: https://issues.apache.org/jira/browse/HBASE-22443
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Biju Nair
>Assignee: Biju Nair
>Priority: Minor
> Attachments: HBASE-22443.PATCH
>
>
> In HBASE-21963 [~taklwu] provided {{hbase-vote.sh}} which helps with
> verification of new release. Adding it to the documentation will help anyone
> who would like to understand the verification process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-22443) Add hbase-vote script details to documentation
Biju Nair created HBASE-22443:
-
Summary: Add hbase-vote script details to documentation
Key: HBASE-22443
URL: https://issues.apache.org/jira/browse/HBASE-22443
Project: HBase
Issue Type: Improvement
Components: documentation
Reporter: Biju Nair
Assignee: Biju Nair
In HBASE-21963 [~taklwu] provided {{hbase-vote.sh}} which helps with
verification of new release. Adding it to the documentation will help anyone
who would like to understand the verification process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-9964) start-hbase.sh returns success exit status and emits "starting master" message when master failed to start
[ https://issues.apache.org/jira/browse/HBASE-9964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843197#comment-16843197 ] Biju Nair commented on HBASE-9964: -- Hi [~ashish singhi], looks like this issue is still relevant. > start-hbase.sh returns success exit status and emits "starting master" > message when master failed to start > -- > > Key: HBASE-9964 > URL: https://issues.apache.org/jira/browse/HBASE-9964 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.13 > Environment: Linux x86_64 (Centos 6.4) >Reporter: Jason Vas Dias >Assignee: Ashish Singhi >Priority: Major > Attachments: HBASE-9964.patch > > > The bin/start-hbase.sh script returns an exit status indicating success, and > emits the message : 'starting master, logging to ...' , even though the > master failed to start: > $ ./bin/start-hbase.sh > starting master, logging to > /home/jason/3P/hbase-0.94.13/logs/hbase-jason-master-jvds.out > $ echo $? > 0 > # ^- this indicates successful exit status on Linux / UNIX-like systems > $ egrep 'start master|RuntimeException' logs/hbase-jason-master-jvds.log > 2013-11-13 14:15:28,309 ERROR > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master > java.lang.RuntimeException: Failed suppression of fs shutdown hook: > Thread[Thread-27,5,main] > The start-base.sh script should IMHO in the above scenario emit a message > like > 'Failed to start the hbase master: java.lang.RuntimeException: Failed > suppression of fs shutdown hook' > and return a non-zero exit status. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-9964) start-hbase.sh returns success exit status and emits "starting master" message when master failed to start
[ https://issues.apache.org/jira/browse/HBASE-9964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair reassigned HBASE-9964: Assignee: Ashish Singhi > start-hbase.sh returns success exit status and emits "starting master" > message when master failed to start > -- > > Key: HBASE-9964 > URL: https://issues.apache.org/jira/browse/HBASE-9964 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.13 > Environment: Linux x86_64 (Centos 6.4) >Reporter: Jason Vas Dias >Assignee: Ashish Singhi >Priority: Major > Attachments: HBASE-9964.patch > > > The bin/start-hbase.sh script returns an exit status indicating success, and > emits the message : 'starting master, logging to ...' , even though the > master failed to start: > $ ./bin/start-hbase.sh > starting master, logging to > /home/jason/3P/hbase-0.94.13/logs/hbase-jason-master-jvds.out > $ echo $? > 0 > # ^- this indicates successful exit status on Linux / UNIX-like systems > $ egrep 'start master|RuntimeException' logs/hbase-jason-master-jvds.log > 2013-11-13 14:15:28,309 ERROR > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master > java.lang.RuntimeException: Failed suppression of fs shutdown hook: > Thread[Thread-27,5,main] > The start-base.sh script should IMHO in the above scenario emit a message > like > 'Failed to start the hbase master: java.lang.RuntimeException: Failed > suppression of fs shutdown hook' > and return a non-zero exit status. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-9820) Method Replication.decorateMasterConfiguration can fail with NPE in case that property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
[
https://issues.apache.org/jira/browse/HBASE-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-9820:
-
Resolution: Invalid
Release Note: Please re-open if this still need to be pursued further.
Status: Resolved (was: Patch Available)
> Method Replication.decorateMasterConfiguration can fail with NPE in case that
> property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
> -
>
> Key: HBASE-9820
> URL: https://issues.apache.org/jira/browse/HBASE-9820
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Jarek Jarcec Cecho
>Assignee: Jean-Marc Spaggiari
>Priority: Major
> Attachments: HBASE-9820-v0-trunk.patch
>
>
> While upgrading HBase dependency on Pig via PIG-3529, I've noticed that
> method {{Replication.decorateMasterConfiguration()}} can throw
> {{NullPointerException}}
> ([code|https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java])
> in case when property HBASE_MASTER_LOGCLEANER_PLUGINS won't be defined.
> The issue was more on a pig side where we weren't propagating default HBase
> configuration resources (such as {{hbase-default.xml}}), but I was curious if
> it's expected that this property will be always defined? We might want to
> tweak the code a bit if this property can be null.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-9820) Method Replication.decorateMasterConfiguration can fail with NPE in case that property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
[
https://issues.apache.org/jira/browse/HBASE-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843195#comment-16843195
]
Biju Nair commented on HBASE-9820:
--
Checked current master code for {{Replication.java}} and there are no instances
of {{HBASE_MASTER_LOGCLEANER_PLUGINS}} which required a default value as per
the patch. Will close since this issue since is pretty old.
> Method Replication.decorateMasterConfiguration can fail with NPE in case that
> property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
> -
>
> Key: HBASE-9820
> URL: https://issues.apache.org/jira/browse/HBASE-9820
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Jarek Jarcec Cecho
>Assignee: Jean-Marc Spaggiari
>Priority: Major
> Attachments: HBASE-9820-v0-trunk.patch
>
>
> While upgrading HBase dependency on Pig via PIG-3529, I've noticed that
> method {{Replication.decorateMasterConfiguration()}} can throw
> {{NullPointerException}}
> ([code|https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java])
> in case when property HBASE_MASTER_LOGCLEANER_PLUGINS won't be defined.
> The issue was more on a pig side where we weren't propagating default HBase
> configuration resources (such as {{hbase-default.xml}}), but I was curious if
> it's expected that this property will be always defined? We might want to
> tweak the code a bit if this property can be null.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22441) BucketCache NullPointerException in cacheBlock
[
https://issues.apache.org/jira/browse/HBASE-22441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843178#comment-16843178
]
Biju Nair commented on HBASE-22441:
---
Hi [~aoxiang], any specific scenarios causing this issue and do you see an
impact on the client? Also this is confined to v2.2.0 only? Thanks.
> BucketCache NullPointerException in cacheBlock
> --
>
> Key: HBASE-22441
> URL: https://issues.apache.org/jira/browse/HBASE-22441
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 2.2.0
>Reporter: binlijin
>Priority: Major
>
> There is no synchronized in the check in cacheBlock, and wen see
> NullPointerException in production cluster.
> {code}
> 2019-05-17 18:17:21,299 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=20,queue=7,port=16020] ipc.RpcServer:
> Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.returnBlock(BucketCache.java:1665)
> at
> org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:250)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:426)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:412)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:67)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.lambda$readBlock$2(HFileReaderImpl.java:1501)
> at java.util.Optional.ifPresent(Optional.java:159)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1499)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:340)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:577)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekBefore(HFileReaderImpl.java:869)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekToPreviousRow(StoreFileScanner.java:515)
> at
> org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:135)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:644)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6612)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6776)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6549)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3183)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3428)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-16290) Dump summary of callQueue content; can help debugging
[ https://issues.apache.org/jira/browse/HBASE-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-16290: -- Component/s: Scheduler > Dump summary of callQueue content; can help debugging > - > > Key: HBASE-16290 > URL: https://issues.apache.org/jira/browse/HBASE-16290 > Project: HBase > Issue Type: Bug > Components: Operability, Scheduler >Affects Versions: 2.0.0 >Reporter: stack >Assignee: Sreeram Venkatasubramanian >Priority: Major > Labels: beginner > Fix For: 2.0.0 > > Attachments: DebugDump_screenshot.png, HBASE-16290.master.001.patch, > HBASE-16290.master.002.patch, HBASE-16290.master.003.patch, > HBASE-16290.master.004.patch, HBASE-16290.master.005.patch, > HBASE-16290.master.006.patch, HBASE-16290.master.007.patch, > HBASE-16290.master.008.patch, Sample Summary.txt > > > Being able to get a clue what is in a backedup callQueue could give insight > on what is going on on a jacked server. Just needs to summarize count, sizes, > call types. Useful debugging. In a servlet? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-12790) Support fairness across parallelized scans
[ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-12790: -- Component/s: Scheduler > Support fairness across parallelized scans > -- > > Key: HBASE-12790 > URL: https://issues.apache.org/jira/browse/HBASE-12790 > Project: HBase > Issue Type: New Feature > Components: Scheduler >Reporter: James Taylor >Assignee: ramkrishna.s.vasudevan >Priority: Major > Labels: Phoenix > Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, > HBASE-12790_1.patch, HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, > HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip > > > Some HBase clients parallelize the execution of a scan to reduce latency in > getting back results. This can lead to starvation with a loaded cluster and > interleaved scans, since the RPC queue will be ordered and processed on a > FIFO basis. For example, if there are two clients, A & B that submit largish > scans at the same time. Say each scan is broken down into 100 scans by the > client (broken down into equal depth chunks along the row key), and the 100 > scans of client A are queued first, followed immediately by the 100 scans of > client B. In this case, client B will be starved out of getting any results > back until the scans for client A complete. > One solution to this is to use the attached AbstractRoundRobinQueue instead > of the standard FIFO queue. The queue to be used could be (maybe it already > is) configurable based on a new config parameter. Using this queue would > require the client to have the same identifier for all of the 100 parallel > scans that represent a single logical scan from the clients point of view. > With this information, the round robin queue would pick off a task from the > queue in a round robin fashion (instead of a strictly FIFO manner) to prevent > starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17088) Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor
[ https://issues.apache.org/jira/browse/HBASE-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-17088: -- Component/s: Scheduler > Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor > > > Key: HBASE-17088 > URL: https://issues.apache.org/jira/browse/HBASE-17088 > Project: HBase > Issue Type: Improvement > Components: rpc, Scheduler >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 1.4.0, 2.0.0 > > Attachments: HBASE-17088-branch-1.patch, HBASE-17088-v1.patch, > HBASE-17088-v2.patch, HBASE-17088-v3.patch, HBASE-17088-v3.patch, > HBASE-17088-v4.patch, HBASE-17088-v4.patch > > > 1. The RWQueueRpcExecutor has eight constructor method and the longest one > has ten parameters. But It is only used in SimpleRpcScheduler and easy to > confused when read the code. > 2. There are duplicate method implement in RWQueueRpcExecutor and > BalancedQueueRpcExecutor. They can be implemented in their parent class > RpcExecutor. > 3. SimpleRpcScheduler read many configs to new RpcExecutor. But the > CALL_QUEUE_SCAN_SHARE_CONF_KEY is only needed by RWQueueRpcExecutor. And > CALL_QUEUE_CODEL_TARGET_DELAY, CALL_QUEUE_CODEL_INTERVAL and > CALL_QUEUE_CODEL_LIFO_THRESHOLD are only needed by AdaptiveLifoCoDelCallQueue. > So I thought we can refactor it. Suggestions are welcome. > Review board: https://reviews.apache.org/r/53726/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-11724) Add to RWQueueRpcExecutor the ability to split get and scan handlers
[ https://issues.apache.org/jira/browse/HBASE-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-11724: -- Component/s: Scheduler > Add to RWQueueRpcExecutor the ability to split get and scan handlers > > > Key: HBASE-11724 > URL: https://issues.apache.org/jira/browse/HBASE-11724 > Project: HBase > Issue Type: New Feature > Components: IPC/RPC, Scheduler >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-11724-v0.patch, HBASE-11724-v1.patch, > HBASE-11724-v2.patch > > > RWQueueRpcExecutor has the devision between reads and writes requests, but we > can split also small-reads and long-reads. This can be useful to force a > deprioritization of scans on the RS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-14331:
--
Component/s: Scheduler
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance, Scheduler
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Major
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch,
> HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-16341) Missing bit on "Regression: Random Read/WorkloadC slower in 1.x than 0.98"
[
https://issues.apache.org/jira/browse/HBASE-16341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-16341:
--
Component/s: Scheduler
> Missing bit on "Regression: Random Read/WorkloadC slower in 1.x than 0.98"
> --
>
> Key: HBASE-16341
> URL: https://issues.apache.org/jira/browse/HBASE-16341
> Project: HBase
> Issue Type: Bug
> Components: rpc, Scheduler
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HBASE-16341.master.001.patch, HBASE-16341.patch,
> HBASE-16341.patch
>
>
> [~larsgeorge] found a missing bit in HBASE-15971 "Regression: Random
> Read/WorkloadC slower in 1.x than 0.98" Let me fix here. Let me quote the man:
> {code}
> BTW, in constructor we do this
> ```String callQueueType = conf.get(CALL_QUEUE_TYPE_CONF_KEY,
> CALL_QUEUE_TYPE_FIFO_CONF_VALUE);
> ```
> (edited)
> [8:19]
> but in `onConfigurationChange()` we do
> ```String callQueueType = conf.get(CALL_QUEUE_TYPE_CONF_KEY,
> CALL_QUEUE_TYPE_DEADLINE_CONF_VALUE);
> ```
> (edited)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-15703) Deadline scheduler needs to return to the client info about skipped calls, not just drop them
[ https://issues.apache.org/jira/browse/HBASE-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-15703: -- Component/s: Scheduler > Deadline scheduler needs to return to the client info about skipped calls, > not just drop them > - > > Key: HBASE-15703 > URL: https://issues.apache.org/jira/browse/HBASE-15703 > Project: HBase > Issue Type: Bug > Components: IPC/RPC, Scheduler >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0, 2.0.0 > > Attachments: HBASE-15703-branch-1.3.v1.patch, > HBASE-15703-branch-1.3.v2.patch > > > In AdaptiveLifoCodelCallQueue we drop the calls when we think we're > overloaded, we should instead return CallDroppedException to the cleent or > something. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-15971) Regression: Random Read/WorkloadC slower in 1.x than 0.98
[ https://issues.apache.org/jira/browse/HBASE-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-15971: -- Component/s: Scheduler > Regression: Random Read/WorkloadC slower in 1.x than 0.98 > - > > Key: HBASE-15971 > URL: https://issues.apache.org/jira/browse/HBASE-15971 > Project: HBase > Issue Type: Sub-task > Components: rpc, Scheduler >Affects Versions: 1.3.0, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 1.3.0, 2.0.0 > > Attachments: 098.hits.png, 098.png, HBASE-15971.branch-1.001.patch, > HBASE-15971.branch-1.002.patch, Screen Shot 2016-06-10 at 5.08.24 PM.png, > Screen Shot 2016-06-10 at 5.08.26 PM.png, branch-1.hits.png, branch-1.png, > flight_recording_10172402220203_28.branch-1.jfr, > flight_recording_10172402220203_29.09820.0.98.20.jfr, handlers.fp.png, > hits.fp.png, hits.patched1.0.vs.unpatched1.0.vs.098.png, run_ycsb.sh > > > branch-1 is slower than 0.98 doing YCSB random read/workloadC. It seems to be > doing about 1/2 the throughput of 0.98. > In branch-1, we have low handler occupancy compared to 0.98. Hacking in > reader thread occupancy metric, is about the same in both. In parent issue, > hacking out the scheduler, I am able to get branch-1 to go 3x faster so will > dig in here. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16063) Race condition in new FIFO fastpath from HBASE-16023
[
https://issues.apache.org/jira/browse/HBASE-16063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-16063:
--
Component/s: Scheduler
> Race condition in new FIFO fastpath from HBASE-16023
>
>
> Key: HBASE-16063
> URL: https://issues.apache.org/jira/browse/HBASE-16063
> Project: HBase
> Issue Type: Sub-task
> Components: Scheduler
>Affects Versions: 1.3.0
>Reporter: stack
>Priority: Major
>
> From [~ikeda] over in HBASE-16023 at
> https://issues.apache.org/jira/browse/HBASE-16023?focusedCommentId=15331172=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15331172
> {quote}
> An concrete example of the race condition:
> 1. Worker checks no task.
> 2. Reader checks no ready handler.
> 3. Worker pushes itself as a ready handler and waits on the semaphore.
> 4. Reader queues a task to the queue, without directly passing it to the
> ready handler nor releasing the semaphore.
> (1,3) and (2,4) should be exclusively executed. That depends on luck, and it
> might be not severe
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-17808) FastPath for RWQueueRpcExecutor
[ https://issues.apache.org/jira/browse/HBASE-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-17808: -- Component/s: Scheduler > FastPath for RWQueueRpcExecutor > --- > > Key: HBASE-17808 > URL: https://issues.apache.org/jira/browse/HBASE-17808 > Project: HBase > Issue Type: Improvement > Components: rpc, Scheduler >Affects Versions: 2.0.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-17808.patch, HBASE-17808.v2.patch > > > FastPath for the FIFO rpcscheduler was introduced in HBASE-16023. But it is > not implemented for RW queues. In this issue, I use > FastPathBalancedQueueRpcExecutor in RW queues. So anyone who want to isolate > their read/write requests can also benefit from the fastpath. > I haven't test the performance yet. But since I haven't change any of the > core implemention of FastPathBalancedQueueRpcExecutor, it should have the > same performance in HBASE-16023. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16089) Add on FastPath for CoDel
[ https://issues.apache.org/jira/browse/HBASE-16089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-16089: -- Component/s: Scheduler regionserver > Add on FastPath for CoDel > - > > Key: HBASE-16089 > URL: https://issues.apache.org/jira/browse/HBASE-16089 > Project: HBase > Issue Type: Improvement > Components: regionserver, Scheduler >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Major > Fix For: 1.3.0, 2.0.0 > > Attachments: HBASE-16089.patch, HBASE-16089.v1.patch, > HBASE-16089.v2.patch, HBASE-16089.v3.patch, > baselineFifo_codel_codelPlusPatch.png, v3.png > > > If this is all that awesome, so we should have it on CoDel too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-10993) Deprioritize long-running scanners
[ https://issues.apache.org/jira/browse/HBASE-10993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-10993: -- Component/s: regionserver IPC/RPC > Deprioritize long-running scanners > -- > > Key: HBASE-10993 > URL: https://issues.apache.org/jira/browse/HBASE-10993 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC, regionserver, Scheduler >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10993-v0.patch, HBASE-10993-v1.patch, > HBASE-10993-v2.patch, HBASE-10993-v3.patch, HBASE-10993-v4.patch, > HBASE-10993-v4.patch, HBASE-10993-v5.patch > > > Currently we have a single call queue that serves all the "normal user" > requests, and the requests are executed in FIFO. > When running map-reduce jobs and user-queries on the same machine, we want to > prioritize the user-queries. > Without changing too much code, and not having the user giving hints, we can > add a “vtime” field to the scanner, to keep track from how long is running. > And we can replace the callQueue with a priorityQueue. In this way we can > deprioritize long-running scans, the longer a scan request lives the less > priority it gets. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-10993) Deprioritize long-running scanners
[ https://issues.apache.org/jira/browse/HBASE-10993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-10993: -- Component/s: Scheduler > Deprioritize long-running scanners > -- > > Key: HBASE-10993 > URL: https://issues.apache.org/jira/browse/HBASE-10993 > Project: HBase > Issue Type: Sub-task > Components: Scheduler >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10993-v0.patch, HBASE-10993-v1.patch, > HBASE-10993-v2.patch, HBASE-10993-v3.patch, HBASE-10993-v4.patch, > HBASE-10993-v4.patch, HBASE-10993-v5.patch > > > Currently we have a single call queue that serves all the "normal user" > requests, and the requests are executed in FIFO. > When running map-reduce jobs and user-queries on the same machine, we want to > prioritize the user-queries. > Without changing too much code, and not having the user giving hints, we can > add a “vtime” field to the scanner, to keep track from how long is running. > And we can replace the callQueue with a priorityQueue. In this way we can > deprioritize long-running scans, the longer a scan request lives the less > priority it gets. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-11737) Document callQueue improvements from HBASE-11355 and HBASE-11724
[ https://issues.apache.org/jira/browse/HBASE-11737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-11737: -- Component/s: Scheduler > Document callQueue improvements from HBASE-11355 and HBASE-11724 > > > Key: HBASE-11737 > URL: https://issues.apache.org/jira/browse/HBASE-11737 > Project: HBase > Issue Type: Sub-task > Components: documentation, Scheduler >Reporter: Misty Linville >Assignee: Misty Linville >Priority: Major > Fix For: 0.99.0 > > Attachments: HBASE-11737.patch, HBASE-11737.patch, HBASE_11737.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16023) Fastpath for the FIFO rpcscheduler
[ https://issues.apache.org/jira/browse/HBASE-16023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-16023: -- Component/s: Scheduler > Fastpath for the FIFO rpcscheduler > -- > > Key: HBASE-16023 > URL: https://issues.apache.org/jira/browse/HBASE-16023 > Project: HBase > Issue Type: Sub-task > Components: Performance, rpc, Scheduler >Affects Versions: 1.3.0, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 1.3.0, 2.0.0 > > Attachments: HBASE-16023.addenum.patch, > HBASE-16023.branch-1.001.patch, hits.nofifo.fifoplusfp.fifownofp.hacks.png > > > This is an idea copied from kudu where we skip queuing a request if there is > a handler ready to go; we just do a direct handoff from reader to handler. > Makes for close to a %20 improvement in random read workloadc testing moving > the bottleneck to HBASE-15716 and to returning the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-15136) Explore different queuing behaviors while busy
[ https://issues.apache.org/jira/browse/HBASE-15136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-15136: -- Component/s: Scheduler > Explore different queuing behaviors while busy > -- > > Key: HBASE-15136 > URL: https://issues.apache.org/jira/browse/HBASE-15136 > Project: HBase > Issue Type: New Feature > Components: IPC/RPC, Scheduler >Reporter: Elliott Clark >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0, 2.0.0 > > Attachments: HBASE-15136-1.2.v1.patch, HBASE-15136-v2.patch, > deadline_scheduler_v_0_2.patch > > > http://queue.acm.org/detail.cfm?id=2839461 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16046) [UMBRELLA] Make our default rpc scheduler the adaptive LIFO+CoDel added in HBASE-15136
[ https://issues.apache.org/jira/browse/HBASE-16046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-16046: -- Component/s: Scheduler > [UMBRELLA] Make our default rpc scheduler the adaptive LIFO+CoDel added in > HBASE-15136 > -- > > Key: HBASE-16046 > URL: https://issues.apache.org/jira/browse/HBASE-16046 > Project: HBase > Issue Type: Task > Components: rpc, Scheduler >Reporter: stack >Priority: Critical > > Our current rpc scheduler default is FIFO. Good for speed but as soon as the > server gets overloaded, our behavior goes south. We can do better, or a plan > is better than none. Our FB brothers and sisters added an rpc scheduler that > is smarter around overloading over in HBASE-15136. The implementation is the > codification of FIFO when all is dandy flipping to LIFO with CoDel and load > shedding when under duress. The algorithm is used elsewhere at FB w/ a nice > writeup on why it 'works' and why. Makes sense to me. Has nice feature that > tuning knobs are few and we should be able to nail defaults easy enough that > will work for majority of cases. Lets adopt it as our default going forward. > This is an umbrella issue for enabling it in 2.0.0. We need to do a few > things as subissues. The mighty [~mantonov] and crew want to get some > production experience first w/ their particular implementation -- makes sense > -- and we need to add the speedup from HBASE-16023 (currently the codel > scheduler is much slower than our default). Another consideration for here or > as a follow on is the suggestion that when all is running smooth, let the > Reader thread run the request rather than do handoff to handler (From > [~ikeda] over on HBASE-14479) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22437) HBOSS: Add Hadoop 2 / 3 profiles
[ https://issues.apache.org/jira/browse/HBASE-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-22437: -- Component/s: hboss > HBOSS: Add Hadoop 2 / 3 profiles > > > Key: HBASE-22437 > URL: https://issues.apache.org/jira/browse/HBASE-22437 > Project: HBase > Issue Type: Improvement > Components: hboss >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Labels: HBOSS > Attachments: 0001-HBASE-22437-HBOSS-Add-Hadoop-2-3-profiles.patch > > > Original discussion on HBASE-22149 indicated interest running HBOSS on Hadoop > 2, and HBase itself maintains profiles for Hadoop 2 and 3. There's no > fundamental reason we can't - there are some minor incompatibilities in the > code, but no fundamental mismatch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-11355) a couple of callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-11355: -- Component/s: Scheduler > a couple of callQueue related improvements > -- > > Key: HBASE-11355 > URL: https://issues.apache.org/jira/browse/HBASE-11355 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance, Scheduler >Affects Versions: 0.99.0, 0.94.20 >Reporter: Liang Xie >Assignee: Matteo Bertozzi >Priority: Major > Fix For: 0.99.0, 0.98.4 > > Attachments: HBASE-11355-v0.patch, HBASE-11355-v1.patch, > HBASE-11355-v2.patch, gets.png > > > In one of my in-memory read only testing(100% get requests), one of the top > scalibility bottleneck came from the single callQueue. A tentative sharing > this callQueue according to the rpc handler number showed a big throughput > improvement(the original get() qps is around 60k, after this one and other > hotspot tunning, i got 220k get() qps in the same single region server) in a > YCSB read only scenario. > Another stuff we can do is seperating the queue into read call queue and > write call queue, we had done it in our internal branch, it would helpful in > some outages, to avoid all read or all write requests ran out of all handler > threads. > One more stuff is changing the current blocking behevior once the callQueue > is full, considering the full callQueue almost means the backend processing > is slow somehow, so a fail-fast here should be more reasonable if we using > HBase as a low latency processing system. see "callQueue.put(call)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22418) Separate queue/executor for Multi get requests
[
https://issues.apache.org/jira/browse/HBASE-22418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22418:
--
Component/s: Scheduler
IPC/RPC
> Separate queue/executor for Multi get requests
> --
>
> Key: HBASE-22418
> URL: https://issues.apache.org/jira/browse/HBASE-22418
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, regionserver, Scheduler
>Reporter: Biju Nair
>Priority: Major
> Attachments: HBASE-22418.DRAFT
>
>
> Using {{RWQueueRpcExecutor}}, {{SimpleRpcScheduler}} lets us define separate
> queue/handlers to process write, read and scan requests. If this can be
> extended to have separate queue/handlers for multi get requests, it will help
> processing single get requests quickly where there is a mix of multi and
> single get workload. This is the case for multi put/single put as well. Any
> drawbacks in doing this segregation?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22418) Separate queue/executor for Multi get requests
[
https://issues.apache.org/jira/browse/HBASE-22418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841472#comment-16841472
]
Biju Nair commented on HBASE-22418:
---
Summary of the proposed change. Currently {{SimpleRpcScheduler}} uses
[RWQueueRpcExecutor|https://github.com/apache/hbase/blob/e984515b7426e8df9e0e50abc6a998f23d5e6e9e/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L91]
when {{hbase.ipc.server.callqueue.read.ratio}} property is set to a value > 0.
{{RWQueueRpcExecutor}} splits the queues and handlers for {{read}} and
{{write}} requests. It also provides the option to split the {{read}}
queue/handlers further to handle {{get}} and {{scan}} requests by setting the
{{hbase.ipc.server.callqueue.scan.ratio}} property. The proposed change is to
add a new property {{hbase.ipc.server.callqueue.multiread.ratio}} which the
users can set to define the percentage of the {{read}} queue/handlers to serve
Multi get/batch requests. This will let users to define separate {{read}}
queue/handlers for {{get}}, {{scan}}, {{multi get/batch get}} requests.
> Separate queue/executor for Multi get requests
> --
>
> Key: HBASE-22418
> URL: https://issues.apache.org/jira/browse/HBASE-22418
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Biju Nair
>Priority: Major
> Attachments: HBASE-22418.DRAFT
>
>
> Using {{RWQueueRpcExecutor}}, {{SimpleRpcScheduler}} lets us define separate
> queue/handlers to process write, read and scan requests. If this can be
> extended to have separate queue/handlers for multi get requests, it will help
> processing single get requests quickly where there is a mix of multi and
> single get workload. This is the case for multi put/single put as well. Any
> drawbacks in doing this segregation?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22418) Separate queue/executor for Multi get requests
[
https://issues.apache.org/jira/browse/HBASE-22418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22418:
--
Attachment: HBASE-22418.DRAFT
> Separate queue/executor for Multi get requests
> --
>
> Key: HBASE-22418
> URL: https://issues.apache.org/jira/browse/HBASE-22418
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Biju Nair
>Priority: Major
> Attachments: HBASE-22418.DRAFT
>
>
> Using {{RWQueueRpcExecutor}}, {{SimpleRpcScheduler}} lets us define separate
> queue/handlers to process write, read and scan requests. If this can be
> extended to have separate queue/handlers for multi get requests, it will help
> processing single get requests quickly where there is a mix of multi and
> single get workload. This is the case for multi put/single put as well. Any
> drawbacks in doing this segregation?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22418) Separate queue/executor for Multi get requests
[
https://issues.apache.org/jira/browse/HBASE-22418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22418:
--
Attachment: (was: HBASE-22418.DRAFT)
> Separate queue/executor for Multi get requests
> --
>
> Key: HBASE-22418
> URL: https://issues.apache.org/jira/browse/HBASE-22418
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Biju Nair
>Priority: Major
>
> Using {{RWQueueRpcExecutor}}, {{SimpleRpcScheduler}} lets us define separate
> queue/handlers to process write, read and scan requests. If this can be
> extended to have separate queue/handlers for multi get requests, it will help
> processing single get requests quickly where there is a mix of multi and
> single get workload. This is the case for multi put/single put as well. Any
> drawbacks in doing this segregation?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-10399) Add documentation for VerifyReplication to refguide
[
https://issues.apache.org/jira/browse/HBASE-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-10399:
--
Resolution: Implemented
Status: Resolved (was: Patch Available)
{{VerifyReplication}} is documented in HBase book (at least briefly) which
helps with user awareness. Closing. Please reopen if this change need to be
pursued further.
> Add documentation for VerifyReplication to refguide
> ---
>
> Key: HBASE-10399
> URL: https://issues.apache.org/jira/browse/HBASE-10399
> Project: HBase
> Issue Type: Improvement
> Components: documentation, Replication
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
>Priority: Minor
> Attachments: HBASE-10399.patch
>
>
> HBase refguide currently doesn't document how VerifyReplication is used for
> comparing local table with remote table.
> Document for VerifyReplication should be added so that users know how to use
> it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-8772) Separate Replication from HBase RegionServer process
[ https://issues.apache.org/jira/browse/HBASE-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair resolved HBASE-8772. -- Resolution: Incomplete Please re-open if this featured need to be pursued. > Separate Replication from HBase RegionServer process > > > Key: HBASE-8772 > URL: https://issues.apache.org/jira/browse/HBASE-8772 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Sameer Vaishampayan >Priority: Major > Labels: performance > > Replication is a separate functionality than managing regions and should be > able to be managed separately as a service rather than rolled into > RegionServer. Load on RegionServer, gc etc shouldn't affect the replication > service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22425) Balance shell command broken in HBase-3.0.0
[
https://issues.apache.org/jira/browse/HBASE-22425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840445#comment-16840445
]
Biju Nair commented on HBASE-22425:
---
Changes look fine.
> Balance shell command broken in HBase-3.0.0
> ---
>
> Key: HBASE-22425
> URL: https://issues.apache.org/jira/browse/HBASE-22425
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 3.0.0
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HBASE-22425.v1.patch
>
>
> Please see:
> https://issues.apache.org/jira/browse/HBASE-22387?focusedCommentId=16837386=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16837386
> {code}
> hbase(main):001:0> balancer
> (eval):3: warning: instance variable @shell not initialized
> Exception `NoMethodError' at (eval):2 - undefined method `command' for
> nil:NilClass
> ERROR: undefined method `command' for nil:NilClass
> Backtrace: (eval):2:in `balancer'
>
> /home/huzheng/.minos/packages/hbase/7b5bccdecd0600d98ad9a28d826e289fb6d58f46-20190510-165336/hbase-3.0.0-SNAPSHOT/lib/ruby/hbase/admin.rb:192:in
> `balancer'
>
> /home/huzheng/.minos/packages/hbase/7b5bccdecd0600d98ad9a28d826e289fb6d58f46-20190510-165336/hbase-3.0.0-SNAPSHOT/lib/ruby/shell/commands/balancer.rb:47:in
> `command'
>
> /home/huzheng/.minos/packages/hbase/7b5bccdecd0600d98ad9a28d826e289fb6d58f46-20190510-165336/hbase-3.0.0-SNAPSHOT/lib/ruby/shell/commands.rb:49:in
> `block in command_safe'
>
> /home/huzheng/.minos/packages/hbase/7b5bccdecd0600d98ad9a28d826e289fb6d58f46-20190510-165336/hbase-3.0.0-SNAPSHOT/lib/ruby/shell/commands.rb:122:in
> `translate_hbase_exceptions'
>
> /home/huzheng/.minos/packages/hbase/7b5bccdecd0600d98ad9a28d826e289fb6d58f46-20190510-165336/hbase-3.0.0-SNAPSHOT/lib/ruby/shell/commands.rb:49:in
> `command_safe'
>
> /home/huzheng/.minos/packages/hbase/7b5bccdecd0600d98ad9a28d826e289fb6d58f46-20190510-165336/hbase-3.0.0-SNAPSHOT/lib/ruby/shell.rb:148:in
> `internal_command'
>
> /home/huzheng/.minos/packages/hbase/7b5bccdecd0600d98ad9a28d826e289fb6d58f46-20190510-165336/hbase-3.0.0-SNAPSHOT/lib/ruby/shell.rb:140:in
> `command'
>(eval):2:in `balancer'
>(hbase):1:in `'
>org/jruby/RubyKernel.java:994:in `eval'
>
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/workspace.rb:87:in
> `evaluate'
>
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/context.rb:380:in
> `evaluate'
>uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:489:in
> `block in eval_input'
>uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:623:in
> `signal_status'
>uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:486:in
> `block in eval_input'
>
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:246:in
> `block in each_top_level_statement'
>org/jruby/RubyKernel.java:1292:in `loop'
>
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:232:in
> `block in each_top_level_statement'
>org/jruby/RubyKernel.java:1114:in `catch'
>
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:231:in
> `each_top_level_statement'
>uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:485:in
> `eval_input'
>
> /home/huzheng/.minos/packages/hbase/7b5bccdecd0600d98ad9a28d826e289fb6d58f46-20190510-165336/hbase-3.0.0-SNAPSHOT/bin/hirb.rb:214:in
> `block in start'
>org/jruby/RubyKernel.java:1114:in `catch'
>
> /home/huzheng/.minos/packages/hbase/7b5bccdecd0600d98ad9a28d826e289fb6d58f46-20190510-165336/hbase-3.0.0-SNAPSHOT/bin/hirb.rb:213:in
> `start'
>
> /home/huzheng/.minos/packages/hbase/7b5bccdecd0600d98ad9a28d826e289fb6d58f46-20190510-165336/hbase-3.0.0-SNAPSHOT/bin/hirb.rb:219:in
> `'
> For usage try 'help "balancer"'
> Took 0.1378 seconds
>
>
> hbase(main):002:0>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22422) Retain an ByteBuff with refCnt=0 when getBlock from LRUCache
[ https://issues.apache.org/jira/browse/HBASE-22422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-22422: -- Component/s: BlockCache > Retain an ByteBuff with refCnt=0 when getBlock from LRUCache > > > Key: HBASE-22422 > URL: https://issues.apache.org/jira/browse/HBASE-22422 > Project: HBase > Issue Type: Sub-task > Components: BlockCache >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: LRUBlockCache-getBlock.png, > image-2019-05-15-12-00-03-641.png > > > After runing YCSB scan/get benchmark in our XiaoMi cluster, we found the get > QPS dropped from 25000/s to hunderds per second in a cluster with five > nodes. > After enable the debug log at YCSB client side, I found the following > stacktrace , see > https://issues.apache.org/jira/secure/attachment/12968745/image-2019-05-15-12-00-03-641.png > Need a patch to fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22418) Separate queue/executor for Multi get requests
[
https://issues.apache.org/jira/browse/HBASE-22418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839897#comment-16839897
]
Biju Nair commented on HBASE-22418:
---
Attached changes for feedback which based on the new property
{{hbase.ipc.server.callqueue.multiread.ratio}} value will use a fraction of the
read queue/handler to service {{multi gets}}.
> Separate queue/executor for Multi get requests
> --
>
> Key: HBASE-22418
> URL: https://issues.apache.org/jira/browse/HBASE-22418
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Biju Nair
>Priority: Major
> Attachments: HBASE-22418.DRAFT
>
>
> Using {{RWQueueRpcExecutor}}, {{SimpleRpcScheduler}} lets us define separate
> queue/handlers to process write, read and scan requests. If this can be
> extended to have separate queue/handlers for multi get requests, it will help
> processing single get requests quickly where there is a mix of multi and
> single get workload. This is the case for multi put/single put as well. Any
> drawbacks in doing this segregation?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22418) Separate queue/executor for Multi get requests
[
https://issues.apache.org/jira/browse/HBASE-22418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22418:
--
Attachment: HBASE-22418.DRAFT
> Separate queue/executor for Multi get requests
> --
>
> Key: HBASE-22418
> URL: https://issues.apache.org/jira/browse/HBASE-22418
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Biju Nair
>Priority: Major
> Attachments: HBASE-22418.DRAFT
>
>
> Using {{RWQueueRpcExecutor}}, {{SimpleRpcScheduler}} lets us define separate
> queue/handlers to process write, read and scan requests. If this can be
> extended to have separate queue/handlers for multi get requests, it will help
> processing single get requests quickly where there is a mix of multi and
> single get workload. This is the case for multi put/single put as well. Any
> drawbacks in doing this segregation?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-22418) Separate queue/executor for Multi get requests
Biju Nair created HBASE-22418:
-
Summary: Separate queue/executor for Multi get requests
Key: HBASE-22418
URL: https://issues.apache.org/jira/browse/HBASE-22418
Project: HBase
Issue Type: Improvement
Reporter: Biju Nair
Using {{RWQueueRpcExecutor}}, {{SimpleRpcScheduler}} lets us define separate
queue/handlers to process write, read and scan requests. If this can be
extended to have separate queue/handlers for multi get requests, it will help
processing single get requests quickly where there is a mix of multi and single
get workload. This is the case for multi put/single put as well. Any drawbacks
in doing this segregation?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22418) Separate queue/executor for Multi get requests
[
https://issues.apache.org/jira/browse/HBASE-22418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22418:
--
Component/s: regionserver
> Separate queue/executor for Multi get requests
> --
>
> Key: HBASE-22418
> URL: https://issues.apache.org/jira/browse/HBASE-22418
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Biju Nair
>Priority: Major
>
> Using {{RWQueueRpcExecutor}}, {{SimpleRpcScheduler}} lets us define separate
> queue/handlers to process write, read and scan requests. If this can be
> extended to have separate queue/handlers for multi get requests, it will help
> processing single get requests quickly where there is a mix of multi and
> single get workload. This is the case for multi put/single put as well. Any
> drawbacks in doing this segregation?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22415) HBOSS: Reduce log verbosity in ZKTreeLockManager when waiting on a parent/child node lock
[ https://issues.apache.org/jira/browse/HBASE-22415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839891#comment-16839891 ] Biju Nair commented on HBASE-22415: --- Changes are pretty straight forward. > HBOSS: Reduce log verbosity in ZKTreeLockManager when waiting on a > parent/child node lock > - > > Key: HBASE-22415 > URL: https://issues.apache.org/jira/browse/HBASE-22415 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Labels: HBOSS > Attachments: HBASE-22415.master.001.patch > > > On normal read/write workloads, it will be very often that requests will > reach an temporary "locked" node by some other requests, causing process logs > to flood with the messages been currently logged as WARN. To improve > supportability, am proposing a patch that reduces these logging to DEBUG. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22415) HBOSS: Reduce log verbosity in ZKTreeLockManager when waiting on a parent/child node lock
[ https://issues.apache.org/jira/browse/HBASE-22415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-22415: -- Labels: HBOSS (was: ) > HBOSS: Reduce log verbosity in ZKTreeLockManager when waiting on a > parent/child node lock > - > > Key: HBASE-22415 > URL: https://issues.apache.org/jira/browse/HBASE-22415 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Labels: HBOSS > Attachments: HBASE-22415.master.001.patch > > > On normal read/write workloads, it will be very often that requests will > reach an temporary "locked" node by some other requests, causing process logs > to flood with the messages been currently logged as WARN. To improve > supportability, am proposing a patch that reduces these logging to DEBUG. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22386) HBOSS: Limit depth that listing locks check for other locks
[ https://issues.apache.org/jira/browse/HBASE-22386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-22386: -- Labels: HBOSS (was: ) > HBOSS: Limit depth that listing locks check for other locks > --- > > Key: HBASE-22386 > URL: https://issues.apache.org/jira/browse/HBASE-22386 > Project: HBase > Issue Type: Bug >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Labels: HBOSS > Fix For: hbase-filesystem-1.0.0-alpha1 > > Attachments: HBASE-22386.001.patch, HBASE-22386.002.patch > > > treeWriteLock will check all the way up and down the tree for locks. This is > more aggressive than it needs to be, and integration testing has shown that > there's significant contention when listing tables, and this is one of > numerous operations that doesn't need to recursively lock the whole subtree. > There's actually a number of operations that only need to lock up or down 1 > level only, so let's start with listing: non-recursive listings don't need to > care about what's going on more than 1 level below them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22416) HBOSS: unit tests fail with ConnectionLoss when IPv6 enabled and not set up locally
[
https://issues.apache.org/jira/browse/HBASE-22416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-22416:
--
Labels: HBOSS (was: )
> HBOSS: unit tests fail with ConnectionLoss when IPv6 enabled and not set up
> locally
> ---
>
> Key: HBASE-22416
> URL: https://issues.apache.org/jira/browse/HBASE-22416
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
> Labels: HBOSS
> Fix For: hbase-filesystem-1.0.0-alpha1
>
> Attachments: HBASE-22416.001.patch
>
>
> I've seen that the HBOSS unit tests will fail for me with a ConnectionLoss
> exception. This is resolved when I force ipv4. My hunch is that we don't
> block for curator to become connected, and because of my local setup, the
> ipv6 lookup happens first and has to time-out. The test starts running but
> Curator hasn't yet become connected, so we fail with a ConnectionLoss in the
> test itself.
> We should wait for Curator to become connected and maybe add
> {{-Djava.net.preferIPv4Stack=true}} to surefire opts
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-11853) Provide an alternative to the apache build for developers (like me) who aren't committers
[ https://issues.apache.org/jira/browse/HBASE-11853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-11853: -- Resolution: Invalid Status: Resolved (was: Patch Available) Please see the previous comment. If this is something which need to be pursued, re-open. > Provide an alternative to the apache build for developers (like me) who > aren't committers > - > > Key: HBASE-11853 > URL: https://issues.apache.org/jira/browse/HBASE-11853 > Project: HBase > Issue Type: Sub-task >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Major > Attachments: HBASE-11853-testing-v0.patch, > HBASE-11853-testing-v1.patch, HBASE-11853-v0.patch, HBASE-11853-v3.patch, > HBASE-11853-v4.patch, HBASE-11853-v5.patch, HBASE-11853-v6.patch, > HBASE-11853-v7.patch > > > Travis CI and Circle-CI now provide free builds for open source projects. I > created the capability to run builds this way. Although they are closed > source (and thus not a replacement for jenkins IMHO), they are super > convenient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-9741) Remove hbase.regions.slop from hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair resolved HBASE-9741. -- Resolution: Incomplete Please refer the previous comment. If this is still an issue, re-open. > Remove hbase.regions.slop from hbase-default.xml > > > Key: HBASE-9741 > URL: https://issues.apache.org/jira/browse/HBASE-9741 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Elliott Clark >Priority: Major > Labels: beginner > Attachments: HBASE-9741-v0.patch, HBASE-9741-v1.patch, > HBASE-9741-v3.patch, HBASE-9741-v4.patch > > > Different balancers have different slop default values. We should remove > hbase.regions.slop from hbase-default.xml -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-9806) Add PerfEval tool for BlockCache
[ https://issues.apache.org/jira/browse/HBASE-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-9806: - Component/s: BlockCache > Add PerfEval tool for BlockCache > > > Key: HBASE-9806 > URL: https://issues.apache.org/jira/browse/HBASE-9806 > Project: HBase > Issue Type: Test > Components: BlockCache, Performance, test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Attachments: HBASE-9806.00.patch, HBASE-9806.01.patch, > HBASE-9806.02.patch, conf_20g.patch, conf_3g.patch, test1_run1_20g.pdf, > test1_run1_3g.pdf, test1_run2_20g.pdf, test1_run2_3g.pdf > > > We have at least three different block caching layers with myriad > configuration settings. Let's add a tool for evaluating memory allocations > and configuration combinations with different access patterns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-11853) Provide an alternative to the apache build for developers (like me) who aren't committers
[ https://issues.apache.org/jira/browse/HBASE-11853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839027#comment-16839027 ] Biju Nair commented on HBASE-11853: --- Based on the last comment and how Jenkins build is happening for patch submissions, this may be a non issue. Can we close this? [~apurtell], [~busbey] > Provide an alternative to the apache build for developers (like me) who > aren't committers > - > > Key: HBASE-11853 > URL: https://issues.apache.org/jira/browse/HBASE-11853 > Project: HBase > Issue Type: Sub-task >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Major > Attachments: HBASE-11853-testing-v0.patch, > HBASE-11853-testing-v1.patch, HBASE-11853-v0.patch, HBASE-11853-v3.patch, > HBASE-11853-v4.patch, HBASE-11853-v5.patch, HBASE-11853-v6.patch, > HBASE-11853-v7.patch > > > Travis CI and Circle-CI now provide free builds for open source projects. I > created the capability to run builds this way. Although they are closed > source (and thus not a replacement for jenkins IMHO), they are super > convenient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
