A new Mutilple-Type Queue idea to handle multiple workloads

[范范欣欣]

Now, the feature HBASE-11355 seperates the single Call Queue into
MutilQueue(get call queue, write call queue and scan call queue), and each
type queue can specify fixed number of handlers. It's helpful in some
outages , to avoid all read or all write requests ran out of handler
threads.

however, there are still several problems :

1. workloads in the same request type(get/write/scan) may influence each
other as before, consider the following scenario:

(1) both client-1 and client-2 send write requests, the client-1 write the
large objects(100KB record) , and client-2 write the small objects (1KB
record). the client -1 will ran out of all handler threads of the
write-queue, and decrease the client-2 throughput

(2) both client-3 and client-4 send get requests, the client-3 search all
data from lots of hfiles( all search key are equally popular), read latency
is high. the client-4 do not require any I/O resources(say, data is
cached). the client-3 will ran out of all handler threads of the get-queue,
and increase the read latency of client-4

2. administor can't increate/decrease the handler number for the specified
queue easily


we are trying to implement a new Mutilple-Typed Queue, administor can
create a queue with a specified number of handler for specified table and
specified request type(get/write/scan), as:

create_queue 'queue1' ,{'handler' => 100}

grant_queue 'table1','scan','queue1'

grant_queue 'tableN','scan','queue1'


create_queue 'queue2' ,{'handler' => 50}

grant_queue 'table2','write','queue2'


the idea based on the fact that the workload for a specified table and
request type will be unique.

in addition, administor can manager the queue with commands:

//easily increase/decrease handlers

alter_queue 'queue1' ,{'handler' => 50}

//list all queues

list_queues

//drop the specified queue

drop_queue 'queue1'


I am wondering if the developers could look at the idea and let me know if
anything is incorrect or inaccurate, or if I have missed anything.


Thanks a lot for your help!

[jira] [Reopened] (HBASE-17895) TestAsyncProcess#testAction fails if unsafe support is false

[Chia-Ping Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai reopened HBASE-17895:


After attaching the patch, you should click the "submit patch" to trigger QA

> TestAsyncProcess#testAction fails if unsafe support is false
> 
>
> Key: HBASE-17895
> URL: https://issues.apache.org/jira/browse/HBASE-17895
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: Chia-Ping Tsai
>Assignee: Wen-Hsiu,Chang
>Priority: Trivial
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17895-v0.patch
>
>
> {noformat}
> assertEquals(-1, action_0.compareTo(action_1));
> assertEquals(1, action_1.compareTo(action_0));
> {noformat}
> The correction is shown below.
> {noformat}
> assertTrue(action_0.compareTo(action_1) < 0);
> assertTrue(action_1.compareTo(action_0) > 0);
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HBASE-17895) TestAsyncProcess#testAction fails if unsafe support is false

[Wen-Hsiu,Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wen-Hsiu,Chang resolved HBASE-17895.

Resolution: Fixed

> TestAsyncProcess#testAction fails if unsafe support is false
> 
>
> Key: HBASE-17895
> URL: https://issues.apache.org/jira/browse/HBASE-17895
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: Chia-Ping Tsai
>Assignee: Wen-Hsiu,Chang
>Priority: Trivial
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17895-v0.patch
>
>
> {noformat}
> assertEquals(-1, action_0.compareTo(action_1));
> assertEquals(1, action_1.compareTo(action_0));
> {noformat}
> The correction is shown below.
> {noformat}
> assertTrue(action_0.compareTo(action_1) < 0);
> assertTrue(action_1.compareTo(action_0) > 0);
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HBASE-17128) Find Cause of a Write Perf Regression in branch-1.2

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-17128.
---
Resolution: Implemented

Resolving as solved by the compound of issues referenced herein.

> Find Cause of a Write Perf Regression in branch-1.2
> ---
>
> Key: HBASE-17128
> URL: https://issues.apache.org/jira/browse/HBASE-17128
> Project: HBase
>  Issue Type: Task
>Reporter: stack
> Attachments: 17072.5.9.txt
>
>
> As reported by [~gbaecher] up on the mailing list, there is a regression in 
> 1.2. The regression is in a CDH version of 1.2 actually but the CDH hbase is 
> a near pure 1.2. This is a working issue to figure which of the below changes 
> brought on slower writes (The list comes from doing the following...git log 
> --oneline  
> remotes/origin/cdh5-1.2.0_5.8.0_dev..remotes/origin/cdh5-1.2.0_5.9.0_dev ... 
> I stripped the few CDH specific changes, packaging and tagging only, and then 
> made two groupings; candidates and the unlikelies):
> {code}
>   1 bbc6762 HBASE-16023 Fastpath for the FIFO rpcscheduler Adds an executor 
> that does balanced queue and fast path handing off requests directly to 
> waiting handlers if any present. Idea taken from Apace Kudu (incubating). See 
> https://gerr#
>   2 a260917 HBASE-16288 HFile intermediate block level indexes might recurse 
> forever creating multi TB files
>   3 5633281 HBASE-15811 Batch Get after batch Put does not fetch all Cells We 
> were not waiting on all executors in a batch to complete. The test for 
> no-more-executors was damaged by the 0.99/0.98.4 fix "HBASE-11403 Fix race 
> conditions aro#
>   4 780f720 HBASE-11625 - Verifies data before building HFileBlock. - Adds 
> HFileBlock.Header class which contains information about location of fields. 
> Testing: Adds CorruptedFSReaderImpl to TestChecksum. (Apekshit)
>   5 d735680 HBASE-12133 Add FastLongHistogram for metric computation (Yi Deng)
>   6 c4ee832 HBASE-15222 Use less contended classes for metrics
>   7
>   8 17320a4 HBASE-15683 Min latency in latency histograms are emitted as 
> Long.MAX_VALUE
>   9 283b39f HBASE-15396 Enhance mapreduce.TableSplit to add encoded region 
> name
>  10 39db592 HBASE-16195 Should not add chunk into chunkQueue if not using 
> chunk pool in HeapMemStoreLAB
>  11 5ff28b7 HBASE-16194 Should count in MSLAB chunk allocation into heap size 
> change when adding duplicate cells
>  12 5e3e0d2 HBASE-16318 fail build while rendering velocity template if 
> dependency license isn't in whitelist.
>  13 3ed66e3 HBASE-16318 consistently use the correct name for 'Apache 
> License, Version 2.0'
>  14 351832d HBASE-16340 exclude Xerces iplementation jars from coming in 
> transitively.
>  15 b6aa4be HBASE-16321 ensure no findbugs-jsr305
>  16 4f9dde7 HBASE-16317 revert all ESAPI changes
>  17 71b6a8a HBASE-16284 Unauthorized client can shutdown the cluster (Deokwoo 
> Han)
>  18 523753f HBASE-16450 Shell tool to dump replication queues
>  19 ca5f2ee HBASE-16379 [replication] Minor improvement to 
> replication/copy_tables_desc.rb
>  20 effd105 HBASE-16135 PeerClusterZnode under rs of removed peer may never 
> be deleted
>  21 a5c6610 HBASE-16319 Fix TestCacheOnWrite after HBASE-16288
>  22 1956bb0 HBASE-15808 Reduce potential bulk load intermediate space usage 
> and waste
>  23 031c54e HBASE-16096 Backport. Cleanly remove replication peers from 
> ZooKeeper.
>  24 60a3b12 HBASE-14963 Remove use of Guava Stopwatch from HBase client code 
> (Devaraj Das)
>  25 c7724fc HBASE-16207 can't restore snapshot without "Admin" permission
>  26 8322a0b HBASE-16227 [Shell] Column value formatter not working in scans. 
> Tested : manually using shell.
>  27 8f86658 HBASE-14818 user_permission does not list namespace permissions 
> (li xiang)
>  28 775cd21 HBASE-15465 userPermission returned by getUserPermission() for 
> the selected namespace does not have namespace set (li xiang)
>  29 8d85aff HBASE-16093 Fix splits failed before creating daughter regions 
> leave meta inconsistent
>  30 bc41317 HBASE-16140 bump owasp.esapi from 2.1.0 to 2.1.0.1
>  31 6fc70cd HBASE-16035 Nested AutoCloseables might not all get closed (Sean 
> Mackrory)
>  32 fe28fe84 HBASE-15891. Closeable resources potentially not getting closed 
> if exception is thrown.
>  33 1d2bf3c HBASE-14644 Region in transition metric is broken -- addendum 
> (Huaxiang Sun)
>  34 fd5f56c HBASE-16056 Procedure v2 - fix master crash for FileNotFound
>  35 10cd038 HBASE-16034 Fix ProcedureTestingUtility#LoadCounter.setMaxProcId()
>  36 dae4db4 HBASE-15872 Split TestWALProcedureStore
>  37 e638d86 HBASE-14644 Region in transition metric is broken (Huaxiang Sun)
>  38 f01b01d HBASE-15496 Throw RowTooBigException only for user scan/get 
> (Guanghao Zhang)
>  39 cc0ce66 HBASE-15746 Remove extra RegionCoprocessor 

Re: [DISCUSS] Status of the 0.98 release line

[Gary Helmling]

+1 to EOL, and thanks to Andrew for all of the RM'ing.

On Mon, Apr 10, 2017 at 12:27 PM Ted Yu  wrote:

> +1
>
> Andrew has done tremendous work.
>
> On Mon, Apr 10, 2017 at 12:17 PM, Mikhail Antonov 
> wrote:
>
> > +1 to EOL 0.98.
> >
> > Thanks Andrew for all the work maintaining it!
> >
> > -Mikhail
> >
> > On Mon, Apr 10, 2017 at 12:10 PM, Dima Spivak 
> > wrote:
> >
> > > +1
> > >
> > > -Dima
> > >
> > > On Mon, Apr 10, 2017 at 12:08 PM, Stack  wrote:
> > >
> > > > I agree we should EOL 0.98.
> > > > St.Ack
> > > >
> > > > On Mon, Apr 10, 2017 at 11:43 AM, Andrew Purtell <
> apurt...@apache.org>
> > > > wrote:
> > > >
> > > > > Please speak up if it is incorrect to interpret the lack of
> responses
> > > as
> > > > > indicating consensus on declaring 0.98 EOL.
> > > > >
> > > > > I believe we should declare 0.98 EOL.
> > > > >
> > > > >
> > > > > On Wed, Mar 29, 2017 at 6:56 AM, Sean Busbey 
> > > wrote:
> > > > >
> > > > > > Hi Folks!
> > > > > >
> > > > > > Back in January our Andrew Purtell stepped down as the release
> > > > > > manager for the 0.98 release line.
> > > > > >
> > > > > > On the resultant dev@hbase thread[1] folks seemed largely in
> favor
> > > of
> > > > > > declaring end-of-maintenance for the 0.98 line.
> > > > > >
> > > > > > Now that it's been a couple of months, does anyone have concerns
> > > about
> > > > > > pushing forward on that?
> > > > > >
> > > > > > Do folks who listen on user@hbase but not dev@hbase have any
> > > concerns?
> > > > > >
> > > > > > As with any end-of-maintenance branch, the PMC would consider on
> a
> > > > > > case-by-case basis doing a future release of the branch should a
> > > > > > critical security vulnerability show up.
> > > > > >
> > > > > >
> > > > > > [1]: https://s.apache.org/DjCi
> > > > > >
> > > > > > -busbey
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > >- Andy
> > > > >
> > > > > If you are given a choice, you believe you have acted freely. -
> > Raymond
> > > > > Teller (via Peter Watts)
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Michael Antonov
> >
>

Re: [DISCUSS] Status of the 0.98 release line

[Ted Yu]

+1

Andrew has done tremendous work.

On Mon, Apr 10, 2017 at 12:17 PM, Mikhail Antonov 
wrote:

> +1 to EOL 0.98.
>
> Thanks Andrew for all the work maintaining it!
>
> -Mikhail
>
> On Mon, Apr 10, 2017 at 12:10 PM, Dima Spivak 
> wrote:
>
> > +1
> >
> > -Dima
> >
> > On Mon, Apr 10, 2017 at 12:08 PM, Stack  wrote:
> >
> > > I agree we should EOL 0.98.
> > > St.Ack
> > >
> > > On Mon, Apr 10, 2017 at 11:43 AM, Andrew Purtell 
> > > wrote:
> > >
> > > > Please speak up if it is incorrect to interpret the lack of responses
> > as
> > > > indicating consensus on declaring 0.98 EOL.
> > > >
> > > > I believe we should declare 0.98 EOL.
> > > >
> > > >
> > > > On Wed, Mar 29, 2017 at 6:56 AM, Sean Busbey 
> > wrote:
> > > >
> > > > > Hi Folks!
> > > > >
> > > > > Back in January our Andrew Purtell stepped down as the release
> > > > > manager for the 0.98 release line.
> > > > >
> > > > > On the resultant dev@hbase thread[1] folks seemed largely in favor
> > of
> > > > > declaring end-of-maintenance for the 0.98 line.
> > > > >
> > > > > Now that it's been a couple of months, does anyone have concerns
> > about
> > > > > pushing forward on that?
> > > > >
> > > > > Do folks who listen on user@hbase but not dev@hbase have any
> > concerns?
> > > > >
> > > > > As with any end-of-maintenance branch, the PMC would consider on a
> > > > > case-by-case basis doing a future release of the branch should a
> > > > > critical security vulnerability show up.
> > > > >
> > > > >
> > > > > [1]: https://s.apache.org/DjCi
> > > > >
> > > > > -busbey
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > >- Andy
> > > >
> > > > If you are given a choice, you believe you have acted freely. -
> Raymond
> > > > Teller (via Peter Watts)
> > > >
> > >
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>

Re: Move the 'stable' pointer for HBase in dist.apache.org to 1.3.0 ?

[Mikhail Antonov]

Yeah, I also would prefer to wait until 1.3.1 is released.

On Mon, Apr 10, 2017 at 11:30 AM, Andrew Purtell 
wrote:

> Ok, thanks.
>
>
> On Mon, Apr 10, 2017 at 11:01 AM, Sean Busbey  wrote:
>
> > Okay, so rephrasing.
> >
> > I'd rather not mark 1.3.0 as the stable release since HBASE-17069 is in
> the
> > current stable pointer (1.2.5) and hasn't been fixed in a 1.3.z release
> > line version yet.
> >
> > I think we should revisit the question if and when there's a 1.3.1
> release.
> >
> > -busbey
> >
> > On Mon, Apr 10, 2017 at 12:53 PM Andrew Purtell 
> > wrote:
> >
> > > I can't ask about 1.3.1 (and neither can you Ted) because 1.3.1 is not
> > > released yet. I need to bring this up now because I made a provisional
> > > remark on the board report about 1.3.0. If possible let's keep this on
> > > topic. Thanks.
> > >
> > >
> > > On Mon, Apr 10, 2017 at 10:50 AM, Ted Yu  wrote:
> > >
> > > > Release 1.3.1 seems to have high chance of passing.
> > > >
> > > > How about using 1.3.1 as stable release ?
> > > >
> > > > On Mon, Apr 10, 2017 at 10:28 AM, Andrew Purtell <
> apurt...@apache.org>
> > > > wrote:
> > > >
> > > > > Shall we move the 'stable' pointer for HBase in dist.apache.org to
> > > > release
> > > > > 1.3.0 ?
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > >- Andy
> > > > >
> > > > > If you are given a choice, you believe you have acted freely. -
> > Raymond
> > > > > Teller (via Peter Watts)
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > If you are given a choice, you believe you have acted freely. - Raymond
> > > Teller (via Peter Watts)
> > >
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>



-- 
Thanks,
Michael Antonov

Re: [DISCUSS] Status of the 0.98 release line

[Mikhail Antonov]

+1 to EOL 0.98.

Thanks Andrew for all the work maintaining it!

-Mikhail

On Mon, Apr 10, 2017 at 12:10 PM, Dima Spivak  wrote:

> +1
>
> -Dima
>
> On Mon, Apr 10, 2017 at 12:08 PM, Stack  wrote:
>
> > I agree we should EOL 0.98.
> > St.Ack
> >
> > On Mon, Apr 10, 2017 at 11:43 AM, Andrew Purtell 
> > wrote:
> >
> > > Please speak up if it is incorrect to interpret the lack of responses
> as
> > > indicating consensus on declaring 0.98 EOL.
> > >
> > > I believe we should declare 0.98 EOL.
> > >
> > >
> > > On Wed, Mar 29, 2017 at 6:56 AM, Sean Busbey 
> wrote:
> > >
> > > > Hi Folks!
> > > >
> > > > Back in January our Andrew Purtell stepped down as the release
> > > > manager for the 0.98 release line.
> > > >
> > > > On the resultant dev@hbase thread[1] folks seemed largely in favor
> of
> > > > declaring end-of-maintenance for the 0.98 line.
> > > >
> > > > Now that it's been a couple of months, does anyone have concerns
> about
> > > > pushing forward on that?
> > > >
> > > > Do folks who listen on user@hbase but not dev@hbase have any
> concerns?
> > > >
> > > > As with any end-of-maintenance branch, the PMC would consider on a
> > > > case-by-case basis doing a future release of the branch should a
> > > > critical security vulnerability show up.
> > > >
> > > >
> > > > [1]: https://s.apache.org/DjCi
> > > >
> > > > -busbey
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > If you are given a choice, you believe you have acted freely. - Raymond
> > > Teller (via Peter Watts)
> > >
> >
>



-- 
Thanks,
Michael Antonov

Re: [DISCUSS] Status of the 0.98 release line

[Dima Spivak]

+1

-Dima

On Mon, Apr 10, 2017 at 12:08 PM, Stack  wrote:

> I agree we should EOL 0.98.
> St.Ack
>
> On Mon, Apr 10, 2017 at 11:43 AM, Andrew Purtell 
> wrote:
>
> > Please speak up if it is incorrect to interpret the lack of responses as
> > indicating consensus on declaring 0.98 EOL.
> >
> > I believe we should declare 0.98 EOL.
> >
> >
> > On Wed, Mar 29, 2017 at 6:56 AM, Sean Busbey  wrote:
> >
> > > Hi Folks!
> > >
> > > Back in January our Andrew Purtell stepped down as the release
> > > manager for the 0.98 release line.
> > >
> > > On the resultant dev@hbase thread[1] folks seemed largely in favor of
> > > declaring end-of-maintenance for the 0.98 line.
> > >
> > > Now that it's been a couple of months, does anyone have concerns about
> > > pushing forward on that?
> > >
> > > Do folks who listen on user@hbase but not dev@hbase have any concerns?
> > >
> > > As with any end-of-maintenance branch, the PMC would consider on a
> > > case-by-case basis doing a future release of the branch should a
> > > critical security vulnerability show up.
> > >
> > >
> > > [1]: https://s.apache.org/DjCi
> > >
> > > -busbey
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > If you are given a choice, you believe you have acted freely. - Raymond
> > Teller (via Peter Watts)
> >
>

Re: [DISCUSS] Status of the 0.98 release line

[Stack]

I agree we should EOL 0.98.
St.Ack

On Mon, Apr 10, 2017 at 11:43 AM, Andrew Purtell 
wrote:

> Please speak up if it is incorrect to interpret the lack of responses as
> indicating consensus on declaring 0.98 EOL.
>
> I believe we should declare 0.98 EOL.
>
>
> On Wed, Mar 29, 2017 at 6:56 AM, Sean Busbey  wrote:
>
> > Hi Folks!
> >
> > Back in January our Andrew Purtell stepped down as the release
> > manager for the 0.98 release line.
> >
> > On the resultant dev@hbase thread[1] folks seemed largely in favor of
> > declaring end-of-maintenance for the 0.98 line.
> >
> > Now that it's been a couple of months, does anyone have concerns about
> > pushing forward on that?
> >
> > Do folks who listen on user@hbase but not dev@hbase have any concerns?
> >
> > As with any end-of-maintenance branch, the PMC would consider on a
> > case-by-case basis doing a future release of the branch should a
> > critical security vulnerability show up.
> >
> >
> > [1]: https://s.apache.org/DjCi
> >
> > -busbey
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

Re: [DISCUSS] Status of 0.94 release line

[Andrew Purtell]

I also think we should declare 0.94 EOL. If no further comment, we will
take this to be consensus.


On Wed, Mar 29, 2017 at 8:15 AM, Ted Yu  wrote:

> Yes, I think so.
>
> On Wed, Mar 29, 2017 at 6:54 AM, Sean Busbey  wrote:
>
> > Hi Folks!
> >
> > It's been 2 years since there's been a release on the 0.94 line. Shall
> > we consider it end-of-maintenance?
> >
> > -busbey
> >
>



-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: [DISCUSS] Status of the 0.98 release line

[Andrew Purtell]

Please speak up if it is incorrect to interpret the lack of responses as
indicating consensus on declaring 0.98 EOL.

I believe we should declare 0.98 EOL.


On Wed, Mar 29, 2017 at 6:56 AM, Sean Busbey  wrote:

> Hi Folks!
>
> Back in January our Andrew Purtell stepped down as the release
> manager for the 0.98 release line.
>
> On the resultant dev@hbase thread[1] folks seemed largely in favor of
> declaring end-of-maintenance for the 0.98 line.
>
> Now that it's been a couple of months, does anyone have concerns about
> pushing forward on that?
>
> Do folks who listen on user@hbase but not dev@hbase have any concerns?
>
> As with any end-of-maintenance branch, the PMC would consider on a
> case-by-case basis doing a future release of the branch should a
> critical security vulnerability show up.
>
>
> [1]: https://s.apache.org/DjCi
>
> -busbey
>



-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: [VOTE] Second release candidate for HBase 1.3.1 (RC1) is available

[Ted Yu]

+1

Checked sums and signatures: Okay
Ran test suite (1.8.0_91): Okay
RAT check: Okay

On Sun, Apr 9, 2017 at 9:40 AM, Andrew Purtell  wrote:

> +1
>
> Checked sums and signatures: ok
> Built from source: ok (7u80)
> RAT check passes: ok
> Unit test suite passes: ok (7u80)
> 1M row LTT: ok (8u102)
> 200M row ITBLL: ok (8u102)
> Phoenix master branch (+PHOENIX-3603): Compiles, most tests pass. IndexIT
> hangs. Probably a Phoenix thing.
>
>
> On Fri, Apr 7, 2017 at 1:31 AM, Mikhail Antonov 
> wrote:
>
> > Hello everyone,
> >
> > I'm pleased to announce that the second release candidate for Apache
> HBase
> > 1.3.1 is available to download and testing.
> >
> > The first release candidate (RC0) was taken down as it contained
> > compatibility issue (thanks Yu Li for finding it and getting it fixed).
> The
> > only difference from RC0 is HBASE-17886.
> >
> > Artifacts are available here:
> >
> > https://dist.apache.org/repos/dist/dev/hbase/1.3.1RC1/
> >
> > Maven artifacts are available in the staging repository:
> >
> > https://repository.apache.org/content/repositories/orgapachehbase-1166
> >
> > All artifacts are signed with my code signing key 35A4ABE2, which is also
> > in the project KEYS file at
> >
> > http://www.apache.org/dist/hbase/KEYS
> >
> > these artifacts correspond to commit hash
> >
> > 930b9a55528fe45d8edce7af42fef2d35e77677a tagged as 1.3.1RC1.
> >
> > HBase 1.3.1 is the first maintenance release in the HBase 1.3.z release
> > line,
> > continuing on the theme of bringing a stable, reliable database to the
> > Hadoop and NoSQL communities. This release includes 68 resolved issues
> > since the initial 1.3.0 release that was 3 month ago.
> >
> > The full list of issues addressed is available at
> >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12310753=12335746
> >
> > and also in the CHANGES.txt file in the root directory of the source
> > tarball.
> >
> > Please take a few minutes to verify the release and vote on
> > releasing it:
> >
> > [ ] +1 Release this package as Apache HBase 1.3.1
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
> > This Vote will run for one week and close Fri Apr 14, 2017 02:00 PST.
> >
> > Thanks !
> > -Mikhail Antonov
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

Re: Move the 'stable' pointer for HBase in dist.apache.org to 1.3.0 ?

[Andrew Purtell]

Ok, thanks.


On Mon, Apr 10, 2017 at 11:01 AM, Sean Busbey  wrote:

> Okay, so rephrasing.
>
> I'd rather not mark 1.3.0 as the stable release since HBASE-17069 is in the
> current stable pointer (1.2.5) and hasn't been fixed in a 1.3.z release
> line version yet.
>
> I think we should revisit the question if and when there's a 1.3.1 release.
>
> -busbey
>
> On Mon, Apr 10, 2017 at 12:53 PM Andrew Purtell 
> wrote:
>
> > I can't ask about 1.3.1 (and neither can you Ted) because 1.3.1 is not
> > released yet. I need to bring this up now because I made a provisional
> > remark on the board report about 1.3.0. If possible let's keep this on
> > topic. Thanks.
> >
> >
> > On Mon, Apr 10, 2017 at 10:50 AM, Ted Yu  wrote:
> >
> > > Release 1.3.1 seems to have high chance of passing.
> > >
> > > How about using 1.3.1 as stable release ?
> > >
> > > On Mon, Apr 10, 2017 at 10:28 AM, Andrew Purtell 
> > > wrote:
> > >
> > > > Shall we move the 'stable' pointer for HBase in dist.apache.org to
> > > release
> > > > 1.3.0 ?
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > >- Andy
> > > >
> > > > If you are given a choice, you believe you have acted freely. -
> Raymond
> > > > Teller (via Peter Watts)
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > If you are given a choice, you believe you have acted freely. - Raymond
> > Teller (via Peter Watts)
> >
>



-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: Move the 'stable' pointer for HBase in dist.apache.org to 1.3.0 ?

[Sean Busbey]

Okay, so rephrasing.

I'd rather not mark 1.3.0 as the stable release since HBASE-17069 is in the
current stable pointer (1.2.5) and hasn't been fixed in a 1.3.z release
line version yet.

I think we should revisit the question if and when there's a 1.3.1 release.

-busbey

On Mon, Apr 10, 2017 at 12:53 PM Andrew Purtell  wrote:

> I can't ask about 1.3.1 (and neither can you Ted) because 1.3.1 is not
> released yet. I need to bring this up now because I made a provisional
> remark on the board report about 1.3.0. If possible let's keep this on
> topic. Thanks.
>
>
> On Mon, Apr 10, 2017 at 10:50 AM, Ted Yu  wrote:
>
> > Release 1.3.1 seems to have high chance of passing.
> >
> > How about using 1.3.1 as stable release ?
> >
> > On Mon, Apr 10, 2017 at 10:28 AM, Andrew Purtell 
> > wrote:
> >
> > > Shall we move the 'stable' pointer for HBase in dist.apache.org to
> > release
> > > 1.3.0 ?
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > If you are given a choice, you believe you have acted freely. - Raymond
> > > Teller (via Peter Watts)
> > >
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

Re: Move the 'stable' pointer for HBase in dist.apache.org to 1.3.0 ?

[Ted Yu]

I brought up 1.3.1 because of HBASE-17886 which is marked as a blocker.

FYI

On Mon, Apr 10, 2017 at 10:53 AM, Andrew Purtell 
wrote:

> I can't ask about 1.3.1 (and neither can you Ted) because 1.3.1 is not
> released yet. I need to bring this up now because I made a provisional
> remark on the board report about 1.3.0. If possible let's keep this on
> topic. Thanks.
>
>
> On Mon, Apr 10, 2017 at 10:50 AM, Ted Yu  wrote:
>
> > Release 1.3.1 seems to have high chance of passing.
> >
> > How about using 1.3.1 as stable release ?
> >
> > On Mon, Apr 10, 2017 at 10:28 AM, Andrew Purtell 
> > wrote:
> >
> > > Shall we move the 'stable' pointer for HBase in dist.apache.org to
> > release
> > > 1.3.0 ?
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > If you are given a choice, you believe you have acted freely. - Raymond
> > > Teller (via Peter Watts)
> > >
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

Re: Move the 'stable' pointer for HBase in dist.apache.org to 1.3.0 ?

[Sean Busbey]

Given the number of outstanding fixes coming in on 1.3.1, I'd rather wait
for it as well.



On Mon, Apr 10, 2017 at 12:50 PM Ted Yu  wrote:

> Release 1.3.1 seems to have high chance of passing.
>
> How about using 1.3.1 as stable release ?
>
> On Mon, Apr 10, 2017 at 10:28 AM, Andrew Purtell 
> wrote:
>
> > Shall we move the 'stable' pointer for HBase in dist.apache.org to
> release
> > 1.3.0 ?
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > If you are given a choice, you believe you have acted freely. - Raymond
> > Teller (via Peter Watts)
> >
>

Re: Move the 'stable' pointer for HBase in dist.apache.org to 1.3.0 ?

[Andrew Purtell]

I can't ask about 1.3.1 (and neither can you Ted) because 1.3.1 is not
released yet. I need to bring this up now because I made a provisional
remark on the board report about 1.3.0. If possible let's keep this on
topic. Thanks.


On Mon, Apr 10, 2017 at 10:50 AM, Ted Yu  wrote:

> Release 1.3.1 seems to have high chance of passing.
>
> How about using 1.3.1 as stable release ?
>
> On Mon, Apr 10, 2017 at 10:28 AM, Andrew Purtell 
> wrote:
>
> > Shall we move the 'stable' pointer for HBase in dist.apache.org to
> release
> > 1.3.0 ?
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > If you are given a choice, you believe you have acted freely. - Raymond
> > Teller (via Peter Watts)
> >
>



-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: Move the 'stable' pointer for HBase in dist.apache.org to 1.3.0 ?

[Ted Yu]

Release 1.3.1 seems to have high chance of passing.

How about using 1.3.1 as stable release ?

On Mon, Apr 10, 2017 at 10:28 AM, Andrew Purtell 
wrote:

> Shall we move the 'stable' pointer for HBase in dist.apache.org to release
> 1.3.0 ?
>
>
> --
> Best regards,
>
>- Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

Re: Is HBase RPC-Handling idempotent for reads?

[Yu Li]

I see, some priority-based preemptive scheduling.

bq. if it requires I/O resources that are not allocated to it
Easy to tell whether the request misses the cache and requires IO
operation, but what's the standard of "not allocated"? Some kind of
timeout? Anyway, interesting topic and let us know if you work it out
(smile).

Best Regards,
Yu

On 11 April 2017 at 01:09, 杨苏立 Yang Su Li  wrote:

> On Sun, Apr 9, 2017 at 11:14 PM, Yu Li  wrote:
>
> > Correct me if I'm wrong, but I think we should assume no other but the
> > single operation when checking whether it's idempotent. Similar to the
> > wikipedia
> > example : "A
> function
> > looking up a customer's name and address in a database
> >  is typically idempotent, since
> > this will not cause the database to change", I think all
> Get/MultiGet/Scan
> > operations in hbase are idempotent.
> >
> > About "speculative rpc handling", I doubt whether it benefits in hbase.
> > Normally if a request already arrives at server side but with slow
> > execution, the problem might be:
> > 1. The server is too busy and request get queued
> > 2. The processing itself is slow due to the request pattern or some
> > hardware failure
> > I don't think a speculative execution of the request could help in any of
> > the above cases. It's different from the speculative task execution in
> MR,
> > there we could choose another node to execute the task while here we have
> > no choice.
> >
>
>
> We have a different use case here. Basically we are trying to enforce
> scheduling at HBase.
> Consider the following scenario: both client-1 and client-2 are competing
> for I/O resources.
> But client-2 are also issuing a bunch of requests that do not require any
> I/O resources (say, data is cached).
> Since we have idle CPU/memory, we want to serve these cached requests for
> client-2, but we do not want client-2 to use more than its fair share of
> I/O.
>
> Unfortunately, at the time we pick RPC call to handle, we don't know
> whether an RPC would cause I/O or not.
> So we think we can abort a request if it requires I/O resources that are
> not allocated to it, and re-schedule it later based on our scheduling
> policy.
>
>
>
>
>
>
> >
> > OTOH, we already have timeout mechanism to make sure server resource
> won't
> > be wasted:
> > 1. For scan
> > - When a request handling timeouts, server will stop further
> > processing, refer to RSRpcServices#getTimeLimit and
> > ScannerContext#checkTimeLimit
> > - If the client went away during processing, server will also stop
> > processing, check the SimpleRpcServer#disconnectSince and
> > RegionScannerImpl#nextInternal methods for more details.
> >
> > 2. For single Get
> > - Controlled by rpc and operation timeout
> >
> > 3. For MultiGet
> > - I think this is something we could improve. On client side we have
> > timeout mechanism but on server side there seems to be no relative
> > interrupt logic.
> >
> >
> > Best Regards,
> > Yu
> >
> > On 10 April 2017 at 11:12, Jerry He  wrote:
> >
> > > Again, it depends on how you abort and 'idempotent' can have different
> > > definitions.
> > >
> > > For example, even if you are only concerned about read,
> > > there are resources on the HRegion that the read touches or acquires
> > > (scanner, lock, mvcc etc) that hopefully will be cleaned/releases with
> > the
> > > abort.
> > > Or you may have it in a bad/inconsistent state.
> > >
> > > Thanks.
> > >
> > > Jerry
> > >
> > >
> > > On Sun, Apr 9, 2017 at 7:14 PM, 张铎(Duo Zhang) 
> > > wrote:
> > >
> > > > I think this depends on how you model the problem. At server side, if
> > you
> > > > re-execute a read operation with a new mvcc, then you may read a
> value
> > > that
> > > > should not be visible if you use the old mvcc. If you define this as
> an
> > > > error then I think there will be conflicts.
> > > >
> > > > But at client side, there is guarantee that the request you send
> first
> > > will
> > > > be executed first. So as long as the read request does not return, I
> > > think
> > > > it is OK to read a value which is written by a write request which is
> > > sent
> > > > after the read request?
> > > >
> > > > Thanks.
> > > >
> > > > 2017-04-10 9:52 GMT+08:00 杨苏立 Yang Su Li :
> > > >
> > > > > We are only concerned about read operations here. Are you
> suggesting
> > > they
> > > > > are completely idempotent?
> > > > > Are there any read-after-write conflicts?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Sui
> > > > >
> > > > > On Sun, Apr 9, 2017 at 8:48 PM, 张铎(Duo Zhang) <
> palomino...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > It depends on how you about the rpc request. For hbase, there
> will
> > be
> > > > no
> > > > > > write conflict, but a write operation can only be finished iff
> all
> > > the
> > 

Move the 'stable' pointer for HBase in dist.apache.org to 1.3.0 ?

[Andrew Purtell]

Shall we move the 'stable' pointer for HBase in dist.apache.org to release
1.3.0 ?


-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: Is HBase RPC-Handling idempotent for reads?

[杨苏立 Yang Su Li]

On Sun, Apr 9, 2017 at 11:14 PM, Yu Li  wrote:

> Correct me if I'm wrong, but I think we should assume no other but the
> single operation when checking whether it's idempotent. Similar to the
> wikipedia
> example : "A function
> looking up a customer's name and address in a database
>  is typically idempotent, since
> this will not cause the database to change", I think all Get/MultiGet/Scan
> operations in hbase are idempotent.
>
> About "speculative rpc handling", I doubt whether it benefits in hbase.
> Normally if a request already arrives at server side but with slow
> execution, the problem might be:
> 1. The server is too busy and request get queued
> 2. The processing itself is slow due to the request pattern or some
> hardware failure
> I don't think a speculative execution of the request could help in any of
> the above cases. It's different from the speculative task execution in MR,
> there we could choose another node to execute the task while here we have
> no choice.
>


We have a different use case here. Basically we are trying to enforce
scheduling at HBase.
Consider the following scenario: both client-1 and client-2 are competing
for I/O resources.
But client-2 are also issuing a bunch of requests that do not require any
I/O resources (say, data is cached).
Since we have idle CPU/memory, we want to serve these cached requests for
client-2, but we do not want client-2 to use more than its fair share of
I/O.

Unfortunately, at the time we pick RPC call to handle, we don't know
whether an RPC would cause I/O or not.
So we think we can abort a request if it requires I/O resources that are
not allocated to it, and re-schedule it later based on our scheduling
policy.






>
> OTOH, we already have timeout mechanism to make sure server resource won't
> be wasted:
> 1. For scan
> - When a request handling timeouts, server will stop further
> processing, refer to RSRpcServices#getTimeLimit and
> ScannerContext#checkTimeLimit
> - If the client went away during processing, server will also stop
> processing, check the SimpleRpcServer#disconnectSince and
> RegionScannerImpl#nextInternal methods for more details.
>
> 2. For single Get
> - Controlled by rpc and operation timeout
>
> 3. For MultiGet
> - I think this is something we could improve. On client side we have
> timeout mechanism but on server side there seems to be no relative
> interrupt logic.
>
>
> Best Regards,
> Yu
>
> On 10 April 2017 at 11:12, Jerry He  wrote:
>
> > Again, it depends on how you abort and 'idempotent' can have different
> > definitions.
> >
> > For example, even if you are only concerned about read,
> > there are resources on the HRegion that the read touches or acquires
> > (scanner, lock, mvcc etc) that hopefully will be cleaned/releases with
> the
> > abort.
> > Or you may have it in a bad/inconsistent state.
> >
> > Thanks.
> >
> > Jerry
> >
> >
> > On Sun, Apr 9, 2017 at 7:14 PM, 张铎(Duo Zhang) 
> > wrote:
> >
> > > I think this depends on how you model the problem. At server side, if
> you
> > > re-execute a read operation with a new mvcc, then you may read a value
> > that
> > > should not be visible if you use the old mvcc. If you define this as an
> > > error then I think there will be conflicts.
> > >
> > > But at client side, there is guarantee that the request you send first
> > will
> > > be executed first. So as long as the read request does not return, I
> > think
> > > it is OK to read a value which is written by a write request which is
> > sent
> > > after the read request?
> > >
> > > Thanks.
> > >
> > > 2017-04-10 9:52 GMT+08:00 杨苏立 Yang Su Li :
> > >
> > > > We are only concerned about read operations here. Are you suggesting
> > they
> > > > are completely idempotent?
> > > > Are there any read-after-write conflicts?
> > > >
> > > > Thanks
> > > >
> > > > Sui
> > > >
> > > > On Sun, Apr 9, 2017 at 8:48 PM, 张铎(Duo Zhang)  >
> > > > wrote:
> > > >
> > > > > It depends on how you about the rpc request. For hbase, there will
> be
> > > no
> > > > > write conflict, but a write operation can only be finished iff all
> > the
> > > > > write operations with a lower mvcc number have been finished. So if
> > you
> > > > > just stop a write operation without recovering the mvcc(I do not
> know
> > > how
> > > > > to recover but I think you need to something...) then the writes
> will
> > > be
> > > > > stuck.
> > > > >
> > > > > And one more thing, for read operation you may interrupt it at any
> > > time,
> > > > > but for write operation, I do not think you can re-execute it with
> a
> > > new
> > > > > mvcc number if the WAL entry has already been flushed out. That
> > means,
> > > > the
> > > > > re-execution process will be different if you about the write
> > operation
> > > > at
> > > > > 

Failure: HBase Generate Website

[Apache Jenkins Server]

Build status: Failure

If successful, the website and docs have been generated and the site has been 
updated automatically.
If you're getting this email, it probably failed and you may need to update the 
site manually if you can't get Jenkins fixed.

If failed, see https://builds.apache.org/job/hbase_generate_website/952/console

YOU DO NOT NEED TO DO THE FOLLOWING ANYMORE! It is here for informational 
purposes and shows what the Jenkins job does to push the site.

  git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
  cd hbase-site
  wget -O- 
https://builds.apache.org/job/hbase_generate_website/952/artifact/website.patch.zip
 | funzip > 82d554e3783372cc6b05489452c815b57c06f6cd.patch
  git fetch
  git checkout -b asf-site-82d554e3783372cc6b05489452c815b57c06f6cd 
origin/asf-site
  git am --whitespace=fix 82d554e3783372cc6b05489452c815b57c06f6cd.patch
  git push origin asf-site-82d554e3783372cc6b05489452c815b57c06f6cd:asf-site
  git commit --allow-empty -m "INFRA-10751 Empty commit"
  git push origin asf-site
  git checkout asf-site
  git branch -D asf-site-82d554e3783372cc6b05489452c815b57c06f6cd

[jira] [Created] (HBASE-17898) Update dependencies

[stack (JIRA)]

stack created HBASE-17898:
-

 Summary: Update dependencies
 Key: HBASE-17898
 URL: https://issues.apache.org/jira/browse/HBASE-17898
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Balazs Meszaros


General issue to cover updating old, stale dependencies for hbase2 release. 
Lets make subissues doing each.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Is HBase RPC-Handling idempotent for reads?

[Jerry He]

Yes.  In the context to the underlying physical region or database,. read
is idempotent.


Thanks

Jerry

On Apr 9, 2017 9:15 PM, "Yu Li"  wrote:

> Correct me if I'm wrong, but I think we should assume no other but the
> single operation when checking whether it's idempotent. Similar to the
> wikipedia
> example : "A function
> looking up a customer's name and address in a database
>  is typically idempotent, since
> this will not cause the database to change", I think all Get/MultiGet/Scan
> operations in hbase are idempotent.
>
> About "speculative rpc handling", I doubt whether it benefits in hbase.
> Normally if a request already arrives at server side but with slow
> execution, the problem might be:
> 1. The server is too busy and request get queued
> 2. The processing itself is slow due to the request pattern or some
> hardware failure
> I don't think a speculative execution of the request could help in any of
> the above cases. It's different from the speculative task execution in MR,
> there we could choose another node to execute the task while here we have
> no choice.
>
> OTOH, we already have timeout mechanism to make sure server resource won't
> be wasted:
> 1. For scan
> - When a request handling timeouts, server will stop further
> processing, refer to RSRpcServices#getTimeLimit and
> ScannerContext#checkTimeLimit
> - If the client went away during processing, server will also stop
> processing, check the SimpleRpcServer#disconnectSince and
> RegionScannerImpl#nextInternal methods for more details.
>
> 2. For single Get
> - Controlled by rpc and operation timeout
>
> 3. For MultiGet
> - I think this is something we could improve. On client side we have
> timeout mechanism but on server side there seems to be no relative
> interrupt logic.
>
>
> Best Regards,
> Yu
>
> On 10 April 2017 at 11:12, Jerry He  wrote:
>
> > Again, it depends on how you abort and 'idempotent' can have different
> > definitions.
> >
> > For example, even if you are only concerned about read,
> > there are resources on the HRegion that the read touches or acquires
> > (scanner, lock, mvcc etc) that hopefully will be cleaned/releases with
> the
> > abort.
> > Or you may have it in a bad/inconsistent state.
> >
> > Thanks.
> >
> > Jerry
> >
> >
> > On Sun, Apr 9, 2017 at 7:14 PM, 张铎(Duo Zhang) 
> > wrote:
> >
> > > I think this depends on how you model the problem. At server side, if
> you
> > > re-execute a read operation with a new mvcc, then you may read a value
> > that
> > > should not be visible if you use the old mvcc. If you define this as an
> > > error then I think there will be conflicts.
> > >
> > > But at client side, there is guarantee that the request you send first
> > will
> > > be executed first. So as long as the read request does not return, I
> > think
> > > it is OK to read a value which is written by a write request which is
> > sent
> > > after the read request?
> > >
> > > Thanks.
> > >
> > > 2017-04-10 9:52 GMT+08:00 杨苏立 Yang Su Li :
> > >
> > > > We are only concerned about read operations here. Are you suggesting
> > they
> > > > are completely idempotent?
> > > > Are there any read-after-write conflicts?
> > > >
> > > > Thanks
> > > >
> > > > Sui
> > > >
> > > > On Sun, Apr 9, 2017 at 8:48 PM, 张铎(Duo Zhang)  >
> > > > wrote:
> > > >
> > > > > It depends on how you about the rpc request. For hbase, there will
> be
> > > no
> > > > > write conflict, but a write operation can only be finished iff all
> > the
> > > > > write operations with a lower mvcc number have been finished. So if
> > you
> > > > > just stop a write operation without recovering the mvcc(I do not
> know
> > > how
> > > > > to recover but I think you need to something...) then the writes
> will
> > > be
> > > > > stuck.
> > > > >
> > > > > And one more thing, for read operation you may interrupt it at any
> > > time,
> > > > > but for write operation, I do not think you can re-execute it with
> a
> > > new
> > > > > mvcc number if the WAL entry has already been flushed out. That
> > means,
> > > > the
> > > > > re-execution process will be different if you about the write
> > operation
> > > > at
> > > > > different stages.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 2017-04-10 6:47 GMT+08:00 杨苏立 Yang Su Li :
> > > > >
> > > > > > We are trying to implement speculative rpc handling for our
> > > workloads.
> > > > So
> > > > > > we want allow RPC Handler to stop executing an RPC call, put it
> > back
> > > to
> > > > > the
> > > > > > queue, and later re-execute it.
> > > > > >
> > > > > > If at time t1, we execute and RPC call half way, aborts, and put
> > the
> > > > call
> > > > > > back to the queue.
> > > > > > Then at time t2 another RPC handler picks the call and re-execute
> > it.
> >