[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095185#comment-16095185 ] Ivan Bella commented on ACCUMULO-4643: -- I was working to ensure that the merge from 1.8 into master was clean and did not undo anything. I will close things up now. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 18h 20m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095167#comment-16095167 ] Christopher Tubbs commented on ACCUMULO-4643: - Can this issue be closed, now that it has been merged into master? Is there more work to be done? > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 18h > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045563#comment-16045563 ] Ivan Bella commented on ACCUMULO-4643: -- [~kturner] It is interesting in test 1 that the standard deviation is so high. With 128 threads and 127 scans, I would have expected all of the scans to get a fair amount of processing time. I am wondering if the max open files is causing issues there. That might explain why the mean and standard deviations is lower in test 2 since the files get closed when one scan yields. These types of results I would expect when the number of tserver threads is smaller than the number of long running scans and would be exactly what I would want to see. I am not sure what to do with test 3. I am guessing there is a test 4 coming which is comparable to test 3 but with yielding. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 17h 10m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[
https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044884#comment-16044884
]
Keith Turner commented on ACCUMULO-4643:
I created a [test suite|https://github.com/keith-turner/yield-test] and ran the
following test.
Test 1 : 128 tserver threads, 127 non-yielding long running scans, 1 thread
doing 50 batch scans of 5 random ranges. Below are the times in ms for the
batch scans. There were 5,000,000 key/values. Each long runnning scan was
reading all. The total time for all long running scans was : 104 seconds
{noformat}
min: 30.0
max: 2395.0
mean: 882.35999
standard deviation: 653.56063254159
{noformat}
Test 2 : 128 tserver threads, 127 yielding long running scans, 1 thread doing
50 batch scans of 5 random ranges. The long running scans yielded every
10,000. Below are the times in ms for the batch scans. Total time for all
long running scans was : 215 seconds
{noformat}
min: 428.0
max: 827.0
mean: 626.28002
standard deviation: 104.54157263100298
{noformat}
Test 3 : 16 tserver threads, 20 long running scans, 1 thread doing 50 batch
scans of 5 random ranges. Below are the times in ms for the batch scans.
There were 31,750,000 key/values. Each long runnning scan was reading all.
The total time for all long running scans was : 174 seconds
{noformat}
min: 120.0
max: 548.0
mean: 210.540002
standard deviation: 70.64345948031524
{noformat}
> Allow iterators to interrupt themselves
> ---
>
> Key: ACCUMULO-4643
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4643
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
>Affects Versions: 1.8.1, 2.0.0
>Reporter: Ivan Bella
>Assignee: Ivan Bella
> Labels: features
> Fix For: 2.0.0
>
> Time Spent: 16h 50m
> Remaining Estimate: 0h
>
> The idea here is to allow an iterator stack to send back a special key or
> throw a special exception which will allow the tablet server to tear down the
> scan to be rebuilt later. This is to handle the case where an iterator is
> doing a lot of work without returning results to avoid starving out other
> scans.
> There are two thoughts on how to do this:
> 1) A special "interrupt" key is returned from the getTopKey call that is
> detected in the Tablet.nextBatch call, is not added to the results, but is
> used to add an unfinished range and results in the remaining ranges to be
> deemed unfinished.
> 2) An special exception is thrown from the next or seek call that included
> the key of the current position, and the same actions are taken as in 1).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039470#comment-16039470 ] Christopher Tubbs commented on ACCUMULO-4643: - Regarding my above comment about 1.8, it might be better to go ahead and include 1.8, because otherwise anybody needing this feature would have to wait until 2.0.0, or we'd have to create a 1.9.0. I think both of those options might be worse, because 1.9.0 would likely add a new maintenance branch to support, without any consensus to obsolete previous maintenance branches. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 1.8.2, 2.0.0 > > Time Spent: 16h 50m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037191#comment-16037191 ] Keith Turner commented on ACCUMULO-4643: Based on comments [~phrocker] made on the PR, I think it would be nice to create a simple test that contrast the responsiveness and throughput of this approach with lots of threads. Unless someone else wants to take a crack at it, I am thinking of creating a little test that does the following. * Write lots of data to Accumulo. * Create X long running scans that filter alot and return little. * Create a few clients that constantly do random lookups. Measure the time of these. The same test would be run with the following. * X threads in tserver and a non-yielding filter. * X/Y where Y > 1 threads and a yielding filter. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 1.8.2, 2.0.0 > > Time Spent: 15h 20m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036179#comment-16036179 ] Christopher Tubbs commented on ACCUMULO-4643: - Adding this to 1.8 would probably break the spirit of semver. I don't think such a change would be expected in a patch release, because it would break the forwards and backwards compatibility that we strive for within patch releases for a given minor version. It might be better to leave a backported 1.8-compatible patch attached to the issue, in case users want to patch their own 1.8 versions, rather than release it as part of an official bugfix release in 1.8. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 1.8.2, 2.0.0 > > Time Spent: 13.5h > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034793#comment-16034793 ] Ivan Bella commented on ACCUMULO-4643: -- I added a second pull request which back ports this to 1.8 > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 1.8.2, 2.0.0 > > Time Spent: 10.5h > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032011#comment-16032011 ] Ivan Bella commented on ACCUMULO-4643: -- Changed the API to be enableYielding(YieldCallback) instead of separate methods on the SKVI > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 7h > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16031804#comment-16031804 ] Ivan Bella commented on ACCUMULO-4643: -- I reworked the pull request to use interface methods instead of throwing exceptions. This also ensures that the yielding mechanism will not interrupt when using row isolation mode. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16031271#comment-16031271 ] Keith Turner commented on ACCUMULO-4643: Another interesting thing that came out of the conversation was a realization about isolation. Lack of isolation (allowing data source to change) between a yield and continue is not as big of a deal if yielding is stateless. If this yield functionality were designed to be stateful, then isolation would be more important. If yield does lack isolation in the final cut, we will need to clearly document that. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030078#comment-16030078 ] Ivan Bella commented on ACCUMULO-4643: -- I will change the implementation as such in the next few days > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030077#comment-16030077 ] Ivan Bella commented on ACCUMULO-4643: -- After a lengthy discussion with [~kturner], we came to the conclusion that if this mechanism is put in place then iterators that implement a yielding mechanism can only be called from iterators that are aware of that fact. Hence a separate interface (or perhaps changing SKVI) that has methods something like the following would be better: // Called to tell an iterator that yielding is supported. public void setYieldSupported(); // Called after every next and seek call to determine it has yielded public boolean hasYielded(); // Called after hasYielded returns true to determine the key to re-seek to later public Key getYieldPosition(); > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029926#comment-16029926 ] Keith Turner commented on ACCUMULO-4643: For option 3, maybe java default methods could be leveraged to add new methods to SKVI. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[
https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029913#comment-16029913
]
Keith Turner commented on ACCUMULO-4643:
{quote}
The behaviour is expected to be identical to when an iterator is torn down
because the buffer is filled, or the timeout is reached.
{quote}
There is currently isolation of data sources when computing a single key/value.
If using this feature, the computation of single key/value could be done using
different data sources. That's a big difference.
> Allow iterators to interrupt themselves
> ---
>
> Key: ACCUMULO-4643
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4643
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
>Affects Versions: 1.8.1, 2.0.0
>Reporter: Ivan Bella
>Assignee: Ivan Bella
> Labels: features
> Fix For: 2.0.0
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> The idea here is to allow an iterator stack to send back a special key or
> throw a special exception which will allow the tablet server to tear down the
> scan to be rebuilt later. This is to handle the case where an iterator is
> doing a lot of work without returning results to avoid starving out other
> scans.
> There are two thoughts on how to do this:
> 1) A special "interrupt" key is returned from the getTopKey call that is
> detected in the Tablet.nextBatch call, is not added to the results, but is
> used to add an unfinished range and results in the remaining ranges to be
> deemed unfinished.
> 2) An special exception is thrown from the next or seek call that included
> the key of the current position, and the same actions are taken as in 1).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029882#comment-16029882 ] Ivan Bella commented on ACCUMULO-4643: -- [~kturner] The behaviour is expected to be identical to when an iterator is torn down because the buffer is filled, or the timeout is reached. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029864#comment-16029864 ] Keith Turner commented on ACCUMULO-4643: Whats the expected/desired behavior of this feature w.r.t. isolation? When an iterator yields and is later continued, will it be continued with the exact same data sources? Or could it be restarted with a different set of data sources (in which a compaction could have occurred or new data was added)? > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029865#comment-16029865 ] Ivan Bella commented on ACCUMULO-4643: -- [~kturner] You are correct in that whatever key gets sent back with the exception (or the "interrupt" key) would have to be transformed. In fact it would need contain enough information such that the subsequent seek invocation (using that key as the start of the range, non-inclusive) will allow the seek to continue where it left off. Option #2 seemed simpler to implement initially. I am going to implement an option #3 where I create a interface that can be used to extend an SKVI with methods to implement this concept instead of using an exception. Option #1 seemed like a potential performance problem as we would need to check the class of the returned key on every cycle. It is thought that an iterator that yields using an exception would only be implemented with iterators that may do a lot of work for no gain and hence this would happen infrequently. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029824#comment-16029824 ] Keith Turner commented on ACCUMULO-4643: [~ivan.bella] for your PR, what made you decide on option #2 over option #1? > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ACCUMULO-4643) Allow iterators to interrupt themselves
[ https://issues.apache.org/jira/browse/ACCUMULO-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029818#comment-16029818 ] Keith Turner commented on ACCUMULO-4643: I was thinking about the use case of an iterator that does some type of transform. If not handled properly, this feature could lead to that iterator being reseeked with an untransformed key from an iterator below it. Where normally its only reseeked with transformed keys. For option #1, was the idea to add a interrupt boolean field to Key (like the delete boolean)? If so, then the iterator doing transformation would have to ensure it sets the interrupt field on the transformed key. For option 2, any iterator doing transformations should catch the exception, transform the key, and then thrown a new exception with the transformed key. > Allow iterators to interrupt themselves > --- > > Key: ACCUMULO-4643 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4643 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Affects Versions: 1.8.1, 2.0.0 >Reporter: Ivan Bella >Assignee: Ivan Bella > Labels: features > Fix For: 2.0.0 > > Time Spent: 6h > Remaining Estimate: 0h > > The idea here is to allow an iterator stack to send back a special key or > throw a special exception which will allow the tablet server to tear down the > scan to be rebuilt later. This is to handle the case where an iterator is > doing a lot of work without returning results to avoid starving out other > scans. > There are two thoughts on how to do this: > 1) A special "interrupt" key is returned from the getTopKey call that is > detected in the Tablet.nextBatch call, is not added to the results, but is > used to add an unfinished range and results in the remaining ranges to be > deemed unfinished. > 2) An special exception is thrown from the next or seek call that included > the key of the current position, and the same actions are taken as in 1). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
