[jira] [Updated] (PHOENIX-3525) Cap automatic index rebuilding to inactive timestamp
[ https://issues.apache.org/jira/browse/PHOENIX-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-3525: -- Labels: secondary_index (was: ) > Cap automatic index rebuilding to inactive timestamp > > > Key: PHOENIX-3525 > URL: https://issues.apache.org/jira/browse/PHOENIX-3525 > Project: Phoenix > Issue Type: Improvement >Reporter: Ankit Singhal >Assignee: James Taylor > Labels: secondary_index > Fix For: 4.12.0, 4.11.1 > > Attachments: PHOENIX-3525_v1.patch, PHOENIX-3525_wip.patch, > PHOENIX-3525_wip2.patch > > > From [~chrajeshbab...@gmail.com] review comment on > https://github.com/apache/phoenix/pull/210 > For automatic rebuilding ,DISABLED_TIMESTAMP is lower bound but there is no > upper bound so we are going rebuild all the new writes written after > DISABLED_TIMESTAMP even though indexes updated properly. So we can introduce > an upper bound of time where we are going to start a rebuild thread so we can > limit the data to rebuild. In case If there are frequent writes then we can > increment the rebuild period exponentially -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PHOENIX-3525) Cap automatic index rebuilding to inactive timestamp
[ https://issues.apache.org/jira/browse/PHOENIX-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-3525: -- Attachment: PHOENIX-3525_v1.patch Simplified patch (doesn't change any client APIs): - delays rebuild until a configurable period of time after setting index status to INACTIVE - detects whether another write failure occurred while rebuilder is running (through checkAndPut and passing expected INDEX_DISABLE_TIMESTAMP). In this case, it will run again rather than activate it. - sets upper time range on query so that writes can continue on data table without impacting rebuilder. We can improve the handling of the case of leaving the index active in a follow up patch. Please review [~samarthjain]. > Cap automatic index rebuilding to inactive timestamp > > > Key: PHOENIX-3525 > URL: https://issues.apache.org/jira/browse/PHOENIX-3525 > Project: Phoenix > Issue Type: Improvement >Reporter: Ankit Singhal >Assignee: James Taylor > Fix For: 4.12.0, 4.11.1 > > Attachments: PHOENIX-3525_v1.patch, PHOENIX-3525_wip2.patch, > PHOENIX-3525_wip.patch > > > From [~chrajeshbab...@gmail.com] review comment on > https://github.com/apache/phoenix/pull/210 > For automatic rebuilding ,DISABLED_TIMESTAMP is lower bound but there is no > upper bound so we are going rebuild all the new writes written after > DISABLED_TIMESTAMP even though indexes updated properly. So we can introduce > an upper bound of time where we are going to start a rebuild thread so we can > limit the data to rebuild. In case If there are frequent writes then we can > increment the rebuild period exponentially -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PHOENIX-3525) Cap automatic index rebuilding to inactive timestamp
[ https://issues.apache.org/jira/browse/PHOENIX-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-3525: -- Summary: Cap automatic index rebuilding to inactive timestamp (was: Cap automatic index rebuilding to inactive timestamp.) > Cap automatic index rebuilding to inactive timestamp > > > Key: PHOENIX-3525 > URL: https://issues.apache.org/jira/browse/PHOENIX-3525 > Project: Phoenix > Issue Type: Improvement >Reporter: Ankit Singhal >Assignee: James Taylor > Attachments: PHOENIX-3525_wip2.patch, PHOENIX-3525_wip.patch > > > From [~chrajeshbab...@gmail.com] review comment on > https://github.com/apache/phoenix/pull/210 > For automatic rebuilding ,DISABLED_TIMESTAMP is lower bound but there is no > upper bound so we are going rebuild all the new writes written after > DISABLED_TIMESTAMP even though indexes updated properly. So we can introduce > an upper bound of time where we are going to start a rebuild thread so we can > limit the data to rebuild. In case If there are frequent writes then we can > increment the rebuild period exponentially -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PHOENIX-3525) Cap automatic index rebuilding to inactive timestamp.
[
https://issues.apache.org/jira/browse/PHOENIX-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samarth Jain updated PHOENIX-3525:
--
Attachment: PHOENIX-3525_wip2.patch
Updated patch.
I am not sure if using the index disable timestamp + 1 as the upper bound is
the correct thing to do. The index disable timestamp is the minimum timestamp
from all the mutations that failed. What we need is the max timestamp from the
failed mutations to determine the the upper bound of rebuild scan.
Instead I am now setting the upper bound to the current time every time
updateIndexState is called with the index disable timestamp set in the request.
On the MetadataRegionObserver side, before kicking off a rebuild, I am now
piggy backing on updateIndexState to return the upper bound timestamp.
For batched rebuild, I am computing the scan end time using the following:
{code}
long batchTime =
getTimestampForBatch(timeStamp,
batchExecutedPerTableMap.get(dataPTable.getName()));
long scanEndTime =
rebuildScanUpperBound != null
? Math.min(batchTime,
rebuildScanUpperBound)
: batchTime;
{code}
I am also making sure that the scan end time + delta is still greater than the
current time and adjusting if necessary.
{code}
/*
* If it has been too long since we got the
rebuildScanUpperBound from
* the server then we set our actual scan end time
to the current time.
*/
long actualScanEndTime =
System.currentTimeMillis() >
rebuildScanUpperBoundDelta
+ scanEndTime ?
System.currentTimeMillis()
:
rebuildScanUpperBoundDelta + scanEndTime;
{code}
For now, the delta has been configured to a default value of 1 min. We can
change this value if we expect the clock skew between region servers to be
larger.
> Cap automatic index rebuilding to inactive timestamp.
> -
>
> Key: PHOENIX-3525
> URL: https://issues.apache.org/jira/browse/PHOENIX-3525
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Ankit Singhal
> Attachments: PHOENIX-3525_wip2.patch, PHOENIX-3525_wip.patch
>
>
> From [~chrajeshbab...@gmail.com] review comment on
> https://github.com/apache/phoenix/pull/210
> For automatic rebuilding ,DISABLED_TIMESTAMP is lower bound but there is no
> upper bound so we are going rebuild all the new writes written after
> DISABLED_TIMESTAMP even though indexes updated properly. So we can introduce
> an upper bound of time where we are going to start a rebuild thread so we can
> limit the data to rebuild. In case If there are frequent writes then we can
> increment the rebuild period exponentially
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (PHOENIX-3525) Cap automatic index rebuilding to inactive timestamp.
[
https://issues.apache.org/jira/browse/PHOENIX-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samarth Jain updated PHOENIX-3525:
--
Attachment: PHOENIX-3525_wip.patch
Work in progress patch. [~jamestaylor], please review.
Summary of changes:
- I changed the lock acquisition strategy to wait for the lock in
updateIndexState(). This is important I think and following is my reasoning:
when index is getting disabled then it is possible that multiple region servers
are trying to update the index state. In that case, only one region server will
be able to acquire the lock. If we don't wait for the lock then the race losing
region servers will get an exception and they will end up aborting the region
server. This could cause cluster wide region server aborts if for some reason
the code being executed under the lock takes a long time.
- In MetadataEndPointImpl I am setting the upper bound of the scan *only* when
the index state is being switched from disable to inactive. I am setting the
upper bound timestamp either when it is not set or when the index disable
timestamp is newer than the existing upper bound. As far was protocol changes
are concerned I think we are fine backward compatibility wise since it is all
proto-bufed.
- I am a little confused about how to handle the scanEndTime when we are check
pointing. For now, I am always using the batch timestamp returned by
{code}
getTimestampForBatch(timeStamp,
batchExecutedPerTableMap.get(dataPTable.getName()));
{code}
as the scan end time that timestamp is HConstants.LATEST_TIMESTAMP in which
case I am using the timestamp returned by updateIndexState call.
> Cap automatic index rebuilding to inactive timestamp.
> -
>
> Key: PHOENIX-3525
> URL: https://issues.apache.org/jira/browse/PHOENIX-3525
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Ankit Singhal
> Attachments: PHOENIX-3525_wip.patch
>
>
> From [~chrajeshbab...@gmail.com] review comment on
> https://github.com/apache/phoenix/pull/210
> For automatic rebuilding ,DISABLED_TIMESTAMP is lower bound but there is no
> upper bound so we are going rebuild all the new writes written after
> DISABLED_TIMESTAMP even though indexes updated properly. So we can introduce
> an upper bound of time where we are going to start a rebuild thread so we can
> limit the data to rebuild. In case If there are frequent writes then we can
> increment the rebuild period exponentially
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
