[
https://issues.apache.org/jira/browse/PHOENIX-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samarth Jain updated PHOENIX-3525:
----------------------------------
Attachment: PHOENIX-3525_wip2.patch
Updated patch.
I am not sure if using the index disable timestamp + 1 as the upper bound is
the correct thing to do. The index disable timestamp is the minimum timestamp
from all the mutations that failed. What we need is the max timestamp from the
failed mutations to determine the the upper bound of rebuild scan.
Instead I am now setting the upper bound to the current time every time
updateIndexState is called with the index disable timestamp set in the request.
On the MetadataRegionObserver side, before kicking off a rebuild, I am now
piggy backing on updateIndexState to return the upper bound timestamp.
For batched rebuild, I am computing the scan end time using the following:
{code}
long batchTime =
getTimestampForBatch(timeStamp,
batchExecutedPerTableMap.get(dataPTable.getName()));
long scanEndTime =
rebuildScanUpperBound != null
? Math.min(batchTime,
rebuildScanUpperBound)
: batchTime;
{code}
I am also making sure that the scan end time + delta is still greater than the
current time and adjusting if necessary.
{code}
/*
* If it has been too long since we got the
rebuildScanUpperBound from
* the server then we set our actual scan end time
to the current time.
*/
long actualScanEndTime =
System.currentTimeMillis() >
rebuildScanUpperBoundDelta
+ scanEndTime ?
System.currentTimeMillis()
:
rebuildScanUpperBoundDelta + scanEndTime;
{code}
For now, the delta has been configured to a default value of 1 min. We can
change this value if we expect the clock skew between region servers to be
larger.
> Cap automatic index rebuilding to inactive timestamp.
> -----------------------------------------------------
>
> Key: PHOENIX-3525
> URL: https://issues.apache.org/jira/browse/PHOENIX-3525
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Ankit Singhal
> Attachments: PHOENIX-3525_wip2.patch, PHOENIX-3525_wip.patch
>
>
> From [[email protected]] review comment on
> https://github.com/apache/phoenix/pull/210
> For automatic rebuilding ,DISABLED_TIMESTAMP is lower bound but there is no
> upper bound so we are going rebuild all the new writes written after
> DISABLED_TIMESTAMP even though indexes updated properly. So we can introduce
> an upper bound of time where we are going to start a rebuild thread so we can
> limit the data to rebuild. In case If there are frequent writes then we can
> increment the rebuild period exponentially
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
