[
https://issues.apache.org/jira/browse/HBASE-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault updated HBASE-26874:
--------------------------------------
Description:
VerifyReplication includes an option "sleepMsBeforeReCompare". This is useful
for helping work around replication lag. However, adding a sleep in a hadoop
job can drastically slow that job down if there is anything more than a small
number of invalid results.
We can mitigate this by doing the recompare in a separate thread. We can limit
the thread pool and fallback to doing the recompare in the main thread if the
thread pool is full. This way we offload some of the slowness but still retain
the same validation guarantees. A configuration can be added to control how
many threads per mapper.
was:
VerifyReplication includes an option "sleepMsBeforeReCompare". This is useful
for helping work around replication lag. However, adding a sleep in a hadoop
job can drastically slow that job down if there is anything more than a small
number of invalid results.
We can mitigate this by doing the recompare in a separate thread. We can limit
the thread pool and fallback to doing the recompare in the main thread if the
thread pool is full. This way we offload some of the slowness but still retain
the same validation guarantees.
> VerifyReplication recompare async
> ---------------------------------
>
> Key: HBASE-26874
> URL: https://issues.apache.org/jira/browse/HBASE-26874
> Project: HBase
> Issue Type: Improvement
> Reporter: Bryan Beaudreault
> Priority: Major
>
> VerifyReplication includes an option "sleepMsBeforeReCompare". This is useful
> for helping work around replication lag. However, adding a sleep in a hadoop
> job can drastically slow that job down if there is anything more than a small
> number of invalid results.
> We can mitigate this by doing the recompare in a separate thread. We can
> limit the thread pool and fallback to doing the recompare in the main thread
> if the thread pool is full. This way we offload some of the slowness but
> still retain the same validation guarantees. A configuration can be added to
> control how many threads per mapper.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
