[
https://issues.apache.org/jira/browse/PHOENIX-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Gwalani updated PHOENIX-7957:
--------------------------------------
Description:
Based on the performance analysis, the replication log file size with 128 MB
has better replay performance. With standard 31 node cluster, the performance
diff for e2e round completion time for same performance test load was better
with 128 MB. Below are the numbers, first test with 256 MB while later one with
128 MB.
{*}1. p95{*}: 23.7 -> 18 s
{*}2. avg{*}: 8.6s -> 5.7s
{*}3. max{*}: 34.5s -> 30.1s
DOD: Update the default value to 128 MB
was:
*Background*
In Replication Replay Design, when replication is in _sync_ mode, the
consistency point is currently computed as the _minimum timestamp of all files
in the_ _IN-PROGRESS directory_ (when non-empty).
*Issue*
{*}{*}Files within a round are picked randomly and moved to IN-PROGRESS. A file
with a later creation timestamp can be moved to IN-PROGRESS before an older
file from the same round still sitting in the IN directory. This causes the
consistency point to advance past data that has not yet been replayed.
*Example*
{*}{*}Round N contains:
File A — timestamp T+5
File B — timestamp T+30
RS-1 picks file B first and renames it to IN-PROGRESS. File A is still in the
IN directory. IN-PROGRESS has no files from previous rounds.
IN directory: file A (T+5) — not yet replayed
IN-PROGRESS: file B (T+30) — being replayed
Current logic yields consistency point = (T+30) − 1. The correct value should
be (T+5) − 1.
*Proposed* *Fix*
**Consistency points should align to round start times. After computing the
minimum timestamp across IN-PROGRESS files, adjust it to the start time of the
round that the minimum timestamp belongs to. This avoids listing all IN
directories while still preventing the consistency point from advancing past
unreplayed files.
> Update the default size based file rotation threashold to 128 MB
> ----------------------------------------------------------------
>
> Key: PHOENIX-7957
> URL: https://issues.apache.org/jira/browse/PHOENIX-7957
> Project: Phoenix
> Issue Type: Sub-task
> Reporter: Himanshu Gwalani
> Assignee: Himanshu Gwalani
> Priority: Major
>
> Based on the performance analysis, the replication log file size with 128 MB
> has better replay performance. With standard 31 node cluster, the performance
> diff for e2e round completion time for same performance test load was better
> with 128 MB. Below are the numbers, first test with 256 MB while later one
> with 128 MB.
> {*}1. p95{*}: 23.7 -> 18 s
> {*}2. avg{*}: 8.6s -> 5.7s
> {*}3. max{*}: 34.5s -> 30.1s
> DOD: Update the default value to 128 MB
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
