Tanuj Khurana created PHOENIX-7977:
--------------------------------------
Summary: Stage forwarded replication log files in a per-shard
staging subdirectory before atomic rename
Key: PHOENIX-7977
URL: https://issues.apache.org/jira/browse/PHOENIX-7977
Project: Phoenix
Issue Type: Sub-task
Reporter: Tanuj Khurana
Assignee: Tanuj Khurana
When a fallback (standby) cluster forwards a replication log file to the peer
cluster, ReplicationLogDiscoveryForwarder.processFile() copied the bytes
directly onto the file's final replay-eligible name <shard>/<ts>_<origin>.plog
using FileUtil.copy(..., overwrite=false).
FileUtil.copy creates the destination at that final name before close()
completes. The standby replay consumer lists the shard directory on its own
cadence, and can thereforepick up the still-open, half-written file. When it
does, it force-recovers the HDFS lease on that file (single-writer model),
which makes the forwarder's own close() fail with LeaseExpiredException.
That exception propagates out of processFile() before the STORE_AND_FORWARD →
SYNC_AND_FORWARD throughput check runs, so the HA group's transition back
toward SYNC is delayed. The source file also remains in the out-progress
directory and is retried, but each retry can re-lose the same race.
Fix
Publish forwarded files atomically instead of exposing them mid-copy:
1. Add ReplicationShardDirectoryManager.STAGING_SUB_DIRECTORY_NAME = ".staging"
and getStagingPath(finalPath), which returns <shard>/.staging/<file-name>. The
staged file keeps its real .plog name but lives one level down in a
subdirectory.
2. The forwarder copies bytes into the staging path (overwrite=true, to reclaim
any orphan left by a prior crashed attempt), then does a same-shard,
same-FileSystem rename(staging, dst) to publish once the file is fully written.
3. On rename returning false: if dst already exists (a retry raced ahead of
replay) delete the staging copy and treat it as delivered; otherwise throw so
the source is retried. Delivery stays at-least-once, which is safe because
replay is idempotent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
