Manish Bhatt created HADOOP-19497:
-------------------------------------
Summary: Enable rename and create recovery from client transaction
id over DFS endpoint
Key: HADOOP-19497
URL: https://issues.apache.org/jira/browse/HADOOP-19497
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure
Affects Versions: 3.5.0
Reporter: Manish Bhatt
Assignee: Manish Bhatt
We have implemented create and rename recovery using client transaction IDs
over the DFS endpoint ([HADOOP-19450] [ABFS] Rename/Create path idempotency
client-level resolution - ASF JIRA). Since the backend changes were not fully
rolled out, we initially implemented the changes with the flag disabled. With
this update, we aim to enable the flag, which will start sending client
transaction IDs. In case of a failure, we will attempt to recover from the
failed state if possible. Here are the detailed steps and considerations for
this process:
1. **Implementation Overview**: We introduced a mechanism for create and rename
recovery via client transaction IDs to enhance reliability and data integrity
over the DFS endpoint. This change was initially flagged as disabled due to
incomplete backend rollouts.
2. **Current Update**: With the backend changes now in place, we are ready to
enable the flag. This will activate the sending of client transaction IDs,
allowing us to track and manage transactions more effectively.
3. **Failure Recovery**: The primary advantage of enabling this flag is the
potential for recovery from failed states. If a transaction fails, we can use
the client transaction ID to attempt a recovery, minimizing data loss and
ensuring continuity.
4. **Next Steps**: We will proceed with enabling the flag and closely monitor
the system's performance. Any issues or failures will be documented and
addressed promptly to ensure a smooth transition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
