[
https://issues.apache.org/jira/browse/RATIS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haonan Hou updated RATIS-2581:
------------------------------
Description:
Ratis currently removes local storage files using normal filesystem operations
such as Files.delete/deleteIfExists and FileChannel.truncate. This is fine for
the default behavior, but some downstream projects embedding Ratis need to
integrate with their own compliant file destruction mechanism.
For example, a downstream system may need to destroy Raft log segments,
snapshots, metadata temp files, and temporary snapshot installation files using
a custom secure-delete / crypto-erase / audited destruction implementation
instead of plain filesystem unlink/truncate.
This request proposes adding a pluggable file destruction abstraction in Ratis,
with the existing behavior as the default implementation.
*Current Behavior*
Ratis core cleanup paths eventually use normal delete/truncate operations.
Examples include:
- FileUtils.delete/deleteIfExists/deleteFully
- FileUtils.truncateFile
- raft log purge and truncate paths
- snapshot installation cleanup
- atomic metadata temp file cleanup
- group remove storage directory cleanup
There are also a few direct calls such as Files.delete,
File.delete/deleteOnExit,
or FileChannel.truncate that can bypass FileUtils.
*Proposed Change*
Introduce a small pluggable file destruction API, for example:
{code:java}
interface RatisFileDestroyer {
void delete(Path path) throws IOException;
void deleteIfExists(Path path) throws IOException;
void truncate(Path path, long targetLength) throws IOException;
} {code}
The default implementation should preserve the current behavior:
- Files.delete
- Files.deleteIfExists
- FileChannel.truncate
Ratis should route its core local storage deletion/truncation paths through this
abstraction, including:
- recursive storage directory cleanup
- Raft log segment purge
- Raft log truncation
- snapshot temp file cleanup
- snapshot installation replacement cleanup
- atomic output temp file abort cleanup
- zero-size open log segment cleanup
- delete-on-exit paths if possible
The API should be configurable by embedders, e.g. through RaftProperties,
RaftServer builder configuration, ServiceLoader, or another Ratis-approved
extension mechanism.
*Compatibility*
The default behavior remains unchanged.
Existing users should not see behavior changes unless they explicitly configure
a custom file destroyer.
was:
Ratis currently removes local storage files using normal filesystem operations
such as Files.delete/deleteIfExists and FileChannel.truncate. This is fine for
the default behavior, but some downstream projects embedding Ratis need to
integrate with their own compliant file destruction mechanism.
For example, a downstream system may need to destroy Raft log segments,
snapshots, metadata temp files, and temporary snapshot installation files using
a custom secure-delete / crypto-erase / audited destruction implementation
instead of plain filesystem unlink/truncate.
This request proposes adding a pluggable file destruction abstraction in Ratis,
with the existing behavior as the default implementation.
*Current Behavior*
Ratis core cleanup paths eventually use normal delete/truncate operations.
Examples include:
- FileUtils.delete/deleteIfExists/deleteFully
- FileUtils.truncateFile
- raft log purge and truncate paths
- snapshot installation cleanup
- atomic metadata temp file cleanup
- group remove storage directory cleanup
There are also a few direct calls such as Files.delete,
File.delete/deleteOnExit,
or FileChannel.truncate that can bypass FileUtils.
*Proposed Change*
Introduce a small pluggable file destruction API, for example:
interface RatisFileDestroyer {
void delete(Path path) throws IOException;
void deleteIfExists(Path path) throws IOException;
void truncate(Path path, long targetLength) throws IOException;
}
The default implementation should preserve the current behavior:
- Files.delete
- Files.deleteIfExists
- FileChannel.truncate
Ratis should route its core local storage deletion/truncation paths through this
abstraction, including:
- recursive storage directory cleanup
- Raft log segment purge
- Raft log truncation
- snapshot temp file cleanup
- snapshot installation replacement cleanup
- atomic output temp file abort cleanup
- zero-size open log segment cleanup
- delete-on-exit paths if possible
The API should be configurable by embedders, e.g. through RaftProperties,
RaftServer builder configuration, ServiceLoader, or another Ratis-approved
extension mechanism.
*Compatibility*
The default behavior remains unchanged.
Existing users should not see behavior changes unless they explicitly configure
a custom file destroyer.
> Pluggable file destruction hook for Raft log, snapshot, and storage cleanup
> ---------------------------------------------------------------------------
>
> Key: RATIS-2581
> URL: https://issues.apache.org/jira/browse/RATIS-2581
> Project: Ratis
> Issue Type: Improvement
> Components: RaftLog, snapshot
> Reporter: Haonan Hou
> Priority: Major
>
> Ratis currently removes local storage files using normal filesystem operations
> such as Files.delete/deleteIfExists and FileChannel.truncate. This is fine for
> the default behavior, but some downstream projects embedding Ratis need to
> integrate with their own compliant file destruction mechanism.
> For example, a downstream system may need to destroy Raft log segments,
> snapshots, metadata temp files, and temporary snapshot installation files
> using
> a custom secure-delete / crypto-erase / audited destruction implementation
> instead of plain filesystem unlink/truncate.
> This request proposes adding a pluggable file destruction abstraction in
> Ratis,
> with the existing behavior as the default implementation.
> *Current Behavior*
> Ratis core cleanup paths eventually use normal delete/truncate operations.
> Examples include:
> - FileUtils.delete/deleteIfExists/deleteFully
> - FileUtils.truncateFile
> - raft log purge and truncate paths
> - snapshot installation cleanup
> - atomic metadata temp file cleanup
> - group remove storage directory cleanup
> There are also a few direct calls such as Files.delete,
> File.delete/deleteOnExit,
> or FileChannel.truncate that can bypass FileUtils.
> *Proposed Change*
> Introduce a small pluggable file destruction API, for example:
> {code:java}
> interface RatisFileDestroyer {
> void delete(Path path) throws IOException;
> void deleteIfExists(Path path) throws IOException;
> void truncate(Path path, long targetLength) throws IOException;
> } {code}
> The default implementation should preserve the current behavior:
> - Files.delete
> - Files.deleteIfExists
> - FileChannel.truncate
> Ratis should route its core local storage deletion/truncation paths through
> this
> abstraction, including:
> - recursive storage directory cleanup
> - Raft log segment purge
> - Raft log truncation
> - snapshot temp file cleanup
> - snapshot installation replacement cleanup
> - atomic output temp file abort cleanup
> - zero-size open log segment cleanup
> - delete-on-exit paths if possible
> The API should be configurable by embedders, e.g. through RaftProperties,
> RaftServer builder configuration, ServiceLoader, or another Ratis-approved
> extension mechanism.
> *Compatibility*
> The default behavior remains unchanged.
> Existing users should not see behavior changes unless they explicitly
> configure
> a custom file destroyer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
