Haonan Hou created RATIS-2681:
---------------------------------
Summary: Add a pluggable listener for gRPC Raft peer data transfer
outcomes
Key: RATIS-2681
URL: https://issues.apache.org/jira/browse/RATIS-2681
Project: Ratis
Issue Type: Improvement
Components: gRPC
Reporter: Haonan Hou
h3. Motivation
Applications embedding Ratis may need to generate audit records for user data
transmitted between Ratis peers that are deployed as physically separated
parts of a distributed TOE.
In Common Criteria terminology, this is relevant to FDP_ITT.1 Basic internal
transfer protection together with FAU_GEN.1 Security audit data generation.
CC:2022 Part 2, section 11.10.5 describes the following audit levels for
FDP_ITT.1:
* minimal: successful user data transfers and the protection method used;
* basic: all user data transfer attempts, the protection method used, and any
errors that occurred.
Reference:
https://www.commoncriteriaportal.org/files/ccfiles/CC2022PART2R1.pdf
Ratis itself does not need to implement application-specific audit logging or
claim Common Criteria conformance. It only needs to expose enough transfer
information for an embedding application to generate its own audit events.
h3. Current limitations
Ratis currently exposes metrics and log messages for AppendEntries and
InstallSnapshot operations, but it does not provide a stable programmatic
callback covering both successful and failed data transfer attempts between
Raft peers.
RATIS-2638 adds a callback for TLS handshake failures. That callback operates
at the physical connection and TLS negotiation layer and cannot report:
* successful AppendEntries or InstallSnapshot transfers;
* transfers using a non-TLS connection;
* request timeouts;
* errors occurring after the TLS handshake;
* the outcome of a particular data-bearing Ratis request.
Application code cannot reliably reconstruct these events from Ratis log
messages or metrics.
h3. Proposed API
Ratis could provide an optional Consumer for gRPC Raft peer data transfer
events.
For example:
{code:java}
Consumer<GrpcDataTransferEvent> dataTransferEventConsumer;
{code}
The Consumer could be registered through Parameters or GrpcConfigKeys,
following an approach similar to the ServerCredentials extension introduced
by RATIS-2638.
A possible event shape is:
{code:java}
public final class GrpcDataTransferEvent {
public Instant getTimestamp();
public RaftPeerId getSource();
public RaftPeerId getDestination();
public ProtectionMethod getProtectionMethod();
public Result getResult();
public Throwable getError();
}
public enum ProtectionMethod {
TLS,
NONE
}
public enum Result {
SUCCESS,
FAILURE
}
{code}
The exact API shape and names are open for discussion.
The dedicated callback implicitly represents a user data transfer event. An
embedding application may map it to the following minimal audit record:
{noformat}
timestamp
event_type = USER_DATA_TRANSFER
subject
source_node
destination_node
protection_method = TLS | NONE
result = success | failure
error
{noformat}
The event type and subject are application-level values. For an outbound peer
transfer, the application may use the local Ratis service or local Raft peer as
the subject.
No request payload, log contents, credentials, term/index, correlation ID,
byte count, or other application data needs to be exposed.
h3. Initial scope
The initial implementation may cover the following outbound gRPC transfers
between Ratis peers:
* AppendEntries requests containing state-machine log entries;
* InstallSnapshot transfers containing snapshot data.
The following requests should not generate data transfer events:
* heartbeat-only AppendEntries requests;
* AppendEntries requests that do not contain state-machine data;
* notification-only InstallSnapshot requests.
Ratis cannot determine the complete application-level classification of user
data. The embedding application remains responsible for deciding whether the
reported transfer is within its audit scope.
h3. Event granularity
For AppendEntries, one event should represent one outbound data-bearing
AppendEntries request attempt.
For InstallSnapshot, one event should represent the complete logical snapshot
transfer rather than each individual snapshot chunk.
Each retry is a separate transfer attempt and may therefore generate a
separate event.
Events should be generated only on the sending peer. This avoids duplicate
audit records from both the source and destination peers.
h3. Result semantics
The result represents the outcome of the data transfer, rather than the
application or Raft state transition performed after the data was received.
For AppendEntries:
* SUCCESS means that a valid AppendEntries response was received for the
corresponding request;
* FAILURE means that the transfer ended because of a local send error,
connection failure, timeout, or response-stream error.
A valid AppendEntries response confirms that the peer communication completed.
Raft protocol results such as INCONSISTENCY or NOT_LEADER should not by
themselves be classified as data transfer failures.
For InstallSnapshot:
* SUCCESS means that the logical snapshot transfer completed and all expected
snapshot responses were received;
* FAILURE means that the logical snapshot transfer ended because of a local
send error, connection failure, timeout, response-stream error, or an
unsuccessful terminal snapshot response.
Intermediate snapshot responses must not generate duplicate success events.
h3. Required semantics
The listener should have the following behavior:
* Generate one terminal event for each data transfer attempt.
* Generate events on the sending peer only.
* Treat every retry as a separate transfer attempt.
* Report the source and destination Raft peers.
* Report TLS when the outbound peer connection uses TLS.
* Report NONE when the outbound peer connection does not use TLS.
* Preserve the best available Throwable for a failed transfer.
* Leave the error empty for a successful transfer.
* Exclude heartbeat-only and other non-data-bearing requests.
* When a stream failure causes pending requests to be discarded, report every
affected data-bearing request as failed exactly once.
* An exception thrown by the Consumer must not affect replication, retries,
leader state, pending requests, or channel lifecycle.
* The Consumer execution context should be documented.
* If invoked on a Ratis or gRPC internal thread, the Consumer must perform only
non-blocking work, such as enqueueing the event for another thread.
* Existing behavior must remain unchanged when no Consumer is configured.
The TLS handshake failure callback introduced by RATIS-2638 remains a separate
transport-level event. It reports failure to establish a TLS channel, while
the proposed listener reports the outcome of an actual data transfer attempt
between Raft peers.
If a data transfer attempt fails because its outbound connection cannot
complete the TLS handshake, the sender-side transfer listener may report that
attempt as failed using the best error available from the gRPC request path.
This does not replace the connection-level event provided by RATIS-2638.
h3. Implementation considerations
For AppendEntries, the relevant sender-side completion paths are in
GrpcLogAppender:
* AppendLogResponseHandler.onNext;
* timeoutAppendRequest;
* AppendLogResponseHandler.onError;
* synchronous exceptions while sending a request.
A request should be marked as data-bearing when it contains at least one
state-machine log entry. This information may be stored together with the
pending AppendEntries request without retaining or copying its payload.
The event should be emitted when the request reaches a terminal state.
The current response-stream error path may clear multiple pending requests.
Before those requests are discarded, every affected data-bearing request
should be reported as failed exactly once.
For InstallSnapshot, the relevant paths include:
* completion after all expected snapshot responses are received;
* InstallSnapshotResponseHandler.onError;
* timeout handling;
* synchronous exceptions while creating or writing the request stream.
The snapshot transfer should have a single terminal state so that an error
cannot be followed by a duplicate success or failure event.
The protection method should be derived from the actual outbound gRPC peer
connection configuration.
Ratis should only invoke the configured Consumer. Serialization, queueing,
filtering, persistence, and audit log formatting remain responsibilities of
the embedding application.
h3. Compatibility and performance
The feature should be opt-in.
When no Consumer is configured:
* existing APIs and constructors should remain compatible;
* AppendEntries and InstallSnapshot behavior should remain unchanged;
* no audit-specific dependency should be introduced;
* no request or snapshot payload should be copied;
* no additional application-visible event should be generated;
* the normal replication path should have negligible additional overhead.
The configured Consumer is expected to be non-blocking. Applications requiring
persistent audit records should enqueue the event and perform serialization
and I/O asynchronously.
Listener failures should be isolated in the same way as the listener added for
RATIS-2638: an exception thrown by application code must not replace the
original transfer result or interfere with Ratis processing.
h3. Suggested tests
Tests should cover:
* successful data-bearing AppendEntries over TLS generates one SUCCESS event;
* successful data-bearing AppendEntries without TLS generates one SUCCESS
event with protection method NONE;
* heartbeat-only AppendEntries does not generate an event;
* AppendEntries without state-machine data does not generate an event;
* AppendEntries timeout generates one FAILURE event;
* a synchronous send exception generates one FAILURE event;
* a response-stream failure reports every affected pending data-bearing
request exactly once;
* a valid INCONSISTENCY response is treated as a completed transfer;
* a valid NOT_LEADER response is treated as a completed transfer;
* successful InstallSnapshot generates one SUCCESS event for the complete
logical snapshot transfer;
* failed InstallSnapshot generates one FAILURE event;
* intermediate snapshot chunk responses do not generate duplicate events;
* notification-only InstallSnapshot does not generate an event;
* TLS and NONE protection methods are reported correctly;
* an exception thrown by the Consumer does not affect replication;
* no event is generated when no Consumer is configured.
I can contribute an implementation and tests after confirming the preferred
API and event granularity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
