[
https://issues.apache.org/jira/browse/RATIS-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haonan Hou reassigned RATIS-2681:
---------------------------------
Assignee: Haonan Hou
> Add a pluggable listener for gRPC Raft peer data transfer outcomes
> ------------------------------------------------------------------
>
> Key: RATIS-2681
> URL: https://issues.apache.org/jira/browse/RATIS-2681
> Project: Ratis
> Issue Type: Improvement
> Components: gRPC
> Reporter: Haonan Hou
> Assignee: Haonan Hou
> Priority: Major
>
> h3. Motivation
> Applications embedding Ratis may need to generate audit records for user data
> transmitted between Ratis peers that are deployed as physically separated
> parts of a distributed TOE.
> In Common Criteria terminology, this is relevant to FDP_ITT.1 Basic internal
> transfer protection together with FAU_GEN.1 Security audit data generation.
> CC:2022 Part 2, section 11.10.5 describes the following audit levels for
> FDP_ITT.1:
> * minimal: successful user data transfers and the protection method used;
> * basic: all user data transfer attempts, the protection method used, and any
> errors that occurred.
> Reference:
> https://www.commoncriteriaportal.org/files/ccfiles/CC2022PART2R1.pdf
> Ratis itself does not need to implement application-specific audit logging or
> claim Common Criteria conformance. It only needs to expose enough transfer
> information for an embedding application to generate its own audit events.
> h3. Current limitations
> Ratis currently exposes metrics and log messages for AppendEntries and
> InstallSnapshot operations, but it does not provide a stable programmatic
> callback covering both successful and failed data transfer attempts between
> Raft peers.
> RATIS-2638 adds a callback for TLS handshake failures. That callback operates
> at the physical connection and TLS negotiation layer and cannot report:
> * successful AppendEntries or InstallSnapshot transfers;
> * transfers using a non-TLS connection;
> * request timeouts;
> * errors occurring after the TLS handshake;
> * the outcome of a particular data-bearing Ratis request.
> Application code cannot reliably reconstruct these events from Ratis log
> messages or metrics.
> h3. Proposed API
> Ratis could provide an optional Consumer for gRPC Raft peer data transfer
> events.
> For example:
> {code:java}
> Consumer<GrpcDataTransferEvent> dataTransferEventConsumer;
> {code}
> The Consumer could be registered through Parameters or GrpcConfigKeys,
> following an approach similar to the ServerCredentials extension introduced
> by RATIS-2638.
> A possible event shape is:
> {code:java}
> public final class GrpcDataTransferEvent {
> public Instant getTimestamp();
> public RaftPeerId getSource();
> public RaftPeerId getDestination();
> public ProtectionMethod getProtectionMethod();
> public Result getResult();
> public Throwable getError();
> }
> public enum ProtectionMethod {
> TLS,
> NONE
> }
> public enum Result {
> SUCCESS,
> FAILURE
> }
> {code}
> The exact API shape and names are open for discussion.
> The dedicated callback implicitly represents a user data transfer event. An
> embedding application may map it to the following minimal audit record:
> {noformat}
> timestamp
> event_type = USER_DATA_TRANSFER
> subject
> source_node
> destination_node
> protection_method = TLS | NONE
> result = success | failure
> error
> {noformat}
> The event type and subject are application-level values. For an outbound peer
> transfer, the application may use the local Ratis service or local Raft peer
> as
> the subject.
> No request payload, log contents, credentials, term/index, correlation ID,
> byte count, or other application data needs to be exposed.
> h3. Initial scope
> The initial implementation may cover the following outbound gRPC transfers
> between Ratis peers:
> * AppendEntries requests containing state-machine log entries;
> * InstallSnapshot transfers containing snapshot data.
> The following requests should not generate data transfer events:
> * heartbeat-only AppendEntries requests;
> * AppendEntries requests that do not contain state-machine data;
> * notification-only InstallSnapshot requests.
> Ratis cannot determine the complete application-level classification of user
> data. The embedding application remains responsible for deciding whether the
> reported transfer is within its audit scope.
> h3. Event granularity
> For AppendEntries, one event should represent one outbound data-bearing
> AppendEntries request attempt.
> For InstallSnapshot, one event should represent the complete logical snapshot
> transfer rather than each individual snapshot chunk.
> Each retry is a separate transfer attempt and may therefore generate a
> separate event.
> Events should be generated only on the sending peer. This avoids duplicate
> audit records from both the source and destination peers.
> h3. Result semantics
> The result represents the outcome of the data transfer, rather than the
> application or Raft state transition performed after the data was received.
> For AppendEntries:
> * SUCCESS means that a valid AppendEntries response was received for the
> corresponding request;
> * FAILURE means that the transfer ended because of a local send error,
> connection failure, timeout, or response-stream error.
> A valid AppendEntries response confirms that the peer communication completed.
> Raft protocol results such as INCONSISTENCY or NOT_LEADER should not by
> themselves be classified as data transfer failures.
> For InstallSnapshot:
> * SUCCESS means that the logical snapshot transfer completed and all expected
> snapshot responses were received;
> * FAILURE means that the logical snapshot transfer ended because of a local
> send error, connection failure, timeout, response-stream error, or an
> unsuccessful terminal snapshot response.
> Intermediate snapshot responses must not generate duplicate success events.
> h3. Required semantics
> The listener should have the following behavior:
> * Generate one terminal event for each data transfer attempt.
> * Generate events on the sending peer only.
> * Treat every retry as a separate transfer attempt.
> * Report the source and destination Raft peers.
> * Report TLS when the outbound peer connection uses TLS.
> * Report NONE when the outbound peer connection does not use TLS.
> * Preserve the best available Throwable for a failed transfer.
> * Leave the error empty for a successful transfer.
> * Exclude heartbeat-only and other non-data-bearing requests.
> * When a stream failure causes pending requests to be discarded, report every
> affected data-bearing request as failed exactly once.
> * An exception thrown by the Consumer must not affect replication, retries,
> leader state, pending requests, or channel lifecycle.
> * The Consumer execution context should be documented.
> * If invoked on a Ratis or gRPC internal thread, the Consumer must perform
> only
> non-blocking work, such as enqueueing the event for another thread.
> * Existing behavior must remain unchanged when no Consumer is configured.
> The TLS handshake failure callback introduced by RATIS-2638 remains a separate
> transport-level event. It reports failure to establish a TLS channel, while
> the proposed listener reports the outcome of an actual data transfer attempt
> between Raft peers.
> If a data transfer attempt fails because its outbound connection cannot
> complete the TLS handshake, the sender-side transfer listener may report that
> attempt as failed using the best error available from the gRPC request path.
> This does not replace the connection-level event provided by RATIS-2638.
> h3. Implementation considerations
> For AppendEntries, the relevant sender-side completion paths are in
> GrpcLogAppender:
> * AppendLogResponseHandler.onNext;
> * timeoutAppendRequest;
> * AppendLogResponseHandler.onError;
> * synchronous exceptions while sending a request.
> A request should be marked as data-bearing when it contains at least one
> state-machine log entry. This information may be stored together with the
> pending AppendEntries request without retaining or copying its payload.
> The event should be emitted when the request reaches a terminal state.
> The current response-stream error path may clear multiple pending requests.
> Before those requests are discarded, every affected data-bearing request
> should be reported as failed exactly once.
> For InstallSnapshot, the relevant paths include:
> * completion after all expected snapshot responses are received;
> * InstallSnapshotResponseHandler.onError;
> * timeout handling;
> * synchronous exceptions while creating or writing the request stream.
> The snapshot transfer should have a single terminal state so that an error
> cannot be followed by a duplicate success or failure event.
> The protection method should be derived from the actual outbound gRPC peer
> connection configuration.
> Ratis should only invoke the configured Consumer. Serialization, queueing,
> filtering, persistence, and audit log formatting remain responsibilities of
> the embedding application.
> h3. Compatibility and performance
> The feature should be opt-in.
> When no Consumer is configured:
> * existing APIs and constructors should remain compatible;
> * AppendEntries and InstallSnapshot behavior should remain unchanged;
> * no audit-specific dependency should be introduced;
> * no request or snapshot payload should be copied;
> * no additional application-visible event should be generated;
> * the normal replication path should have negligible additional overhead.
> The configured Consumer is expected to be non-blocking. Applications requiring
> persistent audit records should enqueue the event and perform serialization
> and I/O asynchronously.
> Listener failures should be isolated in the same way as the listener added for
> RATIS-2638: an exception thrown by application code must not replace the
> original transfer result or interfere with Ratis processing.
> h3. Suggested tests
> Tests should cover:
> * successful data-bearing AppendEntries over TLS generates one SUCCESS event;
> * successful data-bearing AppendEntries without TLS generates one SUCCESS
> event with protection method NONE;
> * heartbeat-only AppendEntries does not generate an event;
> * AppendEntries without state-machine data does not generate an event;
> * AppendEntries timeout generates one FAILURE event;
> * a synchronous send exception generates one FAILURE event;
> * a response-stream failure reports every affected pending data-bearing
> request exactly once;
> * a valid INCONSISTENCY response is treated as a completed transfer;
> * a valid NOT_LEADER response is treated as a completed transfer;
> * successful InstallSnapshot generates one SUCCESS event for the complete
> logical snapshot transfer;
> * failed InstallSnapshot generates one FAILURE event;
> * intermediate snapshot chunk responses do not generate duplicate events;
> * notification-only InstallSnapshot does not generate an event;
> * TLS and NONE protection methods are reported correctly;
> * an exception thrown by the Consumer does not affect replication;
> * no event is generated when no Consumer is configured.
> I can contribute an implementation and tests after confirming the preferred
> API and event granularity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
