[
https://issues.apache.org/jira/browse/NIP-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne reassigned NIP-38:
-----------------------------
Assignee: Mark Payne
> Pluggable Flow Persistence and Audit Storage
> --------------------------------------------
>
> Key: NIP-38
> URL: https://issues.apache.org/jira/browse/NIP-38
> Project: NiFi Improvement Proposal
> Issue Type: Improvement
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> h2. Motivation
> Two parts of the NiFi framework are hardcoded to read and write local files,
> with no way to redirect that storage elsewhere.
> The first is the dataflow itself, which is always persisted to a local file
> as GZIP-compressed JSON ({{{}flow.json.gz{}}}). Some deployments would
> benefit from the ability to store the flow somewhere other than a local file,
> and there is currently no extension point that lets the mechanism for loading
> and saving the flow be replaced. This coupling also appears in components
> that only need to read a small part of the flow. For example,
> {{FileAccessPolicyProvider}} parses {{flow.json.gz}} directly to obtain the
> root process group identifier when seeding initial access policies.
> The second is the flow-change history (the audit trail of who changed what in
> the flow), which is written to an embedded database under
> {{{}nifi.database.directory{}}}. This history is stored by the
> {{{}AuditService{}}}. An {{AuditService}} interface already exists, but the
> framework constructs the {{EntityStoreAuditService}} implementation directly,
> so there is no way to store this history anywhere other than the local
> embedded database.
> This NIP makes both storage mechanisms pluggable, using the same extension
> pattern for each.
> h2. Proposed Solution
> h3. Flow Persistence Provider
> Introduce a {{FlowPersistenceProvider}} extension point in
> {{{}nifi-framework-api{}}}. A single provider is selected per instance and is
> responsible for loading and saving the dataflow. The provider should operate
> on {{{}VersionedDataflow{}}}; the encoding of the flow and the mechanism used
> to store it are entirely the provider's concern.
> {code:java}
> /**
> * Persists and restores the dataflow for a NiFi instance. A single provider
> is selected per
> * instance and is responsible for durably storing the flow so that it can be
> recovered when
> * NiFi restarts. The encoding of the flow and the mechanism used to store it
> are entirely the
> * provider's concern.
> */
> public interface FlowPersistenceProvider {
> /**
> * Called once after instantiation to provide the provider with its
> configuration.
> */
> void initialize(FlowPersistenceProviderInitializationContext
> initializationContext) throws IOException;
> /**
> * Loads the currently persisted flow, or an empty Optional when no flow
> has been persisted.
> */
> Optional<VersionedDataflow> loadFlow() throws IOException;
> /**
> * Persists the flow described by the given context, replacing any
> previously persisted flow.
> */
> void saveFlow(FlowPersistenceContext context) throws IOException;
> }
> {code}
> {{saveFlow}} accepts a {{FlowPersistenceContext}} rather than a bare
> {{VersionedDataflow}} so that additional detail about the save — for example
> a human-readable description of what changed — can be added later without
> breaking existing providers.
> {{VersionedDataflow}} and {{VersionedFlowEncodingVersion}} move from
> {{nifi-framework-core-api}} into {{nifi-framework-api}} so the extension
> point can expose a typed API.
> A default {{StandardFlowPersistenceProvider}} in {{nifi-framework-core}}
> persists the flow as GZIP-compressed JSON to the configured flow
> configuration file.
> h3. Audit Service
> Make the {{AuditService}} a selectable extension point. The interface moves
> from {{nifi-administration}} into {{nifi-framework-api}} so that
> implementations can be delivered in NARs, and it gains a default
> {{initialize}} method that receives the implementation's configuration.
> {code:java}
> /**
> * Called once after instantiation to provide the Audit Service with its
> configuration. The
> * default implementation does nothing so that existing implementations
> continue to work
> * unchanged.
> */
> default void initialize(AuditServiceInitializationContext
> initializationContext) throws IOException {
> }
> {code}
> The implementation to use is selected in {{nifi.properties}} and defaults to
> {{{}EntityStoreAuditService{}}}, which stores the flow-change history in the
> existing embedded database. The directory the default implementation uses is
> supplied through its initialization context, so the storage location is
> configuration rather than a hardcoded file reference.
> h2. Framework Changes
> * New {{FlowPersistenceProvider}} extension point and supporting context
> types in {{{}nifi-framework-api{}}}.
> * {{VersionedDataflow}} and {{VersionedFlowEncodingVersion}} move from
> {{nifi-framework-core-api}} to {{{}nifi-framework-api{}}}.
> * Default {{StandardFlowPersistenceProvider}} plus its extension-discovery
> and dependency-injection wiring in {{{}nifi-framework-core{}}}.
> * Every point at which the framework saves the flow, and every point at
> which it restores the flow, goes through the provider.
> * {{FileAccessPolicyProvider}} obtains the root process group identifier
> from the provider rather than reading {{flow.json.gz}} directly.
> * The previous {{FlowConfigurationDAO}} / {{StandardFlowConfigurationDAO}} /
> {{FlowParser}} / {{FlowInfo}} code path is removed.
> * {{AuditService}} (and its supporting types {{{}History{}}},
> {{{}HistoryQuery{}}}, {{{}PreviousValue{}}}) move from
> {{nifi-administration}} to {{{}nifi-framework-api{}}}, and the interface
> gains a default {{initialize}} method plus an
> {{{}AuditServiceInitializationContext{}}}.
> * The {{AuditService}} implementation is created through extension-discovery
> and dependency-injection wiring in {{nifi-framework-core}} instead of being
> constructed directly.
> * {{EntityStoreAuditService}} reads its storage directory from its
> initialization context rather than from a hardcoded location.
> h2. nifi.properties Configuration
> * {{nifi.flow.persistence.provider.implementation}} — fully qualified class
> name of the {{FlowPersistenceProvider}} to use. Default:
> {{{}org.apache.nifi.controller.flow.StandardFlowPersistenceProvider{}}}.
> * {{nifi.flow.persistence.provider.properties.<name>}} — provider-specific
> configuration, passed to the provider through its initialization context.
> * {{nifi.audit.service.implementation}} — fully qualified class name of the
> {{AuditService}} to use. Default:
> {{{}org.apache.nifi.admin.service.EntityStoreAuditService{}}}.
> * {{nifi.audit.service.properties.<name>}} — implementation-specific
> configuration, passed to the Audit Service through its initialization context.
> The existing {{{}nifi.flow.configuration.file{}}},
> {{{}nifi.flow.configuration.archive.*{}}}, and {{nifi.database.directory}}
> properties are honored as fallbacks.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
