This is an automated email from the ASF dual-hosted git repository.
pvillard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/main by this push:
new e3ac4089c5 NIFI-13909 Refreshed and Streamlined Project README
e3ac4089c5 is described below
commit e3ac4089c5d7849052e8b9bd9d3aa8167cadc48b
Author: exceptionfactory <[email protected]>
AuthorDate: Sat Oct 19 19:39:12 2024 -0500
NIFI-13909 Refreshed and Streamlined Project README
Signed-off-by: Pierre Villard <[email protected]>
This closes #9426.
---
README.md | 368 ++++++++++++++++++++++----------------------------------------
1 file changed, 130 insertions(+), 238 deletions(-)
diff --git a/README.md b/README.md
index 2d76fac0a2..0557eb8171 100644
--- a/README.md
+++ b/README.md
@@ -12,7 +12,12 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-[<img src="https://nifi.apache.org/assets/images/apache-nifi-logo.svg";
width="300" height="126" alt="Apache NiFi"/>][nifi]
+
+# Apache NiFi
+
+<img src="https://nifi.apache.org/assets/images/apache-nifi-logo.svg";
width="300" alt="Apache NiFi"/>
+
+### Status
[](https://github.com/apache/nifi/actions/workflows/ci-workflow.yml)
[](https://github.com/apache/nifi/actions/workflows/system-tests.yml)
@@ -20,289 +25,180 @@
[](https://github.com/apache/nifi/actions/workflows/docker-tests.yml)
[](https://github.com/apache/nifi/actions/workflows/code-coverage.yml)
[](https://codecov.io/gh/apache/nifi)
-[](https://hub.docker.com/r/apache/nifi/)
-[](https://nifi.apache.org/download.html)
-[](https://s.apache.org/nifi-community-slack)
-
-[Apache NiFi](https://nifi.apache.org/) is an easy to use, powerful, and
-reliable system to process and distribute data.
-## Table of Contents
+### Resources
-- [Features](#features)
-- [Requirements](#requirements)
-- [Getting Started](#getting-started)
-- [MiNiFi subproject](#minifi-subproject)
-- [Registry subproject](#registry-subproject)
-- [Getting Help](#getting-help)
-- [Documentation](#documentation)
-- [License](#license)
-- [Export Control](#export-control)
+[](https://central.sonatype.com/artifact/org.apache.nifi/nifi-api)
+[](https://central.sonatype.com/artifact/org.apache.nifi/nifi-nar-maven-plugin)
+[](https://nifi.apache.org/download/)
+[](https://hub.docker.com/r/apache/nifi/)
+[](https://github.com/apache/nifi/blob/main/LICENSE)
+[](https://javadoc.io/doc/org.apache.nifi/nifi-api)
-## Features
+### Contacts
-Apache NiFi was made for dataflow. It supports highly configurable directed
graphs of data routing, transformation, and system mediation logic. Some of its
key features include:
+[](https://issues.apache.org/jira/browse/NIFI)
+[](https://s.apache.org/nifi-community-slack)
+[](https://lists.apache.org/[email protected])
+[](https://lists.apache.org/[email protected])
-- Web-based user interface
- - Seamless experience for design, control, and monitoring
- - Multi-tenant user experience
-- Highly configurable
- - Loss tolerant vs guaranteed delivery
- - Low latency vs high throughput
- - Dynamic prioritization
- - Flows can be modified at runtime
- - Back pressure
- - Scales up to leverage full machine capability
- - Scales out with zero-leader clustering model
-- Data Provenance
- - Track dataflow from beginning to end
-- Designed for extension
- - Build your own processors and more
- - Enables rapid development and effective testing
-- Secure
- - SSL, SSH, HTTPS, encrypted content, etc...
- - Pluggable fine-grained role-based authentication/authorization
- - Multiple teams can manage and share specific portions of the flow
-
-## Minimum Requirements
-* JDK 21
-* Apache Maven 3.9.6
+### Community
-## Getting Started
+[](https://join.slack.com/t/apachenifi/shared_invite/zt-11njbtkdx-ZRU8FKYSWoEHRJetidy0zA)
+[](https://www.linkedin.com/company/apache-nifi/)
+[](https://x.com/apachenifi)
-Read through the [quickstart guide for
development](https://nifi.apache.org/quickstart.html).
-It will include information on getting a local copy of the source, give
pointers on issue
-tracking, and provide some warnings about common problems with development
environments.
+## Features
-For a more comprehensive guide to development and information about
contributing to the project
-read through the [NiFi Developer's
Guide](https://nifi.apache.org/developer-guide.html).
+[Apache NiFi](https://nifi.apache.org/) is an easy to use, powerful, and
reliable system to process and distribute data.
-### Building
+NiFi automates cybersecurity, observability, event streams, and generative AI
data pipelines and distribution
+for thousands of companies worldwide across every industry.
-Run the following Maven command to build standard project modules using
parallel execution:
+- Browser User Interface
+ - Seamless experience for design, control, and monitoring
+ - Runtime management and versioned pipelines
+ - Secure by default with HTTPS
+- Scalable Processing
+ - Configurable prioritization for throughput and latency
+ - Guaranteed delivery with retry and backoff strategies
+ - Horizontal scaling with clustering
+- Provenance Tracking
+ - Searchable history with configurable attributes
+ - Graph data lineage from source to destination
+ - Metadata and content for each processing decision
+- Extensible Design
+ - Plugin interface for Processors and Controller Services
+ - Support for Processors in native Python
+ - REST API for orchestration and monitoring
+- Secure Configuration
+ - Single sign-on with OpenID Connect or SAML 2
+ - Flexible authorization policies for role-based access
+ - Encrypted communication with TLS and SFTP
- ./mvnw clean install -T2C
+## Requirements
-Run the following Maven command to build project modules with static analysis
to confirm compliance with code and licensing requirements:
+NiFi supports modern operating systems and requires recent language versions
for developing and running the application.
- ./mvnw clean install -T2C -P contrib-check
+### Platform Requirements
-Building on Microsoft Windows requires using `mvnw.cmd` instead of `mwnw` to
run the Maven Wrapper.
+- Java 21
-### Deploying
+### Optional Dependencies
-Change directories to `nifi-assembly`. The `target` directory contains binary
archives.
+- Python 3.10 or higher
- laptop:nifi myuser$ cd nifi-assembly
- laptop:nifi-assembly myuser$ ls -lhd target/nifi*
- drwxr-xr-x 3 myuser mygroup 102B Apr 30 00:29
target/nifi-1.0.0-SNAPSHOT-bin
- -rw-r--r-- 1 myuser mygroup 144M Apr 30 00:30
target/nifi-1.0.0-SNAPSHOT-bin.tar.gz
- -rw-r--r-- 1 myuser mygroup 144M Apr 30 00:30
target/nifi-1.0.0-SNAPSHOT-bin.zip
+## Projects
-Copy the `nifi-VERSION-bin.tar.gz` or `nifi-VERSION-bin.zip` to a separate
deployment directory.
-Extracting the distribution will create a new directory named for the version.
+The source repository includes several component projects.
- laptop:nifi-assembly myuser$ mkdir ~/example-nifi-deploy
- laptop:nifi-assembly myuser$ tar xzf target/nifi-*-bin.tar.gz -C
~/example-nifi-deploy
- laptop:nifi-assembly myuser$ ls -lh ~/example-nifi-deploy/
- total 0
- drwxr-xr-x 10 myuser mygroup 340B Apr 30 01:06 nifi-1.0.0-SNAPSHOT
+Please review individual project documentation for additional details.
-### Starting
+- [Apache NiFi](https://nifi.apache.org/documentation/)
+- [Apache NiFi
Registry](https://github.com/apache/nifi/blob/main/nifi-registry/nifi-registry-assembly/README.md)
+- [Apache NiFi
MiNiFi](https://github.com/apache/nifi/blob/main/minifi/minifi-assembly/README.md)
-Change directories to the deployment location and run the following command to
start NiFi.
+## Getting Started
- laptop:~ myuser$ cd ~/example-nifi-deploy/nifi-*
- laptop:nifi-1.0.0-SNAPSHOT myuser$ ./bin/nifi.sh start
-
-Running `bin/nifi.sh start` starts NiFi in the background and exits. Use
`--wait-for-init` with an optional timeout in
-seconds to wait for a complete startup before exiting.
+Project guides provide extensive documentation for installing and extending
the application.
- laptop:nifi-1.0.0-SNAPSHOT myuser$ ./bin/nifi.sh start --wait-for-init 120
-
-### Authenticating
+- [Getting
Started](https://nifi.apache.org/documentation/nifi-2.0.0-M4/html/getting-started.html)
+- [User
Guide](https://nifi.apache.org/documentation/nifi-2.0.0-M4/html/user-guide.html)
+- [Administrator
Guide](https://nifi.apache.org/documentation/nifi-2.0.0-M4/html/administration-guide.html)
+- [Developer
Guide](https://nifi.apache.org/documentation/nifi-2.0.0-M4/html/developer-guide.html)
-The default configuration generates a random username and password on startup.
NiFi writes the generated credentials
-to the application log located in `logs/nifi-app.log` under the NiFi
installation directory.
+## Developing
-The following command can be used to find the generated credentials on
operating systems with `grep` installed:
+NiFi uses the [Maven Wrapper](https://maven.apache.org/wrapper/) for project
development. The Maven Wrapper provides
+shell scripts that download and cache a selected version of [Apache
Maven](https://maven.apache.org/) for running build
+commands.
- laptop:nifi-1.0.0-SNAPSHOT myuser$ grep Generated logs/nifi-app*log
+Developing on Microsoft Windows requires using `mvnw.cmd` instead of `mvnw` to
run Maven commands.
-NiFi logs the generated credentials as follows:
-
- Generated Username [USERNAME]
- Generated Password [PASSWORD]
+### Building
-The `USERNAME` will be a random UUID composed of 36 characters. The `PASSWORD`
will be a random string composed of
-32 characters. The generated credentials will be stored in
`conf/login-identity-providers.xml` with the password stored
-using bcrypt hashing. Record these credentials in a secure location for access
to NiFi.
+Run the following command to build project modules using parallel execution:
-The random username and password can be replaced with custom credentials using
the following command:
+```shell
+./mvnw install -T1C
+```
- ./bin/nifi.sh set-single-user-credentials <username> <password>
-
-### Running
+Run the following command to build project modules using parallel execution
with static analysis to confirm compliance
+with code and licensing requirements:
-Open the following link in a web browser to access NiFi:
https://localhost:8443/nifi
+```shell
+./mvnw install -T1C -P contrib-check
+```
-The web browser will display a warning message indicating a potential security
risk due to the self-signed
-certificate NiFi generated during initialization. Accepting the potential
security risk and continuing to load
-the interface is an option for initial development installations. Production
deployments should provision a certificate
-from a trusted certificate authority and update the NiFi keystore and
truststore configuration.
+Run the following command to build the application binaries without building
other optional modules:
-Accessing NiFi after accepting the self-signed certificate will display the
login screen.
-
+```shell
+./mvnw install -T1C -am -pl :nifi-assembly
+```
-Using the generated credentials, enter the generated username in the `User`
field
-and the generated password in the `Password` field, then select `LOG IN` to
access the system.
-
+### Binaries
-### Configuring
+The `nifi-assembly` module contains the binary distribution.
-The [NiFi User
Guide](https://nifi.apache.org/docs/nifi-docs/html/user-guide.html) describes
how to build a data flow.
+```shell
+ls nifi-assembly/target/nifi-*-bin.zip
+```
-### Stopping
+The `nifi-assembly` module includes the binary distribution in a directory for
local development and testing.
-Run the following command to stop NiFi:
+```shell
+cd nifi-assembly/target/nifi-*-bin/nifi-*/
+```
- laptop:~ myuser$ cd ~/example-nifi-deploy/nifi-*
- laptop:nifi-1.0.0-SNAPSHOT myuser$ ./bin/nifi.sh stop
+## Running
-## MiNiFi subproject
+NiFi provides shell scripts for starting and stopping the system.
-MiNiFi is a child project effort of Apache NiFi. It is a complementary data
collection approach that supplements the core tenets of
[NiFi](https://nifi.apache.org/) in dataflow management, focusing on the
collection of data at the source of its creation.
+Running on Microsoft Windows requires using `nifi.cmd` instead of `nifi.sh`
for system commands.
-Specific goals for MiNiFi are comprised of:
-- small and lightweight footprint
-- central management of agents
-- generation of data provenance
-- integration with NiFi for follow-on dataflow management and full chain of
custody of information
+### Starting
-Perspectives of the role of MiNiFi should be from the perspective of the agent
acting immediately at, or directly adjacent to, source sensors, systems, or
servers.
+Run the following command to start NiFi from the distribution directory:
-To run:
-- Change directory to 'minifi-assembly'. In the target directory, there should
be a build of minifi.
+```shell
+./bin/nifi.sh start
+```
- $ cd minifi-assembly
- $ ls -lhd target/minifi*
- drwxr-xr-x 3 user staff 102B Jul 6 13:07
minifi-1.14.0-SNAPSHOT-bin
- -rw-r--r-- 1 user staff 39M Jul 6 13:07
minifi-1.14.0-SNAPSHOT-bin.tar.gz
- -rw-r--r-- 1 user staff 39M Jul 6 13:07
minifi-1.14.0-SNAPSHOT-bin.zip
+### Accessing
-- For testing ongoing development you could use the already unpacked build
present in the directory
- named "minifi-*version*-bin", where *version* is the current project
version. To deploy in another
- location make use of either the tarball or zipfile and unpack them wherever
you like. The
- distribution will be within a common parent directory named for the version.
+The default configuration generates a random username and password on startup.
NiFi writes the generated credentials
+to the application log located in `logs/nifi-app.log` under the NiFi
installation directory.
- $ mkdir ~/example-minifi-deploy
- $ tar xzf target/minifi-*-bin.tar.gz -C ~/example-minifi-deploy
- $ ls -lh ~/example-minifi-deploy/
- total 0
- drwxr-xr-x 10 user staff 340B Jul 6 01:06 minifi-1.14.0-SNAPSHOT
+The following command can be used to find the generated credentials on
operating systems with `grep` installed:
-To run MiNiFi:
-- Change directory to the location where you installed MiNiFi and run it.
+```shell
+grep Generated logs/nifi-app*log
+```
- $ cd ~/example-minifi-deploy/minifi-*
- $ ./bin/minifi.sh start
+NiFi logs the generated credentials as follows:
-- View the logs located in the logs folder
- $ tail -F ~/example-minifi-deploy/logs/minifi-app.log
+```shell
+Generated Username [USERNAME]
+Generated Password [PASSWORD]
+```
-- For help building your first data flow and sending data to a NiFi instance
see the System Admin Guide located in the docs folder or making use of the
minifi-toolkit.
+The `USERNAME` will be a random UUID composed of 36 characters. The `PASSWORD`
will be a random string.
-- If you are testing ongoing development, you will likely want to stop your
instance.
+The username and password can be replaced with custom credentials using the
following command:
- $ cd ~/example-minifi-deploy/minifi-*
- $ ./bin/minifi.sh stop
+```shell
+./bin/nifi.sh set-single-user-credentials <username> <password>
+```
-### Docker Build
+NiFi defaults to running on the `localhost` address with HTTPS on port `8443`
at the following URL:
-To build:
-- Run a full NiFi build (see above for instructions). Then from the minifi/
subdirectory, execute `mvn -P docker clean install`. This will run the full
build, create a docker image based on it, and run docker-compose integration
tests. After it completes successfully, you should have an
apacheminifi:${minifi.version} image that can be started with the following
command (replacing ${minifi.version} with the current maven version of your
branch):
```
-docker run -d -v
YOUR_CONFIG.YML:/opt/minifi/minifi-${minifi.version}/conf/config.yml
apacheminifi:${minifi.version}
+https://localhost:8443/nifi
```
-## Registry subproject
-
-Registry—a subproject of Apache NiFi—is a complementary application that
provides a central location for storage and management of shared resources
across one or more instances of NiFi and/or MiNiFi.
-
-### Getting Registry Started
-
-1) Build NiFi (see [Getting Started for NiFi](#getting-started) )
-
-or
-
-Build only the Registry subproject:
-
- cd nifi/nifi-registry
- mvn clean install
-
-
-If you wish to enable style and license checks, specify the contrib-check
profile:
-
- mvn clean install -Pcontrib-check
-
-
-2) Start Registry
-
- cd
nifi-registry/nifi-registry-assembly/target/nifi-registry-<VERSION>-bin/nifi-registry-<VERSION>/
- ./bin/nifi-registry.sh start
-
-Note that the application web server can take a while to load before it is
accessible.
-
-3) Accessing the application web UI
-
-With the default settings, the application UI will be available at
[http://localhost:18080/nifi-registry](http://localhost:18080/nifi-registry)
-
-4) Accessing the application REST API
-
-If you wish to test against the application REST API, you can access the REST
API directly. With the default settings, the base URL of the REST API will be
at `http://localhost:18080/nifi-registry-api`. A UI for testing the REST API
will be available at
[http://localhost:18080/nifi-registry-api/swagger/ui.html](http://localhost:18080/nifi-registry-api/swagger/ui.html)
-
-5) Accessing the application logs
-
-Logs will be available in `logs/nifi-registry-app.log`
-
-### Database Testing
-
-In order to ensure that NiFi Registry works correctly against different
relational databases,
-the existing integration tests can be run against different databases by
leveraging the [Testcontainers framework](https://www.testcontainers.org/).
-
-Spring profiles are used to control the DataSource factory that will be made
available to the Spring application context.
-DataSource factories are provided that use the Testcontainers framework to
start a Docker container for a given database and create a corresponding
DataSource.
-If no profile is specified then an H2 DataSource will be used by default and
no Docker containers are required.
-
-Assuming Docker is running on the system where the build is running, then the
following commands can be run:
-
-| Target Database | Build Command
|
-|-----------------|--------------------------------------------------------------------|
-| All supported | `mvn verify -Ptest-all-dbs`
|
-| H2 (default) | `mvn verify`
|
-| MariaDB 10.3 | `mvn verify -Pcontrib-check
-Dspring.profiles.active=mariadb-10-3` |
-| MySQL 8 | `mvn verify -Pcontrib-check
-Dspring.profiles.active=mysql-8` |
-| PostgreSQL 10 | `mvn verify -Dspring.profiles.active=postgres-10`
|
-
-For a full list of the available DataSource factories, consult the
`nifi-registry-test` module.
-
-## Getting Help
-If you have questions, you can reach out to our mailing list:
[email protected]
-([archive](https://lists.apache.org/[email protected])). For more
interactive discussions, community members can often be found in the following
locations:
-
-- Apache NiFi Slack Workspace: https://apachenifi.slack.com/
-
- New users can join the workspace using the following [invite
link](https://join.slack.com/t/apachenifi/shared_invite/zt-11njbtkdx-ZRU8FKYSWoEHRJetidy0zA).
-
-To submit a feature request or bug report, please file a Jira at
[https://issues.apache.org/jira/projects/NIFI/issues](https://issues.apache.org/jira/projects/NIFI/issues).
If this is a **security vulnerability report**, please email
[[email protected]](mailto:[email protected]) directly and review
the [Apache NiFi Security Vulnerability
Disclosure](https://nifi.apache.org/security.html) and [Apache Software
Foundation Security](https://www.apache.org/security/committers.ht [...]
-
-## Documentation
-
-See https://nifi.apache.org/ for the latest NiFi documentation.
-
-See https://nifi.apache.org/minifi and
https://cwiki.apache.org/confluence/display/MINIFI for the latest
MiNiFi-specific documentation.
-
-See https://nifi.apache.org/registry for the latest Registry-specific
documentation.
+Browsers will display a warning message indicating a potential security risk
due to the self-signed certificate
+generated during initialization. Production deployments should provision a
certificate from a trusted certificate
+authority and update the NiFi keystore and truststore configuration.
## License
@@ -328,7 +224,7 @@ currently reside may have restrictions on the import,
possession, use, and/or
re-export to another country, of encryption software. BEFORE using any
encryption software, please check your country's laws, regulations and
policies concerning the import, possession, or use, and re-export of encryption
-software, to see if this is permitted. See <https://www.wassenaar.org/> for
more
+software, to see if this is permitted. See https://www.wassenaar.org for more
information.
The U.S. Government Department of Commerce, Bureau of Industry and Security
@@ -342,15 +238,11 @@ source code.
The following provides more details on the included cryptographic software:
-Apache NiFi uses BouncyCastle, JCraft Inc., and the built-in
-Java cryptography libraries for SSL, SSH, and the protection
-of sensitive configuration parameters. See
-
-- https://bouncycastle.org/about.html
-- http://www.jcraft.com/c-info.html
--
https://www.oracle.com/corporate/security-practices/corporate/governance/global-trade-compliance.html
-
-for more details on each of these libraries cryptography features.
+Apache NiFi uses the following libraries and frameworks for encrypted
+communication and storage of sensitive information:
-[nifi]: https://nifi.apache.org/
-[logo]: https://nifi.apache.org/assets/images/apache-nifi-logo.svg
+- [Apache MINA SSHD](https://mina.apache.org/sshd-project/)
+- [Bouncy Castle](https://www.bouncycastle.org)
+- [Jagged](https://github.com/exceptionfactory/jagged)
+- [Java Cryptography
Architecture](https://docs.oracle.com/en/java/javase/21/security/java-cryptography-architecture-jca-reference-guide.html)
+- [SSHJ](https://github.com/hierynomus/sshj)
![https://nifi.apache.org/assets/images/apache-nifi-logo.svg"](https://nifi.apache.org/assets/images/apache-nifi-logo.svg"){kind=link}
