This is an automated email from the ASF dual-hosted git repository. riemer pushed a commit to branch dev in repository https://gitbox.apache.org/repos/asf/incubator-streampipes-website.git
commit eb95a4b08035d17a269ca15777c955677a620891 Author: Dominik Riemer <[email protected]> AuthorDate: Sun May 2 22:12:12 2021 +0200 [STREAMPIPES-197] Copy CLI docs from Github --- documentation/docs/06_extend-archetypes.md | 2 +- documentation/docs/06_extend-cli.md | 186 +++++++++++++++++++++++++++++ documentation/docs/06_extend-setup.md | 97 ++------------- documentation/website/i18n/en.json | 4 + documentation/website/sidebars.json | 1 + 5 files changed, 200 insertions(+), 90 deletions(-) diff --git a/documentation/docs/06_extend-archetypes.md b/documentation/docs/06_extend-archetypes.md index df98143..71f7675 100644 --- a/documentation/docs/06_extend-archetypes.md +++ b/documentation/docs/06_extend-archetypes.md @@ -23,7 +23,7 @@ We use ``groupId``: ``org.example`` and ``artifactId``: ``ExampleProcessor``. You can keep the default values for the other settings, confirm them by hitting enter. Now, a new folder with the name ``ExampleProcessor`` is generated. -The current {sp.version} is 0.66.0 +The current {sp.version} is 0.67.0 ```bash mvn archetype:generate \ diff --git a/documentation/docs/06_extend-cli.md b/documentation/docs/06_extend-cli.md new file mode 100644 index 0000000..546df8d --- /dev/null +++ b/documentation/docs/06_extend-cli.md @@ -0,0 +1,186 @@ +--- +id: extend-cli +title: StreamPipes CLI +sidebar_label: StreamPipes CLI +--- + +The StreamPipes command-line interface (CLI) is focused on developers in order to provide an easy entrypoint to set up a suitable dev environment, either planning on developing + +* new extensions such as **connect adapters, processors, sinks** or, +* new core features for **backend** and **ui**. + +## TL;DR + +```bash +streampipes env --list +[INFO] Available StreamPipes environment templates: +pipeline-element +... +streampipes env --set pipeline-element +streampipes up -d +``` +> **NOTE**: use `./streampipes` if you haven't add it to the PATH and sourced it (see section "Run `streampipes` from anywhere?"). + +## Prerequisites +The CLI is basically a wrapper around multiple `docker` and `docker-compose` commands plus some additional sugar. + +* Docker >= 17.06.0 +* Docker-Compose >= 1.26.0 (Compose file format: 3.4) +* Google Chrome (recommended), Mozilla Firefox, Microsoft Edge +* For Windows Developer: GitBash only + + +Tested on: **macOS**, **Linux**, **Windows***) + +> **NOTE**: *) If you're using Windows the CLI only works in combination with GitBash - CMD, PowerShell won't work. + + +## CLI commands overview + +``` +StreamPipes CLI - Manage your StreamPipes environment with ease + +Usage: streampipes COMMAND [OPTIONS] + +Options: + --help, -h show help + --version, -v show version + +Commands: + clean Remove StreamPipes data volumes, dangling images and network + down Stop and remove StreamPipes containers + env Inspect and select StreamPipes environments + info Get information + logs Get container logs for specific container + ps List all StreamPipes container for running environment + pull Download latest images from Dockerhub + restart Restart StreamPipes environment + up Create and start StreamPipes container environment + +Run 'streampipes COMMAND --help' for more info on a command. +``` + +## Usage: Along dev life-cycle + +**List** available environment templates. +```bash +streampipes env --list +``` + +**Inspect** services in an available environment to know what kind of services it is composed of. +```bash +streampipes env --inspect pipeline-element +``` + +**Set** environment, e.g. `pipeline-element`, if you want to write a new pipeline element. +```bash +streampipes env --set pipeline-element +``` + +**Start** environment ( default: `dev` mode). Here the service definition in the selected environment is used to start the multi-container landscape. +> **NOTE**: `dev` mode is enabled by default since we rely on open ports to core service such as `consul`, `couchdb`, `kafka` etc. to reach from the IDE when developing. If you don't want to map ports (except the UI port), then use the `--no-ports` flag. + +```bash +streampipes up -d +# start in production mode with unmapped ports +# streampipes up -d --no-ports +``` +Now you're good to go to write your new pipeline element :tada: :tada: :tada: + +> **HINT for extensions**: Use our [Maven archetypes](https://streampipes.apache.org/docs/docs/dev-guide-archetype/) to setup a project skeleton and use your IDE of choice for development. However, we do recommend using IntelliJ. + +> **HINT for core**: To work on `backend` or `ui` features you need to set the template to `backend` and clone the core repository [incubator-streampipes](https://github.com/apache/incubator-streampipes) - check the prerequisites there for more information. + +**Stop** environment and remove docker container +```bash +streampipes down +# want to also clean docker data volumes when stopping the environment? +# streampipes down -v +``` + +## Additionally, useful commands + +**Start individual services only?** We got you! You chose a template that suits your needs and now you only want to start individual services from it, e.g. only Kafka and Consul. + +> **NOTE**: the service names need to be present and match your current `.spenv` environment. + +```bash +streampipes up -d kafka consul +``` + +**Get current environment** (if previously set using `streampipes env --set <environment>`). +```bash +streampipes env +``` + +**Get logs** of specific service and use optional `--follow` flag to stay attached to the logs. +```bash +streampipes logs --follow backend +``` + +**Update** all services of current environment +```bash +streampipes pull +``` + +**Restart** all services of current environment or specific services +```bash +streampipes restart +# restart backend & consul +# streampipes restart backend consul +``` + +**Clean** your system and remove created StreamPipes Docker volumes, StreamPipes docker network and dangling StreamPipes images of old image layers. +```bash +streampipes clean +# remove volumes, network and dangling images +# streampipes clean --volumes +``` + +## Modify/Create an environment template +As of now, this step has to be done **manually**. All environments are located in `environments/`. + +```bash +├── adapter # developing a new connect adapter +├── backend # developing core backend features +├── basic # wanna run core, UI, connect etc from the IDE? +├── full # full version containing more pipeline elements +├── lite # few pipeline elements, less memory +├── pipeline-element # developing new pipeline-elements +└── ui # developing UI features +``` +**Modifying an existing environment template**. To modify an existing template, you can simply add a `<YOUR_NEW_SERVICE>` to the template. +> **NOTE**: You need to make sure, that the service your are adding exists in `deploy/standalone/service/<YOUR_NEW_SERVICE>`. If your're adding a completely new service take a look at existing ones, create a new service directory and include a `docker-compose.yml` and `docker-compose.dev.yml` file. + +``` +[environment:backend] +activemq +kafka +... +<YOUR_NEW_SERVICE> +``` + +**Creating a new** environment template. To create a new environment template, place a new file `environments/<YOUR_NEW_ENVIRONMENT>` in the template directory. Open the file and use the following schema. +> **IMPORTANT**: Please make sure to have `[environment:<YOUR_NEW_ENVIRONMENT>]` header in the first line of your new template matching the name of the file. Make sure to use small caps letters (lowercase) only. + +``` +[environment:<YOUR_NEW_ENVIRONMENT>] +<SERVICE_1> +<SERVICE_2> +... +``` + +## Run `streampipes` from anywhere? No problem +Simply add the path to this cli directory to your `$PATH` (on macOS, Linux) variable, e.g. in your `.bashrc` or `.zshrc`, or `%PATH%` (on Windows). + +For **macOS**, or **Linux**: + +```bash +export PATH="/path/to/incubator-streampipes-installer/cli:$PATH" +``` + +For **Windows 10**, e.g. check this [documentation](https://helpdeskgeek.com/windows-10/add-windows-path-environment-variable/). + + +## Upgrade to new version +To upgrade to a new version, simply edit the version tag `SP_VERSION` in the `.env` file. diff --git a/documentation/docs/06_extend-setup.md b/documentation/docs/06_extend-setup.md index eabaf81..75fecfe 100644 --- a/documentation/docs/06_extend-setup.md +++ b/documentation/docs/06_extend-setup.md @@ -4,18 +4,19 @@ title: Development Setup sidebar_label: Development Setup --- +Pipeline elements in StreamPipes are provided as standalone microservices. New pipeline elements can be easily developed using the provided Maven archetypes and can be installed in StreamPipes at runtime. + In this section, we describe our recommended minimum setup for locally setting up a development instance of StreamPipes needed to develop, run and test new pipeline elements. ## IDE & required dev tools StreamPipes does not have specific requirements on the IDE - so feel free to choose the IDE of your choice. The only requirements in terms of development tools are that you have Java 8 and Maven installed. -## Docker-based local StreamPipes instance +## StreamPipes CLI: Docker-based local StreamPipes instance In order to quickly test developed pipeline elements without needing to install all services required by StreamPipes, we provide a CLI tool that allows you to selectively start StreamPipes components. -The CLI tool allows to switch to several templates (based on docker-compose) depending on the role. For instance, if you are developing a pipeline element, use the template ``pe-developer``. This will start backend and ui components in a Docker container, while you can easily test your pipeline element in your IDE. - -For now, we refer to the Github Readme for instructions on how to use the CLI tool: [https://github.com/apache/incubator-streampipes-installer](https://github.com/apache/incubator-streampipes-installer) +The CLI tool allows to switch to several templates (based on docker-compose) depending on the role. +The documentation on the usage of the CLI tool is available [here](extend-cli). ## Starter projects Now, once you've started the development instance, you are ready to develop your very first pipeline element. @@ -23,90 +24,8 @@ Instead of starting from scratch, we recommend using our provided maven archetyp ### Maven archetypes -Create the Maven archetype as described in the [Getting Started](/docs/dev-guide-archetype) guide. - -### Starting from scratch - -In order to develop a new pipeline element from scratch, you need to create a new Maven project and import the following dependencies: - -<details class="info"> -<summary>pom.xml</summary> -``` -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-container-standalone</artifactId> - <version>0.64.0</version> -</dependency> - -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-sdk</artifactId> - <version>0.64.0</version> -</dependency> - -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-vocabulary</artifactId> - <version>0.64.0</version> -</dependency> - -<!-- This dependency needs to be imported if you plan to develop a new data processor or data sink using the Apache Flink wrapper --> -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-wrapper-flink</artifactId> - <version>0.64.0</version> -</dependency> - -<!-- This dependency needs to be imported if you plan to develop a new data processor or data sink which is running directly on the JVM --> -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-wrapper-standalone</artifactId> - <version>0.64.0</version> -</dependency> - -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-dataformat-json</artifactId> - <version>0.64.0</version> -</dependency> -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-dataformat-smile</artifactId> - <version>0.64.0</version> -</dependency> -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-dataformat-cbor</artifactId> - <version>0.64.0</version> -</dependency> -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-dataformat-fst</artifactId> - <version>0.64.0</version> -</dependency> -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-messaging-jms</artifactId> - <version>0.64.0</version> -</dependency> -<dependency> - <groupId>org.streampipes</groupId> - <artifactId>streampipes-messaging-kafka</artifactId> - <version>0.64.0</version> -</dependency> -``` -</details> - -The following three dependencies are mandatory: - -* `streampipes-container-standalone`, defines that we are going to create a new pipeline element where the description will be accessible through an embedded web server. -* `streampipes-sdk` imports the SDK which provides many convencience functions to create new pipeline elements. -* `streampipes-vocabulary` imports various RDF vocabularies which are used by the SDK to auto-generate the semantic description of pipeline elements. - -The following three dependencies might be optional depending on the pipeline element type you plan to create: - -* `streampipes-wrapper-flink` should be used in case you plan to connect a new data processor or data sink that uses Apache Flink for processing events at runtime. -* `streampipes-wrapper-standalone` should be used in case you plan to connect a new data processor or data sink that does not use an external processing engine. Events are directly processed in a single-host fashion. +Create the Maven archetype as described in the [Getting Started](extend-archetypes) guide. +### Examples -Finally, this dependency will provide abstract classes to define data sources and streams. +We provide several examples that explain the usage of some concepts in this [Github repo](https://github.com/apache/incubator-streampipes-examples). diff --git a/documentation/website/i18n/en.json b/documentation/website/i18n/en.json index fc28944..0622388 100644 --- a/documentation/website/i18n/en.json +++ b/documentation/website/i18n/en.json @@ -77,6 +77,10 @@ "title": "Maven Archetypes", "sidebar_label": "Maven Archetypes" }, + "extend-cli": { + "title": "StreamPipes CLI", + "sidebar_label": "StreamPipes CLI" + }, "extend-sdk-event-model": { "title": "SDK Guide: Event Model", "sidebar_label": "SDK: Event Model" diff --git a/documentation/website/sidebars.json b/documentation/website/sidebars.json index 559dbc7..1eaa5d3 100644 --- a/documentation/website/sidebars.json +++ b/documentation/website/sidebars.json @@ -163,6 +163,7 @@ ], "\uD83D\uDCBB Extend StreamPipes": [ "extend-setup", + "extend-cli", "extend-archetypes", "extend-tutorial-data-sources", "extend-tutorial-data-processors",
