This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/master by this push:
new 9777320 HOP-3667 initial adoc version of Stefan Klaffke's tutorial
new cdd00dc Merge pull request #1281 from bamaer/HOP-3667
9777320 is described below
commit 97773206e0fb97be6b7bcc5cfe49a51b4266f838
Author: Bart Maertens <[email protected]>
AuthorDate: Wed Jan 19 21:04:20 2022 +0100
HOP-3667 initial adoc version of Stefan Klaffke's tutorial
---
.../web-services-in-docker/json-output.png | Bin 0 -> 175410 bytes
.../web-services-in-docker/metadata.png | Bin 0 -> 128946 bytes
.../web-services-in-docker/pipeline.png | Bin 0 -> 26079 bytes
.../web-services-in-docker/result.png | Bin 0 -> 19962 bytes
.../simplified-architecture.png | Bin 0 -> 244385 bytes
docs/hop-user-manual/modules/ROOT/nav.adoc | 2 +
.../apache-hop-web-services-docker.adoc | 175 +++++++++++++++++++++
.../modules/ROOT/pages/how-to-guides/index.adoc | 25 +++
8 files changed, 202 insertions(+)
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/json-output.png
b/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/json-output.png
new file mode 100644
index 0000000..af31432
Binary files /dev/null and
b/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/json-output.png
differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/metadata.png
b/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/metadata.png
new file mode 100644
index 0000000..6bb9581
Binary files /dev/null and
b/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/metadata.png
differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/pipeline.png
b/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/pipeline.png
new file mode 100644
index 0000000..04140b4
Binary files /dev/null and
b/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/pipeline.png
differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/result.png
b/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/result.png
new file mode 100644
index 0000000..bd8c870
Binary files /dev/null and
b/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/result.png
differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/simplified-architecture.png
b/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/simplified-architecture.png
new file mode 100644
index 0000000..3d6787d
Binary files /dev/null and
b/docs/hop-user-manual/modules/ROOT/assets/images/how-to-guides/web-services-in-docker/simplified-architecture.png
differ
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc
b/docs/hop-user-manual/modules/ROOT/nav.adoc
index 8141d6e..821c264 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -424,3 +424,5 @@ under the License.
* xref:best-practices/index.adoc[Best Practices]
* xref:protips/index.adoc[Pro Tips]
* xref:hop-usps.adoc[Unique Selling Propositions]
+* xref:how-to-guides/index.adoc[How-to guides]
+** xref:how-to-guides/apache-hop-web-services-docker.adoc[Hop web services in
Docker]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/apache-hop-web-services-docker.adoc
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/apache-hop-web-services-docker.adoc
new file mode 100644
index 0000000..13365d8
--- /dev/null
+++
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/apache-hop-web-services-docker.adoc
@@ -0,0 +1,175 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[HopServer]]
+:imagesdir: ../../assets/images
+:description: This tutorial explains how to run Apache Hop web services from a
Docker container
+
+= Web Services in Apache Hop
+
+== Introduction
+
+Apache Hop is a very flexible tool for developing data flows (so-called
pipelines) with the help of an easy-to-use UI, with which data can be extracted
from different sources, consolidated and loaded into a new system (ETL).
+
+The tool is not only excellently suited for these classic data warehouse tasks
or for database migrations. In the context of low-code software development,
Apache Hop can also be used to provide fast calculations and data queries as a
web service without much programming effort and to link these calculations to
your own web services. Hundreds of lines of source code are eliminated, as the
functionality can be conveniently implemented as a pipeline with its dozens of
transforms (processin [...]
+
+The following illustration shows an example of where Apache HOP can be
integrated with web services in a full-stack architecture (multi-container
application).
+
+image:how-to-guides/web-services-in-docker/simplified-architecture.png[Hop web
services in Docker - simplified architecture, width="45%"]
+
+In order to call the created Hop web services from a Java application, for
example, the Apache Hop server on which the pipelines are executed is required.
The Hop server provides a web interface (servlet) to return the results of the
execution.
+
+It can be run on a dedicated server or inside a Docker container. The latter
option offers the advantage of scaling quickly when needed.
+
+== Running Pipelines as a Web services in a Docker container
+
+This guide covers the steps to create an example web service and subsequent
deployment in a Docker container.
+
+=== Step 0: Requirements for the web service
+
+The task of the fictitious example web service in this tutorial is to provide
data in real time for a project dashboard (specifically: the number of
active/inactive teams per team type), so that after processing in the API
layer, the data can be displayed in the frontend as a stacked bar chart. The
data is available in normalised form in the database and must first be prepared
by the web service.
+
+=== Step 1: Creation of the pipeline
+
+After creating the project and setting up the database connection in HOP, the
following pipeline is created (file name: countTeams.hpl):
+
+image:how-to-guides/web-services-in-docker/pipeline.png[Hop web services in
Docker - pipeline, width="90%"]
+
+The pipeline expects a project ID as an input parameter in order to be able to
read out the teams belonging to the project from the database. The data is
grouped, then the number is calculated and finally denormalised by team type.
The result of the execution should be returned in JSON format.
+
+=== Step 2: Meta Data Configuration
+
+To make the pipeline available as a web service, metadata must be created for
it and later made known to the Hop server. The use of metadata has the
advantage that the calling application only needs to know the web service name,
the details of the underlying implementation (especially the location of the
pipeline) remain hidden.
+
+The following metadata are created for the pipeline under the Web Service tab:
+
+image:how-to-guides/web-services-in-docker/metadata.png[Hop web services in
Docker - metadata, width="75%"]
+
+The **Name** field contains the name of the web service under which the
calling application will call the service. The **Filename** field contains the
assignment to the storage location of the pipeline. The **Output transform**
field contains the name of the transform that provides the result in the output
field to be specified. Content type specifies the output format of the return
stream. In addition, you can further specify in the metadata whether the web
service should be disabled (* [...]
+
+In the pipeline listed above, the content of the output field outputValue of
the Transform JSON Output is to be returned as the result:
+
+image:how-to-guides/web-services-in-docker/json-output.png[Hop web services in
Docker - JSON Output, width="60%"]
+
+After saving the metadata, everything is ready to run the web service in a
Docker container.
+
+=== Step 3: Set up configurations for different runtime environments
+
+In Apache Hop it is possible to set up runtime environments for different
environments (e.g. Development Local, Docker Single, Docker Multi-App) via
so-called configurations and to pass them to the pipeline without having to
adapt or duplicate the pipeline itself (e.g. one pipeline for each environment).
+
+Connection details (e.g. DB server URL) for database connections, etc. can be
stored as variables in configuration files.
+
+For example, to set up a new database connection whose connection details may
differ depending on the environment, enter the name of the environment variable
(e.g. `${DB_HOST}`) instead of a concrete server URL. +
+
+As soon as you select an environment and the variable is contained in its
configuration file, the variable in the DB configuration is replaced by the
value from the environment configuration.
+
+This functionality is very helpful, for example, to test the pipeline against
different environments before you start the deployment on the Hop Server.
+
+This functionality is also essential for multi-container applications (see
above: full-stack architecture), whose service results from the interaction of
different and externally isolated containers.
+
+A separate environment configuration is therefore necessary if you want to
start your application with Docker Compose in addition to your development
environment (in this example, the `${DB_HOST}` variable would have the service
name of the DB Container instead of the IP address of the DB Server).
+
+=== Step 4: Set up and start Docker Container
+
+To start the Hop Server inside a Docker container, you have the following
options for building the container:
+
+* Using the image provided by Apache Hop
+* Use the Apache Hop image as a starting image and add application-specific
customisations in the form of a new image (e.g. include the Hop project with
its workflows and pipelines as well as third-party APIs).
+* Creation of a complete custom image (possibly based on the DOCKERFILES
provided in the Apache Hop Github repository).
+
+
+Depending on the deployment strategy, it may be advisable to include the
project and its source code in the image, e.g. if only containers may be
deployed in a production environment.
+
+
+
+The first option is sufficient for this guide, as all adjustments can be
transferred to the Docker container via environment variables for the example. +
+In addition, changes to pipelines should be made visible immediately without
having to rebuild the image first. +
+For this purpose, it is necessary to mount the location of the project (incl.
subdirectories for JDBC drivers and the metadata configuration) into the Docker
container.
+
+First download the Docker image:
+
+[source, shell script]
+----
+docker pull apache/hop:<tag>
+Example: docker pull apache/hop:1.1.0-SNAPSHOT
+----
+
+To execute the containers afterwards, the following environment variables or
parameters must be set as a minimum:
+
+[%header, width="90%"]
+|===
+|Variable/Docker Parameter|Example value|Description
+|HOP_SERVER_USER|admin|Admin user to log in to the UI of the Hop Server
+|HOP_SERVER_PASS|admin|Password for logging into the UI of the Hop Server
+|HOP_SERVER_PORT|8182|Port of the server (internal)
+|HOP_SERVER_HOSTNAME|0.0.0.0|Docker internal IP address of the host
+|HOP_PROJECT_NAME|Proj1|When the container is created, a project with all
necessary configurations is initially created, therefore the project to be
specified serves as a placeholder.
+|HOP_PROJECT_FOLDER|/files|Root folder containing the project
+|HOP_ENVIRONMENT_NAME|Local|Name of the environment with which the container
is to be started (e.g. local, prod) - see step 3
+|HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS|/files/config/localTestConfig.json|Comma
separated list of all environment files (see step 3)
+|HOP_SERVER_METADATA_FOLDER|/files/metadata|Directory containing the metadata
information (in this case, the metadata associated with the web service).
+|HOP_SHARED_JDBC_FOLDER|/files/jdbc|Directory containing all required JDBC
drivers (e.g. for MySQL, Oracle) that are not included in the standard delivery.
+|p|8182:8182|Mapping of the internal Docker port to the port of the Docker host
+|V|/my/path/to/location:/files|Mount path and mapping to a Docker internal path
+|Name||Name of the Docker container
+|===
+
+In the technical documentation for Apache HOP running as Docker Container (pls
see https://hop.apache.org/tech-manual/latest/docker-container.html[docs]),
further parameters are listed (e.g. when using SSL), which have been omitted
here for simplification.
+
+Example call:
+
+[source, shell script]
+----
+docker run -it --rm \
+ --env HOP_SERVER_USER=admin \
+ --env HOP_SERVER_PASS=admin \
+ --env HOP_SERVER_PORT=8182 \
+ --env HOP_SERVER_HOSTNAME=0.0.0.0 \
+ --env HOP_PROJECT_NAME=proj1 \
+ --env HOP_PROJECT_FOLDER=/files \
+ --env HOP_ENVIRONMENT_NAME=Local \
+ --env
HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS=/files/config/localTestConfig.json \
+ --env HOP_SERVER_METADATA_FOLDER=/files/metadata \
+ --env HOP_SHARED_JDBC_FOLDER=/files/jdbc \
+ -p 8182:8182 \
+ -v /my/path/to/location:/files \
+ --name test-hop-container \
+ apache/hop:1.1.0-SNAPSHOT
+----
+
+=== Step 5: Call Web Service
+
+If the container has been started successfully, the web server GUI can be
opened via http://<IP_HOST>:<PORT_HOST>/hop/status/ which gives an overview of
the pipelines and workflows called up after logging in (using the administrator
IDs passed).
+
+To call a web service with the name <NameofService>, the following request is
necessary:
+http://<IP_HOST>:<PORT_HOST>/hop/webService?service=<NameofService>&Param1=Value&Param2=Value2....
+
+The WebService **countTeams** from this tutorial expects the parameter
PARAM_PROJECT_ID.
+
+The call is made as follows:
+
+*http://localhost:8182/hop/webService/?service=countTeams&PARAM_PROJECT_ID=63*
+
+image:how-to-guides/web-services-in-docker/result.png[Hop web services in
Docker - result, width="80%"]
+
+As mentioned at the beginning of the tutorial, the API layer above can now
process the JSON output and pass it to the front end, which then uses the
processed data to create the chart.
+
+== Conclusion
+
+Apache Hop Pipelines as web services are easy to set up via meta-data
configurations and can be easily deployed via Docker. They are suitable
everywhere to synchronously return results quickly that do not require a long
processing time. As of version 1.1, Apache Hop also supports the use of
workflows as web services.
+
+Here, the call is asynchronous, i.e. when the service is called, a unique ID
of the workflow to be executed is returned immediately and the execution starts
in the background on the HOP server (for details check
https://hop.apache.org/manual/latest/hop-server/async-web-service.html[the
docs]).
+
diff --git a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
new file mode 100644
index 0000000..67bff8b
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
@@ -0,0 +1,25 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[HopServer]]
+:imagesdir: ../../assets/images
+:description: This page contains a collection of how-to guides to perform a
variety of tasks, configurations etc with Apache Hop.
+
+== How-to guides
+
+This page contains a collection of how-to guides to perform a variety of
tasks, configurations etc with Apache Hop.
+
+* xref:how-to-guides/apache-hop-web-services-docker.adoc[using Apache Hop web
services in Docker]
\ No newline at end of file
