This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-hop.git
The following commit(s) were added to refs/heads/master by this push:
new 097b6b6 HOP-2713, HOP-2714, HOP-2818 technology doc updates for
Azure, Cassandra, Google, Neo4j, added missing plugin placeholders for Neo4j
transforms
new 27cf008 Merge pull request #765 from bamaer/HOP-2714
097b6b6 is described below
commit 097b6b62a78f9455df6f2d499f85fdcf0ef2ef4b
Author: Bart Maertens <[email protected]>
AuthorDate: Sat Apr 24 08:17:06 2021 +0200
HOP-2713, HOP-2714, HOP-2818 technology doc updates for Azure, Cassandra,
Google, Neo4j, added missing plugin placeholders for Neo4j transforms
---
docs/hop-user-manual/modules/ROOT/nav.adoc | 13 +++++--
.../ROOT/pages/best-practices/best-practices.adoc | 2 +-
.../transforms/neo4j-gencsv.adoc} | 6 +--
.../transforms/neo4j-getloginfo.adoc} | 6 +--
.../transforms/neo4j-import.adoc} | 6 +--
.../transforms/neo4j-split-graph.adoc} | 6 +--
.../modules/ROOT/pages/technology/azure/index.adoc | 37 ++++++++++++++++++
.../ROOT/pages/technology/cassandra/index.adoc | 45 ++++++++++++++++++++++
.../google/{google-info.adoc => index.adoc} | 18 ++++-----
.../neo4j/{neo4j-info.adoc => index.adoc} | 24 +++++++++++-
.../modules/ROOT/pages/technology/technology.adoc | 7 ++++
11 files changed, 141 insertions(+), 29 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc
b/docs/hop-user-manual/modules/ROOT/nav.adoc
index dff29d8..391418b 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -160,10 +160,12 @@ under the License.
*** xref:pipeline/transforms/mqttsubscriber.adoc[MQTT Subscriber]
*** xref:pipeline/transforms/multimerge.adoc[Multiway Merge Join]
*** xref:pipeline/transforms/neo4j-cypher.adoc[Neo4j Cypher]
+*** xref:pipeline/transforms/neo4j-gencsv.adoc[Neo4j Generate CSVs]
+*** xref:pipeline/transforms/neo4j-getloginfo.adoc[Neo4j Get Logging Info]
*** xref:pipeline/transforms/neo4j-graphoutput.adoc[Neo4j Graph Output]
-*** xref:pipeline/transforms/neo4j-cypher.adoc[Neo4j Cypher]
-*** xref:pipeline/transforms/neo4j-graphoutput.adoc[Neo4j Graph Output]
+*** xref:pipeline/transforms/neo4j-import.adoc[Neo4j Import]
*** xref:pipeline/transforms/neo4j-output.adoc[Neo4j Output]
+*** xref:pipeline/transforms/neo4j-split-graph.adoc[Neo4j Split Graph]
*** xref:pipeline/transforms/nullif.adoc[Null If]
*** xref:pipeline/transforms/numberrange.adoc[Number range]
*** xref:pipeline/transforms/pgpdecryptstream.adoc[PGP decrypt stream]
@@ -234,6 +236,7 @@ under the License.
*** xref:workflow/actions/abort.adoc[Abort]
*** xref:workflow/actions/addresultfilenames.adoc[Add filenames to result]
*** xref:workflow/actions/as400command.adoc[AS/400 Command]
+*** xref:workflow/actions/cassandra-exec-cql.adoc[Cassandra Exec Cql]
*** xref:workflow/actions/checkdbconnection.adoc[Check Db connections]
*** xref:workflow/actions/checkfilelocked.adoc[Check Files Locked]
*** xref:workflow/actions/webserviceavailable.adoc[Check if Webservice is
available]
@@ -385,8 +388,10 @@ under the License.
* xref:samples/samples.adoc[Samples]
** xref:samples/dummy-sample.adoc[Dummy Sample]
* xref:technology/technology.adoc[Technology]
-** xref:technology/neo4j/neo4j-info.adoc[Neo4j]
-** xref:technology/google/google-info.adoc[Google]
+** xref:technology/azure/index.adoc[Azure]
+** xref:technology/cassandra/index.adoc[Cassandra]
+** xref:technology/neo4j/index.adoc[Neo4j]
+** xref:technology/google/index.adoc[Google]
* xref:hop-server/index.adoc[Hop Server]
** xref:hop-server/rest-api.adoc[REST api]
** xref:hop-server/web-service.adoc[Web Service]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/best-practices/best-practices.adoc
b/docs/hop-user-manual/modules/ROOT/pages/best-practices/best-practices.adoc
index 8b09ecb..34c8618 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/best-practices/best-practices.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/best-practices/best-practices.adoc
@@ -77,7 +77,7 @@ xref:variables.adoc[Variables] provide an easy way to avoid
hard-coding all sort
== Logging
-Take some time to capture the logging of your workflows and pipelines. Any
time you run anything you want to have a trace of it. Things tend to go wrong
when you least expect it and at that point you like being able to see what
happened. See xref:logging/logging-basics.adoc[Logging Basics],
xref:logging/logging-reflection.adoc[Logging Reflection] or consider logging to
a xref:technology/neo4j/neo4j-info.adoc[Neo4j] graph database. This last one
allows you to browse the logging result [...]
+Take some time to capture the logging of your workflows and pipelines. Any
time you run anything you want to have a trace of it. Things tend to go wrong
when you least expect it and at that point you like being able to see what
happened. See xref:logging/logging-basics.adoc[Logging Basics],
xref:logging/logging-reflection.adoc[Logging Reflection] or consider logging to
a xref:technology/neo4j/index.adoc[Neo4j] graph database. This last one allows
you to browse the logging results in [...]
== Mappings
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-gencsv.adoc
similarity index 72%
copy from docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
copy to
docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-gencsv.adoc
index 66fa753..5417050 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-gencsv.adoc
@@ -1,4 +1,3 @@
-[[Technology]]
////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -15,6 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-= Technology
+:documentationPath: /pipeline/transforms/
+:language: en_US
-This section of the documentation contains technology-stack specific
information. The goal is to inform you about how Apache Hop offers support for
the technology in question. It also provides starting points for you if you
want to learn more about the technology.
+= Neo4j Generate CSVs
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-getloginfo.adoc
similarity index 72%
copy from docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
copy to
docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-getloginfo.adoc
index 66fa753..a063abe 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-getloginfo.adoc
@@ -1,4 +1,3 @@
-[[Technology]]
////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -15,6 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-= Technology
+:documentationPath: /pipeline/transforms/
+:language: en_US
-This section of the documentation contains technology-stack specific
information. The goal is to inform you about how Apache Hop offers support for
the technology in question. It also provides starting points for you if you
want to learn more about the technology.
+= Neo4j Get Logging Info
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-import.adoc
similarity index 72%
copy from docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
copy to
docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-import.adoc
index 66fa753..a01f42a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-import.adoc
@@ -1,4 +1,3 @@
-[[Technology]]
////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -15,6 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-= Technology
+:documentationPath: /pipeline/transforms/
+:language: en_US
-This section of the documentation contains technology-stack specific
information. The goal is to inform you about how Apache Hop offers support for
the technology in question. It also provides starting points for you if you
want to learn more about the technology.
+= Neo4j Import
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-split-graph.adoc
similarity index 72%
copy from docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
copy to
docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-split-graph.adoc
index 66fa753..c2551e8 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-split-graph.adoc
@@ -1,4 +1,3 @@
-[[Technology]]
////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -15,6 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-= Technology
+:documentationPath: /pipeline/transforms/
+:language: en_US
-This section of the documentation contains technology-stack specific
information. The goal is to inform you about how Apache Hop offers support for
the technology in question. It also provides starting points for you if you
want to learn more about the technology.
+= Neo4j Split Graph
\ No newline at end of file
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/technology/azure/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/technology/azure/index.adoc
new file mode 100644
index 0000000..3a4c049
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/azure/index.adoc
@@ -0,0 +1,37 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /technology/azure/
+:language: en_US
+
+= Azure
+
+From https://en.wikipedia.org/wiki/Microsoft_Azure[Wikipedia]:
+
+Microsoft Azure, commonly referred to as Azure (/ˈæʒər/), is a cloud computing
service created by Microsoft for building, testing, deploying, and managing
applications and services through Microsoft-managed data centers. It provides
software as a service (SaaS), platform as a service (PaaS) and infrastructure
as a service (IaaS) and supports many different programming languages, tools,
and frameworks, including both Microsoft-specific and third-party software and
systems.
+
+Hop supports Azure through a number of metadata types:
+
+== Pipeline Transforms
+
+* xref:pipeline/transforms/azure-event-hubs-listener.adoc[Azure Event Hubs
Listener]: listens indefinitely to an Event Hub on the Microsoft Azure cloud
platform.
+* xref:pipeline/transforms/azure-event-hubs-writer.adoc[Azure Event Hubs
Writer]: allows you to write messages (events) to a streaming service bus
called Event Hubs on the Microsoft Azure cloud platform.
+
+== VFS
+
+Apache xref:vfs.adoc[VFS Support] in Hop allows you to directly read from a
multitude of file systems and protocols, including Azure:
+
+* xref:vfs/azure-blob-storage-vfs.adoc[Azure Blog Storage]: read from and
write data directly in Azure Blob Storage
\ No newline at end of file
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/technology/cassandra/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/technology/cassandra/index.adoc
new file mode 100644
index 0000000..a9db7e8
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/cassandra/index.adoc
@@ -0,0 +1,45 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /technology/cassandra/
+:language: en_US
+
+= Cassandra Technology
+
+From the https://cassandra.apache.org/[Apache Cassandra] website:
+
+The database is the right choice when you need scalability and high
availability without compromising performance.
http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html[Linear
scalability] and proven fault-tolerance on commodity hardware or cloud
infrastructure make it the perfect platform for mission-critical data.
Cassandra's support for replicating across multiple datacenters is
best-in-class, providing lower latency for your users and the peace of mind of
kno [...]
+
+From https://en.wikipedia.org/wiki/Apache_Cassandra[Wikipedia]:
+
+Apache Cassandra is a free and open-source, distributed, wide-column store,
NoSQL database management system designed to handle large amounts of data
across many commodity servers, providing high availability with no single point
of failure. Cassandra offers support for clusters spanning multiple
datacenters,[2] with asynchronous masterless replication allowing low latency
operations for all clients. Cassandra was designed to implement a combination
of Amazon's Dynamo distributed storage [...]
+
+== Cassandra Support in Hop
+
+Hop supports Cassandra in the following metadata objects:
+
+=== Metadata Types
+
+* xref:metadata-types/cassandra/cassandra-connection.adoc[Cassandra
Connection]: Create a connection to your Cassandra database cluster.
+
+=== Workflow Actions
+
+* xref:workflow/actions/cassandra-exec-cql.adoc[Cassandra Exec CQL]: Execute
Cassandra https://cassandra.apache.org/doc/latest/cql/[CQL]
+
+=== Pipeline Transforms
+
+* xref:pipeline/transforms/cassandra-input.adoc[Cassandra Input]: Reads from a
Cassandra cluster through a CQL query.
+* xref:pipeline/transforms/cassandra-output.adoc[Cassandra Output]: Write data
to a table in a Cassandra cluster.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc
b/docs/hop-user-manual/modules/ROOT/pages/technology/google/index.adoc
similarity index 51%
rename from
docs/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc
rename to docs/hop-user-manual/modules/ROOT/pages/technology/google/index.adoc
index 69c28be..4b17f97 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/google/index.adoc
@@ -23,19 +23,15 @@ under the License.
The Google technology stack is supported in Hop through a number of plugins.
We briefly touch upon them below.
-=== Google Analytics
+== Pipeline Transforms
-You can use the `Google Analytics` transform to get information from this
service.
+* xref:pipeline/transforms/googleanalytics.adoc[Google Analytics]: You can use
the `Google Analytics` transform to get information from this service.
+* xref:pipeline/transforms/googlesheetsinput.adoc[Google Sheets Input] and
xref:pipeline/transforms/googlesheetsoutput.adoc[Google Sheets Output]: It can
be very convenient to save data into a Google sheet or indeed to read from it.
You can do this using the `Google Sheets Input` and `Google Sheets Output`
transforms.
-== Google Sheets
+== VFS
-It can be very convenient to save data into a Google sheet or indeed to read
from it. You can do this using the `Google Sheets Input` and `Google Sheets
Output` transforms.
+Apache xref:vfs.adoc[VFS Support] in Hop allows you to directly read from a
multitude of file systems and protocols, including Google:
-== Google Drive
-
-Hop has a Virtual File System (VFS) plugin for Google Drive. For more
information see the VFS section of the documentation.
-
-== Google Cloud Storage
-
-Hop has a Virtual File System (VFS) plugin for Google Cloud Storage. For more
information see the VFS section of the documentation.
+* xref:vfs/google-drive-vfs.adoc[Google Drive]: read and write data directly
from and to Google Drive files and folders.
+* xref:vfs/google-cloud-storage-vfs.adoc[Google Cloud Storage]: read and write
data directly from and to files and folders in Google Cloud Storage buckets
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/neo4j-info.adoc
b/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/index.adoc
similarity index 58%
rename from
docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/neo4j-info.adoc
rename to docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/index.adoc
index e3c10a0..45a7d38 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/neo4j-info.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/index.adoc
@@ -25,10 +25,32 @@ Neo4j is an open source graph database which you can
download from **https://neo
You can use it to represent information with nodes and relationships in a
property graph. Neo4j doesn't use indexes which allows it to traverse large
graphs really quickly with so-called graph algorithms. For more information on
these unique graph algorithms see:
https://neo4j.com/docs/graph-data-science/current/algorithms/#algorithms[Neo4j
Graph Algorithms]
-== Execution lineage
+Hop supports Neo4j through the following metadata objects
+
+== Perspective: Execution lineage
You can use Neo4j to store logging and execution lineage of your workflows and
pipelines.
The way you do this is simply by setting the variable NEO_LOGGING_CONNECTION
to the name of the Neo4j Connection where you want the logging and lineage to
be written to.
The Neo4j plugin offers a separate perspective to query this logging and
lineage information. For example, it allows you to quickly jump to the place
where an error occurred. This neat trick is performed by asking the database
to find the shortest path between and execution node where an error occurred
and without children and the "grand parent" node. The path you get is the
exact path that was followed from for example the "grand parent" workflow to
the exact transform where an error [...]
+== Metadata Types
+
+* xref:metadata-types/neo4j/neo4j-connection.adoc[Neo4j Connection]: create or
manage a connection to a Neo4j database
+* xref:metadata-types/neo4j/neo4j-graphmodel.adoc[Neo4j Graph Model]: crearte
or manage a Neo4j (sub)graph model
+
+== Workflow Actions
+
+* xref:workflow/actions/neo4j-checkconnections.adoc[Neo4j Check Connection]:
check a connection to a Neo4j database
+* xref:workflow/actions/neo4j-cypherscript.adoc[Neo4j Cypher Script]: execute
a Cypher script or query
+
+== Pipeline Transforms
+
+* xref:pipeline/transforms/neo4j-cypher.adoc[Neo4j Cypher]: execute a Cypher
query on a Neo4j database using parameter info from input fields
+* xref:pipeline/transforms/neo4j-gencsv.adoc[Neo4j Generate CSVs]: Generate
CSV files for nodes and relationships to be used with neo4j-import
+* xref:pipeline/transforms/neo4j-getloginfo.adoc[Neo4j Get Log Info]: Queries
the Neo4j logging graph for execution info
+* xref:pipeline/transforms/neo4j-graphoutput.adoc[Neo4j Graph Output]: Writes
to a Neo4j graph using an input fields mapping.
+* xref:pipeline/transforms/neo4j-import.adoc[Neo4j Import]: Runs a Neo4j
import command using the provided CSV files
+* xref:pipeline/transforms/neo4j-output.adoc[Neo4j Output]: Write nodes and/or
relationships to a Neo4j graph
+* xref:pipeline/transforms/neo4j-split-graph[Neo4j Split Graph]: Splits the
nodes and relationships of a Neo4j graph
+
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
b/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
index 66fa753..91ce1f0 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
@@ -18,3 +18,10 @@ under the License.
= Technology
This section of the documentation contains technology-stack specific
information. The goal is to inform you about how Apache Hop offers support for
the technology in question. It also provides starting points for you if you
want to learn more about the technology.
+
+The technologies supported by Hop currently include:
+
+* xref:technology/azure/index.adoc[Azure]
+* xref:technology/cassandra/index.adoc[Apache Cassandra]
+* xref:technology/neo4j/index.adoc[Neo4j]
+* xref:technology/google/index.adoc[Google]
\ No newline at end of file
