This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-hop.git
The following commit(s) were added to refs/heads/master by this push:
new b46e4ca HOP-2328, HOP-2710, HOP-2712: token replacement, passwords,
samples, hop-search
new 22791e6 Merge pull request #772 from bamaer/HOP-2328
b46e4ca is described below
commit b46e4ca3285402368416821bb122aee7578d993d
Author: Bart Maertens <[email protected]>
AuthorDate: Wed Apr 28 12:58:25 2021 +0200
HOP-2328, HOP-2710, HOP-2712: token replacement, passwords, samples,
hop-search
---
docs/hop-user-manual/modules/ROOT/nav.adoc | 2 +-
.../modules/ROOT/pages/hop-tools/hop-encrypt.adoc | 64 ++++++++++++++++++++++
.../modules/ROOT/pages/hop-tools/hop-search.adoc | 62 ++++++++++++++++++++-
.../modules/ROOT/pages/hop-tools/index.adoc | 10 ++--
.../modules/ROOT/pages/password/passwords.adoc | 52 +++++++++++++++++-
.../pipeline/transforms/tokenreplacement.adoc | 51 +++++++++++++++++
.../modules/ROOT/pages/samples/dummy-sample.adoc | 21 -------
.../modules/ROOT/pages/samples/samples.adoc | 8 ++-
8 files changed, 240 insertions(+), 30 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc
b/docs/hop-user-manual/modules/ROOT/nav.adoc
index ef45290..68a290b 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -391,7 +391,6 @@ under the License.
//::=END AUTO GENERATED LINKS
** xref:plugins/external-plugins.adoc[External Plugins]
* xref:samples/samples.adoc[Samples]
-** xref:samples/dummy-sample.adoc[Dummy Sample]
* xref:technology/technology.adoc[Technology]
** xref:technology/azure/index.adoc[Azure]
** xref:technology/cassandra/index.adoc[Cassandra]
@@ -402,4 +401,5 @@ under the License.
** xref:hop-server/web-service.adoc[Web Service]
* xref:hop-tools/index.adoc[Hop Tools]
** xref:hop-tools/hop-conf/hop-conf.adoc[hop-conf]
+** xref:hop-tools/hop-encrypt.adoc[hop-encrypt]
** xref:hop-tools/hop-search.adoc[hop-search]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-encrypt.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-encrypt.adoc
new file mode 100644
index 0000000..76ea056
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-encrypt.adoc
@@ -0,0 +1,64 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+= Hop Encrypt
+
+Hop Encrypt is a command line tool to encrypt (obfuscate) passwords for use in
XML, password or Hop metadata files.
+
+== Usage
+
+[source, bash]
+----
+hop-encrypt usage:
+
+ encr <-hop|-server> <password>
+ Options:
+ -hop: generate an obfuscated or encrypted password
+ -server : generate an obfuscated password to include in the hop-server
password file 'pwd/hop.pwd'
+
+This command line tool obfuscates or encrypts a plain text password for use in
XML, password or metadata files.
+Make sure to also copy the password encryption prefix to indicate the
obfuscated nature of the password.
+Hop will then be able to make the distinction between regular plain text
passwords and obfuscated ones.
+----
+
+== Options
+
+[options="header", width="90%"]
+|===
+|Option|Description
+|<password>|The password to obfuscate
+|-hop|generate an obfuscated or encrypted password
+|-server|generate an obfuscated password to include in the hop-server password
file 'pwd/hop.pwd'
+|===
+
+== Examples
+
+=== Hop Example
+
+Encrypt (obfuscate) the password 'abcd' for use in Hop workflows, pipelines
and other metadata files.
+
+`./hop-encrypt.sh -hop abcd`
+
+Output: `Encrypted 2be98afc86aa7f2e4cb79ce10df90acde`
+
+
+=== Hop Server Example
+
+Encrypt (obfuscate) the password 'abcd' for use in Hop server.
+
+``./hop-encrypt.sh -hop abcd`
+
+Output: `OBF:1s3g1s3i1s3k1s3m`
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-search.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-search.adoc
index b53a12e..25e57bc 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-search.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-search.adoc
@@ -16,4 +16,64 @@ under the License.
////
= Hop Search
-coming soon
+Hop Search is a command line tool to search for metadata in Hop or a Hop
project.
+
+== Usage
+
+[source, bash]
+----
+Usage: <main class> [-hilx] [-e=<environmentOption>] [-j=<projectOption>]
+ <searchString>
+ <searchString> The string to search for
+ -e, --environment=<environmentOption>
+ The name of the lifecycle environment to use
+ -h, --help Displays this help message and quits.
+ -i, --case-insensitive Perform a case insensitive search
+ -j, --project=<projectOption>
+ The name of the project to use
+ -l, --print-locations Print which locations are being looked at
+ -x, --regular-expression The specified search string is a regular
expression
+./hop-search.sh 9.08s user 0.44s system 338% cpu 2.808 total
+----
+
+== Options
+
+[options="header", width="90%"]
+|===
+|Option|Description
+|<searchString>|The string to search for
+|-e|The name of the lifecycle environment to use
+|-h|Displays this help message and quits.
+|-i|Perform a case insensitive search
+|-j|The name of the project to use
+|-l|Print which locations are being looked at
+|-x|The specified search string is a regular expression
+|===
+
+== Examples
+
+Example command: search for the string 'json' (case insensitive) in the
'hop-samples' project.
+
+Command: `./hop-search.sh -j hop-samples -i json `
+
+Output:
+
+[source, bash]
+----
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : null : matching property value: Enhanced JSON -
Generate complex nested structure from table
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Final Complex Data structure : matching property
value: EnhancedJsonOutput
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Final Complex Data structure : matching property
value: Enhanced JSON Output
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Final Complex Data structure : pipeline transform
property : jsonBloc
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Final Complex Data structure : pipeline transform
property : jsonPrittified (Boolean)
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Final Complex Data structure : pipeline transform
property : isJSONFragment (Boolean)
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Final Complex Data structure : pipeline transform
property : isJSONFragment (Boolean)
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Final Complex Data structure : pipeline transform
property : isJSONFragment (Boolean)
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Intermediate Structures : matching property value:
EnhancedJsonOutput
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Intermediate Structures : matching property value:
Enhanced JSON Output
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Intermediate Structures : pipeline transform
property : jsonPrittified (Boolean)
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : Intermediate Structures : pipeline transform
property : isJSONFragment (Boolean)
+<PROJECT_PATH>/samples/transforms/Enhanced JSON - Generate complex nested
structure from table.hpl : null : matching property value: This sample gets a
table of values and generate a complex nested structure as output
+by looping over a predefined key. Second Enhanced JSON Transform als gets a
JSON fragment produced
+by furst Enhanced JSON Transform and use it as value of an attribute of the
resulting final
+JSON stream.
+----
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/index.adoc
index 3e22341..f9a91fe 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/index.adoc
@@ -21,9 +21,9 @@ Even though Hop developers will spend the majority of their
time in Hop Gui, Hop
This page provides an overview of the tools that are available in the platform.
* xref:hop-tools/hop-conf/hop-conf.adoc[Hop Conf]
-* Hop Encrypt
-* Hop Gui
-* Hop Run
+* xref:hop-tools/hop-encrypt.adoc[Hop Encrypt]
+* xref:hop-gui/[Hop Gui]
+* xref:hop-run/index.adoc[Hop Run]
* xref:hop-tools/hop-search.adoc[Hop Search]
-* Hop Server
-* Hop Translator
\ No newline at end of file
+* xref:hop-server/index.adoc[Hop Server]
+*
https://hop.apache.org/community/contribution-guides/translation-contribution-guide/[Hop
Translator]
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/password/passwords.adoc
b/docs/hop-user-manual/modules/ROOT/pages/password/passwords.adoc
index 4873491..c569b64 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/password/passwords.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/password/passwords.adoc
@@ -15,7 +15,57 @@ specific language governing permissions and limitations
under the License.
////
[[password-plugins]]
-= Password Plugins
+= Passwords and Password Plugins
+
+Hop provides password obfuscation as a way to prevent storing passwords as
clear text.
+
+Obfuscation is the deliberate act of creating source or machine code that is
difficult for humans to
understand[https://en.wikipedia.org/wiki/Obfuscation_(software)[1]].
+
+With obfuscated passwords, Hop provides a way to store passwords in a way the
makes them hard (but not impossible!) to read.
+Obfuscated passwords are read by Hop and are converted to the original
password as late as possible (e.g. to pass to a database when creating a
connection).
+
+== Password obfuscation in Hop
+
+Obfuscate passwords are stored in Hop metadata with the 'Encrypted' prefix.
The example below shows a database connection with an obfuscated password 'abcd.
+
+[source,json]
+----
+{
+ "rdbms": {
+ "POSTGRESQL": {
+ "databaseName": "DBNAME",
+ "pluginId": "POSTGRESQL",
+ "indexTablespace": "",
+ "dataTablespace": "",
+ "accessType": 0,
+ "hostname": "localhost",
+ "password": "Encrypted 2be98afc86aa7f2e4cb79ce10df90acde",
+ "port": "5432",
+ "pluginName": "PostgreSQL",
+ "servername": "",
+ "attributes": {},
+ "username": "dbuser"
+ }
+ },
+ "name": "DBCONN"
+}
+----
+
+== Password obfuscation in Hop Server
+
+Obfuscated passwords for Hop Server can be stored in the server's .pwd file.
+
+In the example below, Hop's default server hop.pwd contains the username
'cluster' and the default obfuscated passowrd 'cluster'
+, with the 'OBF:' prefix.
+
+[source,bash]
+cluster: OBF:1v8w1uh21z7k1ym71z7i1ugo1v9q,default
+
+== Password Encryption (Obfuscation) with Hop Encrypt
+
+Check xref:hop-tools/hop-encrypt.adoc[Hop Encrypt] for more information on how
to obfuscate or encrypt password for use in Hop or Hop Server.
+
+== Password plugins
Password handling in Hop can be implemented as plugins.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tokenreplacement.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tokenreplacement.adoc
index de07b82..531672d 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tokenreplacement.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/tokenreplacement.adoc
@@ -21,4 +21,55 @@ under the License.
== Description
+This transform replaces tokens in an input string or file. The transform can
then output this data either to a file or a field on the stream.
+
+A token contains a start string, a name, and an end string. For example
${my_token} could be a token. The start string, and end string are configurable
and can be any series of characters.
+
+When replacing tokens in a file, this transform reads the entire file. It does
not operate on a single line at a time. When replacing tokens in a file it is a
best practice to output to a file also to prevent Hop from having to read the
entire file into memory.
+
+
+
== Options
+
+=== Input Tab
+
+* Input Type - Where to read the field to do the token replacement from.
Either text, field, or file.
+* Input Type Text
+** Input Text - The text to token replace
+* Input Type Field
+** Input Field - The input field to token replace
+* Input Type File
+** Input filename - The name of the file to token replace
+** Filename is in field? - Is the name of the file to token replace in a field?
+** Input filename field - The field the name of the file is in.
+** Add Input filename to result? - Add the input filename(s) to the result
files list.
+
+=== Output Tab
+
+* Output Type - Where to put the token replaced string. Either field or file.
+* Output Type Field
+** Output field name - The name of the field to put the token replaced string
in.
+* Output Type File
+** Output filename - The name of the file to write to.
+** Filename is in field? - Is the name of the output file in a field?
+** Output filename field - The name of the field the output filename is in.
+** Append output file? - If the output file already exists should it be
appended to. If not checked, Pentaho will overwrite the file if it exists.
+** Create parent folder? - Should Pentaho create the parent folder?
+** Output format - The format of the new line delimiter for the output file.
+** Output encoding - The character encoding to use when writing the file.
+** Split every - Split the output file into a new file every n rows.
+** Include stepnr in filename? - Should the step number be included in the
output filename?
+** Include partition nr in filename? - Should the partition number be included
in the output filename?
+** Include date in filename? - Should the curren date be included in the
output filename?
+** Include time in filename? - Should the current time be included in the
output filename?
+** Specify date format? - Do you want to specify the date format to include in
the output filename?
+** Date time format - The date/time format to include in the output filename.
+** Add output filenames to result? - Add the output filename(s) to the result
files list.
+
+=== Tokens Tab
+
+* Token start string - The string that indicates the start of a token.
+* Token end string - The string that indicates the end of a token. Everything
between the token start string and the token end string is the token name.
+* Stream name - The name of the field on the stream containing the value to
replace the token with.
+* Token name - The name of the token to replace.
+* Get Fields button - Gets the list of input fields, and tries to map them to
an Avro field by an exact name match.
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/samples/dummy-sample.adoc
b/docs/hop-user-manual/modules/ROOT/pages/samples/dummy-sample.adoc
deleted file mode 100644
index 5b8d9b4..0000000
--- a/docs/hop-user-manual/modules/ROOT/pages/samples/dummy-sample.adoc
+++ /dev/null
@@ -1,21 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
-////
-[[Samples-DummySample]]
-= Dummy sample
-
-Coming soon
-
diff --git a/docs/hop-user-manual/modules/ROOT/pages/samples/samples.adoc
b/docs/hop-user-manual/modules/ROOT/pages/samples/samples.adoc
index 4d70037..f637674 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/samples/samples.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/samples/samples.adoc
@@ -18,5 +18,11 @@ under the License.
= Samples
-Coming soon.
+Hop comes with a set of samples for workflows, pipelines, actions, transforms
and other metadata objects.
+
+This `samples` project is included in your Hop installation in the
`config/projects/samples` folder. You can open individual sample workflows or
pipelines, or add a xref:projects/index.adoc[Hop project], pointing the
project's home directory to the `samples` folder.
+
+The samples project is work in progress. More samples will gradually be added
until feature completeness.
+
+The project's folder structure may change over time, but is intended to be
self-explanatory: you'll find sample workflows and pipelines along with samples
that have been organized in folder by action or transform name. For example,
the 'workflowexecutor' folder contains a workflow and pipeline that show how to
use the workflow executor transform (to execute workflows) in a pipeline.
