This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 489330434a Issue #3185 : Add Fields, parameters, and variables
documentation page (#7960)
489330434a is described below
commit 489330434a175fe86dee3e1a8387b6a1ef8fe3e5
Author: Matt Casters <[email protected]>
AuthorDate: Sat Aug 15 10:19:55 2026 +0200
Issue #3185 : Add Fields, parameters, and variables documentation page
(#7960)
Keep the useful parts of the community draft, drop overlap with the
existing Variables reference, and link the comparison from Concepts,
Getting Started, and the executor / Set-Get Variables pages.
---
docs/hop-user-manual/modules/ROOT/nav.adoc | 1 +
.../modules/ROOT/pages/concepts.adoc | 2 +
.../ROOT/pages/fields-parameters-variables.adoc | 162 +++++++++++++++++++++
.../ROOT/pages/getting-started/hop-concepts.adoc | 5 +
.../ROOT/pages/getting-started/hop-next-steps.adoc | 1 +
.../pages/pipeline/transforms/getvariable.adoc | 2 +
.../pipeline/transforms/pipeline-executor.adoc | 15 +-
.../pages/pipeline/transforms/setvariable.adoc | 4 +-
.../pipeline/transforms/workflow-executor.adoc | 6 +-
.../pages/snippets/best-practices/variables.adoc | 1 +
.../hop-concepts/fields-parameters-variables.adoc | 20 +++
.../modules/ROOT/pages/variables.adoc | 42 ++----
.../ROOT/pages/workflow/actions/pipeline.adoc | 2 +
.../ROOT/pages/workflow/actions/workflow.adoc | 4 +-
14 files changed, 225 insertions(+), 42 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc
b/docs/hop-user-manual/modules/ROOT/nav.adoc
index 5f888f9711..fc4d99eb10 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -401,6 +401,7 @@ under the License.
*** xref:workflow/actions/xslt.adoc[XSL pipeline]
*** xref:workflow/actions/zipfile.adoc[Zip File]
//::=END AUTO GENERATED LINKS ACTIONS
+* xref:fields-parameters-variables.adoc[Fields, parameters, and variables]
* xref:variables.adoc[Variables]
* xref:vfs.adoc[Virtual File System]
** xref:vfs/aws-s3-vfs.adoc[Amazon Web Services S3]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/concepts.adoc
b/docs/hop-user-manual/modules/ROOT/pages/concepts.adoc
index b39d30ffd9..f14b4ecee5 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/concepts.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/concepts.adoc
@@ -39,6 +39,8 @@ include::snippets/hop-concepts/data-types.adoc[]
The following items are an alphabetically ordered list of concepts that are
used throughout Hop and will be mentioned at various locations in the Hop tools
and documentation.
+Fields, parameters, and variables::
+include::snippets/hop-concepts/fields-parameters-variables.adoc[]
Lazy Loading::
include::snippets/hop-concepts/lazy-loading.adoc[]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/fields-parameters-variables.adoc
b/docs/hop-user-manual/modules/ROOT/pages/fields-parameters-variables.adoc
new file mode 100644
index 0000000000..ccc152824e
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/fields-parameters-variables.adoc
@@ -0,0 +1,162 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[FieldsParametersVariables]]
+:imagesdir: ../assets/images
+:openvar: ${
+:closevar: }
+:description: Fields, parameters, and variables are the three ways Apache Hop
moves data and configuration. This page compares them, how they flow between
pipelines and workflows, and the pitfalls that come up when they share a name.
+
+= Fields, parameters, and variables
+
+Apache Hop moves information in three related but different ways:
+
+* *Fields* are the typed columns on a data row that flow between transforms.
+* *Parameters* are named inputs declared on a pipeline or workflow. They
become variables of the same name when that pipeline or workflow starts.
+* *Variables* are named string values in a scope (JVM, environment, project,
run configuration, workflow, or pipeline).
+
+They can look similar in the UI, especially when a dialog column is titled
*Parameters/Variables*, but they do not behave the same way.
+
+Fields, parameters, and variables are available downstream when they are in
scope. You can pass them again through as many nesting levels as you need. The
*Get Parameters* / *Get Variables* buttons only look one level up: they read
the pipeline or workflow the action or transform points at, not the whole
ancestor chain.
+
+== At a glance
+
+[cols="1,2,2,2", options="header"]
+|===
+| |Fields |Parameters |Variables
+
+|What they are
+|Named, typed columns on a data row
+|Named inputs declared on a pipeline or workflow
+|Named string values in a scope
+
+|Typical analogy
+|Columns in a result set
+|Function arguments
+|Environment or configuration
+
+|Type
+|Hop data types (String, Integer, Date, and so on)
+|Always a string; activated as a variable of the same name
+|Always a string
+
+|Scope
+|The stream between transforms, and result rows between a parent and a child
+|The pipeline or workflow that declared them
+|JVM, environment, project, run configuration, workflow, or pipeline
+
+|Direction
+|Flow with the data. A child can return rows to its parent
+|Passed downstream when the child starts
+|Inherited downstream. Can be set "upward" only with a wider scope
+
+|When they become visible
+|After a row has been produced
+|After the child pipeline or workflow has started
+|After the pipeline or workflow that should read them has started
+|===
+
+For the full variable reference (how to write `{openvar}NAME{closevar}`,
hierarchy, resolvers, and the built-in names), see
xref:variables.adoc[Variables].
+
+== Fields
+
+A field is a column on a data row. Transforms read incoming fields, add or
change fields, and send the row on.
+
+After a pipeline run, click the small table icon on a transform to preview the
cached rows and see the fields that were in scope. That preview is described in
xref:pipeline/run-preview-debug-pipeline.adoc[Run, Preview and Debug a
Pipeline].
+
+Fields do not travel as variables. To use a field value as a variable in a
child pipeline or workflow, map it on the *Parameters* tab of a
xref:pipeline/transforms/pipeline-executor.adoc[Pipeline Executor],
xref:pipeline/transforms/workflow-executor.adoc[Workflow Executor],
xref:workflow/actions/pipeline.adoc[Pipeline] action, or
xref:workflow/actions/workflow.adoc[Workflow] action. To turn a variable back
into a field, use xref:pipeline/transforms/getvariable.adoc[Get variables].
+
+A child can send fields back to its parent. Typical pattern:
+
+. In the child pipeline, end with
xref:pipeline/transforms/copyrowstoresult.adoc[Copy rows to result].
+. On the parent xref:pipeline/transforms/pipeline-executor.adoc[Pipeline
Executor], open the *Result rows* tab and declare the field names and types you
expect back.
+. Use the executor hop *This output will contain the result rows after
execution*.
+
+The samples project includes this pattern in
`samples/loops/pipeline-executor.hpl`.
+
+NOTE: Copy rows to result as a way to build loops from a workflow is
xref:how-to-guides/loops-in-apache-hop.adoc[deprecated]. Prefer a pipeline or
workflow executor. Returning result rows from an executor child is still the
supported way to send fields back to the parent.
+
+== Parameters
+
+Think of parameters as function arguments. Declaring them on a pipeline is
like writing `MyPipeline(parameter1, parameter2, ...)`. Each parameter has a
name, an optional default, and an optional description (pipeline or workflow
properties, *Parameters* tab).
+
+When the pipeline or workflow starts, each parameter is activated as a
variable of the same name. From that point you can write
`{openvar}MY_PARAM{closevar}` anywhere a variable is accepted. Use
xref:pipeline/transforms/getvariable.adoc[Get variables] only when you need
that value as a field in the stream.
+
+Rules that matter in practice:
+
+* Declare the parameter on the *receiving* pipeline or workflow. If a Pipeline
Executor (or Pipeline action) sets `CUSTOMER_ID`, the child must list
`CUSTOMER_ID` in its own properties. Leave the child's default empty when you
want the caller to supply the value.
+* Parameters travel *downstream* when the child is started. They are not sent
back to the parent. To return information, send *fields* (result rows, above)
or set a xref:pipeline/transforms/setvariable.adoc[variable] with a wider scope.
+* A value you pass on the caller's *Parameters* tab wins over the child's
default. A column titled *Parameters/Variables* means: set this name in the
child, overriding whatever was there before. That is how you override a child
default for one execution.
+* *Pass parameter values to sub pipeline* (Pipeline / Workflow action) and
*Inherit all variables from pipeline* (executors) control implicit inheritance.
If the child declares a parameter, also list that name on the caller's
*Parameters* tab when you want a specific value, even if a same-named variable
already exists in the parent. Otherwise the child's empty default can replace
the inherited value.
+* *Get Parameters* fills the tab from the child you selected. It does not walk
further up the call stack.
+
+See xref:pipeline/create-pipeline.adoc[Create a Pipeline] for the properties
dialog.
+
+== Variables
+
+Variables are more global than fields, and you choose the scope: the Java
virtual machine, the root workflow, the parent, and so on. They are always
strings.
+
+Two practical kinds:
+
+* *Configuration variables* are set before execution: system properties,
xref:projects/projects-environments.adoc[environment] files, project variables,
run configurations, or `hop-run -p`. Any downstream workflow or pipeline can
use `{openvar}myVariable{closevar}` directly. You do not need Get variables
unless the value must become a field.
+* *Runtime variables* are set while a pipeline or workflow is running, usually
with xref:pipeline/transforms/setvariable.adoc[Set Variables] or the
xref:workflow/actions/setvariables.adoc[Set Variables] action. A running
pipeline cannot pick up a new value that another transform in the *same*
pipeline just set: all transforms start together. Set the variable, then start
another pipeline or workflow (or let a Pipeline Executor start the child once
per row). A nested pipeline that was alre [...]
+
+A variable can be visible "upstream" only when it was set with a wider scope
than the child (for example *Valid in the parent* or *Valid in the root*).
+Set Variables also crawls nested execution engines so a value set in a child
can reach a mixed workflow/pipeline hierarchy when the chosen scope allows it.
+
+The places you can define variables, and which level inherits from which, are
listed in xref:variables.adoc#_how_can_i_define_variables[Variables: How can I
define variables] and xref:variables.adoc#_hierarchy[Variables: Hierarchy].
+
+== Passing values between pipelines and workflows
+
+[cols="1,3", options="header"]
+|===
+|You want to... |Do this
+
+|Pass a field into a child as a parameter
+|Map the field on the caller's *Parameters* tab. Declare the same name on the
child.
+
+|Pass a configuration value into everything
+|Set an environment, project, or run-configuration variable. Refer to
`{openvar}NAME{closevar}` in any descendant.
+
+|Use a variable as a field
+|xref:pipeline/transforms/getvariable.adoc[Get variables] in the pipeline that
needs the field.
+
+|Return fields from a child to the parent
+|Copy rows to result in the child, and *Result rows* on the parent executor.
+
+|Return a single value toward a parent
+|xref:pipeline/transforms/setvariable.adoc[Set Variables] with a parent,
grand-parent, or root scope.
+
+|Loop with a new parameter value each time
+|Pipeline Executor or Workflow Executor (once per row, or grouped). Each child
start sees the new parameter. See
xref:how-to-guides/loops-in-apache-hop.adoc[Loops in Apache Hop].
+|===
+
+== Common pitfalls
+
+* *Same name, different thing.* A field `customer_id`, a parameter
`CUSTOMER_ID`, and a variable `CUSTOMER_ID` can all exist at once. The field is
not the variable. Map it explicitly when you need the value in the other form.
+* *Declaring a parameter can hide an inherited variable.* If the child lists
`HOSTNAME` with an empty default and the caller does not pass `HOSTNAME`, Hop
may set the variable to empty instead of keeping a project or environment
value. Pass the parameter from the caller, or give the child a non-empty
default so an already-set variable of the same name is kept.
+* *Do not set and read a variable in the same pipeline.* Transforms run in
parallel. There is no guaranteed order, so the reader can see a previous run's
value or nothing. This is the most common variable surprise.
+* *A pipeline that is already running will not see new variables.* Start a new
pipeline (or a new executor iteration) after Set Variables. Parameters are the
alternative when you need a value at start-up of each child.
+* *Get Parameters only sees one level.* If the name is declared two levels up,
type it or copy it yourself.
+* *Workflows have no data stream.* You cannot read fields in a workflow action
the way a transform does. Pass parameters into the next pipeline, or use Get
variables there.
+
+== See also
+
+* xref:variables.adoc[Variables] — syntax, hierarchy, resolvers, and built-in
names
+* xref:pipeline/transforms/setvariable.adoc[Set Variables] and
xref:pipeline/transforms/getvariable.adoc[Get variables]
+* xref:pipeline/transforms/pipeline-executor.adoc[Pipeline Executor] and
xref:pipeline/transforms/workflow-executor.adoc[Workflow Executor]
+* xref:workflow/actions/pipeline.adoc[Pipeline] and
xref:workflow/actions/workflow.adoc[Workflow] actions
+* xref:how-to-guides/loops-in-apache-hop.adoc[Loops in Apache Hop]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-concepts.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-concepts.adoc
index 04002f3dca..ba0acffb3e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-concepts.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-concepts.adoc
@@ -40,6 +40,11 @@ All actions in a workflow run sequentially by default.
* **Projects** are logical collections of hop code and configuration.
**Environments** contain the environment-specific (e.g. dev, uat, prd)
metadata.
+* **Fields** are the typed columns on a data row.
+**Parameters** are named inputs on a pipeline or workflow.
+**Variables** are named string values in a scope.
+They look similar in the UI but they do not behave the same way — see
xref:fields-parameters-variables.adoc[Fields, parameters, and variables].
+
include::../snippets/hop-concepts/item-types.adoc[]
include::../snippets/hop-concepts/hop-projects-environments.adoc[]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
index f8ed66a5ed..d4ab225f79 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
@@ -29,6 +29,7 @@ Here are a couple of topics you may want to look into:
* xref:pipeline/pipelines.adoc[Pipelines] takes closer look at the various
aspects of creating and running pipelines, and contains the entire list of
transforms that are at your disposal
* xref:workflow/workflows.adoc[Workflows] takes a closer look at the various
aspects of create and running workflows, and contains the entire list of
actions that are at your disposal
+* xref:fields-parameters-variables.adoc[Fields, parameters, and variables]
compares the three ways Hop moves data and configuration
* xref:best-practices/index.adoc[Best Practices] covers a number of things you
might want to think about while using Apache Hop.
* xref:projects/index.adoc[Projects] explains how to work with projects and
environments
* xref:hop-server/deploying.adoc[Deploying Hop Server] covers the three common
ways to get a project onto a running server
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getvariable.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getvariable.adoc
index 39098ae21e..6b010b6171 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getvariable.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getvariable.adoc
@@ -45,6 +45,8 @@ TIP: If you need to refer to a previous pipeline’s data
row(s) fields, then u
== Usage
Note that workflow/environment variables are set only once. A pipeline needs
to start to get any new variables. A running or sequential or nested (nested
pipelines are technically the same pipeline) pipeline can't fetch new variable
values. A pipeline is considered started when a pipeline starts for every row
in a pipeline executor.
+See xref:fields-parameters-variables.adoc[Fields, parameters, and variables]
for when to use a field, a parameter, or a variable.
+
Refer to parameters/variables using the syntax: {openvar}myVariable{closevar},
for example from a previous pipeline.
*Two ways to pass fields, parameters to variables downstream:*
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
index e81c0b66ec..ed50603192 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
@@ -80,6 +80,7 @@ See also:
* The xref:workflow/actions/pipeline.adoc[Pipeline action] that executes a
pipeline from a workflow.
* The xref:pipeline/transforms/workflow-executor.adoc[Workflow Executor
transform] that executes a workflow from a pipeline.
* The xref:how-to-guides/loops-in-apache-hop.adoc[Loops in Apache Hop] how-to
guide.
+* xref:fields-parameters-variables.adoc[Fields, parameters, and variables] for
how values move between parent and child.
Samples (samples project):
@@ -111,15 +112,17 @@ The filename may contain variables (for example, you can
use the built-in Intern
In this tab you can specify which field to use to set a certain parameter or
variable value. If multiple rows are passed to the workflow, the first row is
taken to set the parameters or variables.
TIP: If you leave the "Inherit all variables from pipeline" option checked (it
is by default), all the variables defined in the current pipeline are passed to
the child pipeline.
-You can pass *parameters and variables* downstream only. You can’t pass
params/variables between pipelines unless they are started new. For example,
you can pass params/variables between pipelines, when each named pipeline is
started in a pipeline executor – once per row.
+Parameters and variables are passed *downstream* when the child pipeline
starts (once per executor iteration).
+They are not sent back to the parent.
+To return information, send *fields* as result rows, or set a variable with a
wider scope.
+See xref:fields-parameters-variables.adoc[Fields, parameters, and variables]
for the comparison and inheritance rules.
-Though you cannot pass parameters and variables upstream (in nested or
sequential pipelines) you can pass data rows back up a pipeline via the
following pattern. See project: samples/loops/pipeline-executor.hpl
+To return fields from the child (samples project:
`samples/loops/pipeline-executor.hpl`):
-* The *parent pipeline executor* specifies the row field name defined in the
child pipeline row under the tab “Result rows”. The output of the parent
pipeline executor is option “result rows after execution”.
+* On the *parent* executor, list the expected field names and types on the
*Result rows* tab, and use the hop *This output will contain the result rows
after execution*.
+* In the *child*, produce a row with those same names and types, and end with
xref:pipeline/transforms/copyrowstoresult.adoc[Copy rows to result].
-* *Child pipeline*: A data row is generated with the same field name and type
that is defined in the parent pipeline executor tab “Results rows” in the child
pipeline. The last transform of the child pipeline is “copy rows to result”.
-
-Remember that all parameters must be defined (in edit pipeline/workflow
properties) at least once in each pipeline or workflow.
+Declare every parameter you pass on the receiving pipeline (pipeline
properties → *Parameters*).
[options="header"]
|===
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvariable.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvariable.adoc
index 5abd78ce01..5c804ec6cc 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvariable.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/setvariable.adoc
@@ -46,7 +46,9 @@ Variables have no type in Hop (casted as strings). If working
with formulas for
Note that variables cannot be passed upstream between pipelines. Parameters
are best passed downstream to avoid threading issues. A nested pipeline is
technically the same pipeline, so variables are inherited in the initialization
phase.
-Though you cannot pass parameters and variables upstream (in nested or
sequential pipelines) you can pass data rows back up a pipeline via the
following pattern in the samples project: samples/loops/pipeline-executor.hpl
+You cannot pass parameters back to a parent.
+To return data, send fields as result rows (samples project:
`samples/loops/pipeline-executor.hpl`) or set a variable with a wider scope.
+See xref:fields-parameters-variables.adoc[Fields, parameters, and variables].
A variable can be set in one pipeline and be available in the next pipeline
(named pipeline) that is in the loop of a pipeline executor. If you are using
a pipeline executor child, the parent pipeline does not restart and does not
get any set variables. The new variable name to set in a child pipeline is
shown below in the second column.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc
index 2ca1dccfd6..eea35fd6ab 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc
@@ -44,7 +44,10 @@ The Workflow Executor transform execute a Hop workflow from
within a pipeline.
By default, the specified workflow will be executed once for each input row
(this can be changed in the Row Grouping tab).
-Fields from the data row(s) can be used to set parameters and variables and it
is passed to the workflow in the form of a result row. Remember that all
parameters must be defined at least once in each pipeline or workflow (Edit
pipeline/workflow properties). You are passing a parameter when you send a
field/parameter/variable to a workflow executor from the Parameters tab.
+Fields from the data row(s) can be used to set parameters and variables and
are passed to the workflow in the form of a result row.
+Declare every parameter you pass on the receiving workflow (workflow
properties → *Parameters*).
+You are passing a parameter when you send a field, parameter, or variable to a
workflow executor from the *Parameters* tab.
+See xref:fields-parameters-variables.adoc[Fields, parameters, and variables]
for how fields, parameters, and variables differ and how they flow between
parent and child.
You can also allow a group of records to be passed based on the value in a
field (when the value changes the workflow is executed) or on time. In these
cases, the first row of the group or rows is used to set parameters or
variables in the workflow.
@@ -56,6 +59,7 @@ See also:
* The xref:workflow/actions/pipeline.adoc[Pipeline action] that executes a
pipeline from a workflow.
* The xref:pipeline/transforms/pipeline-executor.adoc[Pipeline Executor
transform] that executes a sub-pipeline from a pipeline.
* The xref:how-to-guides/loops-in-apache-hop.adoc[Loops in Apache Hop] how-to
guide.
+* xref:fields-parameters-variables.adoc[Fields, parameters, and variables] for
how values move between parent and child.
Samples (samples project)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/variables.adoc
b/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/variables.adoc
index f1ddfc790d..3fa72d1811 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/variables.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/variables.adoc
@@ -21,6 +21,7 @@ under the License.
:imagesdir: ../../assets/images
Parameterize everything! xref:variables.adoc[Variables] provide an easy way to
avoid hard-coding all sorts of things in your system, environment or project.
+When you also use fields and pipeline/workflow parameters, see
xref:fields-parameters-variables.adoc[Fields, parameters, and variables] so the
three are not mixed up.
* Put environment specific settings in one or more
xref:projects/projects-environments.adoc[environment] configuration files. This
allows you to deploy your project to another environment (dev/uat/prod) without
changing your project, you'll only need to configure another set of
configuration files.
* When referencing file locations, prefer `{openvar}PROJECT_HOME{closevar}`
over expressions like `{openvar}Internal.Entry.Current.Directory{closevar}` or
`{openvar}Internal.Pipeline.Filename.Directory{closevar}`
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-concepts/fields-parameters-variables.adoc
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-concepts/fields-parameters-variables.adoc
new file mode 100644
index 0000000000..98c37cdba1
--- /dev/null
+++
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-concepts/fields-parameters-variables.adoc
@@ -0,0 +1,20 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+Fields are the typed columns on a data row.
+Parameters are named inputs declared on a pipeline or workflow; they become
variables of the same name when that file starts.
+Variables are named string values in a scope (environment, project, run
configuration, workflow, or pipeline).
+See xref:fields-parameters-variables.adoc[Fields, parameters, and variables]
for the comparison, how values flow, and the common pitfalls.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/variables.adoc
b/docs/hop-user-manual/modules/ROOT/pages/variables.adoc
index a14cd93af1..6c605f6fea 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/variables.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/variables.adoc
@@ -31,36 +31,8 @@ It's simply bad form to hardcode host names, user names,
passwords, directories
Variables allow your solutions to adapt to a changing environment.
If for example the database server is different when developing than it is
when running in production, you set it as a variable.
-TIP: Fields, parameters, and variables implicitly are available downstream if
in scope. You can pass them again any number of levels, but the "get
parameters/variables" button only retrieves from one level above.
-
-== Fields
-Fields are columns in data row(s) and are viewable in some transform field
textboxes and columns and you can see the fields in scope and its value when
looking at transform row results after executing a workflow/pipeline (click the
little grid icon on the bottom-right of a transform to see a preview of the
cached results). Field values can be passed upstream for example if you use a
pipeline executor and fill in the Result rows tab and use it in conjunction
with a “copy rows to result” t [...]
-
-== Parameters
-Think of parameters as function arguments, which turn into variables with the
same name. When adding parameters you are basically creating
MyPipeline(parameter1, parameter2,..). Parameters (e.g. in pipeline or workflow
properties) need to be declared at least once in every pipeline or workflow.
-
-For example, if you set parameters on a pipeline executor, the pipeline that
is being called must declare the same parameter names in its pipeline
properties (set to NULL if you want to inherit values). Parameters can not be
“sent” upstream, but you can use a Set variables transform, but they become
variables in the scope they are defined in.
-
-**Explicit vs Implicit:** If you set a parameter to a default value, it
becomes explicit (e.g. when editing a pipeline in pipeline properties), and it
will take precedence over the same named implicit variable (a passed-in
variable), but not take precedence over the same named explicit parameter. So
whenever you see a transform with a column title “Parameters/Variables” that
means an explicit parameter is sent to a function that will then set a variable
name overriding any previous set p [...]
-
-To change implicit parameter behaviour to explicit, you can also disable “pass
all parameters” on the pipeline action. Parameters are simply variables which
are explicitly defined in a workflow or pipeline to make them recognizable from
outside those objects. They can also have a description and a default value.
-
-You cannot combine implicit variable inheritance with explicit parameter
definitions. So, if you add parameters to the pipeline definition (or to a
pipeline executor) and you want it set, you must add it to the parameters tab
of the child pipeline action/transform even if the same variable already exists.
-
-**Advanced:** there are multiple layers in hop where you can set variables
that do or do not get overwritten downstream (Java -> hop environment ->
project -> run configuration -> workflow -> pipeline).
-
-== Variables
-Variables are more global and the scope can be targeted (entire Java VM,
grandparent workflow, etc.) whereas fields are the data flowing between the
transforms. Local variables can be set in a pipeline but should not be used in
the same pipeline as they are not thread-safe and can inherit a previous value.
It is better to send or return variables to another pipeline/workflow before
using them. Variables can be passed upstream if the variable was set to a
larger scope. E.g.: if a variabl [...]
-
-Think of variables in 2 scopes: runtime variables and environment/workflow
variables:
-
-* Runtime variables – Runtime variables depend on pipeline information to
generate, so they cannot be set beforehand, you need to declare those
differently to be able to use them.
-* Environment/workflow variables or parameters – Environment
variables/parameters are set once and used when needed in any downstream
workflow/pipeline and there is no need to use Get Variables , you can refer to
them directly like {openvar}myVariable{closevar} unless you need it in a
field/data stream.
- - E.g.: Define a parameter only once, even just in the Pipeline Executor (no
need to define in receiving pipeline)
-
-A pipeline needs to start to get new variables. A running or nested pipeline
can't fetch new variable values. A pipeline is considered started when a
pipeline starts for every row in a pipeline executor. An alternative is to use
parameters.
-
-
+Fields, parameters, and variables are related but they are not the same thing.
+For a comparison, how values flow between pipelines and workflows, and the
common pitfalls, see xref:fields-parameters-variables.adoc[Fields, parameters,
and variables].
== How do I use a variable?
@@ -174,11 +146,15 @@ This also means that's important to know where variables
can be set and used and
== Parameters
-Pipeline and workflows can (optionally) accept parameters.
+Pipelines and workflows can (optionally) accept parameters.
+
+Parameters are a special type of variable: they are declared on the pipeline
or workflow (name, optional default, optional description) so callers can pass
values in.
+When the pipeline or workflow starts, each parameter is activated as a
variable of the same name, available only in that execution.
-Workflow and pipeline parameters are similar, and are a special type of
variable that is only available within the current workflow or pipeline.
+Pass parameters from a parent through the *Parameters* tab of a
xref:workflow/actions/pipeline.adoc[Pipeline] action,
xref:workflow/actions/workflow.adoc[Workflow] action,
xref:pipeline/transforms/pipeline-executor.adoc[Pipeline Executor], or
xref:pipeline/transforms/workflow-executor.adoc[Workflow Executor].
+Declare the same names on the receiving pipeline or workflow.
-Workflow and pipeline parameters can have a default value and a description,
and can be passed on from workflows and pipelines to other workflows and
pipelines through a variety of workflow actions and pipeline transforms.
+How parameters differ from fields and from other variables, including
inheritance and override rules, is covered in
xref:fields-parameters-variables.adoc[Fields, parameters, and variables].
=== System properties
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc
index f74a585b35..917e006ca2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc
@@ -96,6 +96,8 @@ See xref:logging/logging-basics.adoc[Logging] for more
details.
=== Parameters tab
+See xref:fields-parameters-variables.adoc[Fields, parameters, and variables]
for how parameters relate to fields and variables.
+
*Pass params downstream*: On the Parameters tab, select the pipeline transform
checkbox to `Pass parameter values to sub pipeline`. The parameter must already
exist in the pipeline (in pipeline properties for example) or alternatively, on
the Parameters tab, you can specify new parameters.
The Parameters tab allows you to override existing parameter values or NULL
them by leaving the value empty.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc
index bb9f86a0b3..e33fb08572 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc
@@ -101,7 +101,9 @@ See also the logging window in Logging.
=== Parameters Tab
-Specify which parameters will be passed to the sub-workflow:
+Specify which parameters will be passed to the sub-workflow.
+
+See xref:fields-parameters-variables.adoc[Fields, parameters, and variables]
for how parameters relate to fields and variables.
[options="header",cols="1,3", width="90%"]
|===
