This is an automated email from the ASF dual-hosted git repository.

hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git


The following commit(s) were added to refs/heads/main by this push:
     new 3c9513615e Beam Flink pipeline engine page: Flink 1.19, one Flink job 
ID section, no Javadoc leftovers. #9048 (#9222)
3c9513615e is described below

commit 3c9513615ed2a743f3d0088ac9ee6e714913025e
Author: Adalennis <[email protected]>
AuthorDate: Tue Nov 3 09:01:52 2026 +0100

    Beam Flink pipeline engine page: Flink 1.19, one Flink job ID section, no 
Javadoc leftovers. #9048 (#9222)
---
 .../beam-flink-pipeline-engine.adoc                | 23 +++++++++++-----------
 1 file changed, 11 insertions(+), 12 deletions(-)

diff --git 
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
 
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
index c6610c74ba..dce4a9944f 100644
--- 
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
+++ 
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
@@ -22,7 +22,7 @@ under the License.
 
 == Beam Flink
 
-This runner allows you to run Hop pipelines on https://flink.apache.org[Apache 
Flink] version 1.13.
+This runner allows you to run Hop pipelines on https://flink.apache.org[Apache 
Flink] version 1.19. See 
xref:pipeline/beam/getting-started-with-beam.adoc[Getting started with Apache 
Beam] for the Beam and Flink versions of each Hop release.
 
 The Flink runner supports two modes: Local Direct Flink Runner and Flink 
Runner.
 
@@ -60,7 +60,7 @@ If this is set to false, the task will only decline a the 
checkpoint and continu
 A value of zero effectively disables fault tolerance.
 A value of -1 indicates that the system default value (as defined in the 
configuration) should be used.|-1
 |Execution retry delay (ms)|Sets the delay in milliseconds between executions.
-A value of {@code -1} indicates that the default value should be used.|-1
+A value of -1 indicates that the default value should be used.|-1
 |Object re-use|Sets the behavior of reusing objects.
 Enabling the object reuse mode will instruct the runtime to reuse user objects 
for better performance.|false
 |Disable metrics|Disable Beam metrics in Flink Runner|-1
@@ -76,7 +76,7 @@ See 
https://issues.apache.org/jira/browse/FLINK-2491[FLINK-2491] for progress on
 Interval value <= 0 disables the feature.|0
 |Auto watermark interval|The interval in milliseconds for automatic watermark 
emission.|
 |Batch execution mode|Flink mode for data exchange of batch pipelines.
-Reference {@link org.apache.flink.api.common.ExecutionMode}.
+The values are those of Flink's `org.apache.flink.api.common.ExecutionMode`.
 Set this to BATCH_FORCED if pipelines get blocked, see 
https://issues.apache.org/jira/browse/FLINK-10672[FLINK-10672]|P
 |User agent|A user agent string as per 
https://tools.ietf.org/html/rfc2616[RFC2616], describing the pipeline to 
external services.|
 |Temp location|Path for temporary files.|
@@ -88,9 +88,10 @@ Set this to BATCH_FORCED if pipelines get blocked, see 
https://issues.apache.org
 |Fat jar file location|Fat jar location.|
 |===
 
-== Job id
+== Flink job ID
 
-Beam does not return the Flink job id on the pipeline result. Hop chooses the 
id before submitting the job. After `prepareExecution()` it is available from 
the engine:
+Beam doesn't return the Flink job ID on the pipeline result, so Hop chooses 
the ID itself before it submits the job, when the Flink master is `[local]` or 
`host:port`.
+After `prepareExecution()` it is available from the engine:
 
 [source,java]
 ----
@@ -99,7 +100,11 @@ if (pipeline instanceof BeamFlinkPipelineEngine 
flinkEngine) {
 }
 ----
 
-The same value is written to the pipeline log. When the run configuration has 
an execution information location, it is stored as `flink.job.id` and shown on 
the execution viewer info tab. A master of `[collection]` does not submit a 
cluster job, so no id is assigned.
+The same value is written to the pipeline log. When the run configuration has 
an execution information location, it is stored as `flink.job.id` and shown on 
the execution viewer info tab.
+
+With a master of `[auto]`, which is also used when the master is empty, Flink 
takes its environment from `flink run` or the Kubernetes operator and ignores 
the ID Hop would set, and a master of `[collection]` doesn't submit a cluster 
job: in those cases Hop has no Flink job ID.
+Use the Beam job name instead: Hop derives it from the pipeline name, and you 
can find the job by that name in the Flink UI or with the Flink REST API.
+The Dataflow runner is different: Hop reports that job ID through the 
execution detail `dataflow.job.id`.
 
 == Running with Flink Run
 
@@ -170,9 +175,3 @@ bin/flink run \
 
 You can specify a master of `[local]` to run using an embedded Flink engine.
 It's primarily used for testing locally.
-
-== Flink job ID
-
-Hop does not show a Flink job ID after a Beam run. Beam 2.74 keeps the 
attached result as accumulators and a runtime, and the detached result keeps 
its job client in a private field with no accessor. There is no supported way 
to read the job ID from `PipelineResult`.
-
-The Beam job name is still available. Hop derives it from the pipeline name. 
On a Flink cluster, use that name in the Flink UI or the Flink REST API to find 
the job. The Dataflow runner is different: Hop reports that job ID through the 
execution detail `dataflow.job.id`.

Reply via email to