This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hop-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f3e6a5e2d6 Website updated to 45a48e9
f3e6a5e2d6 is described below
commit f3e6a5e2d6235ccf73cfb8cb7d0c3c5c879cd732
Author: jenkins <[email protected]>
AuthorDate: Mon Sep 21 18:37:04 2026 +0000
Website updated to 45a48e9
---
manual/next/hop-gui/canvas-mouse.html | 5 ++-
manual/next/hop-gui/perspective-configuration.html | 5 +++
.../pipeline/beam/getting-started-with-beam.html | 12 +++++-
manual/next/pipeline/transforms/groupby.html | 43 +++++++++++++++++-----
manual/next/pipeline/transforms/memgroupby.html | 18 +++++++++
manual/next/pipeline/transforms/sort.html | 23 ++++++++++++
.../pipeline/transforms/uniquerowsbyhashset.html | 21 +++++++++++
7 files changed, 115 insertions(+), 12 deletions(-)
diff --git a/manual/next/hop-gui/canvas-mouse.html
b/manual/next/hop-gui/canvas-mouse.html
index b2190b752a..f2efcdcacb 100644
--- a/manual/next/hop-gui/canvas-mouse.html
+++ b/manual/next/hop-gui/canvas-mouse.html
@@ -2771,7 +2771,7 @@
<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
-<p>The pipeline and workflow canvas react to where you click and with which
button. This page is the reference for all of it. Two options in the <a
href="perspective-configuration.html" class="xref page">configuration
perspective</a> (General tab, Canvas section) change the behaviour:</p>
+<p>The pipeline and workflow canvas react to where you click and with which
button. This page is the reference for all of it. Three options in the <a
href="perspective-configuration.html" class="xref page">configuration
perspective</a> (General tab, Canvas section) change the behaviour:</p>
</div>
<div class="ulist">
<ul>
@@ -2781,6 +2781,9 @@
<li>
<p><strong>Use right click for the context dialog</strong> — off by default.
When on, a left click never opens the <a href="hop-gui-popup-dialog.html"
class="xref page">context dialog</a>; a right click does. Everything else a
left click does (open properties from a name, press a badge, select, drag)
stays the same.</p>
</li>
+<li>
+<p><strong>Use menus instead of the context dialog</strong> — off by default.
When on, wherever this page says "context dialog" for a transform, action, hop
or note, a pop-up menu with a submenu per category opens instead. The empty
canvas keeps the context dialog while the <a
href="hop-gui-popup-dialog.html#palette-tree" class="xref page">palette
tree</a> is hidden (that is where you search for transforms and actions to
add); with the palette shown it gets a menu of the pipeline/workflo [...]
+</li>
</ul>
</div>
<div class="admonitionblock tip">
diff --git a/manual/next/hop-gui/perspective-configuration.html
b/manual/next/hop-gui/perspective-configuration.html
index ae217d9cdc..9e28b8d5f5 100644
--- a/manual/next/hop-gui/perspective-configuration.html
+++ b/manual/next/hop-gui/perspective-configuration.html
@@ -2939,6 +2939,11 @@ disabled (default): (single) click on a transform or
action icon to open the con
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
</tr>
<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use menus
instead of the context dialog</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Transforms, actions, hops and notes show their actions in a
pop-up menu with a submenu per category instead of the <a
href="hop-gui-popup-dialog.html" class="xref page">context dialog</a>. A click
on the empty canvas keeps the context dialog while the <a
href="hop-gui-popup-dialog.html#palette-tree" class="xref page">palette
tree</a> is hidden, since that is where new transforms and actions are searched
for; with the pal [...]
+<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
+</tr>
+<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Draw
border around names on canvas?</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">If this
option is enabled, a border will be drawn around the names of transforms and
actions on the canvas.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
diff --git a/manual/next/pipeline/beam/getting-started-with-beam.html
b/manual/next/pipeline/beam/getting-started-with-beam.html
index 7ff0c7a700..a0e347dd8f 100644
--- a/manual/next/pipeline/beam/getting-started-with-beam.html
+++ b/manual/next/pipeline/beam/getting-started-with-beam.html
@@ -3053,12 +3053,17 @@ When using the Beam engines it uses
<code>org.apache.beam.sdk.io.synthetic.Synth
<div class="sect2">
<h3 id="_unsupported_transforms"><a class="link"
href="#_unsupported_transforms">Unsupported transforms</a></h3>
<div class="paragraph">
-<p>A few transforms are simply not supported because we haven’t found a
good way to do this on Beam yet:</p>
+<p>A few transforms are simply not supported because we haven’t found a
good way to do this on Beam yet.
+They all depend on seeing the rows in a particular order, or on holding state
for the whole data set in one place, and a Beam pipeline re-shuffles rows
across workers to maximize parallelism.
+Each of these transforms would therefore only ever see the slice of the data
one worker happens to hold, and would quietly produce a wrong result, so the
Beam engines refuse them with an error instead:</p>
</div>
<div class="ulist">
<ul>
<li>
-<p><a href="../transforms/uniquerows.html" class="xref page">Unique
Rows</a></p>
+<p><a href="../transforms/uniquerows.html" class="xref page">Unique Rows</a> :
Use the <code>Memory Group By</code> instead</p>
+</li>
+<li>
+<p><a href="../transforms/uniquerowsbyhashset.html" class="xref page">Unique
Rows (HashSet)</a> : Use the <code>Memory Group By</code> instead</p>
</li>
<li>
<p><a href="../transforms/groupby.html" class="xref page">Group By</a> : Use
the <code>Memory Group By</code> instead</p>
@@ -3069,6 +3074,9 @@ When using the Beam engines it uses
<code>org.apache.beam.sdk.io.synthetic.Synth
</ul>
</div>
<div class="paragraph">
+<p>The Hop GUI knows about this list: when you set the canvas palette filter
to a Beam engine these transforms are hidden, and running a pipeline that still
contains one is blocked with the reason shown above.</p>
+</div>
+<div class="paragraph">
<p>The <a href="../transforms/rowdenormaliser.html" class="xref
page">Denormaliser</a> transform works technically correct on Apache Beam in
release 1.1.0 and later.
Even so you need to consider that the aggregation of the key-value pairs in
that transform (in the general case) only happens on a sub-set of the rows.
That is because in a Beam pipeline the order in which rows arrive is lost
because they are continuously re-shuffled to maximize parallelism.
diff --git a/manual/next/pipeline/transforms/groupby.html
b/manual/next/pipeline/transforms/groupby.html
index 0a1160a4a5..94def79fd0 100644
--- a/manual/next/pipeline/transforms/groupby.html
+++ b/manual/next/pipeline/transforms/groupby.html
@@ -2798,35 +2798,35 @@
Native Spark
<span class="visually-hidden">not supported</span>
</li>
- <li class="engine engine--supported" data-engine="beam-spark">
+ <li class="engine engine--unsupported" data-engine="beam-spark">
<svg class="engine__mark" width="12" height="12" viewBox="0 0 12 12"
aria-hidden="true" focusable="false">
- <path d="M2 6.4 4.6 9 10 3.2" fill="none" stroke="currentColor"
stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"/>
+ <path d="M3 3l6 6M9 3l-6 6" fill="none" stroke="currentColor"
stroke-width="1.8" stroke-linecap="round"/>
</svg>
Beam Spark
- <span class="visually-hidden">supported</span>
+ <span class="visually-hidden">not supported</span>
</li>
- <li class="engine engine--supported" data-engine="beam-flink">
+ <li class="engine engine--unsupported" data-engine="beam-flink">
<svg class="engine__mark" width="12" height="12" viewBox="0 0 12 12"
aria-hidden="true" focusable="false">
- <path d="M2 6.4 4.6 9 10 3.2" fill="none" stroke="currentColor"
stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"/>
+ <path d="M3 3l6 6M9 3l-6 6" fill="none" stroke="currentColor"
stroke-width="1.8" stroke-linecap="round"/>
</svg>
Beam Flink
- <span class="visually-hidden">supported</span>
+ <span class="visually-hidden">not supported</span>
</li>
- <li class="engine engine--supported" data-engine="beam-dataflow">
+ <li class="engine engine--unsupported" data-engine="beam-dataflow">
<svg class="engine__mark" width="12" height="12" viewBox="0 0 12 12"
aria-hidden="true" focusable="false">
- <path d="M2 6.4 4.6 9 10 3.2" fill="none" stroke="currentColor"
stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"/>
+ <path d="M3 3l6 6M9 3l-6 6" fill="none" stroke="currentColor"
stroke-width="1.8" stroke-linecap="round"/>
</svg>
Beam Dataflow
- <span class="visually-hidden">supported</span>
+ <span class="visually-hidden">not supported</span>
</li>
</ul>
<article class="doc">
@@ -2855,6 +2855,31 @@
<div class="paragraph">
<p>You can use the <a href="memgroupby.html" class="xref page">Memory Group
By</a> transform to handle non-sorted input.</p>
</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>Group By does not run on the distributed pipeline engines.
+Both the Beam engines and the native Spark engine refuse the transform and
stop the pipeline with an error.</p>
+</div>
+<div class="paragraph">
+<p>The reason is the sorted input this transform relies on.
+A distributed engine spreads the rows over many workers and re-shuffles them
to maximize parallelism, so a sort upstream only ever orders the rows one
worker happens to hold.
+The groups would be formed per worker and the aggregates would be silently
wrong.</p>
+</div>
+<div class="paragraph">
+<p>Use <a href="memgroupby.html" class="xref page">Memory Group By</a> instead.
+It needs no sorted input, and both engines map it onto their own shuffle
(<code>GroupByKey</code> on Beam) so the grouping covers the whole data set.
+See <a href="../beam/getting-started-with-beam.html#_unsupported_transforms"
class="xref page">Getting started with Beam</a> for the other transforms in
this situation.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
</div>
</div>
<div class="sect1">
diff --git a/manual/next/pipeline/transforms/memgroupby.html
b/manual/next/pipeline/transforms/memgroupby.html
index c971e9d119..4d8e0c94ba 100644
--- a/manual/next/pipeline/transforms/memgroupby.html
+++ b/manual/next/pipeline/transforms/memgroupby.html
@@ -2853,6 +2853,24 @@ When the number of rows is too large to fit into memory,
use a combination of <a
</tr>
</table>
</div>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>That fallback to <a href="sort.html" class="xref page">Sort Rows</a> and <a
href="groupby.html" class="xref page">Group By</a> only applies to the Hop
engines.
+Neither of those transforms runs on the Beam engines or on the native Spark
engine.</p>
+</div>
+<div class="paragraph">
+<p>On those engines Memory Group By is the aggregation to reach for: it is
translated into the engine’s own grouping operation
(<code>org.apache.beam.sdk.transforms.GroupByKey</code> on Beam, a Spark
shuffle on native Spark) rather than run as a Hop transform per worker, so
neither the sorting nor the memory limit described above applies.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
</div>
</div>
<div class="sect1">
diff --git a/manual/next/pipeline/transforms/sort.html
b/manual/next/pipeline/transforms/sort.html
index 51de0d80bd..3c99f1ae14 100644
--- a/manual/next/pipeline/transforms/sort.html
+++ b/manual/next/pipeline/transforms/sort.html
@@ -2852,6 +2852,29 @@ You use can use multiple copies of the Sort Rows
transform to speed up large sor
</tr>
</table>
</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>Sort Rows is not supported on the Beam engines.
+A Beam pipeline spreads rows over many workers and re-shuffles them to
maximize parallelism, so this transform can only order the rows a single worker
happens to hold at that moment, never the data set as a whole.</p>
+</div>
+<div class="paragraph">
+<p>If you need sorted input for a downstream transform, that transform will
not work on Beam either.
+Use an alternative that does not depend on row order, such as <a
href="memgroupby.html" class="xref page">Memory Group By</a> in place of <a
href="groupby.html" class="xref page">Group By</a>.
+See <a href="../beam/getting-started-with-beam.html#_unsupported_transforms"
class="xref page">Getting started with Beam</a>.</p>
+</div>
+<div class="paragraph">
+<p>The native Spark engine does support this transform: it is mapped onto a
Spark sort across the full data set instead of being run per partition.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
</div>
</div>
<div class="sect1">
diff --git a/manual/next/pipeline/transforms/uniquerowsbyhashset.html
b/manual/next/pipeline/transforms/uniquerowsbyhashset.html
index 31b1d94a0f..a13bab7c7a 100644
--- a/manual/next/pipeline/transforms/uniquerowsbyhashset.html
+++ b/manual/next/pipeline/transforms/uniquerowsbyhashset.html
@@ -2852,6 +2852,27 @@ When using this transform on larger sets of data there
is a risk of hash collisi
</tr>
</table>
</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>This transform is not supported on the distributed pipeline engines:
neither the Beam engines nor the native Spark engine.
+The hash set it uses to spot duplicates lives in the memory of a single
worker, and these engines spread the rows over many workers, so duplicates that
end up on different workers would survive.
+Both refuse the transform with an error rather than hand you that incomplete
result.</p>
+</div>
+<div class="paragraph">
+<p>Use a <a href="memgroupby.html" class="xref page">Memory Group By</a> to
get distinct rows instead: it is translated into the engine’s own
grouping operation and covers the whole data set.
+On the native Spark engine <a href="uniquerows.html" class="xref page">Unique
Rows</a> is an option too, because that engine maps it onto its own distinct
operation; on the Beam engines it is not, as it is refused there as well.
+See <a href="../beam/getting-started-with-beam.html#_unsupported_transforms"
class="xref page">Getting started with Beam</a>.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
</div>
</div>
<div class="sect1">
