This is an automated email from the ASF dual-hosted git repository.
fhueske pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 07b4ff26f54 [hotfix][docs] Document LATERAL SNAPSHOT load-phase
handling for static build sides
07b4ff26f54 is described below
commit 07b4ff26f545e0d9373949fd82573074ae442291
Author: Fabian Hueske <[email protected]>
AuthorDate: Wed Sep 16 14:43:46 2026 +0200
[hotfix][docs] Document LATERAL SNAPSHOT load-phase handling for static
build sides
Add sections for handling static and nearly static build sides and operator
metrics.
Generated-By: Claude Sonnet 5
---
.../content.zh/docs/sql/reference/queries/joins.md | 42 +++++++++++++++++++++
docs/content/docs/sql/reference/queries/joins.md | 43 ++++++++++++++++++++++
2 files changed, 85 insertions(+)
diff --git a/docs/content.zh/docs/sql/reference/queries/joins.md
b/docs/content.zh/docs/sql/reference/queries/joins.md
index eaacc8eee5e..1d83fd7e270 100644
--- a/docs/content.zh/docs/sql/reference/queries/joins.md
+++ b/docs/content.zh/docs/sql/reference/queries/joins.md
@@ -358,6 +358,25 @@ Build-side updates continue to be applied to the state and
become visible to sub
The load phase is what distinguishes a `LATERAL SNAPSHOT` join from the
[processing-time temporal join](#processing-time-temporal-join), which has been
disabled for Flink SQL. The processing-time temporal join starts joining
immediately at query start, so early probe-side rows are joined against
whatever build-side data happens to have been loaded so far, producing missing
or stale results that depend on the order in which the inputs are read. By
first loading the build side, a `LATERAL [...]
+**Completing the load phase for static or nearly-static build sides**
+
+The event-time gate (build-side watermark reaches `load_completed_time`) works
well when the build side keeps producing records and advancing its watermark.
+Build sides that are *static* (loaded once, then silent) or *nearly static*
(infrequent updates) are common for reference data such as currency rates,
product catalogs, or feature tables.
+Since watermarks only advance when new records with larger event times arrive,
such inputs may take a long time to (or never) produce a watermark that reaches
`load_completed_time`.
+This leaves the join **stuck in the load phase indefinitely**: probe-side rows
keep buffering, state keeps growing, and no results are ever emitted.
+
+Picking a `load_completed_time` that both sits above the build side's real
event times and is guaranteed to be reached by its watermark is difficult and
requires insight into the data: set it too low and the join flips before the
build side is fully loaded (probe rows are joined against an incomplete build
side); set it too high and the join never flips.
+
+If the build-side input is truly static and its source connector supports it,
configure the connector to run in **bounded mode**.
+When the source finishes reading, it advances the watermark to its maximum
value, which completes the load phase.
+
+If the build side is not truly static or the connector does not support a
bounded mode, set **`load_completed_idle_timeout`**.
+It is a processing-time (wall-clock) safety net that is independent of
event-time progress: the operator flips from load to join once the build-side
watermark has not advanced for the configured duration, so it triggers the flip
during a genuine stall — exactly the situation a static or idle build side ends
up in once it has been fully read.
+Choose the timeout larger than any expected pause in the build side's changes
during start-up — if it fires while the backlog is still being read, the join
flips against an incomplete build side.
+`load_completed_idle_timeout` is **not set by default**, so you must set it
explicitly if your build side may become static or idle during start-up.
+
+If a `LATERAL SNAPSHOT` join produces no output, first check whether it is
stuck in the load phase using the metrics below, then set
`load_completed_idle_timeout`.
+
**Syntax**
The build side is wrapped in the `SNAPSHOT` table function, which is called
with `LATERAL`. The outer (probe-side) table must be an append-only table.
@@ -395,6 +414,29 @@ Because rows are joined against the build-side state that
is current at processi
The build-side state grows with the number of distinct build-side keys, and
during the load phase the buffered probe-side rows add to the state footprint
until the operator transitions to the join phase. Use `state_ttl` to bound the
build-side state for keys that are no longer updated or joined. You can reduce
the amount of data that is buffered and processed during the load phase by
configuring scan start offsets on the build and probe-side inputs, for example
with a `scan.startup.*` [d [...]
+**Monitoring**
+
+The operator exposes metrics that help you understand which phase the join is
in and whether it is making progress. They are reported under the operator's
[metric group]({{< ref "docs/ops/metrics" >}}) alongside the standard operator
metrics.
+
+| Metric | Type | Description |
+| --- | --- | --- |
+| `currentPhase` | Gauge | The current phase: `0` = load phase, `1` = join
phase. A value stuck at `0` means the load phase has not completed. |
+| `currentBuildSideWatermark` | Gauge | The highest build-side watermark
observed (epoch milliseconds). In the load phase the flip happens once this
reaches `load_completed_time`. A value that stops advancing below
`load_completed_time` is the signature of a static or idle build side that
might not complete the load phase on its own. |
+| `currentProbeSideWatermark` | Gauge | The latest probe-side watermark
observed (epoch milliseconds). Probe-side watermarks are held back during the
load phase and forwarded during the join phase. |
+| `numProbeSideRecordsBuffered` | Gauge | The number of probe-side rows
currently buffered. This grows during the load phase and drops to zero shortly
after the flip to the join phase. |
+| `numStateTtlEvictions` | Counter | The number of build-side keys evicted
from state by `state_ttl`. Only advances during the join phase. |
+| `maxJoinFanOut` | Gauge | The largest number of joined rows emitted for a
single probe-side row. |
+| `avgJoinFanOut` | Gauge | The average number of joined rows emitted per
probe-side row. |
+| `numUnmatchedProbeRecords` | Counter | The number of probe-side rows that
found no build-side match. For an `INNER JOIN` these produce no output; for a
`LEFT JOIN` they are emitted null-padded. |
+| `numUnmatchedBuildRetractions` | Counter | The number of build-side
retractions (`-U`/`-D`) for a row that was not present in state. |
+
+To diagnose a join that emits no output, check `currentPhase` first. If it is
`0` while `numProbeSideRecordsBuffered` keeps growing, the join is stuck in the
load phase.
+Compare `currentBuildSideWatermark` against your `load_completed_time`: if the
build-side watermark has stopped advancing below it, the build side has become
static or idle and the event-time gate will never be crossed.
+Set `load_completed_idle_timeout` as described above to let the load phase
complete.
+
+On a query restart, the metric values are not restored: counters restart at
zero and the watermark and buffer gauges are re-populated as new records and
watermarks arrive.
+The phase, however, is restored from state.
+
**Batch mode**
In batch mode, a `LATERAL SNAPSHOT` join is executed as a regular (`INNER` or
`LEFT`) join between the probe side and the complete build side. Batch
execution reads the entire build side before joining, so there is no load phase
and no incremental state build-up. The streaming-specific arguments (`on_time`,
`load_completed_time`, `load_completed_idle_timeout`, and `state_ttl`) are
accepted but have no effect, and the build side does not need to declare a
watermark or provide a `on_time`.
diff --git a/docs/content/docs/sql/reference/queries/joins.md
b/docs/content/docs/sql/reference/queries/joins.md
index c815b0b585f..ba31ded4566 100644
--- a/docs/content/docs/sql/reference/queries/joins.md
+++ b/docs/content/docs/sql/reference/queries/joins.md
@@ -367,6 +367,25 @@ The load phase is what distinguishes a `LATERAL SNAPSHOT`
join from the [process
The processing-time temporal join starts joining immediately at query start,
so early probe-side rows are joined against whatever build-side data happens to
have been loaded so far, producing missing or stale results that depend on the
order in which the inputs are read.
By first loading the build side, a `LATERAL SNAPSHOT` join avoids this problem.
+**Completing the load phase for static or nearly-static build sides**
+
+The event-time gate (build-side watermark reaches `load_completed_time`) works
well when the build side keeps producing records and advancing its watermark.
+Build sides that are *static* (loaded once, then silent) or *nearly static*
(infrequent updates) are common for reference data such as currency rates,
product catalogs, or feature tables.
+Since watermarks only advance when new records with larger event times arrive,
such inputs may take a long time to (or never) produce a watermark that reaches
`load_completed_time`.
+This leaves the join **stuck in the load phase indefinitely**: probe-side rows
keep buffering, state keeps growing, and no results are ever emitted.
+
+Picking a `load_completed_time` that both sits above the build side's real
event times and is guaranteed to be reached by its watermark is difficult and
requires insight into the data: set it too low and the join flips before the
build side is fully loaded (probe rows are joined against an incomplete build
side); set it too high and the join never flips.
+
+If the build-side input is truly static and its source connector supports it,
configure the connector to run in **bounded mode**.
+When the source finishes reading, it advances the watermark to its maximum
value, which completes the load phase.
+
+If the build side is not truly static or the connector does not support a
bounded mode, set **`load_completed_idle_timeout`**.
+It is a processing-time (wall-clock) safety net that is independent of
event-time progress: the operator flips from load to join once the build-side
watermark has not advanced for the configured duration, so it triggers the flip
during a genuine stall — exactly the situation a static or idle build side ends
up in once it has been fully read.
+Choose the timeout larger than any expected pause in the build side's changes
during start-up — if it fires while the backlog is still being read, the join
flips against an incomplete build side.
+`load_completed_idle_timeout` is **not set by default**, so you must set it
explicitly if your build side may become static or idle during start-up.
+
+If a `LATERAL SNAPSHOT` join produces no output, first check whether it is
stuck in the load phase using the metrics below, then set
`load_completed_idle_timeout`.
+
**Syntax**
The build side is wrapped in the `SNAPSHOT` table function, which is called
with `LATERAL`. The outer (probe-side) table must be an append-only table.
@@ -404,6 +423,30 @@ Because rows are joined against the build-side state that
is current at processi
The build-side state grows with the number of distinct build-side keys, and
during the load phase the buffered probe-side rows add to the state footprint
until the operator transitions to the join phase. Use `state_ttl` to bound the
build-side state for keys that are no longer updated or joined. You can reduce
the amount of data that is buffered and processed during the load phase by
configuring scan start offsets on the build and probe-side inputs, for example
with a `scan.startup.*` [d [...]
+**Monitoring**
+
+The operator exposes metrics that help you understand which phase the join is
in and whether it is making progress.
+They are reported under the operator's [metric group]({{< ref
"docs/ops/metrics" >}}) alongside the standard operator metrics.
+
+| Metric | Type | Description
|
+| --- | ---
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `currentPhase` | Gauge | The current phase: `0` = load phase, `1` = join
phase. A value stuck at `0` means the load phase has not completed.
|
+| `currentBuildSideWatermark` | Gauge | The highest build-side watermark
observed (epoch milliseconds). In the load phase the flip happens once this
reaches `load_completed_time`. A value that stops advancing below
`load_completed_time` is the signature of a static or idle build side that
might not complete the load phase on its own. |
+| `currentProbeSideWatermark` | Gauge | The latest probe-side watermark
observed (epoch milliseconds). Probe-side watermarks are held back during the
load phase and forwarded during the join phase.
|
+| `numProbeSideRecordsBuffered` | Gauge | The number of probe-side rows
currently buffered. This grows during the load phase and drops to zero shortly
after the flip to the join phase.
|
+| `numStateTtlEvictions` | Counter | The number of build-side keys evicted
from state by `state_ttl`. Only advances during the join phase.
|
+| `maxJoinFanOut` | Gauge | The largest number of joined rows emitted for a
single probe-side row.
|
+| `avgJoinFanOut` | Gauge | The average number of joined rows emitted per
probe-side row.
|
+| `numUnmatchedProbeRecords` | Counter | The number of probe-side rows that
found no build-side match. For an `INNER JOIN` these produce no output; for a
`LEFT JOIN` they are emitted null-padded.
|
+| `numUnmatchedBuildRetractions` | Counter | The number of build-side
retractions (`-U`/`-D`) for a row that was not present in state.
|
+
+To diagnose a join that emits no output, check `currentPhase` first. If it is
`0` while `numProbeSideRecordsBuffered` keeps growing, the join is stuck in the
load phase.
+Compare `currentBuildSideWatermark` against your `load_completed_time`: if the
build-side watermark has stopped advancing below it, the build side has become
static or idle and the event-time gate will never be crossed.
+Set `load_completed_idle_timeout` as described above to let the load phase
complete.
+
+On a query restart, the metric values are not restored: counters restart at
zero and the watermark and buffer gauges are re-populated as new records and
watermarks arrive.
+The phase, however, is restored from state.
+
**Batch mode**
In batch mode, a `LATERAL SNAPSHOT` join is executed as a regular (`INNER` or
`LEFT`) join between the probe side and the complete build side. Batch
execution reads the entire build side before joining, so there is no load phase
and no incremental state build-up. The streaming-specific arguments (`on_time`,
`load_completed_time`, `load_completed_idle_timeout`, and `state_ttl`) are
accepted but have no effect, and the build side does not need to declare a
watermark or provide a `on_time`.
