This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 242ea805c75a docs(compaction): document the log compaction table
service (#19572)
242ea805c75a is described below
commit 242ea805c75ab9dc5ad0d55fa06df16b32e961ae
Author: deepakpanda93 <[email protected]>
AuthorDate: Tue Aug 11 16:57:25 2026 +0530
docs(compaction): document the log compaction table service (#19572)
* docs(compaction): document the log compaction table service
Log compaction shipped in 0.13.0 but has no user-facing prose anywhere in
the docs. It appears only as a one-line action definition in timeline.md,
passing mentions in hudi_stack.md, storage_layouts.md and metadata.md, three
auto-generated rows in configurations.md, and a paragraph in the 1.0 tech
spec. compaction.md, where a reader looking for compaction actually lands,
never mentions it. Verified across all eleven versioned copies: zero hits
for
"log compaction", "logcompaction" or "minor compaction" in either
compaction.md or write_operations.md.
Add a "Log Compaction" section to compaction.md covering what it does, that
it applies to Merge-on-Read tables, the logcompaction timeline action, the
three configs, and a link to RFC-48. Config names, defaults and sinceVersion
values were taken from HoodieCompactionConfig at release-1.2.0.
The issue asks for this on the write operations page, but log compaction is
a
table service rather than a hoodie.datasource.write.operation value, and
compaction itself is not listed among the operation types either. Putting it
beside UPSERT and INSERT would misfile it. Instead the substance lives in
compaction.md and the Write path step in write_operations.md gains a pointer
to it, so the page named in the issue does cover it.
Two things deliberately not claimed. The description of
hoodie.log.compaction.enable stays as narrow as the upstream config
documentation, which only states the metadata-table effect. And a note
records that log compaction has no SQL procedure, CLI command or standalone
utility, unlike compaction, which has all three -- confirmed against
procedures.md, cli.md and the release-1.2.0 file tree.
Applied to next and to every 1.x versioned copy. compaction.md differs in
each version, so the section was placed against its own surroundings per
file; 1.0.0 has no Related Resources heading, so it is appended there.
Closes #15779.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* docs(compaction): correct the scope of hoodie.log.compaction.enable
Review feedback on apache/hudi#19572 questioned two things I had flagged as
uncertain. Traced both through release-1.2.0.
hoodie.log.compaction.enable is not a user-facing switch at all, and listing
it beside hoodie.log.compaction.inline was misleading. HoodieWriteConfig
exposes it as isLogCompactionEnabled(), whose only caller is
HoodieBackedTableMetadataWriter, reading it off metadataWriteConfig rather
than the data table's config. That value is not the user's either:
HoodieMetadataWriteUtils.createMetadataWriteConfig sets it via
withLogCompactionEnabled(writeConfig.isLogCompactionEnabledOnMetadata()),
i.e. from hoodie.metadata.log.compaction.enable. Setting
hoodie.log.compaction.enable on a data table therefore does nothing: no
data-table code path reads it. The upstream config description, which
mentions only the metadata table, is accurate but easy to misread.
Dropped it from the config table. Documented the two configs that are real
user-facing knobs for metadata-table log compaction --
hoodie.metadata.log.compaction.enable and
hoodie.metadata.log.compaction.blocks.threshold -- and added a caution
explaining what hoodie.log.compaction.enable actually is, so anyone who
finds
it in the configuration reference is not left guessing.
On scheduling, the reviewer was right that "scheduled inline" was
incomplete.
There is no async log compaction service: AsyncCompactService and its Spark
subclasses call compactor.compact() and cover compaction only, and no
AsyncLogCompactService exists. But BaseHoodieTableServiceClient does expose
scheduleLogCompaction and logCompact publicly, so programmatic scheduling is
possible. The note now says inline is the only built-in path, repeats that
there is no async service, procedure, CLI command or utility, and points at
the client methods.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* docs(compaction): restrict log compaction section to 1.2.0 and fix the
claims
Committer review on apache/hudi#19572 (voonhous) found two blockers and
several precision problems. I verified each against the Java source before
acting; all of them hold.
Blocker 1, inline log compaction is broken before 1.0.2. WriteOperationType
serializes LOG_COMPACT as "logcompact" at 1.0.0 and 1.0.1 with no matching
case in fromValue, so it falls through to
default: throw new HoodieException("Invalid value of Type."). Confirmed by
reading the enum at each tag: 1.0.0 and 1.0.1 have LOG_COMPACT("logcompact")
and zero matching cases, 1.0.2 onward has LOG_COMPACT("log_compact") with
the
case present. That is HUDI-9220, fixed in 1.0.2. Documenting
hoodie.log.compaction.inline on 1.0.0 and 1.0.1 told users to switch on
something that throws.
Blocker 2, the read amplification claim only holds from 1.2.0. The
COMPACTED_BLOCK_TIMES skip lives in scanInternalV2, and scanInternal picks
the
path at runtime from a config defaulting to false. Confirmed: 1.0.0, 1.0.2
and
1.1.1 all still branch on enableOptimizedLogBlocksScan with both scan paths
present, while 1.2.0 has zero references to either branch, the skip having
become unconditional.
Both point the same way, so the section is now restricted to next and
version-1.2.0. The 1.0.0, 1.0.1, 1.0.2 and 1.1.1 copies of compaction.md and
write_operations.md are reverted to upstream, byte for byte. Reverting
write_operations.md there matters too: leaving the pointer behind would have
left a link to a #log-compaction anchor that no longer exists on those
pages.
Corrections to the remaining copies:
- The completed instant is a deltacommit, not a logcompaction.
ActiveTimelineV2.transitionLogCompactionInflightToComplete creates the
completed instant with DELTA_COMMIT_ACTION in every affected version, so
someone listing .hoodie for a completed logcompaction would conclude it
never ran. Now says logcompaction covers requested and inflight only.
- Added the cost side. The stitched block is appended, not a rewrite, so the
superseded blocks survive until the next full compaction and clean, and a
single run can emit more than one block. RFC-48 sells the feature on write
amplification; the section previously kept only the read benefit.
- Reworded the blocks.threshold row. It gates scheduling rather than the
trigger, none of its read sites are guarded by inlineLogCompactionEnabled,
and it fires on log file count as well as block count.
- Async does exist for the metadata table from 1.2.0, via
hoodie.metadata.table.service.manager.enabled with .actions=logcompaction,
so the flat "no async" claim was wrong for these two copies. Also noted
that
log compaction is not exposed through Flink options at all.
- Noted that a pending metadata table log compaction blocks metadata table
major compaction, per HUDI-7533.
- write_operations.md now says log compaction runs only when
hoodie.log.compaction.inline is enabled, rather than implying it is on by
default like compaction.
- Linked the LOGCOMPACTION bullet in timeline.md to the new section, which
previously dead-ended.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* docs(compaction): flag that MDT log compaction delegation needs a
deployed TSM
Follow-up review on apache/hudi#19572 (voonhous): the sentence about
delegating metadata table log compaction to an async pipeline was a footgun.
Verified every part of the report against release-1.2.0 before rewording.
Delegation stops execution and nothing in Hudi picks the work up.
HoodieBackedTableMetadataWriter still schedules the log compaction, then
logs "Skipping execution of log compaction on MDT as it is delegated to
table
service manager" instead of running it, and the pending-services path
likewise skips runAnyPendingLogCompactions on the same condition, so already
pending instants are not picked up on a later commit either.
There is also no executor to pick them up. Only
HoodieTableServiceManagerClient
and its config class exist in the tree; there is no server, and the config
class still carries "TODO: enable docs gen ... after TSM is landed
(HUDI-3475)". The default endpoint is http://localhost:9091.
Combined with the restriction already documented in the note below, the
outcome of following the old sentence without a table service manager
deployed is that logcompaction instants pile up on the metadata table and
metadata table compaction stops being scheduled as well.
Added a caution stating the prerequisite explicitly, and reworded the
paragraph from "delegated to an async pipeline" to "delegated to an external
table service manager", which is what it actually is.
Also noted that Flink does not need any of this. Checked the reviewer's
pointer rather than repeating it: CompactionUtil.scheduleMetadataCompaction
first tries scheduleCompaction and, failing that, falls through to
writeClient.scheduleLogCompaction when log compaction is enabled, so the
Flink compaction pipeline schedules and executes metadata table log
compaction itself with no table service manager involved.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* docs(compaction): correct the MDT table service manager guidance
The section recommended delegating metadata table log compaction to a
table service manager via hoodie.metadata.table.service.manager.actions
=logcompaction, and pointed at hoodie.table.service.manager.uris as the
endpoint to make reachable. Neither holds.
delegateToTableServiceManager in BaseHoodieTableServiceClient is the only
site that constructs a HoodieTableServiceManagerClient, so it is the only
place the uris config is read. It cannot dispatch a log compaction: the
guard asks isEnabledAndActionSupported(ActionType.compaction) regardless
of the service type passed in, and LOG_COMPACT has no case in the switch
below it, falling through to "Not supported delegate to table service
manager". The client exposes only executeCompaction, executeClean and
executeClustering.
shouldDelegateToTableServiceManager is a plain config predicate, so
setting the action still makes HoodieBackedTableMetadataWriter skip
inline execution. The setting therefore stands the writer down without
handing the work anywhere, leaving pending logcompaction instants on the
metadata table, which in turn blocks metadata table compaction from being
scheduled via validateCompactionScheduling.
Reword the paragraph to state that the writer executes it, keep the Flink
note, and replace the caution with one that says the action cannot be
used for this rather than that a table service manager must be deployed.
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Co-authored-by: voon <[email protected]>
---
website/docs/compaction.md | 58 ++++++++++++++++++++++
website/docs/timeline.md | 2 +-
website/docs/write_operations.md | 2 +-
website/versioned_docs/version-1.2.0/compaction.md | 58 ++++++++++++++++++++++
website/versioned_docs/version-1.2.0/timeline.md | 2 +-
.../version-1.2.0/write_operations.md | 2 +-
6 files changed, 120 insertions(+), 4 deletions(-)
diff --git a/website/docs/compaction.md b/website/docs/compaction.md
index 8f37c464a5bc..d3ab480ef876 100644
--- a/website/docs/compaction.md
+++ b/website/docs/compaction.md
@@ -283,6 +283,64 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group, so a file group that receives frequent
+small updates can be kept efficient without paying the cost of rewriting its
base file. Readers skip the log blocks that
+have already been stitched, so there are fewer blocks to merge on read.
+
+The stitched block is appended rather than replacing anything: the superseded
blocks stay on disk until the next full
+compaction and clean. Log compaction therefore trades extra storage for fewer
blocks to merge on read. A single run can
+also emit more than one block if the merged output exceeds the log block size.
+
+Log compaction is scheduled on the timeline as a `logcompaction` action, in
the `requested` and `inflight` states. On
+completion it is committed as a `deltacommit`, so there is no completed
`logcompaction` instant to look for.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once a file slice has at least this many log files, or at
least this many log blocks. Applies to any scheduling attempt, whether
triggered inline or programmatically.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
+
+:::note
+`hoodie.log.compaction.inline` is the only built-in way to schedule log
compaction on a data table. There is no
+asynchronous log compaction service for the data table, and no SQL procedure,
Hudi CLI command or standalone utility,
+unlike compaction. It is also not exposed through Flink options, so Flink
cannot schedule it for a data table.
+Programmatic scheduling is available through the write client's
`scheduleLogCompaction` and `logCompact` methods.
+:::
+
+The metadata table runs its own log compaction, controlled by a separate pair
of configs. It is executed by the writer
+that maintains the metadata table. On Flink no extra configuration is needed:
the compaction pipeline schedules and
+executes metadata table log compaction directly.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.metadata.log.compaction.enable` | `false` (Optional) | Enables log
compaction for the metadata table.<br /><br />`Config Param:
ENABLE_LOG_COMPACTION_ON_METADATA_TABLE`<br />`Since Version: 0.14.0` |
+| `hoodie.metadata.log.compaction.blocks.threshold` | `5` (Optional) | Number
of log blocks above which log compaction is scheduled on the metadata table.<br
/><br />`Config Param: LOG_COMPACT_BLOCKS_THRESHOLD`<br />`Since Version:
0.14.0` |
+
+:::caution
+`hoodie.metadata.table.service.manager.actions` lists `logcompaction` among
its supported actions, but it cannot be used
+to move metadata table log compaction off the writer. Setting it stops the
writer from executing log compaction inline
+without handing the work anywhere: Hudi has no dispatch path that delegates a
log compaction to a table service manager,
+and does not ship a table service manager in any case. Pending `logcompaction`
instants would then accumulate on the
+metadata table and, per the note below, metadata table compaction would stop
being scheduled as well.
+:::
+
+:::note
+While a log compaction is pending on the metadata table, major compaction of
the metadata table is not scheduled, since
+metadata partitions such as the record level index rely on processing-time
ordering. See
+[HUDI-7533](https://issues.apache.org/jira/browse/HUDI-7533).
+:::
+
+:::caution
+`hoodie.log.compaction.enable` also appears in the configuration reference,
but it is not a switch to set on your table.
+Hudi applies it internally to the metadata table's own write config, deriving
its value from
+`hoodie.metadata.log.compaction.enable`. Setting it on a data table has no
effect: use
+`hoodie.log.compaction.inline` for the data table, and
`hoodie.metadata.log.compaction.enable` for the metadata table.
+:::
+
+See [RFC-48](https://github.com/apache/hudi/blob/master/rfc/rfc-48/rfc-48.md)
for the design behind this table service.
+
## Related Resources
<h3>Blogs</h3>
diff --git a/website/docs/timeline.md b/website/docs/timeline.md
index 0da85c7fddff..8246d28f02be 100644
--- a/website/docs/timeline.md
+++ b/website/docs/timeline.md
@@ -30,7 +30,7 @@ Following are the valid action types.
like clustering.
* **CLEANS** - Table service that removes older file slices that are no longer
needed from the table, by deleting those files.
* **COMPACTION** - Table service to reconcile differential data between base
and delta files, by merging delta files into base files.
-* **LOGCOMPACTION** - Table service to merge multiple small log files into a
bigger log file in the same file slice.
+* **LOGCOMPACTION** - Table service to merge multiple small log files into a
bigger log file in the same file slice. See [log
compaction](compaction.md#log-compaction).
* **CLUSTERING** - Table service to rewrite existing file groups with
optimized sort order or storage layouts, as new file groups in the table.
* **INDEXING** - Table service to build an index of a requested type on a
column of the table, consistent with the state of the table at the completed
instant in face of ongoing writes.
* **ROLLBACK** - Indicates that an unsuccessful write operation was rolled
back, removing any partial/uncommitted files produced during such a write from
storage.
diff --git a/website/docs/write_operations.md b/website/docs/write_operations.md
index 6c8eb699ce1d..f2117b8a2124 100644
--- a/website/docs/write_operations.md
+++ b/website/docs/write_operations.md
@@ -128,7 +128,7 @@ The following is an inside look on the Hudi write path and
the sequence of event
6. Update [Index](indexes.md): Now that the write is performed, we will go
back and update the index.
7. Commit: Finally we commit all of these changes atomically. ([Post-commit
callback](platform_services_post_commit_callback.md) can be configured.)
8. [Clean](cleaning.md) (if needed): Following the commit, cleaning is invoked
if needed.
-9. [Compaction](compaction.md): If you are using MOR tables, compaction will
either run inline, or be scheduled asynchronously
+9. [Compaction](compaction.md): If you are using MOR tables, compaction will
either run inline, or be scheduled asynchronously. If
`hoodie.log.compaction.inline` is enabled, [log
compaction](compaction.md#log-compaction) may also run, stitching small log
blocks together without rewriting the base file.
10. Archive : Lastly, we perform an archival step which moves old
[timeline](timeline.md) items to an archive folder.
Here is a diagramatic representation of the flow.
diff --git a/website/versioned_docs/version-1.2.0/compaction.md
b/website/versioned_docs/version-1.2.0/compaction.md
index ada7a66e0ced..8579e11e44e7 100644
--- a/website/versioned_docs/version-1.2.0/compaction.md
+++ b/website/versioned_docs/version-1.2.0/compaction.md
@@ -283,6 +283,64 @@ Offline compaction needs to submit the Flink task on the
command line. The progr
The retry options (`--retry`, `--retry-last-failed-job`,
`--job-max-processing-time-ms`) are only effective in single-run mode, not in
service mode. Service mode has implicit retry semantics via its continuous
monitoring loop. A warning will be logged if `--retry-last-failed-job` is
enabled but `--job-max-processing-time-ms` is not set to a positive value.
:::
+## Log Compaction
+
+Log compaction is a minor compaction for Merge-on-Read tables. Rather than
merging log files into a new base file, it
+stitches several small log blocks into a larger one within the same file
group, so a file group that receives frequent
+small updates can be kept efficient without paying the cost of rewriting its
base file. Readers skip the log blocks that
+have already been stitched, so there are fewer blocks to merge on read.
+
+The stitched block is appended rather than replacing anything: the superseded
blocks stay on disk until the next full
+compaction and clean. Log compaction therefore trades extra storage for fewer
blocks to merge on read. A single run can
+also emit more than one block if the merged output exceeds the log block size.
+
+Log compaction is scheduled on the timeline as a `logcompaction` action, in
the `requested` and `inflight` states. On
+completion it is committed as a `deltacommit`, so there is no completed
`logcompaction` instant to look for.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.log.compaction.inline` | `false` (Optional) | When set to true, the
log compaction service is triggered after each write. While being simpler
operationally, this adds extra latency on the write path.<br /><br />`Config
Param: INLINE_LOG_COMPACT`<br />`Since Version: 0.13.0` |
+| `hoodie.log.compaction.blocks.threshold` | `5` (Optional) | Log compaction
can be scheduled once a file slice has at least this many log files, or at
least this many log blocks. Applies to any scheduling attempt, whether
triggered inline or programmatically.<br /><br />`Config Param:
LOG_COMPACTION_BLOCKS_THRESHOLD`<br />`Since Version: 0.13.0` |
+
+:::note
+`hoodie.log.compaction.inline` is the only built-in way to schedule log
compaction on a data table. There is no
+asynchronous log compaction service for the data table, and no SQL procedure,
Hudi CLI command or standalone utility,
+unlike compaction. It is also not exposed through Flink options, so Flink
cannot schedule it for a data table.
+Programmatic scheduling is available through the write client's
`scheduleLogCompaction` and `logCompact` methods.
+:::
+
+The metadata table runs its own log compaction, controlled by a separate pair
of configs. It is executed by the writer
+that maintains the metadata table. On Flink no extra configuration is needed:
the compaction pipeline schedules and
+executes metadata table log compaction directly.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.metadata.log.compaction.enable` | `false` (Optional) | Enables log
compaction for the metadata table.<br /><br />`Config Param:
ENABLE_LOG_COMPACTION_ON_METADATA_TABLE`<br />`Since Version: 0.14.0` |
+| `hoodie.metadata.log.compaction.blocks.threshold` | `5` (Optional) | Number
of log blocks above which log compaction is scheduled on the metadata table.<br
/><br />`Config Param: LOG_COMPACT_BLOCKS_THRESHOLD`<br />`Since Version:
0.14.0` |
+
+:::caution
+`hoodie.metadata.table.service.manager.actions` lists `logcompaction` among
its supported actions, but it cannot be used
+to move metadata table log compaction off the writer. Setting it stops the
writer from executing log compaction inline
+without handing the work anywhere: Hudi has no dispatch path that delegates a
log compaction to a table service manager,
+and does not ship a table service manager in any case. Pending `logcompaction`
instants would then accumulate on the
+metadata table and, per the note below, metadata table compaction would stop
being scheduled as well.
+:::
+
+:::note
+While a log compaction is pending on the metadata table, major compaction of
the metadata table is not scheduled, since
+metadata partitions such as the record level index rely on processing-time
ordering. See
+[HUDI-7533](https://issues.apache.org/jira/browse/HUDI-7533).
+:::
+
+:::caution
+`hoodie.log.compaction.enable` also appears in the configuration reference,
but it is not a switch to set on your table.
+Hudi applies it internally to the metadata table's own write config, deriving
its value from
+`hoodie.metadata.log.compaction.enable`. Setting it on a data table has no
effect: use
+`hoodie.log.compaction.inline` for the data table, and
`hoodie.metadata.log.compaction.enable` for the metadata table.
+:::
+
+See [RFC-48](https://github.com/apache/hudi/blob/master/rfc/rfc-48/rfc-48.md)
for the design behind this table service.
+
## Related Resources
<h3>Blogs</h3>
diff --git a/website/versioned_docs/version-1.2.0/timeline.md
b/website/versioned_docs/version-1.2.0/timeline.md
index 0da85c7fddff..8246d28f02be 100644
--- a/website/versioned_docs/version-1.2.0/timeline.md
+++ b/website/versioned_docs/version-1.2.0/timeline.md
@@ -30,7 +30,7 @@ Following are the valid action types.
like clustering.
* **CLEANS** - Table service that removes older file slices that are no longer
needed from the table, by deleting those files.
* **COMPACTION** - Table service to reconcile differential data between base
and delta files, by merging delta files into base files.
-* **LOGCOMPACTION** - Table service to merge multiple small log files into a
bigger log file in the same file slice.
+* **LOGCOMPACTION** - Table service to merge multiple small log files into a
bigger log file in the same file slice. See [log
compaction](compaction.md#log-compaction).
* **CLUSTERING** - Table service to rewrite existing file groups with
optimized sort order or storage layouts, as new file groups in the table.
* **INDEXING** - Table service to build an index of a requested type on a
column of the table, consistent with the state of the table at the completed
instant in face of ongoing writes.
* **ROLLBACK** - Indicates that an unsuccessful write operation was rolled
back, removing any partial/uncommitted files produced during such a write from
storage.
diff --git a/website/versioned_docs/version-1.2.0/write_operations.md
b/website/versioned_docs/version-1.2.0/write_operations.md
index 6c8eb699ce1d..f2117b8a2124 100644
--- a/website/versioned_docs/version-1.2.0/write_operations.md
+++ b/website/versioned_docs/version-1.2.0/write_operations.md
@@ -128,7 +128,7 @@ The following is an inside look on the Hudi write path and
the sequence of event
6. Update [Index](indexes.md): Now that the write is performed, we will go
back and update the index.
7. Commit: Finally we commit all of these changes atomically. ([Post-commit
callback](platform_services_post_commit_callback.md) can be configured.)
8. [Clean](cleaning.md) (if needed): Following the commit, cleaning is invoked
if needed.
-9. [Compaction](compaction.md): If you are using MOR tables, compaction will
either run inline, or be scheduled asynchronously
+9. [Compaction](compaction.md): If you are using MOR tables, compaction will
either run inline, or be scheduled asynchronously. If
`hoodie.log.compaction.inline` is enabled, [log
compaction](compaction.md#log-compaction) may also run, stitching small log
blocks together without rewriting the base file.
10. Archive : Lastly, we perform an archival step which moves old
[timeline](timeline.md) items to an archive folder.
Here is a diagramatic representation of the flow.
