This is an automated email from the ASF dual-hosted git repository.
vinothchandar pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4ba23d1275fc docs(blog): explainer wave 1: open table format, upsert,
incremental ETL, ACID (#19269)
4ba23d1275fc is described below
commit 4ba23d1275fcdb0a7434a448b75be2e57e9e73b1
Author: vinoth chandar <[email protected]>
AuthorDate: Tue Jul 14 13:33:14 2026 -0700
docs(blog): explainer wave 1: open table format, upsert, incremental ETL,
ACID (#19269)
---
...18-hudi-incremental-processing-on-data-lakes.md | 4 +
.../2026-07-14-what-is-an-open-table-format.md | 113 +++++++++++++++
.../2026-07-15-what-is-upsert-on-a-data-lake.md | 116 +++++++++++++++
...07-16-what-is-incremental-etl-on-a-data-lake.md | 156 +++++++++++++++++++++
.../blog/2026-07-17-what-is-acid-on-a-data-lake.md | 128 +++++++++++++++++
website/static/llms.txt | 4 +
6 files changed, 521 insertions(+)
diff --git
a/website/blog/2020-08-18-hudi-incremental-processing-on-data-lakes.md
b/website/blog/2020-08-18-hudi-incremental-processing-on-data-lakes.md
index a39e3f612db1..5461c357c128 100644
--- a/website/blog/2020-08-18-hudi-incremental-processing-on-data-lakes.md
+++ b/website/blog/2020-08-18-hudi-incremental-processing-on-data-lakes.md
@@ -17,6 +17,10 @@ Hudi joined the Apache incubator for incubation in January
2019, and was promote
of Hudi to the data lake from the perspective of "incremental processing".
More information about Apache Hudi's framework functions, features, usage
scenarios, and
latest developments can be found at [QCon Global Software Development
Conference (Shanghai Station)
2020](https://qconplus.infoq.cn/2020/shanghai/presentation/2646).
+:::info
+For a current, ground-up explainer of this idea, see [What is Incremental ETL
on a Data Lake?](/blog/2026/07/16/what-is-incremental-etl-on-a-data-lake).
+:::
+
### NOTE: This article is a translation of the infoq.cn article, found
[here](https://www.infoq.cn/article/CAgIDpfJBVcJHKJLSbhe), with minor edits
<!--truncate-->
diff --git a/website/blog/2026-07-14-what-is-an-open-table-format.md
b/website/blog/2026-07-14-what-is-an-open-table-format.md
new file mode 100644
index 000000000000..601c6efdeb22
--- /dev/null
+++ b/website/blog/2026-07-14-what-is-an-open-table-format.md
@@ -0,0 +1,113 @@
+---
+title: "What is an Open Table Format?"
+excerpt: "What open table formats like Apache Iceberg, Apache Hudi and Delta
Lake actually are — table metadata plus a transaction protocol that turns files
on object storage into transactional tables."
+description: "An open table format is table metadata plus a transaction
protocol that turns files on cloud storage into database tables with ACID
commits, schema evolution and time travel — Apache Iceberg, Hudi, Delta Lake
and Paimon compared."
+authors: [vinoth-chandar]
+category: how-to
+image: /assets/images/blog/dlh_1200.png
+tags:
+- table format
+- data lakehouse
+- apache iceberg
+- delta lake
+- open architecture
+- apache xtable
+---
+
+An open table format is an openly governed specification with two inseparable
parts: **table metadata** — which files make up the table, what schema they
follow, what statistics they carry, at every version — and a **transaction
protocol** — the rules by which writers commit changes atomically and readers
see consistent snapshots. Together they turn a collection of files on cloud
object storage into a database-like table with ACID transactions, schema
evolution, and time travel. Apache H [...]
+Because the specification is open, any query engine — Apache Spark, Apache
Flink, Trino, and many others — can read and write the same table without going
through a proprietary system.
+
+Open table formats emerged to fix a specific gap. Cloud object stores such as
Amazon S3, Azure Blob Storage, and Google Cloud Storage made it cheap to store
virtually unlimited data, and columnar file formats made it efficient to scan.
But a pile of Parquet files is not a table: no atomic commits, no record-level
updates or deletes, no history to query, and no protection for a reader
scanning a directory mid-rewrite.
+
+Table formats close that gap by layering a transaction log and rich table
metadata over the files, and in doing so became the foundational component of
the [data lakehouse](/blog/2024/07/11/what-is-a-data-lakehouse) architecture —
warehouse-grade reliability and management over open data on the lake.
+
+This post covers the problem, the anatomy of a table format, the major open
projects, and how the pieces fit together.
+
+## The Problem Before Table Formats
+
+Until roughly a decade ago, the de-facto "table" abstraction on data lakes was
the Apache Hive model: a table was a directory, partitions were subdirectories,
and the Hive Metastore recorded the schema and the list of partitions. This
worked well enough on HDFS for append-only batch workloads, but it had
structural weaknesses that cloud object storage made acute.
+
+- **No atomicity.** A failed job left partial files behind, and readers could
not distinguish committed data from garbage; jobs resorted to fragile rename
tricks that object stores do not support atomically.
+- **No record-level mutations.** Updating or deleting a record meant rewriting
entire partitions, making change data capture (CDC) from operational databases
and GDPR deletions painful and expensive.
+- **Coarse, expensive planning.** Query planning relied on listing directories
— slow and costly on object stores — and on partition values as the only
pruning mechanism; statistics about the actual data in each file lived nowhere.
+- **No isolation or history.** A query could read an inconsistent mix of old
and new files while a pipeline rewrote the table, and with no table versions
there was no time travel, rollback, or incremental "what changed since
yesterday" queries.
+
+The fix was to stop treating a table as a directory listing and treat it
instead as versioned metadata plus a commit protocol: a transactionally
maintained record of the files that make up the table. Apache Hudi pioneered
this idea in production, built at Uber in 2016 for near-real-time ingestion
with updates, deletes, and incremental processing on one of the world's largest
data lakes, and described publicly in [Uber's engineering
blog](https://www.uber.com/blog/hoodie/) in early 2017. [...]
+
+## What a Table Format Actually Consists Of
+
+Strip away the branding and every open table format is a specification for a
handful of cooperating pieces of metadata stored alongside the data files.
+
+
+<p align = "center">Figure: The table format as a metadata layer above open
data file formats</p>
+
+**A log of changes to the table.** The heart of a table format is an ordered
record of every commit — data writes, deletes, schema changes, and maintenance
operations. In Hudi this is the [timeline](/docs/timeline); Delta Lake calls it
the transaction log; Iceberg maintains a chain of snapshots referenced from
table metadata files. Whatever the name, this log is the source of truth: a
change is part of the table if and only if it is committed to the log, and
commit is an atomic operation [...]
+
+**Snapshots and versioning.** Every commit produces a new consistent view of
the table, so readers can query it "as of" any retained version — enabling time
travel for debugging and audits, rollback of bad writes, and, where the format
tracks record-level changes, incremental queries that return only what changed
between two points in time.
+
+**File-level statistics and indexes.** Table formats track per-file and
per-column statistics — min/max values, null counts, record counts — that
engines use to skip files that cannot contain matching rows, often eliminating
most I/O for selective queries. Hudi generalizes this into a [multi-modal
indexing subsystem](/docs/indexes) stored in an internal [metadata
table](/docs/metadata), including column statistics, bloom filters, and
record-level indexes that also accelerate the write pa [...]
+
+**Schema tracking and evolution.** The format stores the table's schema and
defines rules for [evolving it](/docs/schema_evolution) — adding, renaming, or
promoting column types — without rewriting existing data files, plus schema
enforcement on write to catch pipeline breakages before bad data lands in the
table.
+
+**Concurrency control.** Finally, the specification defines how multiple
writers and background jobs coexist. All major formats give readers snapshot
isolation — a query sees a consistent version of the table regardless of
in-flight writes — while for concurrent writers they define mechanisms from
optimistic concurrency control to non-blocking approaches for jobs touching
disjoint or reconcilable data; Hudi's [concurrency
control](/docs/concurrency_control) documentation covers these tra [...]
+
+How data is physically organized under this metadata varies too. Hudi, for
example, defines two [table types](/docs/table_types): Copy-on-Write, which
rewrites base files on update for maximum read performance, and Merge-on-Read,
which writes changes to compact log files that are periodically merged — a
trade-off between write latency and read cost that other formats have since
adopted variants of.
+
+Note, though, that everything above is *specification* — passive metadata and
protocol rules. A table format defines what a valid table looks like; it does
not keep the table healthy. In database terms, that job belongs to a **storage
engine**: the machinery that compacts small files, clusters data, cleans up old
versions, and builds and maintains indexes. Every table on a lake needs this
running somewhere — built into the project, supplied by the query engine, or
bought from a vendor — [...]
+
+## The Major Open Table Formats
+
+Four open source projects dominate the space today. All are openly governed
(three at the Apache Software Foundation, one at the Linux Foundation), and all
deliver the core capabilities above; they differ in design center and scope.
+
+**[Apache Hudi](https://hudi.apache.org)** originated at Uber in 2016 for
fast, mutable ingestion — upserts, deletes, and incremental processing — on the
data lake. Hudi is accurately described as a table format plus a storage
engine: beyond the storage specification (timeline, file layout, metadata
table), it ships a writer runtime with pluggable indexing for fast updates, and
built-in [table services](/docs/hudi_stack) — compaction, clustering, cleaning,
and indexing — that run inline [...]
+
+**[Apache Iceberg](https://iceberg.apache.org)** originated at Netflix and was
donated to the ASF in 2018. Iceberg's design center is an engine-agnostic
specification for table metadata, with immutable snapshots, hidden partitioning
(partition values derived from column transforms), and well-defined metadata
evolution. It deliberately scopes itself to the format and libraries, leaving
table maintenance to engines and vendors, and has attracted the broadest
catalog and vendor ecosystem vi [...]
+
+**[Delta Lake](https://delta.io)** was created at Databricks and open-sourced
in 2019; it is governed under the Linux Foundation. Delta centers on a
JSON-based transaction log with periodic Parquet checkpoints, offering ACID
transactions, schema enforcement, and time travel. Like Hudi, it has always
been more than a format on its home platform, pairing the log with maintenance
operations such as optimize and vacuum. It is the native format of Databricks
and has first-class Spark integrat [...]
+
+**[Apache Paimon](https://paimon.apache.org)** is the newest of the four,
grown out of the Flink Table Store subproject and a top-level Apache project
since 2024. Paimon is narrower in scope than the other three: its design center
is LSM-tree (log-structured merge tree) storage on the lake, optimizing write
speed for high-frequency updates and changelog-producing tables consumed by
Apache Flink, while remaining readable by batch engines like Spark and Trino.
The trade-off is a classic on [...]
+
+An honest summary: the formats have converged substantially on core
capabilities — ACID commits, schema evolution, time travel, statistics-based
pruning — and continue to borrow ideas from one another. But write performance
and read performance remain distinct problems with different trade-offs, even
today: a design that makes writes cheap taxes readers unless compaction and
read-side indexing keep pace, while one that optimizes purely for scans makes
every update a rewrite. Because read [...]
+
+
+## How Query Engines Integrate
+
+A table format is only useful insofar as engines speak it. Integration happens
at two levels.
+
+At the **data plane**, each format ships libraries or specifications that
engines embed. Apache Spark and Apache Flink integrate with all four formats
for both reading and writing; distributed SQL engines such as Trino and Presto
ship native connectors. Cloud warehouses — Amazon Athena and Redshift, Google
BigQuery, Snowflake — can query open table formats directly, with capabilities
varying by format and vendor, and Python-native engines (DuckDB, Daft, Ray)
extend the same reach to ligh [...]
+
+At the **control plane**, a catalog tracks which tables exist and points
engines at each table's current metadata. Options range from the venerable Hive
Metastore and AWS Glue Data Catalog to newer catalogs like Apache Polaris,
Unity Catalog, and Apache Gravitino. The catalog is what turns `SELECT * FROM
orders` into "resolve `orders` to a storage location and metadata pointer, then
plan the scan." Hudi syncs its table metadata to these catalogs through catalog
sync tools, keeping tables [...]
+
+## Interoperability: You Don't Have to Pick One Forever
+
+A common source of anxiety is that choosing a table format is a one-way door.
Increasingly, it is not. [Apache XTable](https://xtable.apache.org)
(incubating) is an open source project that translates table metadata between
Hudi, Iceberg, and Delta Lake — in any direction — without copying or rewriting
the underlying data files. Because all three formats ultimately describe
Parquet files plus metadata, XTable can read one format's metadata and write
out the equivalent for another, as a o [...]
+
+This is how major vendors ship interoperability today. XTable was co-launched
by Microsoft, Google, and Onehouse before its donation to the ASF; [Microsoft
OneLake](https://xtable.apache.org/docs/fabric/) in Fabric uses it under the
hood to surface tables across Delta and Iceberg formats, and Onehouse uses it
to keep customer tables continuously queryable as Hudi, Iceberg, and Delta at
once. Databricks takes the same one-copy approach within Delta Lake through
UniForm, which generates Ic [...]
+
+## Table Format vs File Format vs Lakehouse
+
+These three terms get conflated constantly, so a short disambiguation is worth
the space.
+
+- A **file format** (Apache Parquet, ORC, Avro) defines how records are
encoded *inside a single file* — columnar layout, compression, per-file
statistics. It knows nothing about other files.
+- A **table format** (Hudi, Iceberg, Delta Lake, Paimon) defines how *many
files plus metadata* form a versioned, transactional table. It sits directly
above the file format.
+- A **data lakehouse** is the overall *architecture* that combines cloud
object storage, open file formats, a table format, catalogs, and query engines
into a warehouse-like platform on the lake. The table format is its
load-bearing component, alongside storage engines/table services, catalogs, and
compute.
+
+
+<p align = "center">Figure: Where the table format sits in a lakehouse
architecture</p>
+
+So Parquet is not a table format, and a table format alone is not a lakehouse
— it is the layer that makes one possible. For the architectural view, see the
companion post on [what a data lakehouse
is](/blog/2024/07/11/what-is-a-data-lakehouse).
+
+## Conclusion
+
+An open table format — table metadata plus a transaction protocol — turns
cheap, scalable object storage into a real analytical database substrate: files
become tables, jobs become transactions, and a directory of Parquet becomes
something you can update, evolve, audit, and query as of any point in time,
from any engine that implements the open specification. Apache Hudi proved the
idea in production first while building the transactional data lake; Apache
Iceberg, Delta Lake, and Apache [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {question: 'What is an open table format?', answer: 'An open table format is
an openly governed specification combining table metadata (which files form the
table, their schema and statistics, at every version) with a transaction
protocol (how writers commit atomically and readers see consistent snapshots).
Together they turn files on cloud object storage into a database-like table
with ACID transactions, schema evolution, and time travel. Apache Hudi, Apache
Iceberg, Delta Lake, and A [...]
+ {question: 'Is Parquet an open table format?', answer: 'No. Apache Parquet
is an open file format: it defines how records are encoded inside a single
file. A table format like Apache Hudi or Apache Iceberg sits above Parquet and
defines how many files plus transactional metadata form a versioned table.'},
+ {question: 'What is the difference between a table format and a file
format?', answer: 'A file format defines the byte layout of one file, including
encoding, compression, and per-file statistics. A table format defines how a
collection of such files behaves as a single transactional table, tracking
committed files, schema, versions, and statistics across the whole table. They
are complementary layers, not alternatives.'},
+ {question: 'Which open table format should I use?', answer: 'It depends on
your workloads, and note that write and read performance involve different
trade-offs. Hudi is strongest for mutable, incremental ingestion at high
volume, with built-in indexing for fast writes and automated compaction and
clustering to keep reads fast; Iceberg emphasizes an engine-neutral
specification with broad catalog support; Delta Lake offers deep Spark and
Databricks integration; Paimon targets LSM-based [...]
+ {question: 'Do open table formats support ACID transactions?', answer: 'Yes.
All major open table formats provide atomic commits and snapshot isolation for
readers, so queries always see a consistent version of the table. They differ
in how they handle concurrent writers, using techniques such as optimistic
concurrency control and non-blocking concurrency mechanisms.'},
+ {question: 'Is Apache Hudi just a table format?', answer: 'No. Hudi includes
an open table format, but pairs it with a built-in storage engine: a writer
runtime with pluggable indexing plus table services such as compaction,
clustering, and cleaning that keep tables optimized automatically. Hudi was
also the first open table format in production, built at Uber in 2016 for its
transactional data lake.'},
+]} />
diff --git a/website/blog/2026-07-15-what-is-upsert-on-a-data-lake.md
b/website/blog/2026-07-15-what-is-upsert-on-a-data-lake.md
new file mode 100644
index 000000000000..0c3851ddaf19
--- /dev/null
+++ b/website/blog/2026-07-15-what-is-upsert-on-a-data-lake.md
@@ -0,0 +1,116 @@
+---
+title: "What is Upsert on a Data Lake?"
+excerpt: "An upsert updates a record if it exists and inserts it if it does
not. This post explains why that was historically hard on data lakes, and how
open table formats like Apache Hudi make it fast."
+description: "What an upsert is, why record updates were historically hard on
data lakes, and how table formats like Apache Hudi implement fast, indexed
upserts."
+authors: [vinoth-chandar]
+category: how-to
+image:
/assets/images/blog/2023-05-10-top-3-things-you-can-do-to-get-fast-upsert-performance-in-apache-hudi.png
+tags:
+- upsert
+- dml
+- indexing
+- data lakehouse
+- cow
+- mor
+---
+
+An upsert is a write operation that updates a record if it already exists and
inserts it if it does not — a single atomic operation that combines *update*
and *insert*, keyed on a unique record identifier. Databases have offered this
for decades (`MERGE` in SQL, `ON CONFLICT DO UPDATE` in PostgreSQL, `replace
into` in MySQL). On a data lake, however, upserts were effectively impossible
for most of the technology's history, and making them fast is still what
separates data lake storage sy [...]
+
+## Why Data Lakes Couldn't Update Records
+
+Data lakes are built on files — typically columnar formats like Apache Parquet
or ORC — sitting on distributed file systems or cloud object storage such as
Amazon S3. Two properties of this foundation make record-level updates hard.
+
+First, the files themselves are immutable. A Parquet file is written once,
with column chunks, compression, and statistics computed over the whole file;
there is no way to reach into it and change one row in place. Second, the
storage layer underneath offers no transactional semantics across files. Early
cloud object stores were even only eventually consistent, so a reader listing a
directory mid-write could see a partial, corrupt view of the table. There were
no primary keys, no notion [...]
+
+The Hive-era workaround was coarse-grained rewriting: partition the table
(say, by day), and when any record in a partition changed, recompute and
overwrite the *entire partition*. Pipelines re-read hours or days of data to
apply what might be a handful of changed rows, then swapped directories and
hoped no query was reading at that moment. This pattern wasted enormous
compute, delayed data freshness from minutes to hours or days, and provided no
isolation guarantees. It is precisely the [...]
+
+## How Open Table Formats Make Upserts Possible
+
+Open table formats — [Apache Hudi](https://hudi.apache.org), Apache Iceberg,
and Delta Lake — introduced a transactional metadata layer on top of immutable
files, turning a directory of Parquet files into a proper table. That metadata
layer is what makes upserts possible, through a few key mechanisms (covered
more broadly in [What is a Data
Lakehouse?](/blog/2024/07/11/what-is-a-data-lakehouse)):
+
+- **Atomic commits.** Every write produces new file versions, and a commit
protocol atomically publishes them. In Hudi, each write is an action on a
[timeline](/docs/timeline); queries only ever see fully committed data, so a
failed or in-flight write is invisible to readers.
+- **Snapshot isolation and versioning.** Because old file versions are
retained until cleaned, readers see a consistent snapshot while writers produce
the next one. Updating a record no longer risks corrupting a running query.
+- **Record keys.** Hudi brings the database notion of a primary key to the
lake: every record has a key (record key, plus optionally a partition path)
that uniquely identifies it. This is the precondition for "update if exists" to
even be well defined.
+- **Fine-grained file management.** Instead of rewriting a whole partition,
the system can identify exactly which files contain the affected records and
rewrite — or log changes against — only those.
+
+With these pieces in place, an upsert on a data lake becomes what it is in a
database: hand the system a batch of records, and it transactionally reconciles
them against what is already stored. In Hudi, `upsert` is the [default write
operation](/docs/write_operations#upsert): incoming records are tagged as
inserts or updates via an index lookup, sized into files, written, and
committed atomically. The target table never shows duplicates.
+
+## The Anatomy of an Upsert
+
+Understanding the cost of an upsert requires walking through what actually
happens. Hudi's [write path](/docs/write_operations#write-path) breaks it into
distinct steps.
+
+**1. Key definition and deduplication.** Each incoming record carries a record
key. Since an input batch may itself contain multiple versions of the same key
(common with change streams), records are first combined by key, keeping the
winner according to configured ordering fields.
+
+**2. Index lookup.** The system must answer: *does this key already exist, and
if so, in which file?* Hudi maintains a mapping from each record key to a *file
group* — the unit of storage that holds all versions of a set of records. The
[index lookup](/docs/indexes) tags each incoming record as an update (routed to
its existing file group) or an insert (assigned to a new or under-sized file
group). This step is the heart of the upsert, and — as the next section shows —
the main determina [...]
+
+**3. Merge and write.** Tagged records are then written, and here the [table
type](/docs/table_types) determines the strategy:
+
+- **Copy-on-Write (CoW)** rewrites the affected base files: the writer reads
the current base file of each touched file group, merges in the updates, and
writes a new versioned base file. Queries stay simple and fast — they read pure
columnar data with zero merge overhead — but write amplification is high:
updating even a few records in a file means rewriting the whole file. CoW suits
read-heavy tables with moderate update rates.
+- **Merge-on-Read (MoR)** instead appends the changed records to compact delta
log files attached to each file group, deferring the expensive base-file
rewrite. Writes become lightweight and fast, enabling commits every few
minutes; queries merge the logs with base files at read time, and a background
[compaction](/docs/compaction) process periodically folds logs into new base
files to restore pure-columnar read performance. MoR suits high-frequency
updates and streaming ingestion, tradi [...]
+
+The trade-off is summarized well by the [comparison
table](/docs/table_types#comparison) in the Hudi docs: CoW pays `O(file groups
written)` in write amplification for zero read amplification; MoR pays
`O(records changed)` on both sides.
+
+**4. Commit.** Finally, the write — new base files, new log blocks, and index
updates — is committed atomically to the timeline, at which point queries can
see it.
+
+## Why Indexing Is the Difference Between Minutes and Hours
+
+Step 2 above deserves its own section, because it is where naive upsert
implementations fall down. Without an index, the only way to find which files
contain the incoming keys is to *join the input batch against the entire table*
— reading key columns from every file, every time you write. For a large table
receiving a small trickle of updates, this means the cost of each write is
proportional to the size of the table, not the size of the change.
+
+
+<p align = "center">Figure: Merge cost for updates (dark blue) against base
files (light blue), with and without an index</p>
+
+Hudi treats [indexing](/docs/indexes) as an integral part of its storage
engine and offers several index types tuned to different workloads:
+
+- **Bloom index**: stores bloom filters and key ranges (in file footers, or
centrally in the metadata table) so most files can be pruned without reading
data. Works very well when keys have some ordering — e.g., event tables with
timestamp-prefixed keys — where range pruning eliminates most candidate files.
+- **Simple index**: a lean join against keys extracted from existing files; a
better fit for random-update workloads like dimension tables, where bloom
filters would return true for nearly every file anyway.
+- **Bucket index**: hashes keys directly to file groups, eliminating the
lookup entirely at the cost of a fixed (or consistent-hashed) bucket layout.
+- **[Record Level Index](/blog/2023/11/01/record-level-index)**: a scalable,
exact key-to-file-group mapping stored in Hudi's metadata table, sharded by
hash. Instead of pruning candidate files probabilistically, it answers the
location question with a point lookup, delivering order-of-magnitude speedups
on large deployments where index lookup dominates write latency.
+
+The practical impact is dramatic. In a table with tens of thousands of files,
an untagged upsert must consider all of them; a well-chosen index narrows the
work to only the file groups that actually contain updated records.
Merge-on-Read benefits doubly: the index bounds how many change records any
base file must be merged against at query time, not just at write time. For
concrete tuning guidance, see [Top 3 Things You Can Do to Get Fast Upsert
Performance in Apache Hudi](/blog/2023/05/ [...]
+
+## What Upserts Unlock
+
+Fast record-level mutation is not a niche feature; it changes what a data lake
can be used for.
+
+- **Change data capture (CDC) replication.** Streaming a database's change log
(via Debezium, AWS DMS, or similar) into the lake requires applying inserts,
updates, and deletes in order, continuously. Upserts make it possible to
maintain a query-ready mirror of an operational database with minute-level
freshness, instead of nightly full dumps.
+- **Compliance deletes (GDPR/CCPA).** "Delete this user's data" is a targeted
mutation across potentially every partition of a table. With keyed
[deletes](/docs/write_operations#delete) — implemented on the same indexed path
as upserts — this becomes a routine operation rather than a table-rewriting
project.
+- **Deduplication.** Event pipelines deliver duplicates; upserting by event
key means the table enforces uniqueness on write, so consumers never need to
deduplicate at query time.
+- **Mutable fact and dimension tables.** Late-arriving corrections, order
status changes, transaction settlements, slowly changing dimensions — all
become incremental updates against the affected records, rather than periodic
partition rewrites.
+- **Incremental pipelines.** Because upserts record exactly what changed and
when, downstream jobs can consume just the changes via [incremental
queries](/docs/table_types#query-types), replacing full-table rescans with
streaming-style processing.
+
+## How Hudi, Iceberg, and Delta Lake Handle Mutation
+
+All three major open table formats support upserts today, but with different
designs and different cost profiles. Delta Lake and Iceberg expose mutation
primarily through `MERGE INTO`: the engine joins the source batch against the
target table to find affected files, then either rewrites them (copy-on-write)
or, in newer versions, writes delete files/deletion vectors that mark rows as
removed (merge-on-read style reads). Neither maintains a persistent
record-level index, so locating affe [...]
+
+## Conclusion
+
+An upsert — update if the record exists, insert if it does not — is the
operation that turns a data lake from an append-only archive into a live,
continuously updated foundation for analytics. It was historically out of reach
because lakes stored immutable files with no transactions, keys, or
record-level metadata, forcing wasteful partition rewrites. Open table formats
supplied the transactional layer; what determines real-world performance is
everything layered on top: how records are [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {
+ question: 'What is the difference between upsert and merge?',
+ answer: 'They achieve similar outcomes through different interfaces. MERGE
is a SQL statement where you write an explicit join condition and specify what
happens on match or no match, giving full control including conditional logic.
Upsert is a keyed write operation: the system uses the table\'s record key to
decide automatically whether each incoming record is an update or an insert. In
Apache Hudi, upsert is the default write operation and uses an index to locate
existing records, [...]
+ },
+ {
+ question: 'Can you update a record in a Parquet file?',
+ answer: 'Not in place. Parquet files are immutable once written, so there
is no way to modify a single row inside one. Table formats work around this by
rewriting the affected file with the change applied, or by logging the change
separately and merging it with the base file when the data is read. Either way,
the update is expressed as new files plus a metadata commit, never as an
in-place edit.',
+ },
+ {
+ question: 'Is upsert slow on a data lake?',
+ answer: 'It depends almost entirely on how the system finds existing
records and how it applies changes. Without an index, every upsert must scan or
join against the whole table to locate matching records, so write cost grows
with table size. With an index such as Hudi\'s bloom index or record level
index, the write only touches file groups that actually contain updated
records. Choosing Merge-on-Read further reduces write latency by logging
changes instead of rewriting columnar file [...]
+ },
+ {
+ question: 'What is the difference between upsert and insert in Apache
Hudi?',
+ answer: 'Upsert performs an index lookup to tag each incoming record as an
update or an insert, guaranteeing the table never shows duplicate keys. Insert
skips the index lookup entirely, which makes it faster but allows duplicates
unless the input is pre-deduplicated. Upsert is recommended for change data
capture and other update-heavy workloads, while insert suits append-mostly data
that can tolerate or separately handle duplicates.',
+ },
+ {
+ question: 'How do GDPR deletes work on a data lake?',
+ answer: 'A GDPR or CCPA delete request means removing specific records,
identified by key, from potentially every partition of a table. Table formats
implement this on the same path as upserts: an index lookup finds the files
containing the keys, and the delete is applied either by rewriting those files
or by logging delete markers that compaction later reconciles. Hudi supports
both soft deletes, which null out non-key fields, and hard deletes, which
remove the records entirely.',
+ },
+ {
+ question: 'Do Apache Iceberg and Delta Lake support upserts?',
+ answer: 'Yes. Both support row-level mutation, primarily through the SQL
MERGE INTO statement, with copy-on-write file rewrites or merge-on-read style
delete files and deletion vectors. The main design difference from Apache Hudi
is that they do not maintain a persistent record-level index, so finding
affected files relies on joining the source data against the table with
statistics-based pruning. Hudi instead defines record keys as a table-level
concept and maintains indexes that ma [...]
+ },
+]} />
diff --git a/website/blog/2026-07-16-what-is-incremental-etl-on-a-data-lake.md
b/website/blog/2026-07-16-what-is-incremental-etl-on-a-data-lake.md
new file mode 100644
index 000000000000..c3811c6e6923
--- /dev/null
+++ b/website/blog/2026-07-16-what-is-incremental-etl-on-a-data-lake.md
@@ -0,0 +1,156 @@
+---
+title: "What is Incremental ETL on a Data Lake?"
+excerpt: "Incremental ETL is a pipeline pattern that processes only the data
that changed since the last run. This post explains how it works, what the
storage layer must provide, and how it compares to batch and stream processing."
+description: "Incremental ETL processes only changed data instead of
recomputing entire tables. Learn how it works on a data lake and how Apache
Hudi enables it."
+authors: [vinoth-chandar]
+category: how-to
+image: /assets/images/blog/incr-processing/image7.png
+tags:
+- incremental processing
+- etl
+- streaming
+- data lakehouse
+- cdc
+---
+
+Incremental ETL is a pipeline pattern that processes only the data that
changed since the last run, instead of recomputing entire tables or partitions
on every execution. Each run of an incremental pipeline consumes a delta — the
records inserted, updated, or deleted after a known checkpoint — applies its
transformation logic to just that delta, and merges the results into the target
table. The pipeline's work is proportional to how much the data changed, not
how large the tables have grown.
+
+That distinction matters because most data pipelines today still run as
full-recompute batch jobs: every night (or every hour), they re-scan source
tables end to end and rewrite the derived tables downstream, even when only a
tiny fraction of the rows actually changed. As data volumes grow, that
recomputation gets slower and more expensive at exactly the moment businesses
are asking for fresher data.
+
+Incremental ETL occupies the middle ground between classic batch processing
and full stream processing. It keeps the familiar economics and tooling of
batch — scheduled jobs, SQL or DataFrame transformations, cheap [data
lakehouse](/blog/2024/07/11/what-is-a-data-lakehouse) storage — while borrowing
the core idea of stream processing: treat data as a continuous flow of changes,
and process each change roughly once.
+
+## The Problem with Full Recomputation
+
+Consider a typical medallion-style warehouse or lakehouse: raw data lands in
bronze tables, gets cleaned into silver tables, and is aggregated into gold
tables for reporting. In a full-recompute design, each layer is rebuilt by
scanning its inputs wholesale. Three problems compound as the tables grow:
+
+- **Cost.** A daily job that scans a multi-terabyte fact table to pick up one
day's worth of changes does orders of magnitude more I/O and compute than the
size of the change justifies. Joins are the worst offenders: rewriting a large
joined table means re-shuffling both sides in full, every run. Organizations
routinely spend a large share of their data platform budget on pipelines that
mostly reprocess data they already processed yesterday.
+
+- **Latency.** If rebuilding a table takes four hours, the data in it can
never be fresher than four hours plus the scheduling interval. Teams respond by
running the job less often (making data staler) or throwing more compute at it
(making it more expensive). Neither addresses the root cause: the work grows
with table size rather than change size.
+
+- **Missed SLAs and cascading delays.** Derived pipelines form DAGs. When an
upstream full-recompute job runs long, every downstream job either starts late
or reads stale inputs. Late-arriving data makes this worse: when records for an
old partition show up, the whole partition — and every downstream partition
derived from it — must be recomputed, silently blowing through freshness SLAs.
+
+Stream processing solves the latency problem but replaces batch's cost profile
with a different one: always-on clusters, state stores to manage, and a second
storage system (typically a log store like Kafka) holding another copy of the
data. For the large class of pipelines where "fresh within a few minutes" is
sufficient, that operational overhead buys latency headroom no one uses.
+
+## What Makes ETL "Incremental"
+
+An ETL pipeline is incremental when both its input and its output are
expressed as changes:
+
+1. **Change streams in.** Instead of scanning the full source table, the
pipeline asks the storage layer a question a plain file listing cannot answer:
*"give me all records that changed since commit X."* The result is a delta —
new inserts, the latest values of updated records, and deletes — along with a
new checkpoint to resume from next run.
+
+2. **Incremental writes out.** Instead of rewriting the target table, the
pipeline applies its transformed delta using record-level operations — upserts
and deletes — so only the affected file groups are rewritten. Crucially, this
makes the *output* table itself an incremental source: the next pipeline in the
DAG can consume its changes the same way.
+
+Because deltas flow in and deltas flow out, the pattern composes. A chain of
incremental pipelines behaves like a topology of stream processors — each stage
consuming a change stream and emitting one — except the "streams" are ordinary
tables on cheap cloud storage, queryable by any engine at any time. This is the
[duality between change logs and
tables](/blog/2020/08/18/hudi-incremental-processing-on-data-lakes) that stream
processing literature has long described, realized directly on [...]
+
+The efficiency win is easy to see with numbers. If a 10 TB fact table receives
50 GB of changes per hour, an hourly incremental pipeline processes 50 GB per
run instead of 10 TB — a 200x reduction in data scanned, while *improving*
freshness from daily to hourly.
+
+## What the Storage Layer Must Provide
+
+Incremental ETL is not primarily a compute-engine feature; it is a storage
contract. Plain files in cloud storage — even columnar formats like Parquet —
cannot tell a consumer what changed, and cannot accept record-level updates
without rewriting whole files or partitions. Open table formats close this gap
to varying degrees. Concretely, the storage layer must provide three primitives:
+
+- **Record-level change retrieval.** The table must be able to answer "what
changed between time T1 and T2" efficiently, without scanning everything. This
comes in two flavors: *incremental queries*, which return the latest value of
each changed record (ideal for driving downstream upserts), and *CDC queries*,
which return before/after images of every change (needed when the
transformation must see each individual mutation). Apache Hudi supports [both
query types](/docs/sql_queries#incre [...]
+
+- **Upserts and deletes to apply changes.** Producing a delta is only useful
if the next table can absorb it cheaply. That requires primary keys and indexes
to locate which files contain the affected records, so a small change touches a
small amount of storage. Hudi's [write operations](/docs/write_operations) are
built around indexed upserts and deletes, with merge modes to handle
out-of-order and late-arriving data correctly.
+
+- **Ordering and commit points.** Consumers need a reliable notion of "where I
left off." Hudi's [timeline](/docs/timeline) records every action on the table
as an atomically committed instant; incremental queries use commit completion
times to define exact, repeatable change windows, and checkpoints are managed
automatically by tools like Hudi Streamer's incremental sources. Hudi
additionally tracks record-level metadata (commit time and sequence number on
every record), so change histo [...]
+
+These primitives — upsert and incremental consumption — are exactly the pair
that defines the pattern: with them, you can maintain one table from a change
stream, then derive another table from *its* change stream, end to end. The
[use cases documentation](/docs/use_cases) covers how this plays out across
ingestion, warehouse offload, and derived pipelines.
+
+### Frequent Table Maintenance Becomes the Norm
+
+There is a fourth requirement that follows directly from the first three: the
format must be designed for frequent table maintenance, because incremental
processing fundamentally increases how often the table is written. A table that
once absorbed one nightly batch now commits every few minutes — producing small
files, accumulating deltas in log files, and continuously churning the storage
layout. At that write frequency, maintenance work —
[compaction](/docs/compaction), clustering, cle [...]
+
+And it must run *without blocking the writers it serves*. If compacting or
clustering a table means pausing ingestion, or if maintenance jobs contend with
writers as just another optimistic transaction — conflicting and forcing
retries on exactly the hot file groups that change most — then every
maintenance pass reintroduces the latency and cost the pipeline went
incremental to eliminate. Hudi builds table services into the storage engine
for this reason: compaction, clustering, cleaning [...]
+
+## Incremental ETL vs. Stream Processing
+
+Incremental ETL and stream processing share semantics but differ sharply in
infrastructure and economics:
+
+| | Stream processing (Flink, Kafka Streams) | Incremental ETL (Hudi on a data
lake) |
+|---|---|---|
+| Latency | Sub-second to seconds | Minutes |
+| Compute model | Always-on jobs with managed state | Scheduled or continuous
mini-batch jobs; state lives in tables |
+| Storage | Event log (Kafka) + state stores + eventual sink | One copy:
lakehouse tables on cloud object storage |
+| Reprocessing/backfill | Replay the log, rebuild state | Re-run the job over
a commit range or full table |
+| Query access to intermediate results | Requires sinking to another system |
Every intermediate table is directly queryable |
+
+A useful way to summarize the trade: **incremental ETL delivers streaming
semantics at minute-level batch economics.** If a use case genuinely needs
sub-second reactions — fraud blocking, operational alerting — a stream
processor is the right tool. But a large majority of analytics pipelines have
freshness requirements in the minutes-to-hour range, where paying for always-on
stream infrastructure and a second copy of the data buys nothing. The two also
compose rather than compete: Flink [...]
+
+## A Concrete Pipeline Walkthrough
+
+Here is the pattern end to end, using a familiar CDC scenario: an operational
database (say, PostgreSQL) whose changes must feed an analytics table.
+
+**Step 1 — CDC into a bronze table.** A capture tool such as Debezium streams
the database's change log into Kafka, and an ingestion job — for example [Hudi
Streamer](/docs/hoodie_streaming_ingestion) — continuously upserts those
changes into a bronze Hudi table. The bronze table is now a queryable,
transactional mirror of the source, and every commit on its timeline is a
consumable change point.
+
+**Step 2 — incrementally read the bronze table.** The silver pipeline wakes up
on its schedule (or runs continuously) and pulls only what changed since its
last checkpoint:
+
+```scala
+// Read only the records that changed since the last run
+val ordersDelta = spark.read.format("hudi").
+ option("hoodie.datasource.query.type", "incremental").
+ option("hoodie.datasource.read.begin.instanttime", lastCheckpoint).
+ load(bronzeOrdersPath)
+```
+
+The same read is available in SQL through the `hudi_table_changes`
table-valued function:
+
+```sql
+SELECT * FROM hudi_table_changes('bronze_orders', 'latest_state',
'20260716083000000');
+```
+
+**Step 3 — transform and upsert into a silver table.** The transformation runs
over just the delta, and the result is applied as an upsert keyed on the record
key:
+
+```scala
+val silverDelta = ordersDelta.
+ filter(col("order_status") =!= "CANCELLED").
+ withColumn("order_total", col("quantity") * col("unit_price"))
+
+silverDelta.write.format("hudi").
+ option("hoodie.datasource.write.operation", "upsert").
+ option("hoodie.table.name", "silver_orders").
+ mode("append").
+ save(silverOrdersPath)
+```
+
+The job then persists the new checkpoint (the latest commit completion time it
consumed) for the next run. Because the silver table was written with upserts,
it exposes its own change stream — a gold-layer aggregation job can consume
`silver_orders` incrementally in exactly the same way, re-aggregating only the
keys whose inputs changed.
+
+
+<p align = "center">Figure: Changes are upserted into an upstream table and
incrementally consumed to maintain a derived table</p>
+
+One honest caveat: not every transformation is trivially incremental. Simple
projections, filters, and enrichments map one-to-one onto deltas. Aggregations
require merging new partial results into existing ones, and some full-table
operations (percentiles, complex window functions) may still warrant periodic
full recomputation. A pragmatic design runs the same logic incrementally for
freshness and occasionally in full for correction — on the same tables and
infrastructure.
+
+## A Brief History: Pioneered at Uber
+
+Incremental processing on the data lake is not a recent idea bolted onto table
formats — it is the problem Hudi was created to solve. In 2016, Uber's data
team faced exactly the trade-off described above: petabyte-scale Hadoop tables
that could only be rewritten wholesale, and business use cases (like
driver-partner incentives and trip pricing) that needed data in minutes, not
hours. The case for a third processing style — incremental, on top of batch
storage — was laid out in the O'Reil [...]
+
+## Conclusion
+
+Incremental ETL replaces "recompute everything, always" with "process what
changed, continuously." It requires a storage layer that can serve record-level
change streams, absorb upserts and deletes efficiently, and expose reliable
commit points to checkpoint against — the primitives Apache Hudi has provided
on data lakes since its inception. The payoff is pipelines whose cost tracks
the rate of change in the business rather than the accumulated size of history,
with data freshness measur [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {
+ question: 'What is incremental ETL?',
+ answer: 'Incremental ETL is a data pipeline pattern that processes only
the records that were inserted, updated, or deleted since the pipeline\'s last
run, instead of re-scanning and rewriting entire tables. Each run consumes a
delta from a checkpoint, transforms it, and merges the result into the target
table with upserts and deletes. This makes pipeline cost proportional to the
volume of change rather than total table size.'
+ },
+ {
+ question: 'What is the difference between incremental ETL and streaming?',
+ answer: 'Both process data as a flow of changes, but stream processors
like Apache Flink run always-on jobs with managed state and typically require a
separate event log such as Kafka, delivering sub-second latency. Incremental
ETL runs as scheduled or continuous mini-batch jobs directly against lakehouse
tables on cloud storage, delivering minute-level freshness at much lower
infrastructure cost. A common summary is that incremental ETL provides
streaming semantics at batch economics.'
+ },
+ {
+ question: 'How does incremental ETL reduce cost?',
+ answer: 'It eliminates repeated scans and rewrites of data that did not
change. If a large table receives a small volume of changes between runs, an
incremental pipeline reads and processes only that delta instead of the whole
table, often reducing data scanned per run by orders of magnitude. Compute
clusters also only run for the duration of each small job rather than staying
on permanently.'
+ },
+ {
+ question: 'What does the storage layer need to support incremental ETL?',
+ answer: 'Four things: the ability to efficiently retrieve record-level
changes between two points in time, support for upserts and deletes so
downstream tables can absorb deltas cheaply, an ordered log of commits that
consumers can checkpoint against, and table maintenance that runs without
blocking writers — because incremental processing multiplies write frequency,
compaction, clustering, and cleaning become continuous background work rather
than occasional jobs. Plain Parquet file [...]
+ },
+ {
+ question: 'Can every transformation be made incremental?',
+ answer: 'No. Projections, filters, enrichments, and key-based joins map
naturally onto change streams, and aggregations can be maintained by merging
partial results for the affected keys. Some computations, like exact
percentiles or complex full-table window functions, are hard to update
incrementally, so teams often run those periodically in full while keeping the
rest of the pipeline incremental.'
+ },
+ {
+ question: 'Which Apache Hudi features enable incremental ETL?',
+ answer: 'Hudi provides incremental queries that return the latest values
of changed records after a given commit, CDC queries that return before and
after images of each change, and indexed upsert and delete write operations to
apply deltas downstream. Its timeline records every commit with completion
times that define exact change windows, and record-level metadata preserves
change history across compaction and clustering. Ingestion tools like Hudi
Streamer manage incremental checkp [...]
+ }
+]} />
diff --git a/website/blog/2026-07-17-what-is-acid-on-a-data-lake.md
b/website/blog/2026-07-17-what-is-acid-on-a-data-lake.md
new file mode 100644
index 000000000000..237c64225182
--- /dev/null
+++ b/website/blog/2026-07-17-what-is-acid-on-a-data-lake.md
@@ -0,0 +1,128 @@
+---
+title: "What is ACID on a Data Lake?"
+excerpt: "What atomicity, consistency, isolation and durability actually mean
for tables on object storage, and how lakehouse table formats like Apache Hudi
deliver them."
+description: "What ACID means for tables on S3, GCS or ADLS: how table formats
like Apache Hudi deliver atomic commits, snapshot isolation and safe concurrent
writes."
+authors: [sivabalan]
+category: how-to
+image: /assets/images/blog/acid.png
+tags:
+- acid
+- concurrency control
+- hudi timeline
+- data lakehouse
+---
+
+ACID on a data lake means that writes to tables stored on object storage are
Atomic, Consistent, Isolated and Durable — the same transactional guarantees a
database provides, applied to files in S3, GCS or ADLS. That sentence is easy
to say and surprisingly hard to deliver. A database owns its storage engine end
to end; a data lake is a pile of Parquet files on an object store that offers
durable single-object PUTs and nothing more. There is no multi-file atomic
operation, no rename that [...]
+
+This post walks through what each of the four letters means concretely at the
file level, how lakes worked before table formats existed, how formats like
[Apache Hudi](https://hudi.apache.org) deliver these guarantees, and — just as
important — what ACID on a lake does *not* promise.
+
+## What A, C, I and D Mean on Object Storage
+
+The textbook definitions of ACID were written for databases processing many
small transactions. On a data lake, the transactions are large batch or
streaming writes that produce files, so it is worth restating each property in
file terms.
+
+**Atomicity** means a write either publishes completely or not at all.
Consider a Spark job writing 500 Parquet files into a table and dying at file
320 — an executor is lost, a spot instance is reclaimed, someone kills the job.
Atomicity requires that those 320 orphaned files are never visible to any
query, and that the failed write can be cleanly rolled back. Without it, every
downstream consumer inherits a partially written dataset and no reliable way to
detect it.
+
+**Consistency** means every write moves the table from one valid state to
another valid state. On a lake this covers schema enforcement (a write with an
incompatible schema is rejected rather than quietly producing unreadable
files), key constraints like upsert semantics (a record with the same key
updates in place rather than duplicating), and the invariant that the table's
metadata always describes exactly the set of files that constitute the table.
+
+**Isolation** means concurrent readers and writers do not observe each other's
intermediate states. Two failure modes matter here. First, two writers
targeting the same partition at the same time — say, a backfill job and a
streaming ingest job — must not interleave their files such that the table ends
up with a mix of both writes' partial output. Second, a reader that starts
listing files mid-write must not see half of a commit; it should see the table
exactly as it was at some committe [...]
+
+**Durability** means a committed write survives failures. Object stores
already provide excellent durability for individual objects — S3, GCS and ADLS
replicate objects across zones. The lake-specific part of durability is that
the *commit record itself* is persisted to the same durable storage, so that
"was this write committed?" has a crash-proof answer, and a committed
transaction is never silently rolled back by a subsequent cleanup process.
+
+## Life Before ACID on Data Lakes
+
+The Hadoop and early cloud-lake era answered these problems with conventions,
not guarantees. It is worth remembering how fragile those conventions were,
because they explain why table formats exist.
+
+The classic Hive pattern was the **partition swap**: write data into a staging
directory, then "atomically" move it into place and update the Hive metastore
partition location. On HDFS a directory rename was genuinely atomic, so this
mostly worked. On object stores there are no directories and no atomic rename —
a "rename" is a copy-then-delete of every object — so the same pattern became a
slow, non-atomic window during which readers could see anything.
+
+The **`_SUCCESS` file** convention had MapReduce and Spark drop an empty
marker file when a job finished, and downstream jobs would poll for it. This
signaled completion but protected nothing: readers that listed the directory
directly saw partial data, a rerun of a failed job could mix old and new files,
and there was no rollback story at all — cleanup after a failed job meant a
human with a delete command deciding which files were orphans.
+
+Layered on top of this, early S3 offered only **eventual consistency** for
listings, so even a fully written directory might not be fully *visible* yet.
Whole projects (S3Guard, committers with DynamoDB bookkeeping) existed just to
paper over that. S3 became strongly consistent in 2020, which removed one class
of bugs — but strong single-object consistency still does not give you
multi-file atomicity, isolation between writers, or rollback. Those require a
transaction log.
+
+## How Table Formats Deliver ACID
+
+The shared insight behind Apache Hudi, Apache Iceberg and Delta Lake is that
the table's state should be defined by a **metadata log with an atomic publish
step**, not by whatever files happen to be in a directory. Data files are
written first, invisibly; then a single, small metadata operation makes them
visible all at once. Because that final operation is one atomic object-store
action (a single PUT, or a conditional/compare-and-swap write), the entire
multi-file write inherits its atomicity.
+
+In Hudi, that log is the [timeline](/docs/timeline) — an ordered event log of
every action performed on the table, stored under the table's
`.hoodie/timeline` directory. Each write is an action that moves through
explicit states: `REQUESTED` (the write is planned), `INFLIGHT` (data files are
being produced), and `COMPLETED` (the write is published). The transition to
`COMPLETED` is a single atomic object-store operation, and Hudi timestamps
every action with monotonically increasing, [Tr [...]
+
+
+<p align = "center">Figure: Writes and table services recorded as actions on
the Hudi timeline</p>
+
+This structure delivers each letter directly:
+
+- **Atomicity**: our 500-file Spark job that dies at file 320 leaves an
`INFLIGHT` action on the timeline and 320 uncommitted files that no query will
ever read, because queries resolve visible files through completed timeline
actions — never by listing directories. A rollback action later removes the
orphaned files. Nothing partial is ever served.
+- **Consistency**: the timeline is the single source of truth for table state,
and schema and key constraints are enforced as part of the write before the
commit is published.
+- **Isolation**: readers get **snapshot isolation**. A query binds to the set
of completed commits as of its start and sees exactly that state for its entire
duration, no matter how many writes complete concurrently. Multi-version
concurrency control (MVCC) keeps prior file versions around so long-running
queries are never yanked out from under.
+- **Durability**: the commit metadata lives on the same replicated object
storage as the data, so a completed instant is as durable as the data it
describes.
+
+For a deeper walkthrough of how the timeline underpins each transaction state
transition, see the earlier post [Hoodie Timeline: Foundational pillar for ACID
transactions](/blog/2023/07/09/Hoodie-Timeline-Foundational-pillar-for-ACID-transactions).
+
+To be fair to the broader ecosystem: Apache Iceberg and Delta Lake achieve
ACID through the same fundamental idea with different mechanics. Iceberg builds
immutable snapshot trees of manifest files and commits by atomically swapping a
pointer to the new table metadata (typically via a catalog's compare-and-swap).
Delta Lake maintains a JSON transaction log (`_delta_log`) where writing log
entry *N+1* is the atomic commit point, using conditional writes on the object
store. All three give [...]
+
+## Concurrency Control Models
+
+Atomic commits solve the single-writer case. The harder question — and where
table formats differ most — is what happens when multiple processes write to
the same table. Hudi's [concurrency control](/docs/concurrency_control)
documentation lays out a spectrum of models:
+
+**Single writer with table services.** Most pipelines have one writer, plus
background table services (compaction, clustering, cleaning) that rewrite data
for performance. Hudi distinguishes these by design: table services run
lock-free alongside the writer, coordinated through the timeline via MVCC,
either inline or fully async in the same process. For this very common case, no
external lock infrastructure is needed at all.
+
+**Optimistic concurrency control (OCC).** When genuinely separate jobs write
to one table — a streaming ingest plus a batch backfill, say — Hudi supports
OCC at file-group granularity. Each writer proceeds without blocking, and at
commit time takes a short-lived distributed lock to check whether a
concurrently completed commit touched the same files. Non-overlapping writes
both succeed; overlapping writes cause one to abort and retry. The lock is held
only around the commit-time critical [...]
+
+
+<p align = "center">Figure: Optimistic concurrency control — conflicts
detected at commit time</p>
+
+OCC works well when conflicts are rare, but it has an honest weakness for
streaming: two high-frequency writers that routinely touch the same file groups
will repeatedly abort each other, wasting compute and starving progress.
+
+**Non-blocking concurrency control (NBCC).** For multi-writer streaming, Hudi
supports a mode where multiple writers can write into the *same* file groups
simultaneously without aborting each other. Instead of preventing the conflict,
resolution is deferred: commits are ordered by completion time on the timeline,
and the query reader and the compactor merge overlapping writes
deterministically. A lock is needed only for the instant-metadata write to the
timeline itself. NBCC currently ta [...]
+
+## What ACID on a Data Lake Does Not Mean
+
+Honesty matters here, because "ACID" invites comparisons to OLTP databases
that lakes are not designed to win.
+
+- **It is not OLTP.** Commit latency on a lakehouse is measured in seconds —
an object-store round trip plus metadata work — not the milliseconds of a
transactional database. Lakehouse transactions are designed for batches and
micro-batches of records, not single-row point writes at high QPS.
+- **Transactions span one table.** The unit of atomicity in Hudi (and its
peers, by and large) is a single table's commit. Multi-table transactions,
cross-table foreign keys, and interactive multi-statement transactions with
`BEGIN`/`COMMIT` semantics are not part of the model.
+- **Isolation is snapshot isolation, not serializability of arbitrary
read-write transactions.** Writers are serialized against each other per table,
and readers see consistent snapshots — which is exactly what analytical
workloads need — but you should not port an OLTP application's locking
assumptions onto a lake.
+- **ACID does not fix bad pipelines.** Atomic commits guarantee that what was
written is either fully visible or invisible; they do not validate that the
data itself was correct.
+
+If your workload is high-QPS point reads and writes, use an operational
database and land the changes onto the lake via CDC — which, conveniently, is a
workload where lake ACID shines.
+
+## Why It Matters
+
+These guarantees are not academic; they are what make several load-bearing
patterns possible at all.
+
+**CDC correctness.** Replicating a database table into the lake via change
data capture means a continuous stream of upserts and deletes. Without atomic
commits and key-based upserts, a mid-batch failure leaves the lake copy in a
state the source database never had — and there is no way to reconcile. With
ACID, every commit is an all-or-nothing application of a batch of changes, and
incremental consumers downstream see change batches in a well-defined order,
never partially.
+
+**Concurrent ingestion and table services.** Real tables need continuous
compaction, clustering and cleaning to stay queryable. Running those alongside
ingestion without transactions means either stopping the world or risking
corruption. MVCC on the timeline lets Hudi rewrite data in the background while
writers write and readers read, with each process pinned to a consistent
snapshot.
+
+**Regulatory deletes.** GDPR and CCPA erasure requests require deleting
specific records from immutable files — physically, a read-modify-rewrite of
affected files. ACID makes each deletion pass atomic and auditable: the
timeline records exactly when the delete committed, queries never see a
half-deleted state, and the commit history serves as evidence of compliance.
+
+## Conclusion
+
+ACID on a data lake is a commit protocol over object storage: data files are
written invisibly, then published through one atomic metadata operation, with
snapshot isolation for readers and explicit concurrency control between
writers. Hudi implements this with its [timeline](/docs/timeline) — an ordered,
durable event log of every action on the table — plus a spectrum of
[concurrency control](/docs/concurrency_control) modes ranging from lock-free
single-writer deployments to OCC and no [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {
+ question: 'Does S3 support ACID transactions?',
+ answer: 'Not by itself. S3 provides durable, strongly consistent
operations on individual objects, but no atomic operation spanning multiple
objects and no isolation between concurrent writers. ACID transactions on S3
come from a table format layered on top, such as Apache Hudi, Apache Iceberg or
Delta Lake, which publishes multi-file writes through a single atomic metadata
commit.'
+ },
+ {
+ question: 'Is a data lake ACID compliant?',
+ answer: 'A raw data lake of Parquet or ORC files is not ACID compliant:
readers can observe partial writes, concurrent jobs can corrupt each other, and
failed jobs leave orphaned files. A data lake becomes ACID compliant when
tables are managed by a transactional table format, at which point it is
usually called a lakehouse.'
+ },
+ {
+ question: 'How does Hudi guarantee atomicity?',
+ answer: 'Hudi records every write as an action on its timeline, moving
through requested, inflight and completed states. Data files are written first
but remain invisible; the transition to completed is a single atomic
object-store operation. Queries only read files referenced by completed
actions, so a failed write is never visible and is later rolled back cleanly.'
+ },
+ {
+ question: 'What is snapshot isolation on a data lake?',
+ answer: 'Snapshot isolation means every query reads the table as of a
specific committed state and sees exactly that state for its entire run, even
while new writes commit concurrently. Table formats implement it with
multi-version concurrency control, keeping prior file versions available until
no reader needs them.'
+ },
+ {
+ question: 'Can multiple jobs write to the same lakehouse table at once?',
+ answer: 'Yes, with concurrency control. Hudi supports optimistic
concurrency control, where writers proceed in parallel and conflicts on
overlapping files are detected at commit time using a short-lived lock, and
non-blocking concurrency control, where multiple writers can even target the
same file group and conflicts are resolved at read and compaction time.'
+ },
+ {
+ question: 'Is ACID on a data lake the same as ACID in a database like
PostgreSQL?',
+ answer: 'The guarantees are the same in kind but different in scope and
latency. Lakehouse transactions cover batches of records within one table and
commit in seconds, whereas an OLTP database offers millisecond,
multi-statement, often multi-table transactions. Lakes are built for analytical
and streaming workloads, not high-QPS point reads and writes.'
+ }
+]} />
diff --git a/website/static/llms.txt b/website/static/llms.txt
index 138ed0fcc51f..72ac2077ebd7 100644
--- a/website/static/llms.txt
+++ b/website/static/llms.txt
@@ -39,6 +39,10 @@ It can also synchronize your data to half dozen data
catalogs to keep table cons
- [Apache Hudi: 21 Unique
Differentiators](https://hudi.apache.org/blog/2025/03/05/hudi-21-unique-differentiators):
The technical crux of what sets Hudi apart — indexing, non-blocking
concurrency control, async compaction, read-side indexes and built-in table
services.
- [Apache Hudi 1.2: The Open Lakehouse for AI and Multimodal
Data](https://hudi.apache.org/blog/2026/06/07/apache-hudi-release-1-2-announcement):
First-class support for vectors, blobs/unstructured data, and semi-structured
data on the lakehouse — for Spark pipelines doing unstructured data processing
or generating embeddings.
- [What is a Data Lakehouse & How does it
Work?](https://hudi.apache.org/blog/2024/07/11/what-is-a-data-lakehouse): The
lakehouse architecture explained — evolution, components, use cases and key
technologies.
+- [What is an Open Table
Format?](https://hudi.apache.org/blog/2026/07/14/what-is-an-open-table-format):
How open table formats like Hudi, Iceberg and Delta Lake bring database
semantics to files on object storage.
+- [What is Upsert on a Data
Lake?](https://hudi.apache.org/blog/2026/07/15/what-is-upsert-on-a-data-lake):
Record-level updates and deletes on immutable cloud storage, and how Hudi makes
them fast.
+- [What is Incremental ETL on a Data
Lake?](https://hudi.apache.org/blog/2026/07/16/what-is-incremental-etl-on-a-data-lake):
Processing only the data that changed instead of recomputing whole tables.
+- [What is ACID on a Data
Lake?](https://hudi.apache.org/blog/2026/07/17/what-is-acid-on-a-data-lake):
What atomicity, consistency, isolation and durability mean when your database
is a bucket of files.
## Blog — Migration guides
