wombatu-kun commented on code in PR #19282:
URL: https://github.com/apache/hudi/pull/19282#discussion_r3628602216
##########
website/learn/tech-specs.md:
##########
@@ -216,24 +233,55 @@ meaningful for them.
### Log Files
-The log files contain different type of blocks, that encode delta updates,
deletes or change logs against the base file. They are named with the convention
+Log files encode inserts, updates, deletes, or change logs against a file
group. Hudi supports two physical log formats,
+and a file slice may contain both formats after a table-version upgrade.
+
+**Inline log file format (v0/v1)**
+
+Inline logs use the following hidden-file naming convention:
+
+**_.\[file\_id\]\_\[requested\_instant\_time\].log.\[version\]\_\[write\_token\]_**
+
+An inline log file is appendable and can contain multiple Hudi-framed blocks.
Inline CDC data uses the optional `.cdc`
+suffix after the write token.
-**_.\[file\_id\]\_\[requested\_instant\_time\].log.\[version\]\_\[write\_token\]_**.
+**Native log file format (v2)**
+Native logs use a visible, write-once file with this naming convention:
+**_\[file\_id\]\_\[write\_token\]\_\[requested\_instant\_time\]\_\[version\].\[log\_type\].\[native\_format\]_**
-* **file\_id** - File Id of the base file in the slice
-* **requested\_instant\_time** \- Time at which the write operation that
produced this log file was requested.
-* **version** - Current version of the log file format, to order deltas
against the base file.
-* **write\_token** - Monotonically increasing token for every attempt to
write the log file. This should help uniquely identifying the log file when
there are failures and retries.
- Cleaner can clean-up partial log files if the write token is not the
latest in the file slice.
+| Log type | File-name component | Parquet example |
+| --- | --- | --- |
+| Data | `log` | `<file_id>_<write_token>_<instant>_<version>.log.parquet` |
+| Delete | `deletes` |
`<file_id>_<write_token>_<instant>_<version>.deletes.parquet` |
+| Change data capture | `cdc` |
`<file_id>_<write_token>_<instant>_<version>.cdc.parquet` |
+
+The fields used by both conventions are:
+
+* **file_id** - ID of the file group to which the log belongs.
+* **requested_instant_time** - Begin time of the write action that produced
the log.
+* **version** - Sequence number that orders log files for the same file group
and instant. This is distinct from the
+ physical log-format version stored in the block envelope or native-file
footer.
+* **write_token** - Token identifying the task and attempt that produced the
file. It distinguishes retries and lets
+ cleaning remove unsuccessful attempts.
+* **native_format** - Storage format of a native log file, such as `parquet`,
`orc`, or `hfile`. The current writer uses
+ the effective base-file format; Lance is the exception described below.
+
+Table versions below 10 write inline logs. Writers targeting table version 10
or later write native logs by default,
+except for Lance-backed tables, which continue to use inline logs until native
Lance log support is available.
Review Comment:
`CommonClientUtils.shouldWriteNativeLogs` keys only on the write table
version and carries no base-file-format branch, and Lance is written as a
native log format, so v10 Lance tables do not stay on inline logs. Drop the
Lance exception here, in the `native_format` bullet, the Versioning table row,
the Writer Expectations sentence and the v10 change-log bullet, and add `lance`
to the `native_format` examples.
##########
website/learn/tech-specs.md:
##########
@@ -249,10 +297,53 @@ Hudi Log format specification is as follows.
| **footer** | variable | Similar to Header. Map of footer metadata entries. |
| **total block length** | 8 | Total size of the block including the magic
bytes. This is used to determine if a block is corrupt by comparing to the
block size in the header. Each log block assumes that the block size will be
last data written in a block. Any data if written after is just ignored. |
+#### Native Log Format (v2)
+
+The native log format stores records directly in a standard native file rather
than embedding that file inside a Hudi
+binary envelope. Each native file represents one logical data, delete, or CDC
block. Parquet, the primary example from
+RFC-103, has the following layout:
+
+```text
+Row group 1 (records)
+Row group 2 (records)
+...
+Parquet footer
+ - schema and row-group metadata
+ - key-value metadata
+ hudi.log.format.metadata = <JSON object>
+```
+
+The `hudi.log.format.metadata` footer value is a JSON object keyed by the
symbolic header names listed in
+[Headers](#headers). The writer always adds `VERSION=2` and also persists the
applicable values such as `INSTANT_TIME`,
+`SCHEMA`, `RECORD_POSITIONS`, `BLOCK_IDENTIFIER`, `IS_PARTIAL`, and
Review Comment:
Native logs do not encode `RECORD_POSITIONS` the way the Headers table
describes: `HoodieNativeLogFormatWriter` writes it through
`LogReaderUtils.encodePositionsAsLongList` as a Base64 count-plus-varlong list
in insert order, not as a serialized `Roaring64NavigableMap` bitmap. Worth
spelling the native encoding out here so an independent reader does not decode
a native footer as a bitmap.
##########
website/learn/tech-specs.md:
##########
@@ -622,19 +719,32 @@ to construct file slices out of file groups that are of
interest to the read, ba
Readers need to determine the correct file slice (an optional base file plus
an ordered list of log files) to read in order to construct the snapshot state,
using the following steps:
1. Reader picks an instant in the timeline — the latest completion time on the
timeline or a specific time explicitly specified. Call this the **_snapshot
read time_**.
-2. Reader computes all file groups in the table by listing all files that are
part of the table, grouping them by `file_id`, and eliminating files that don't
belong to any completed write action on the timeline.
+2. Reader computes all file groups in the table by listing all files that are
part of the table, grouping them by `file_id`, and eliminating files that don't
belong to any completed write action on the timeline. File-name parsing must
recognize both inline and native log-file conventions.
3. Reader then eliminates replaced file groups, by removing any file groups
that are part of any **_replacecommit_** actions completed on or before the
snapshot read time.
4. For each remaining file group, the reader determines the correct file slice
as follows:
1. Find the base file with the greatest requested instant time less than or
equal to the snapshot read time.
2. Obtain all log files with completion time less than or equal to the
snapshot read time, sorted by completion time.
5. For each file slice obtained, the reader merges the base and log files:
- 1. Scan log blocks in order. Each block carries an `INSTANT_TIME` header; a
`COMMAND_BLOCK` of type `ROLLBACK_BLOCK`
- causes the immediately preceding block to be discarded (this is how
partial/aborted appends are masked out without rewriting the log file).
- Blocks whose `INSTANT_TIME` no longer corresponds to a completed instant
on the timeline are also skipped. `BLOCK_IDENTIFIER`
- is used to deduplicate blocks produced by task retries within the same
instant.
- 2. When the base file is scanned, for every record the reader has to look
up whether there is a newer version of the record available in the surviving
log blocks and merge it into the record iterator.
- When `RECORD_POSITIONS` is present, this merge can be done positionally
against the base file identified by
`BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS` rather than via key lookup.
- 3. Compaction-produced blocks may set the `COMPACTED_BLOCK_TIMES` header
listing the instant times of the (minor-compacted) blocks they replace; the
reader skips any earlier blocks whose instant time is covered.
+ 1. Detect each log's physical format from its file name and route it to the
inline or native read path. A version-10
Review Comment:
At v10 a single instant emits both a data log and a delete log, and
`HoodieLogFile.LogFileComparator` orders them by instant, log version, write
token, then log-type precedence so data is read before deletes. The
slice-selection step above still says log files are sorted by completion time,
which cannot express that ordering; update it here or leave as a follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
