hudi-agent commented on code in PR #19282:
URL: https://github.com/apache/hudi/pull/19282#discussion_r3575891076
##########
website/learn/tech-specs.md:
##########
@@ -216,24 +233,55 @@ meaningful for them.
### Log Files
-The log files contain different type of blocks, that encode delta updates,
deletes or change logs against the base file. They are named with the convention
+Log files encode inserts, updates, deletes, or change logs against a file
group. Hudi supports two physical log formats,
+and a file slice may contain both formats after a table-version upgrade.
+
+**Inline log file format (v0/v1)**
+
+Inline logs use the following hidden-file naming convention:
+
+**_.\[file\_id\]\_\[requested\_instant\_time\].log.\[version\]\_\[write\_token\]_**
+
+An inline log file is appendable and can contain multiple Hudi-framed blocks.
Inline CDC data uses the optional `.cdc`
+suffix after the write token.
-**_.\[file\_id\]\_\[requested\_instant\_time\].log.\[version\]\_\[write\_token\]_**.
+**Native log file format (v2)**
+Native logs use a visible, write-once file with this naming convention:
+**_\[file\_id\]\_\[write\_token\]\_\[requested\_instant\_time\]\_\[version\].\[log\_type\].\[native\_format\]_**
-* **file\_id** - File Id of the base file in the slice
-* **requested\_instant\_time** \- Time at which the write operation that
produced this log file was requested.
-* **version** - Current version of the log file format, to order deltas
against the base file.
-* **write\_token** - Monotonically increasing token for every attempt to
write the log file. This should help uniquely identifying the log file when
there are failures and retries.
- Cleaner can clean-up partial log files if the write token is not the
latest in the file slice.
+| Log type | File-name component | Parquet example |
+| --- | --- | --- |
+| Data | `log` | `<file_id>_<write_token>_<instant>_<version>.log.parquet` |
+| Delete | `deletes` |
`<file_id>_<write_token>_<instant>_<version>.deletes.parquet` |
+| Change data capture | `cdc` |
`<file_id>_<write_token>_<instant>_<version>.cdc.parquet` |
+
+The fields used by both conventions are:
+
+* **file_id** - ID of the file group to which the log belongs.
+* **requested_instant_time** - Begin time of the write action that produced
the log.
+* **version** - Sequence number that orders log files for the same file group
and instant. This is distinct from the
+ physical log-format version stored in the block envelope or native-file
footer.
+* **write_token** - Token identifying the task and attempt that produced the
file. It distinguishes retries and lets
+ cleaning remove unsuccessful attempts.
+* **native_format** - Storage format of a native log file, such as `parquet`,
`orc`, or `hfile`. The current writer uses
Review Comment:
🤖 This bullet says `native_format` may be `parquet`, `orc`, or `hfile` and
that "the current writer uses the effective base-file format," but the Native
Log Format section states "The current LSM file-group reader expects native
Parquet logs," and RFC-103 is described as defining Parquet as the primary
layout. It might help to clarify whether ORC/HFile native logs are actually
written and readable in table version 10 today, or whether v2 is Parquet-only
for now with ORC/HFile listed as future formats. As written, a user with an
ORC- or HFile-based table could reasonably expect native-log support that may
not yet exist.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
website/learn/tech-specs.md:
##########
@@ -280,11 +372,13 @@ Metadata key mapping from Integer to actual metadata is
as follows:
| BLOCK\_IDENTIFIER | 6 | Block
sequence number used to detect duplicate log blocks (e.g. due to task retries).
|
| IS\_PARTIAL | 7 | Boolean
indicating whether the data block encodes partial updates.
|
| BASE\_FILE\_INSTANT\_TIME\_OF\_RECORD\_POSITIONS | 8 | Begin
(requested) instant time of the base file that the `RECORD_POSITIONS` bitmap is
relative to. Used by the reader to validate that positions still refer to the
same base file when reconstructing a file slice.
|
+| VERSION | 9 | Physical
log-format version. Native log writers always add this value to
`hudi.log.format.metadata`.
|
Review Comment:
🤖 The VERSION row is added to the inline Headers table with Encoding ID `9`,
but the description says it is added by *native* log writers to the
`hudi.log.format.metadata` JSON — and the note just above the table (and the
Block types section) states native logs use symbolic names and "readers must
not interpret the numeric IDs." So Encoding ID 9 would only ever apply to
inline logs. Does the inline v0/v1 format actually serialize a VERSION header?
Note the inline envelope already carries a 4-byte `version` field (see the
Inline Log Format spec table), so a VERSION header there would be redundant. It
might help to clarify whether VERSION is inline-serialized at all, or whether
this row belongs conceptually to the native metadata (in which case an integer
Encoding ID is a bit misleading).
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
website/learn/tech-specs.md:
##########
@@ -216,24 +233,55 @@ meaningful for them.
### Log Files
-The log files contain different type of blocks, that encode delta updates,
deletes or change logs against the base file. They are named with the convention
+Log files encode inserts, updates, deletes, or change logs against a file
group. Hudi supports two physical log formats,
+and a file slice may contain both formats after a table-version upgrade.
+
+**Inline log file format (v0/v1)**
+
+Inline logs use the following hidden-file naming convention:
+
+**_.\[file\_id\]\_\[requested\_instant\_time\].log.\[version\]\_\[write\_token\]_**
+
+An inline log file is appendable and can contain multiple Hudi-framed blocks.
Inline CDC data uses the optional `.cdc`
+suffix after the write token.
-**_.\[file\_id\]\_\[requested\_instant\_time\].log.\[version\]\_\[write\_token\]_**.
+**Native log file format (v2)**
+Native logs use a visible, write-once file with this naming convention:
+**_\[file\_id\]\_\[write\_token\]\_\[requested\_instant\_time\]\_\[version\].\[log\_type\].\[native\_format\]_**
-* **file\_id** - File Id of the base file in the slice
-* **requested\_instant\_time** \- Time at which the write operation that
produced this log file was requested.
-* **version** - Current version of the log file format, to order deltas
against the base file.
-* **write\_token** - Monotonically increasing token for every attempt to
write the log file. This should help uniquely identifying the log file when
there are failures and retries.
- Cleaner can clean-up partial log files if the write token is not the
latest in the file slice.
+| Log type | File-name component | Parquet example |
+| --- | --- | --- |
+| Data | `log` | `<file_id>_<write_token>_<instant>_<version>.log.parquet` |
+| Delete | `deletes` |
`<file_id>_<write_token>_<instant>_<version>.deletes.parquet` |
+| Change data capture | `cdc` |
`<file_id>_<write_token>_<instant>_<version>.cdc.parquet` |
+
+The fields used by both conventions are:
+
+* **file_id** - ID of the file group to which the log belongs.
+* **requested_instant_time** - Begin time of the write action that produced
the log.
+* **version** - Sequence number that orders log files for the same file group
and instant. This is distinct from the
+ physical log-format version stored in the block envelope or native-file
footer.
+* **write_token** - Token identifying the task and attempt that produced the
file. It distinguishes retries and lets
+ cleaning remove unsuccessful attempts.
+* **native_format** - Storage format of a native log file, such as `parquet`,
`orc`, or `hfile`. The current writer uses
+ the effective base-file format; Lance is the exception described below.
+
+Table versions below 10 write inline logs. Writers targeting table version 10
or later write native logs by default,
Review Comment:
🤖 "Writers targeting table version 10 or later write native logs by default,
except for Lance-backed tables" — the metadata table is HFile-based and relies
on inline HFile log blocks with point-lookup merge semantics. Since the default
native format follows the effective base-file format, this rule as stated would
imply the metadata table writes native HFile logs at v10, yet the reader is
described as expecting native Parquet. Could the spec clarify whether the
metadata table is exempt (continues writing inline HFile logs) or migrates to
native logs at v10? @yihua could you confirm the intended metadata-table log
behavior under table version 10 so the exception list here is complete?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
