This is an automated email from the ASF dual-hosted git repository. voonhous pushed a commit to branch release-1.2.1 in repository https://gitbox.apache.org/repos/asf/hudi.git
commit 1025464a3d1e97bd4e5ef1dfa9a1c6d631a84b8f Author: Danny Chan <[email protected]> AuthorDate: Thu Jun 11 12:56:25 2026 +0800 docs: RFC-106 - Record Level and Secondary Index Support for Flink Writers (#17610) (cherry picked from commit 84c1f591481d41bc8ffac313e5ae992db19ed3d5) --- rfc/README.md | 213 ++++++++++++++-------------- rfc/rfc-106/index-compaction-flow.png | Bin 0 -> 329243 bytes rfc/rfc-106/index-write-flow.png | Bin 0 -> 375082 bytes rfc/rfc-106/rfc-106.md | 260 ++++++++++++++++++++++++++++++++++ rfc/rfc-106/rli-access-pattern.png | Bin 0 -> 96912 bytes 5 files changed, 368 insertions(+), 105 deletions(-) diff --git a/rfc/README.md b/rfc/README.md index 20b1cd53529b..512ee1bc9b01 100644 --- a/rfc/README.md +++ b/rfc/README.md @@ -34,108 +34,111 @@ The list of all RFCs can be found here. > Older RFC content is still > [here](https://cwiki.apache.org/confluence/display/HUDI/RFC+Process). -| RFC Number | Title | Status | -|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------| -| 1 | [CSV Source Support for Delta Streamer](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+01+%3A+CSV+Source+Support+for+Delta+Streamer) | :white_check_mark: `COMPLETED` | -| 2 | [ORC Storage in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708439) | :white_check_mark: `COMPLETED` | -| 3 | [Timeline Service with Incremental File System View Syncing](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708965) | :white_check_mark: `COMPLETED` | -| 4 | [Faster Hive incremental pull queries](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115513622) | :white_check_mark: `COMPLETED` | -| 5 | [HUI (Hudi WebUI)](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130027233) | :x: `ABANDONED` | -| 6 | [Add indexing support to the log file](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+06+%3A+Add+indexing+support+to+the+log+file) | :x: `ABANDONED` | -| 7 | [Point in time Time-Travel queries on Hudi table](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+07+%3A+Point+in+time+Time-Travel+queries+on+Hudi+table) | :white_check_mark: `COMPLETED` | -| 8 | [Metadata based Record Index](./rfc-8/rfc-8.md) | :white_check_mark: `COMPLETED` | -| 9 | [Hudi Dataset Snapshot Exporter](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+09+%3A+Hudi+Dataset+Snapshot+Exporter) | :white_check_mark: `COMPLETED` | -| 10 | [Restructuring and auto-generation of docs](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+10+%3A+Restructuring+and+auto-generation+of+docs) | :white_check_mark: `COMPLETED` | -| 11 | [Refactor of the configuration framework of hudi project](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+11+%3A+Refactor+of+the+configuration+framework+of+hudi+project) | :x: `ABANDONED` | -| 12 | [Efficient Migration of Large Parquet Tables to Apache Hudi](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+12+%3A+Efficient+Migration+of+Large+Parquet+Tables+to+Apache+Hudi) | :white_check_mark: `COMPLETED` | -| 13 | [Integrate Hudi with Flink](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=141724520) | :white_check_mark: `COMPLETED` | -| 14 | [JDBC incremental puller](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller) | :white_check_mark: `COMPLETED` | -| 15 | [HUDI File Listing Improvements](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements) | :white_check_mark: `COMPLETED` | -| 16 | [Abstraction for HoodieInputFormat and RecordReader](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+16+Abstraction+for+HoodieInputFormat+and+RecordReader) | :white_check_mark: `COMPLETED` | -| 17 | [Abstract common meta sync module support multiple meta service](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+17+Abstract+common+meta+sync+module+support+multiple+meta+service) | :white_check_mark: `COMPLETED` | -| 18 | [Insert Overwrite API](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+18+Insert+Overwrite+API) | :white_check_mark: `COMPLETED` | -| 19 | [Clustering data for freshness and query performance](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+19+Clustering+data+for+freshness+and+query+performance) | :white_check_mark: `COMPLETED` | -| 20 | [handle failed records](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+20+%3A+handle+failed+records) | :arrows_counterclockwise: `ONGOING` | -| 21 | [Allow HoodieRecordKey to be Virtual](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+21+%3A+Allow+HoodieRecordKey+to+be+Virtual) | :white_check_mark: `COMPLETED` | -| 22 | [Snapshot Isolation using Optimistic Concurrency Control for multi-writers](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+22+%3A+Snapshot+Isolation+using+Optimistic+Concurrency+Control+for+multi-writers) | :white_check_mark: `COMPLETED` | -| 23 | [Hudi Observability metrics collection](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+23+%3A+Hudi+Observability+metrics+collection) | :x: `ABANDONED` | -| 24 | [Hoodie Flink Writer Proposal](https://cwiki.apache.org/confluence/display/HUDI/RFC-24%3A+Hoodie+Flink+Writer+Proposal) | :white_check_mark: `COMPLETED` | -| 25 | [Spark SQL Extension For Hudi](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+25%3A+Spark+SQL+Extension+For+Hudi) | :white_check_mark: `COMPLETED` | -| 26 | [Optimization For Hudi Table Query](https://cwiki.apache.org/confluence/display/HUDI/RFC-26+Optimization+For+Hudi+Table+Query) | :white_check_mark: `COMPLETED` | -| 27 | [Data skipping index to improve query performance](https://cwiki.apache.org/confluence/display/HUDI/RFC-27+Data+skipping+index+to+improve+query+performance) | :white_check_mark: `COMPLETED` | -| 28 | [Support Z-order curve](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181307144) | :white_check_mark: `COMPLETED` | -| 29 | [Hash Index](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+29%3A+Hash+Index) | :white_check_mark: `COMPLETED` | -| 30 | [Batch operation](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+30%3A+Batch+operation) | :x: `ABANDONED` | -| 31 | [Hive integration Improvement](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+31%3A+Hive+integration+Improvment) | :x: `ABANDONED` | -| 32 | [Kafka Connect Sink for Hudi](https://cwiki.apache.org/confluence/display/HUDI/RFC-32+Kafka+Connect+Sink+for+Hudi) | :arrows_counterclockwise: `ONGOING` | -| 33 | [Hudi supports more comprehensive Schema Evolution](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+33++Hudi+supports+more+comprehensive+Schema+Evolution) | :white_check_mark: `COMPLETED` | -| 34 | [Hudi BigQuery Integration](./rfc-34/rfc-34.md) | :white_check_mark: `COMPLETED` | -| 35 | [Make Flink MOR table writing streaming friendly](https://cwiki.apache.org/confluence/display/HUDI/RFC-35%3A+Make+Flink+MOR+table+writing+streaming+friendly) | :white_check_mark: `COMPLETED` | -| 36 | [HUDI Metastore Server](https://cwiki.apache.org/confluence/display/HUDI/%5BWIP%5D+RFC-36%3A+HUDI+Metastore+Server) | :arrows_counterclockwise: `ONGOING` | -| 37 | [Hudi Metadata based Bloom Index](rfc-37/rfc-37.md) | :white_check_mark: `COMPLETED` | -| 38 | [Spark Datasource V2 Integration](./rfc-38/rfc-38.md) | :white_check_mark: `COMPLETED` | -| 39 | [Incremental source for Debezium](./rfc-39/rfc-39.md) | :white_check_mark: `COMPLETED` | -| 40 | [Connector for Trino](./rfc-40/rfc-40.md) | :white_check_mark: `COMPLETED` | -| 41 | [Snowflake Integration](./rfc-41/rfc-41.md), supported via [Apache XTable (Incubating)](https://xtable.apache.org/) | :x: `ABANDONED` | -| 42 | [Consistent Hashing Index](./rfc-42/rfc-42.md) | :arrows_counterclockwise: `ONGOING` | -| 43 | [Table Management Service](./rfc-43/rfc-43.md) | :x: `ABANDONED` | -| 44 | [Hudi Connector for Presto](./rfc-44/rfc-44.md) | :white_check_mark: `COMPLETED` | -| 45 | [Asynchronous Metadata Indexing](./rfc-45/rfc-45.md) | :white_check_mark: `COMPLETED` | -| 46 | [Optimizing Record Payload Handling](./rfc-46/rfc-46.md) | :white_check_mark: `COMPLETED` | -| 47 | [Add Call Produce Command for Spark SQL](./rfc-47/rfc-47.md) | :white_check_mark: `COMPLETED` | -| 48 | [LogCompaction for MOR tables](./rfc-48/rfc-48.md) | :white_check_mark: `COMPLETED` | -| 49 | [Support sync with DataHub](./rfc-49/rfc-49.md) | :white_check_mark: `COMPLETED` | -| 50 | [Improve Timeline Server](./rfc-50/rfc-50.md) | :x: `ABANDONED` | -| 51 | [Change Data Capture](./rfc-51/rfc-51.md) | :arrows_counterclockwise: `ONGOING` | -| 52 | [Introduce Secondary Index to Improve HUDI Query Performance](./rfc-52/rfc-52.md) | :x: `ABANDONED` | -| 53 | [Use Lock-Free Message Queue Improving Hoodie Writing Efficiency](./rfc-53/rfc-53.md) | :white_check_mark: `COMPLETED` | -| 54 | [New Table APIs and Streamline Hudi Configs](./rfc-54/rfc-54.md) | :x: `ABANDONED` | -| 55 | [Improve Hive/Meta sync class design and hierarchies](./rfc-55/rfc-55.md) | :white_check_mark: `COMPLETED` | -| 56 | [Early Conflict Detection For Multi-Writer](./rfc-56/rfc-56.md) | :white_check_mark: `COMPLETED` | -| 57 | [DeltaStreamer Protobuf Support](./rfc-57/rfc-57.md) | :white_check_mark: `COMPLETED` | -| 58 | [Integrate column stats index with all query engines](./rfc-58/rfc-58.md) | :white_check_mark: `COMPLETED` | -| 59 | [Multiple event_time Fields Latest Verification in a Single Table](./rfc-59/rfc-59.md) | :eyes: `UNDER REVIEW` | -| 60 | [Federated Storage Layer](./rfc-60/rfc-60.md) | :eyes: `UNDER REVIEW` | -| 61 | [Snapshot view management](./rfc-61/rfc-61.md) | :eyes: `UNDER REVIEW` | -| 62 | [Diagnostic Reporter](./rfc-62/rfc-62.md) | :eyes: `UNDER REVIEW` | -| 63 | [Expression Indexes](./rfc-63/rfc-63.md) | :arrows_counterclockwise: `ONGOING` | -| 64 | [New Hudi Table Spec API for Query Integrations](./rfc-64/rfc-64.md) | :eyes: `UNDER REVIEW` | -| 65 | [Partition TTL Management](./rfc-65/rfc-65.md) | :white_check_mark: `COMPLETED` | -| 66 | [Non Blocking Concurrency Control](./rfc-66/rfc-66.md) | :white_check_mark: `COMPLETED` | -| 67 | [Hudi Bundle Standards](./rfc-67/rfc-67.md) | :white_check_mark: `COMPLETED` | -| 68 | [A More Effective HoodieMergeHandler for COW Table with Parquet](./rfc-68/rfc-68.md) | :x: `ABANDONED` | -| 69 | [Hudi 1.x](./rfc-69/rfc-69.md) | :white_check_mark: `COMPLETED` | -| 70 | [Hudi Reverse Streamer](./rfc/rfc-70/rfc-70.md) | :eyes: `UNDER REVIEW` | -| 71 | [Enhance OCC conflict detection](./rfc/rfc-71/rfc-71.md) | :eyes: `UNDER REVIEW` | -| 72 | [Redesign Hudi-Spark Integration](./rfc/rfc-72/rfc-72.md) | :arrows_counterclockwise: `ONGOING` | -| 73 | [Multi-Table Transactions](./rfc-73/rfc-73.md) | :eyes: `UNDER REVIEW` | -| 74 | [`HoodieStorage`: Hudi Storage Abstraction and APIs](./rfc-74/rfc-74.md) | :arrows_counterclockwise: `ONGOING` | -| 75 | [Hudi-Native HFile Reader and Writer](./rfc-75/rfc-75.md) | :white_check_mark: `COMPLETED` | -| 76 | [Auto Record key generation](./rfc-76/rfc-76.md) | :white_check_mark: `COMPLETED` | -| 77 | [Secondary Index](./rfc-77/rfc-77.md) | :white_check_mark: `COMPLETED` | -| 78 | [1.0 Migration](./rfc-78/rfc-78.md) | :hammer_and_wrench: `IN PROGRESS` | -| 79 | [Robust handling of spark task retries and failures](./rfc-79/rfc-79.md) | :x: `ABANDONED` | -| 80 | [Column Groups](./rfc-80/rfc-80.md) | :hammer_and_wrench: `IN PROGRESS` | -| 81 | [Introduce Primary Key Sorted Table](./rfc-81/rfc-81.md) | :eyes: `UNDER REVIEW` | -| 82 | [Concurrent schema evolution detection](./rfc-82/rfc-82.md) | :white_check_mark: `COMPLETED` | -| 83 | [Incremental Table Service](./rfc-83/rfc-83.md) | :white_check_mark: `COMPLETED` | -| 84 | [Optimized SerDe of `DataStream` in Flink operators](./rfc-84/rfc-84.md) | :white_check_mark: `COMPLETED` | -| 85 | [Hudi Issue and Sprint Management in Jira](./rfc-85/rfc-85.md) | :white_check_mark: `COMPLETED` | -| 86 | [DataFrame Implementation of HUDI write path](./rfc-86/rfc-86.md) | :eyes: `UNDER REVIEW` | -| 87 | [Avro elimination for Flink writer](./rfc-87/rfc-87.md) | :hammer_and_wrench: `IN PROGRESS` | -| 88 | [New Schema/DataType/Expression Abstractions](./rfc-88/rfc-88.md) | :eyes: `UNDER REVIEW` | -| 89 | [Dynamic Partition Level Bucket Index](./rfc-89/rfc-89.md) | :eyes: `UNDER REVIEW` | -| 90 | Add support for cancellable clustering table service plans | :eyes: `UNDER REVIEW` | -| 91 | Storage-based lock provider using conditional writes | :hammer_and_wrench: `IN PROGRESS` | -| 92 | Support Bitmap Index | :hammer_and_wrench: `IN PROGRESS` | -| 93 | [Pluggable Table Formats in Hudi](./rfc-93/rfc-93.md) | :hammer_and_wrench: `IN PROGRESS` | -| 94 | Hudi Timeline User Interface (UI) | :eyes: `UNDER REVIEW` | -| 95 | Hudi Flink Source Based on FLIP-27 | :eyes: `UNDER REVIEW` | -| 96 | Introduce Unified Bucket Index | :eyes: `UNDER REVIEW` | -| 97 | Deprecate Hudi Payload Class Usage | :eyes: `UNDER REVIEW` | -| 98 | [Spark Datasource V2 Read](./rfc-98/rfc-98.md) | :eyes: `UNDER REVIEW` | -| 99 | [Hudi Type System Redesign](./rfc-99/rfc-99.md) | :eyes: `UNDER REVIEW` | -| 100 | [Unstructured Data Storage in Hudi](./rfc-100/rfc-100.md) | :eyes: `UNDER REVIEW` | -| 101 | [Updates to the HoodieRecordMerger API](./rfc-101/rfc-101.md) | :hammer_and_wrench: `IN PROGRESS` | -| 102 | RLI support for Flink streaming | :eyes: `UNDER REVIEW` | -| 103 | Hudi LSM tree layout | :eyes: `UNDER REVIEW` | \ No newline at end of file +| RFC Number | Title | Status | +| ------------ |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------------------------- | +| 1 | [CSV Source Support for Delta Streamer](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+01+%3A+CSV+Source+Support+for+Delta+Streamer) | :white_check_mark: `COMPLETED` | +| 2 | [ORC Storage in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708439) | :white_check_mark: `COMPLETED` | +| 3 | [Timeline Service with Incremental File System View Syncing](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708965) | :white_check_mark: `COMPLETED` | +| 4 | [Faster Hive incremental pull queries](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115513622) | :white_check_mark: `COMPLETED` | +| 5 | [HUI (Hudi WebUI)](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130027233) | :x: `ABANDONED` | +| 6 | [Add indexing support to the log file](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+06+%3A+Add+indexing+support+to+the+log+file) | :x: `ABANDONED` | +| 7 | [Point in time Time-Travel queries on Hudi table](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+07+%3A+Point+in+time+Time-Travel+queries+on+Hudi+table) | :white_check_mark: `COMPLETED` | +| 8 | [Metadata based Record Index](./rfc-8/rfc-8.md) | :white_check_mark: `COMPLETED` | +| 9 | [Hudi Dataset Snapshot Exporter](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+09+%3A+Hudi+Dataset+Snapshot+Exporter) | :white_check_mark: `COMPLETED` | +| 10 | [Restructuring and auto-generation of docs](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+10+%3A+Restructuring+and+auto-generation+of+docs) | :white_check_mark: `COMPLETED` | +| 11 | [Refactor of the configuration framework of hudi project](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+11+%3A+Refactor+of+the+configuration+framework+of+hudi+project) | :x: `ABANDONED` | +| 12 | [Efficient Migration of Large Parquet Tables to Apache Hudi](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+12+%3A+Efficient+Migration+of+Large+Parquet+Tables+to+Apache+Hudi) | :white_check_mark: `COMPLETED` | +| 13 | [Integrate Hudi with Flink](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=141724520) | :white_check_mark: `COMPLETED` | +| 14 | [JDBC incremental puller](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller) | :white_check_mark: `COMPLETED` | +| 15 | [HUDI File Listing Improvements](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements) | :white_check_mark: `COMPLETED` | +| 16 | [Abstraction for HoodieInputFormat and RecordReader](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+16+Abstraction+for+HoodieInputFormat+and+RecordReader) | :white_check_mark: `COMPLETED` | +| 17 | [Abstract common meta sync module support multiple meta service](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+17+Abstract+common+meta+sync+module+support+multiple+meta+service) | :white_check_mark: `COMPLETED` | +| 18 | [Insert Overwrite API](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+18+Insert+Overwrite+API) | :white_check_mark: `COMPLETED` | +| 19 | [Clustering data for freshness and query performance](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+19+Clustering+data+for+freshness+and+query+performance) | :white_check_mark: `COMPLETED` | +| 20 | [handle failed records](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+20+%3A+handle+failed+records) | :arrows_counterclockwise: `ONGOING` | +| 21 | [Allow HoodieRecordKey to be Virtual](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+21+%3A+Allow+HoodieRecordKey+to+be+Virtual) | :white_check_mark: `COMPLETED` | +| 22 | [Snapshot Isolation using Optimistic Concurrency Control for multi-writers](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+22+%3A+Snapshot+Isolation+using+Optimistic+Concurrency+Control+for+multi-writers) | :white_check_mark: `COMPLETED` | +| 23 | [Hudi Observability metrics collection](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+23+%3A+Hudi+Observability+metrics+collection) | :x: `ABANDONED` | +| 24 | [Hoodie Flink Writer Proposal](https://cwiki.apache.org/confluence/display/HUDI/RFC-24%3A+Hoodie+Flink+Writer+Proposal) | :white_check_mark: `COMPLETED` | +| 25 | [Spark SQL Extension For Hudi](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+25%3A+Spark+SQL+Extension+For+Hudi) | :white_check_mark: `COMPLETED` | +| 26 | [Optimization For Hudi Table Query](https://cwiki.apache.org/confluence/display/HUDI/RFC-26+Optimization+For+Hudi+Table+Query) | :white_check_mark: `COMPLETED` | +| 27 | [Data skipping index to improve query performance](https://cwiki.apache.org/confluence/display/HUDI/RFC-27+Data+skipping+index+to+improve+query+performance) | :white_check_mark: `COMPLETED` | +| 28 | [Support Z-order curve](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181307144) | :white_check_mark: `COMPLETED` | +| 29 | [Hash Index](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+29%3A+Hash+Index) | :white_check_mark: `COMPLETED` | +| 30 | [Batch operation](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+30%3A+Batch+operation) | :x: `ABANDONED` | +| 31 | [Hive integration Improvement](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+31%3A+Hive+integration+Improvment) | :x: `ABANDONED` | +| 32 | [Kafka Connect Sink for Hudi](https://cwiki.apache.org/confluence/display/HUDI/RFC-32+Kafka+Connect+Sink+for+Hudi) | :arrows_counterclockwise: `ONGOING` | +| 33 | [Hudi supports more comprehensive Schema Evolution](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+33++Hudi+supports+more+comprehensive+Schema+Evolution) | :white_check_mark: `COMPLETED` | +| 34 | [Hudi BigQuery Integration](./rfc-34/rfc-34.md) | :white_check_mark: `COMPLETED` | +| 35 | [Make Flink MOR table writing streaming friendly](https://cwiki.apache.org/confluence/display/HUDI/RFC-35%3A+Make+Flink+MOR+table+writing+streaming+friendly) | :white_check_mark: `COMPLETED` | +| 36 | [HUDI Metastore Server](https://cwiki.apache.org/confluence/display/HUDI/%5BWIP%5D+RFC-36%3A+HUDI+Metastore+Server) | :arrows_counterclockwise: `ONGOING` | +| 37 | [Hudi Metadata based Bloom Index](rfc-37/rfc-37.md) | :white_check_mark: `COMPLETED` | +| 38 | [Spark Datasource V2 Integration](./rfc-38/rfc-38.md) | :white_check_mark: `COMPLETED` | +| 39 | [Incremental source for Debezium](./rfc-39/rfc-39.md) | :white_check_mark: `COMPLETED` | +| 40 | [Connector for Trino](./rfc-40/rfc-40.md) | :white_check_mark: `COMPLETED` | +| 41 | [Snowflake Integration](./rfc-41/rfc-41.md), supported via [Apache XTable (Incubating)](https://xtable.apache.org/) | :x: `ABANDONED` | +| 42 | [Consistent Hashing Index](./rfc-42/rfc-42.md) | :arrows_counterclockwise: `ONGOING` | +| 43 | [Table Management Service](./rfc-43/rfc-43.md) | :x: `ABANDONED` | +| 44 | [Hudi Connector for Presto](./rfc-44/rfc-44.md) | :white_check_mark: `COMPLETED` | +| 45 | [Asynchronous Metadata Indexing](./rfc-45/rfc-45.md) | :white_check_mark: `COMPLETED` | +| 46 | [Optimizing Record Payload Handling](./rfc-46/rfc-46.md) | :white_check_mark: `COMPLETED` | +| 47 | [Add Call Produce Command for Spark SQL](./rfc-47/rfc-47.md) | :white_check_mark: `COMPLETED` | +| 48 | [LogCompaction for MOR tables](./rfc-48/rfc-48.md) | :white_check_mark: `COMPLETED` | +| 49 | [Support sync with DataHub](./rfc-49/rfc-49.md) | :white_check_mark: `COMPLETED` | +| 50 | [Improve Timeline Server](./rfc-50/rfc-50.md) | :x: `ABANDONED` | +| 51 | [Change Data Capture](./rfc-51/rfc-51.md) | :arrows_counterclockwise: `ONGOING` | +| 52 | [Introduce Secondary Index to Improve HUDI Query Performance](./rfc-52/rfc-52.md) | :x: `ABANDONED` | +| 53 | [Use Lock-Free Message Queue Improving Hoodie Writing Efficiency](./rfc-53/rfc-53.md) | :white_check_mark: `COMPLETED` | +| 54 | [New Table APIs and Streamline Hudi Configs](./rfc-54/rfc-54.md) | :x: `ABANDONED` | +| 55 | [Improve Hive/Meta sync class design and hierarchies](./rfc-55/rfc-55.md) | :white_check_mark: `COMPLETED` | +| 56 | [Early Conflict Detection For Multi-Writer](./rfc-56/rfc-56.md) | :white_check_mark: `COMPLETED` | +| 57 | [DeltaStreamer Protobuf Support](./rfc-57/rfc-57.md) | :white_check_mark: `COMPLETED` | +| 58 | [Integrate column stats index with all query engines](./rfc-58/rfc-58.md) | :white_check_mark: `COMPLETED` | +| 59 | [Multiple event_time Fields Latest Verification in a Single Table](./rfc-59/rfc-59.md) | :eyes: `UNDER REVIEW` | +| 60 | [Federated Storage Layer](./rfc-60/rfc-60.md) | :eyes: `UNDER REVIEW` | +| 61 | [Snapshot view management](./rfc-61/rfc-61.md) | :eyes: `UNDER REVIEW` | +| 62 | [Diagnostic Reporter](./rfc-62/rfc-62.md) | :eyes: `UNDER REVIEW` | +| 63 | [Expression Indexes](./rfc-63/rfc-63.md) | :arrows_counterclockwise: `ONGOING` | +| 64 | [New Hudi Table Spec API for Query Integrations](./rfc-64/rfc-64.md) | :eyes: `UNDER REVIEW` | +| 65 | [Partition TTL Management](./rfc-65/rfc-65.md) | :white_check_mark: `COMPLETED` | +| 66 | [Non Blocking Concurrency Control](./rfc-66/rfc-66.md) | :white_check_mark: `COMPLETED` | +| 67 | [Hudi Bundle Standards](./rfc-67/rfc-67.md) | :white_check_mark: `COMPLETED` | +| 68 | [A More Effective HoodieMergeHandler for COW Table with Parquet](./rfc-68/rfc-68.md) | :x: `ABANDONED` | +| 69 | [Hudi 1.x](./rfc-69/rfc-69.md) | :white_check_mark: `COMPLETED` | +| 70 | [Hudi Reverse Streamer](./rfc/rfc-70/rfc-70.md) | :eyes: `UNDER REVIEW` | +| 71 | [Enhance OCC conflict detection](./rfc/rfc-71/rfc-71.md) | :eyes: `UNDER REVIEW` | +| 72 | [Redesign Hudi-Spark Integration](./rfc/rfc-72/rfc-72.md) | :arrows_counterclockwise: `ONGOING` | +| 73 | [Multi-Table Transactions](./rfc-73/rfc-73.md) | :eyes: `UNDER REVIEW` | +| 74 | [`HoodieStorage`: Hudi Storage Abstraction and APIs](./rfc-74/rfc-74.md) | :arrows_counterclockwise: `ONGOING` | +| 75 | [Hudi-Native HFile Reader and Writer](./rfc-75/rfc-75.md) | :white_check_mark: `COMPLETED` | +| 76 | [Auto Record key generation](./rfc-76/rfc-76.md) | :white_check_mark: `COMPLETED` | +| 77 | [Secondary Index](./rfc-77/rfc-77.md) | :white_check_mark: `COMPLETED` | +| 78 | [1.0 Migration](./rfc-78/rfc-78.md) | :hammer_and_wrench: `IN PROGRESS` | +| 79 | [Robust handling of spark task retries and failures](./rfc-79/rfc-79.md) | :x: `ABANDONED` | +| 80 | [Column Groups](./rfc-80/rfc-80.md) | :hammer_and_wrench: `IN PROGRESS` | +| 81 | [Introduce Primary Key Sorted Table](./rfc-81/rfc-81.md) | :eyes: `UNDER REVIEW` | +| 82 | [Concurrent schema evolution detection](./rfc-82/rfc-82.md) | :white_check_mark: `COMPLETED` | +| 83 | [Incremental Table Service](./rfc-83/rfc-83.md) | :white_check_mark: `COMPLETED` | +| 84 | [Optimized SerDe of `DataStream` in Flink operators](./rfc-84/rfc-84.md) | :white_check_mark: `COMPLETED` | +| 85 | [Hudi Issue and Sprint Management in Jira](./rfc-85/rfc-85.md) | :white_check_mark: `COMPLETED` | +| 86 | [DataFrame Implementation of HUDI write path](./rfc-86/rfc-86.md) | :eyes: `UNDER REVIEW` | +| 87 | [Avro elimination for Flink writer](./rfc-87/rfc-87.md) | :hammer_and_wrench: `IN PROGRESS` | +| 88 | [New Schema/DataType/Expression Abstractions](./rfc-88/rfc-88.md) | :eyes: `UNDER REVIEW` | +| 89 | [Dynamic Partition Level Bucket Index](./rfc-89/rfc-89.md) | :eyes: `UNDER REVIEW` | +| 90 | Add support for cancellable clustering table service plans | :eyes: `UNDER REVIEW` | +| 91 | [Storage-based lock provider using conditional writes](./rfc-91/rfc-91.md) | :white_check_mark: `COMPLETED` | +| 92 | Support Bitmap Index | :hammer_and_wrench: `IN PROGRESS` | +| 93 | [Pluggable Table Formats in Hudi](./rfc-93/rfc-93.md) | :hammer_and_wrench: `IN PROGRESS` | +| 94 | Hudi Timeline User Interface (UI) | :eyes: `UNDER REVIEW` | +| 95 | [Hudi Flink Source Based on FLIP-27](./rfc-95/rfc-95.md) | :white_check_mark: `COMPLETED` | +| 96 | Introduce Unified Bucket Index | :eyes: `UNDER REVIEW` | +| 97 | Deprecate Hudi Payload Class Usage | :eyes: `UNDER REVIEW` | +| 98 | [Spark Datasource V2 Read](./rfc-98/rfc-98.md) | :eyes: `UNDER REVIEW` | +| 99 | [Hudi Type System Redesign](./rfc-99/rfc-99.md) | :eyes: `UNDER REVIEW` | +| 100 | [Unstructured Data Storage in Hudi](./rfc-100/rfc-100.md) | :eyes: `UNDER REVIEW` | +| 101 | [Updates to the HoodieRecordMerger API](./rfc-101/rfc-101.md) | :hammer_and_wrench: `IN PROGRESS` | +| 102 | [Spark Batch Vector Search in Apache Hudi](./rfc-102/rfc-102/md) | :white_check_mark: `COMPLETED` | +| 103 | Hudi LSM tree layout | :eyes: `UNDER REVIEW` | +| 104 | [Unify schema evolution on schema-on-read](./rfc-104/rfc-104.md) | :eyes: `UNDER REVIEW` | +| 105 | [Trino Hudi Connector — Shim/Bundle Refactor](./rfc-105/rfc-105.md) | :eyes: `UNDER REVIEW` | +| 106 | [Record Level and Secondary Index Support for Flink Writers](./rfc-106/rfc-106.md) | :white_check_mark: `COMPLETED` | diff --git a/rfc/rfc-106/index-compaction-flow.png b/rfc/rfc-106/index-compaction-flow.png new file mode 100644 index 000000000000..a81e1d4f1f23 Binary files /dev/null and b/rfc/rfc-106/index-compaction-flow.png differ diff --git a/rfc/rfc-106/index-write-flow.png b/rfc/rfc-106/index-write-flow.png new file mode 100644 index 000000000000..6259071a55f6 Binary files /dev/null and b/rfc/rfc-106/index-write-flow.png differ diff --git a/rfc/rfc-106/rfc-106.md b/rfc/rfc-106/rfc-106.md new file mode 100644 index 000000000000..d344ccf579f9 --- /dev/null +++ b/rfc/rfc-106/rfc-106.md @@ -0,0 +1,260 @@ + <!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +# RFC-106: Record Level and Secondary Index Support for Flink Writers + +## Proposers + +- @danny0405 + +## Approvers + - @geserdugarov + - @vinothchandar + - @cshuo + +## Status + +GH Discussion: https://github.com/apache/hudi/discussions/17452 + +> Please keep the status updated in `rfc/README.md`. + +## Abstract + +Apache Hudi provides multiple indexing strategies to efficiently locate records during upsert operations. +The **Record Level Index (RLI)** is a global index stored in Hudi's **Metadata Table (MDT)** that maps each record key to its +exact file group location, enabling O(1) lookups. The **Secondary Index (SI)** extends this capability to non-record-key, non-unique-key columns. +Currently, Spark reads/writes support RLI & SI while Flink does not, creating feature disparity between the two engines for Hudi table reads and writes. + +This RFC proposes adding RLI and SI support for Flink streaming writes. Throughout this document, the term **"index"** refers broadly to both RLI and SI; +when discussing behavior specific to one type, the terms "RLI" or "SI" will be used explicitly. + +The goals of this RFC are: + +- Provide reliable and performant write support for RLI/SI using Flink APIs +- Ensure cross-engine compatibility so that Flink can access and utilize indexes written by Spark, and vice versa +- Support global RLI for cross-partition upserts, as well as partition-level RLI for large fact tables +- Enable asynchronous compaction for MDT when indexing is enabled, either within the writer pipeline or via background table services +- Implement smart caching of index data for low-latency access during streaming writes +- Document scale and performance limits for write throughput supported by indexing (based on empirical benchmarks) +- Design the implementation to be extensible for arbitrary secondary indexing on different columns + +## Background + +Apache Hudi uses indexes to determine the location of existing records when processing upserts. Without an efficient index, Hudi would +need to scan the entire table to find whether a record already exists and where it is located. Different index types offer different +trade-offs between write performance, read performance, and resource consumption. + +Currently, Flink Hudi sink does not support RLI or SI, while Hudi Spark datasource does and proven at [massive production scale](https://hudi.apache.org/blog/2023/11/01/record-level-index/). +This inconsistency causes friction for users who migrate tables from Spark to Flink streaming. When migrating, users must switch +the index type from RLI/SI to either `bucket` (a hash-based partitioning scheme) or `flink_state` (which uses Flink's state backend to +maintain record-to-location mappings). This migration overhead complicates production deployments. + +Another key motivation is to provide scalable, efficient support for **cross-partition updates**—scenarios where a record's partition path changes between writes. +Currently, the only option for handling cross-partition updates in Flink is the `flink_state` index, which maintains a global view of all record locations. However, this approach has significant drawbacks: +it consumes substantial memory (proportional to the table size) and cannot be shared across different workloads or job restarts without state migration. + +## High Level Design + +The high-level design introduces the following components: + +- **MDT-based Index backend**: A new index implementation that can replace the current `flink_state` index, storing record-to-location mappings in the MDT rather than in Flink's state backend +- **Index cache with invalidation**: An in-memory cache to accelerate RLI lookups, along with a cache invalidation mechanism to maintain consistency with the committed state of the table +- **New Flink Index operator**: A separate Flink operator (`IndexWrite`) responsible for writing RLI/SI payloads to the MDT +- **Synchronous MDT writes**: The MDT's RLI and SI files are written synchronously with the data table files within the same commit boundary; the metadata is then sent to the coordinator for a final commit to the MDT (after the `FILES` partition update is computed) +- **Asynchronous MDT compaction**: MDT compaction is performed asynchronously, reusing the existing data file compaction pipeline to minimize task slot consumption + + + +### Detailed Design + +### The Index Access + +In Hudi's Flink integration, the `BucketAssigner` operator is responsible for determining where each incoming record should be written. +It must identify whether each record is an insert (new record), update (existing record), or delete. To make this determination, the operator needs to look up whether +the record key already exists in the table and, if so, where it is located. + +With index support, the `BucketAssigner` operator will use the index metadata stored in the MDT as its backend. It will probe the index +with incoming record keys to determine the appropriate operation type (insert, update, or delete). In this design, the index serves the same role +that the `flink_state` index currently serves. Since the existing `BucketAssigner` already supports both **global** and **non-global** index types, +the global RLI will be used for **global** index configurations, while partitioned RLI will be used for **non-global** configurations. + +To optimize index access patterns and avoid caching all index shards in every `BucketAssigner` task, the input records will be shuffled +by `hash(record_key) % num_index_shards`. This uses the same hashing algorithm as the MDT's index partitioner, ensuring that +each `BucketAssigner` task only needs to read from a subset of index shards. + +#### Index Cache + +Streaming workloads require low-latency processing of each record to achieve high throughput. Thus, each record lookup against the index +should complete really fast. Reading a RLI entry each time for each record will incur 10+ms of latency per record and seriously affect throughput. + +**New index mappings cache:** Additionally, a separate memory cache is needed for index mappings created during the current checkpoint. +These mappings are not yet committed to the Hudi table and are therefore invisible to MDT queries. This cache must not be cleared until the +corresponding checkpoint/instant is committed to Hudi, which indicates that the index payloads have also been committed. This ensures multiple +records for the same record key (e,g insert to a key, followed by an update within the same commit boundary) are routed consistently to same +file group, preserving the 1:1 mapping from record key to file group. + +The cache stores `key -> location` mappings at the record level, the items are evicted by checkpoint level when the checkpoints are committed to Hudi. +(Note that the MDT reader also maintains its own native file-level cache.) + +The actual index writes occur in the `IndexWrite` operator and the location from the cache will be propagated downstream from the `BucketAssigner` operator, where cache lookups and MDT queries to determine record locations. +The cache is updated for new records and location changes, while the MDT is queried only for existing key locations. + +The cache update flow is as follows: + +1. Probe the cache for the key. If found, update the cache entry if the location has changed. +2. If the key is not in the cache, fall back to querying the MDT. If the key exists in the MDT, add it to the cache with its location. +3. If the key does not exist in the MDT either, add the new key and its assigned location to the cache. + +#### Index Access Consistency On Fail Cases + +Hudi uses a two-phase commit protocol where each Flink checkpoint corresponds to a Hudi completed instant. During a checkpoint, Flink completes the data writes and collects the Hudi commit metadata. +Once the checkpoint acknowledgment event is received, Flink knows the checkpoint completed successfully, and the corresponding Hudi instant can be committed. +However, the acknowledgment message is sent asynchronously on a best-effort basis and may be lost in corner cases. A Hudi instant cannot be committed without receiving this acknowledgment. + +During job restarts or task failover, there are scenarios where a Flink checkpoint succeeds but the corresponding Hudi instant remains uncommitted due to the two-phase commit mechanism: + +1. **Missing acknowledgment**: The acknowledgment message is lost entirely. +2. **Job restart during commit**: The acknowledgment is received, but the job restarts during the instant commit process: + 1. All write metadata from a checkpoint is collected in the coordinator and is ready for committing. + 2. The acknowledgment message is received. + 3. The coordinator begins committing the instant with the collected write metadata. + 4. The job restarts or crashes. + 5. The instant remains uncommitted even though the checkpoint succeeded. +3. **Task failover before acknowledgment**: A task fails over from a checkpoint before its acknowledgment is received: + 1. All write metadata from checkpoint `ckp_n` is collected in the coordinator and is ready for committing. + 2. The checkpoint succeeds, but the acknowledgment has not been received yet. + 3. A task fails and recovers from `ckp_n`. + 4. The instant remains uncommitted even though the checkpoint succeeded. + 5. The acknowledgment message is eventually received. + 6. During the gap between steps 4 and 5, the `BucketAssigner` must access the uncommitted instant because the checkpoint was successful and the data is valid. +4. **Task failover after acknowledgment but before commit completion**: A task fails over after the acknowledgment is received but before the instant commit completes: + 1. All write metadata from checkpoint `ckp_n` is collected in the coordinator and is ready for committing. + 2. The acknowledgment message is received. + 3. The coordinator begins committing the instant with the collected write metadata. + 4. A task fails and recovers from `ckp_n`. + 5. The instant remains uncommitted even though the checkpoint succeeded. + 6. Step 3 eventually completes and the instant is committed. + 7. During the gap between steps 4 and 6, the `BucketAssigner` must access the uncommitted instant because the checkpoint was successful and the data is valid. + +For data table (DT) metadata, the pipeline will recommit the instant using the recovered table metadata. However, since the `BucketAssigner` operator is upstream of the `StreamWrite` operator, there is a time gap before these inflight instants can be recommitted. +The gap between Flink checkpoint and Hudi instant commit will incur consistency issue on index read view, it is fixed by cases: + +- on task failover: the coordinator would recommit the pending instants with successful Flink checkpoints; +- on job restart: trigger an explicit job failover from coordinator after the recovered pending instant been recommitted. + +#### Add Event Time Ordering Value for RLI Payload +For cross-partition updates or deletes, we can not update the RLI directly based on the existing key-location mappings. Currently, the RLI payload only has the key to location mappings without actual ordering value. +we need to merge the data records to see if the incoming record is a valid update or delete(for valid, it means greater ordering value), that is the behavior for Spark RLI write path, but it is too costly for streaming. + +For e.g, for two records `r1:{key: k1, orderingValue: 2, partition: par1}` and `r2:{key: k1, orderingValue: 1, partition: par2}`, `r2` comes behind `r1` in a different commit, +comparison of just key existence is not enough, we need to also compare the ordering value to see that `r2` is not a valid update and +not sending the retraction record(delete record) into partition `par1` for payload delete. + +The suggested solution is to store the ordering value into the RLI payload, so that we can compare the ordering value when there is a match of exiting key lookup, to make the decision whether the incoming record is a valid +upserts or not. + +The query execution follows this order: first access the in-memory cache, then query the MDT index: + + + +### Shuffling Index Records + +In the `StreamWrite` operator, index records are derived from incoming data records and sent to the `IndexWrite` operator in a streaming +fashion. These index records are shuffled by `hash(record_key) % num_index_shards`, using the same hashing algorithm as the MDT's +index partitioner. This shuffling strategy is critical for avoiding a combinatorial explosion of files written to the MDT partition. +Without it, the number of files would be `N * M`, where `N` is the number of index partition buckets and `M` is the number of data table +buckets involved in the current write. + +To ensure that each data record and its corresponding index record always belong to the same commit/checkpoint, we leverage +Flink's barrier alignment mechanism. In Flink, checkpoint barriers flow together with records through the pipeline (see [how-does-state-snapshotting-work](https://nightlies.apache.org/flink/flink-docs-master/docs/learn-flink/fault_tolerance/#how-does-state-snapshotting-work)). +When the `StreamWrite` operator receives a record, it emits both the data record and its corresponding index record within a single `#processElement` call. +This ensures that the two records are never separated by a checkpoint barrier. + +For example: + +```text +// irN means an index record, drN means a data record +e.g: [r4 r3 r2 r1 ] => BucketAssignor => [ ir4 dr4 ir3 dr3 ir2 dr2 ir1 dr1] +``` + +The barrier propagation algorithm prevents the checkpoint barrier from being placed between an index record and its corresponding data record. +A placement like `[ ir4 dr4 ir3 dr3 ir2 <checkpoint barrier> dr2 ir1 dr1]` cannot occur. + +### The Index Write + +In the `IndexWrite` operator, index records are buffered and then written to the MDT when triggered by a Flink checkpoint. +The write status metadata is then sent to the `coordinator`. This metadata includes two parts: + +- **A**: The written data file paths +- **B**: The written MDT file paths (specifically those under the index partitions) + +#### Committing MDT (including Index Partitions) + +When committing to the data table, the MDT is committed first with the index write metadata (the MDT index partition file handles). +The `RLI` and `SI` partition file handles are committed together in the `FILES` partition. + +During a Flink checkpoint, each index-writing and data-writing task flushes all its records to the index and data files respectively. +This ensures that the index and data files are always consistent. Both are committed together from the Coordinator as a single Hudi commit, +following the current commit protocol. + +To maintain exactly-once semantics during job recovery, the write status metadata must be stored in multiple locations: the `StreamWrite` operator, the `IndexWrite` operator, and the `coordinator`. +This follows the same pattern as the current approach for maintaining data table metadata. + +### The Compaction + +To minimize task slot consumption, the implementation reuses the existing data file compaction sub-pipeline for MDT compaction. +This asynchronous compaction is automatically enabled when indexing is active. + + + +## Implementation Plan + +The umbrella issue: [Support record index for Flink writer](https://github.com/apache/hudi/issues/17647) + +- [Add RLI access cache to support efficient lookup](https://github.com/apache/hudi/issues/17697) +- [Adapter the BucketAssign function with the MDT backed index](https://github.com/apache/hudi/issues/17699) +- [Support mini-batch access to the MDT index for bucket assign function](https://github.com/apache/hudi/issues/17842) +- [Add basic infra to bookeep the mappings between checkpoint id to instant](https://github.com/apache/hudi/issues/17700) +- [Add a new index write function](https://github.com/apache/hudi/issues/17701) +- [Integrate the MDT compaction with existing compaction sub-pipeline and offline job](https://github.com/apache/hudi/issues/17702) + +## Rollout/Adoption Plan + + - What impact (if any) will there be on existing users? + - No impact because this is a new feature. Existing Flink users can continue using their current index types, and + adoption of RLI/SI is optional. + +## Test Plan + +1. Verify that all write and read scenarios work correctly with indexing enabled. +2. Validate that asynchronous compaction works correctly with mixed DT/MDT workloads. +3. Add new tests to cover job recovery scenarios with uncommitted DT/MDT commits. +4. Conduct benchmarks to determine the upper throughput threshold at which indexing is recommended for streaming workloads. + +## Appendix + +### The Job/Task failover + +To maintain exactly-once semantics, the implementation includes infrastructure to support job and task recovery: + +1. In the `coordinator`, historical uncommitted metadata is persisted in the checkpoint state. +2. In the `StreamWrite` operator, the current write metadata list is persisted in the checkpoint state. + +When the job is restarted, the following steps are triggered to recover uncommitted instants: + +1. The `StreamWrite` operator checks for pending instants in the checkpoint state and resends an event to the `coordinator` to collect the uncommitted metadata. +2. The `coordinator` recovers its write metadata list from the checkpoint state. +3. The `coordinator` recommits the uncommitted instants using the combined metadata from steps 1 and 2. diff --git a/rfc/rfc-106/rli-access-pattern.png b/rfc/rfc-106/rli-access-pattern.png new file mode 100644 index 000000000000..00888f9c5a4a Binary files /dev/null and b/rfc/rfc-106/rli-access-pattern.png differ
