HuangZhenQiu opened a new issue, #19598: URL: https://github.com/apache/hudi/issues/19598
### Feature Description **What the feature achieves:** The Dynamic Partitioned Cache introduces a RocksDB-based local index replica that adaptively trades off bootstrap time against RLI lookup latency based on the write patterns of the target dataset. By accounting for workload-specific write characteristics, the cache can determine how much local index state to materialize during bootstrap, balancing initialization cost with the lookup performance achieved once the replica is serving traffic **Why this feature is needed:** In Hudi's Flink upsert pipeline, the BucketAssigner operator must determine whether each incoming record is an insert or an update by looking up its record key in the index. RFC-106 introduces an in-memory cache to accelerate these lookups, but for large-scale streaming workloads, this approach has fundamental limitations: - Unbounded Cost: Each RLI entry requires approximately 50–70 bytes of memory. For a table containing 1 billion records, caching the entire index would consume 50–70 GB of JVM heap. In addition, a record buffer is required to improve RLI lookup efficiency. For CDC workloads with high event throughput (QPS) and large record sizes, maintaining a two-minute buffer can further increase memory consumption significantly. As a result, the compute cost of upsert ingestion workloads can rise substantially. - Cache thrashing: With bounded memory, the cache must evict entries aggressively. For workloads that access records across many partitions, this leads to frequent cache misses and fallback to MDT queries (10+ ms per record), severely degrading throughput. - Cold start latency: On job restart or task failover, the in-memory cache starts empty. Warming the cache through individual MDT lookups creates a prolonged period of degraded performance. ### User Experience **How users will use this feature:** - Configuration changes needed - API changes - Usage examples ### Hudi RFC Requirements **RFC PR link:** (if applicable) https://github.com/apache/hudi/pull/19046/changes **Why RFC is/isn't needed:** - Does this change public interfaces/APIs? (Yes) - Does this change storage format? (No) - Justification: The new design will introduce a new RocksDBPartitionedIndexBackend that needs to be integrated into BucketAssign operator. For some use cases, that the partition info can't be provided in the input record, users may use newly provided classes and interface to compose their Flink upsert pipelines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
