luoyuxia commented on PR #3630: URL: https://github.com/apache/fluss/pull/3630#issuecomment-4941968270
> Thanks for proposing this. @luoyuxia I’m also exploring this area with Fluss + Paimon. > > Regarding the proposed client fallback to server-side historical partition lookup, I’m a bit concerned that this could be risky. The lake lookup operation can be much slower than a local RocksDB lookup, especially when Paimon is backed by HDFS with poor performance. If this lookup is executed on the tablet server side, it may slow down Netty worker threads and affect the latency of other RPC requests. > > For the case without historical data ingestion, I think this could potentially be handled by direct client-side lake lookup instead. On the Fluss side, we could consider introducing a Flink hybrid lake lookup async function for this purpose. For the case with historical data ingestion, client-side lookup could still be used as a fallback when the historical partition + key is not found on the tablet server side. > > Overall, I would prefer using client-side PK lookup where possible, to avoid overloading tablet servers during historical data ingestion bursts. Thanks for sharing your thoughts. First, I’d like to clarify that the expensive Paimon lookup does not run on the Netty worker thread. It is handed off to the TabletServer IO executor through CompletableFuture.supplyAsync. Its concurrency is bounded by a semaphore, and excess requests receive a retriable throttling error with client-side backoff. I agree that historical lookup still consumes additional TabletServer CPU, memory, network IO, and local disk for the Paimon lookup cache. This is a resource trade-off, even though it does not block the Netty event loop. An important motivation for server-side historical lookup is historical data ingestion. We want to support writing into historical partitions while still producing the correct changelog. The server-side write path therefore needs to retrieve the previous value from historical storage before applying a write. Some server-accessible historical lookup infrastructure is required independently of the client fallback in this PR; the client lookup path reuses the same infrastructure. Server-side lookup also keeps the generic Fluss lookup API independent of a particular lake implementation. Clients do not need to manage Paimon/Hadoop dependencies, lake credentials, catalog configuration, key encoding, snapshot selection, and fallback semantics. A Flink hybrid lake lookup async function would be useful for Flink workloads, but Fluss also has Java, Rust, CPP, Python clients. Requiring every client to implement its own lake lookup and fallback logic would duplicate behavior and would be particularly difficult for Rust and Python clients when the lake implementation is JVM-based. I think the two approaches can coexist, similar to Paimon’s local lookup and remote query service models. A remote lookup service can provide generic lookup capability for historical ingestion and all Fluss clients. The current PR runs this capability on TabletServers, but it could potentially be moved to a dedicated service later for better resource isolation. Client-side local lookup can then be introduced as an optional optimization where practical, with Flink async lookup as one possible first integration. For this PR, my preference is to keep the bounded asynchronous server-side path because it also supports historical ingestion semantics, and explore client-side lookup as an optional strategy in a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
