zuston commented on PR #3630: URL: https://github.com/apache/fluss/pull/3630#issuecomment-4933546333
Thanks for proposing this. @luoyuxia I’m also exploring this area with Fluss + Paimon. Regarding the proposed client fallback to server-side historical partition lookup, I’m a bit concerned that this could be risky. The lake lookup operation can be much slower than a local RocksDB lookup, especially when Paimon is backed by HDFS with poor performance. If this lookup is executed on the tablet server side, it may slow down Netty worker threads and affect the latency of other RPC requests. For the case without historical data ingestion, I think this could potentially be handled by direct client-side lake lookup instead. On the Fluss side, we could consider introducing a Flink hybrid lake lookup async function for this purpose. For the case with historical data ingestion, client-side lookup could still be used as a fallback when the historical partition + key is not found on the tablet server side. Overall, I would prefer using client-side PK lookup where possible, to avoid overloading tablet servers during historical data ingestion bursts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
