XiaoHongbo-Hope commented on PR #7143: URL: https://github.com/apache/paimon/pull/7143#issuecomment-3828728740
> Maybe async is enough? Multiple threads may consume too many memory. And has there been any performance test to measure the improvement? Current read is synchronous. Yes, we have the performance test of data loader, that is the code base of performance test from `pai`. single thread 200MB/s vs 260~270MB/s with 16 worker (process) and 10 prefetch threads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
