liuliquan-marshal opened a new issue, #16863: URL: https://github.com/apache/iceberg/issues/16863
### Feature Request / Improvement # background see DISCUSSION: https://lists.apache.org/thread/m702c28cs8pspsv7to4xzp10dczb6m0m ## brief summary ### random read ``` Benchmark (bufferSizeKB) (fileIOClass) (fileSizeKB) Mode Cnt Score Error Units FileIOBenchmark.randomRead 1024 org.apache.iceberg.aws.s3.S3FileIO 131072 avgt 4 1817.108 ± 37.337 ms/op FileIOBenchmark.randomRead 1024 org.apache.iceberg.aliyun.oss.OSSFileIO 131072 avgt 5 27164.064 ± 24437.452 ms/op ``` OSSFileIO is more than 10x slower than S3FileIO. When a random read ends, `OSSInputStream` calls the underlying `close()` method, which continues to consume the remaining TCP data, causing unnecessary waiting. Besides, `OSSInputStream` does not implement `RangeReadable`. ### Sequential Write ``` Benchmark (bufferSizeKB) (fileIOClass) (fileSizeKB) Mode Cnt Score Error Units FileIOBenchmark.sequentialWrite 1024 org.apache.iceberg.aliyun.oss.OSSFileIO 1048576 avgt 5 4162.820 ± 162.809 ms/op FileIOBenchmark.sequentialWrite 1024 org.apache.iceberg.aws.s3.S3FileIO 1048576 avgt 4 1615.085 ± 73.897 ms/op ``` OSSFileIO is about 2x slower. The current OSSFileIO implementation writes data to a local file first, then uploads the entire file via the `PutObject` API. Write and huge file PutObject are serial executions causing bad performance and possible page cache flush. And 5GB size is limited due to PutObject API. # roadmap 1. Contribute the FileIO benchmark. 2. Improve `OSSInputStream` performance: fix `close()` performance bug(consuming all remaining unnecessary data) when seek and read . And implement RangeReadable(random read will actually invoke this after finishing this). 3. Improve `OSSOutputStream`: Upload part every 8MB(default, faster than 32MB) rather than PutObject. So far, daily use can be satisfied. 4. Upgrade OSS SDK version from V1 to V2(https://github.com/aliyun/alibabacloud-oss-java-sdk-v2) for better performance. 5. More performance improvements including vector read(multi range read) , analytics accelerator and etc. ### Query engine None ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
