chitralverma commented on issue #5943: URL: https://github.com/apache/opendal/issues/5943#issuecomment-2858120315
@messense I did some benchmarking [here](https://github.com/apache/opendal/issues/5938) to compare the effect of reader and writer options which are also exposed to the python bindings with #6086 . regarding the comparison of chunk read vs the new `write_from` method, i did a little benchmarking based on the code in #6086 , code and results below. ``` import asyncio import timeit import opendal SRC_LOCATION = "test.pq" DEST_LOCATION = "/test/bench_test.pq" S3_BUCKET = "xxx" AWS_ACCESS_KEY_ID = "xxx" AWS_SECRET_ACCESS_KEY = "xxx" AWS_ENDPOINT = "xxx" async def write_chunk(): src = opendal.AsyncOperator( "fs", root="/", ) dest = opendal.AsyncOperator( "s3", bucket=S3_BUCKET, aws_access_key_id=AWS_ACCESS_KEY_ID, aws_secret_access_key=AWS_SECRET_ACCESS_KEY, endpoint=AWS_ENDPOINT, ) async with ( await src.open(SRC_LOCATION, "rb") as reader, await dest.open(DEST_LOCATION, "wb", chunk=5*1024*1024, concurrent = 1024) as writer, ): while chunk := await reader.read(8192): await writer.write(chunk) async def write_lazy(): src = opendal.AsyncOperator( "fs", root="/", ) dest = opendal.AsyncOperator( "s3", bucket=S3_BUCKET, aws_access_key_id=AWS_ACCESS_KEY_ID, aws_secret_access_key=AWS_SECRET_ACCESS_KEY, endpoint=AWS_ENDPOINT, ) async with ( await src.open(SRC_LOCATION, "rb") as reader, await dest.open(DEST_LOCATION, "wb", chunk=5*1024*1024, concurrent = 1024) as writer, ): await writer.write_from(reader) def naive_run(): asyncio.run(write_chunk()) def lazy_run(): asyncio.run(write_lazy()) def naive_vs_lazy(): print(f"write_chunk: {timeit.timeit(naive_run, number=3)}") print(f"write_from: {timeit.timeit(lazy_run, number=3)}") if __name__ == "__main__": naive_vs_lazy() ``` **Results** ``` ## Run 1: without the concurrent and chunk options on writer write_chunk: 305.1042374999961 write_from: 257.5218124999956 ## Run 2: with the concurrent and chunk options on writer write_chunk: 45.874264957994455 write_from: 35.0909794170002 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
