thomasfrederikhoeck commented on issue #5882: URL: https://github.com/apache/arrow-rs/issues/5882#issuecomment-2288624368
> @Xuanwo I might be network related but I have some feeling that is related to how `object_store` or `delta-rs` handles if there is a lower throughput than within a Azure data center (some connections going stale while waiting for somthing else). > > * Running from local laptop in Europe > * Azure region is Westeurope > * I just benchmarked with `azcopy bench "https://ACCOUNT.blob.core.windows.net/CONTAINER?SAS"; --file-count 20 --size-per-file 10000M`. So 20 files of 10 Gb and here I get a throughput of 145 Mb/s. It runs through with no failures. > > The benchmark took 1+ hours with no failure while the delta-rs call fails within a few minutes. @tustvold The weird thing is that I can run some rather large data opeartions (taking an 1+ hour) with `azcopy bench` without seeing any dropped connection or something like that. I can maybe add: Before [this PR](https://github.com/pola-rs/polars/pull/15083/files ) in polars we sometimes saw similar issues but I'm very far from knowledge-able on networking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
