dosubot[bot] commented on issue #6855: URL: https://github.com/apache/opendal/issues/6855#issuecomment-3605008708
<!-- Greeting --> Hi @seewindcn! I'm [Dosu](https://go.dosu.dev/dosubot) and Iām helping the opendal team. <!-- Answer --> There are a few known issues in opendal Python bindings that could cause hangs or blocking during file.write, especially when using S3-compatible backends like Tencent COS. One common cause is interaction with the underlying Tokio async runtime and Python's multiprocessing or fork behavior, which can lead to resource and lock contention and hang all threads if the GIL is not properly released or if resources are not reinitialized after fork. This is a broader problem affecting all Python crates that depend on Tokio, not just OpenDAL [[source]](https://github.com/apache/opendal/issues/5330). Another area to check is whether the file is being properly closed after writing. In some cases, failing to call close or shutdown on the writer can leave resources hanging or uploads incomplete, especially for multipart uploads [[source]](https://github.com/apache/opendal/issues/4120) [[source]](https://github.com/apache/opendal/issues/4926). To help narrow this down, could you share: - The opendal and Python versions you're using - Whether you're using multiprocessing or forking in your application - If the hang happens with both sync and async APIs - A Python stack trace (using e.g. `faulthandler.dump_traceback_later`) during the hang In the meantime, make sure to always call close/shutdown on the file after writing, and avoid forking after initializing opendal. If you are using multiprocessing, try to initialize opendal objects after the fork, not before. If you can reproduce the issue with a minimal script, that would also help pinpoint the cause. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/39ec1569-5ab2-4353-b47a-f842103eb6e9?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/39ec1569-5ab2-4353-b47a-f842103eb6e9?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/39ec1569-5ab2-4353-b47a-f842103eb6e9?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/39ec1569-5ab2-4353-b47a-f842103eb6e9?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/39ec1569-5ab2-4353-b47a-f842103eb6e9?feedback_type=hallucination) | [Report š](https://app.dosu.dev/response-feedback/39ec1569-5ab2-4353-b47a-f842103eb6e9?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/39ec1569-5ab2-4353-b47a-f842103eb6e9?feedback_type=other)</sup> [](https://app.dosu.dev/997a2c5a-0c54-497b-989d-44651635dd36/ask?utm_source=github) [](https://cursor.com/link/prompt?text=There%20are%20a%20few%20known%20issues%20in%20opendal%20Python%20bindings%20that%20could%20cause%20hangs%20or%20blocking%20during%20file.write%2C%20especially%20when%20using%20S3-compatible%20backends%20like%20Tencent%20COS.%20One%20common%20cause%20is%20interaction%20with%20the%20underlying%20Tokio%20async%20runtime%20and%20Python%27s%20multiprocessing%20or%20fork%20behavior%2C%20which%20can%20lead%20to%20resource%20and%20lock%20contention%20and%20hang%20all%20threads%20if%20the%20GIL%20is%20not%20properly%20released%20or%20if%20resources%20are%20not%20reinitialized%20after%20fork.%20This%20is%20a%20broader%20problem%20affecting%20all%20Python%20crates%20that%20depend%20on%20Tokio%2C%20not%20just%20OpenDAL%20%5B%5Bsource%5D%5D%28https%3A//github.c om/apache/opendal/issues/5330%29.%0A%0AAnother%20area%20to%20check%20is%20whether%20the%20file%20is%20being%20properly%20closed%20after%20writing.%20In%20some%20cases%2C%20failing%20to%20call%20close%20or%20shutdown%20on%20the%20writer%20can%20leave%20resources%20hanging%20or%20uploads%20incomplete%2C%20especially%20for%20multipart%20uploads%20%5B%5Bsource%5D%5D%28https%3A//github.com/apache/opendal/issues/4120%29%20%5B%5Bsource%5D%5D%28https%3A//github.com/apache/opendal/issues/4926%29.%0A%0ATo%20help%20narrow%20this%20down%2C%20could%20you%20share%3A%0A-%20The%20opendal%20and%20Python%20versions%20you%27re%20using%0A-%20Whether%20you%27re%20using%20multiprocessing%20or%20forking%20in%20your%20application%0A-%20If%20the%20hang%20happens%20with%20both%20sync%20and%20async%20APIs%0A-%20A%20Python%20stack%20trace%20%28using%20e.g.%20%60faulthandler.dump_traceback_later%60%29%20during%20the%20hang%0A%0AIn%20the%20meantime%2C%20make%20sure%20to%20always%20call%20close/shutdown%20on%20th e%20file%20after%20writing%2C%20and%20avoid%20forking%20after%20initializing%20opendal.%20If%20you%20are%20using%20multiprocessing%2C%20try%20to%20initialize%20opendal%20objects%20after%20the%20fork%2C%20not%20before.%20If%20you%20can%20reproduce%20the%20issue%20with%20a%20minimal%20script%2C%20that%20would%20also%20help%20pinpoint%20the%20cause.) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/opendal/issues/6855) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
