gaogaotiantian commented on PR #58032:
URL: https://github.com/apache/spark/pull/58032#issuecomment-5311243834
I believe `os.makedirs` is about 2x slower than `os.path.exists("test_dir")`
when the directory actually exists. It's definitely fine for `setup.py` which
is super insensitive to such perf diff. Could you elaborate a bit on other
usages about why they don't need this perf improvement? Profiler dump should be
fine because that's normally a user manual operation and the disk IO already
takes plenty of time. Not sure about shuffle. Basically I want to understand if
this is on some hot paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
