[issue23649] tarfile not re-entrant for multi-threading

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue23649] tarfile not re-entrant for multi-threading

Change by Jeffrey Kintscher : -- nosy: +Jeffrey.Kintscher ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue23649] tarfile not re-entrant for multi-threading

Xavier de Gaye added the comment: extract_from_pkgs() in the attached extract_from_packages.py script extracts /etc files from the tar files in PKG_DIR into WORK_DIR using a ThreadPoolExecutor (a ThreadPoolExecutor, when used to extract all the /etc files from the packages

[issue23649] tarfile not re-entrant for multi-threading

Lars Gustäbel added the comment: I agree with David that there is no need for tarfile to be thread-safe. There is nothing to be gained from distributing one TarFile object among multiple threads because it operates on a single resource which has to be accessed sequentially anyway. So, it

[issue23649] tarfile not re-entrant for multi-threading

STINNER Victor added the comment: Lars Gustäbel added the comment: I agree with David that there is no need for tarfile to be thread-safe. There is nothing to be gained from distributing one TarFile object among multiple threads because it operates on a single resource which has to be

[issue23649] tarfile not re-entrant for multi-threading

New submission from Srdjan Grubor: When running tarfile.extract through multiple threads, the archive reading pointer is not protected from simultaneous seeks and causes various convoluted bugs: some code self.archive_object.extract(member, extraction_path) File

[issue23649] tarfile not re-entrant for multi-threading

Changes by Srdjan Grubor sg...@sgnn7.org: -- type: behavior - enhancement ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23649 ___ ___

[issue23649] tarfile not re-entrant for multi-threading

Srdjan Grubor added the comment: Also, extract_member in tarfile.py is not thread-safe since the check for folder existence might occur during another thread's creation of that same dir causing the code to error out. File /usr/lib/python3.4/concurrent/futures/thread.py, line 54, in run

[issue23649] tarfile not re-entrant for multi-threading

Changes by Srdjan Grubor sg...@sgnn7.org: -- type: enhancement - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23649 ___ ___

[issue23649] tarfile not re-entrant for multi-threading

Srdjan Grubor added the comment: The code around tarfile multi-threading was fixed for me on the user-side with threading.Lock() usage so it might work to use this within the library and the directory creation could be improved by probably doing a try/except around the makedirs() call with

[issue23649] tarfile not re-entrant for multi-threading

R. David Murray added the comment: If you want to use an object that has state in more than one thread you generally have to put some locking around it. Unless I'm missing something (which I might be) I don't think it is tarfile's responsibility to do this. -- nosy: +r.david.murray

[issue23649] tarfile not re-entrant for multi-threading

Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo, lars.gustaebel versions: +Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23649 ___

[issue23649] tarfile not re-entrant for multi-threading

Srdjan Grubor added the comment: I don't know if that's true of core libraries. Why complicate things for end users when those issues could be done in the library itself and be completely transparent to the devs? A simple RLock latch wouldn't pose almost any speed degradation but would work

[issue23649] tarfile not re-entrant for multi-threading

Srdjan Grubor added the comment: After some thinking, for the makedirs it should only need makedirs(exist_ok=True) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23649 ___

[issue23649] tarfile not re-entrant for multi-threading

Srdjan Grubor added the comment: The whole lib still needs the threading locks added but the patch submitted should fix things for people that do the locking from their code. -- ___ Python tracker rep...@bugs.python.org

[issue23649] tarfile not re-entrant for multi-threading

Srdjan Grubor added the comment: Patch for the multithreaded expansion of files and use of makedirs. -- keywords: +patch Added file: http://bugs.python.org/file38462/mutithreading_tarfile.patch ___ Python tracker rep...@bugs.python.org