zhreshold commented on issue #13209: Allow dataloader iterator to be reused URL: https://github.com/apache/incubator-mxnet/pull/13209#issuecomment-441484485 @leezu I currently do have an option to utilize process pools, and it should work elegantly without caching iterators explicitly. However, due to the backward compatibility issue of future module mentioned here https://pypi.org/project/futures/: *"The ProcessPoolExecutor class has known (unfixable) problems on Python 2 and should not be relied on for mission critical work."* Therefore I am not including the update in this PR. Besides, the logic to protect RecordIO handles is moved to `recordio.py`. This PR can improve initialization time significantly (more than 5x) on average according to local benchmark. Please have a second look @leezu @szha @hetong007
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
