I am working on cpu with 80 virtual cores(what cpu_count() tells me) and 6 nvidia1080 gpus. I used mx.io.ImageRecordIter to load .rec and .idx, which contains about 3 million images. During training process, the cpu usage goes up to 3000%. I figured out if I use smaller batches and less preprocessing_threads, the usage could be back to like 600%, however that also reduces the speed of training. Is there a way to load my training data in .rec and .idx files with multiprocessing so that I could separate the workload?
--- [Visit Topic](https://discuss.mxnet.apache.org/t/data-loading-from-rec-with-multiprocessing/6826/1) or reply to this email to respond. You are receiving this because you enabled mailing list mode. To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/47be13a60e0dfdf94beccb468e96a8873fcc703d4f99c0dbfb881a71fb043907).
