ThomasDelteil commented on issue #10042: [MXNET-86] Gluon dataloader crash on speech recognition training URL: https://github.com/apache/incubator-mxnet/issues/10042#issuecomment-374359896 The segfault does not seem to happen with latest master, however latest master does seem to be MUCH slower than 1.1.0. by a factor of 3-4. `make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1` **optimized** refers to the version where `.asscalar()` is only called every 100 batches. | mxnet | 0 worker | 8 workers | 0 worker, optimized | 8 workers, optimized | | ------------------ |:-------------:|:----------|----------------------|-----------------------| | mxnet-cu90: 1.1.0 | 11s | 4s | 8s | seg fault | | master: 1.2.0 | 13.5s | 14s | 13.5 | 14.5 | @Jerryzcn Any idea why that might be? Looks like on current master, the data loading is not the limiting factor in the performance. whilst when using 1.1.0 it was. If I didn't botch anything during my build, that looks like a pretty bad regression for 1.2.0
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
