ptrendx commented on issue #9774: mx.io.ImageRecordIter does not respect dtype argument / FP16 performance on Volta URL: https://github.com/apache/incubator-mxnet/issues/9774#issuecomment-367138608 I was asking about the imagenet script. If you use smaller batch size like 256 for 8 GPUs (the best results you will see with larger batch size like 1024 for 8 GPUs), you may consider turning on the NCCL kvstore (`--kv-store nccl` in the `train_imagenet.py` script). Yes, 2 is a valid (albeit undocumented :-() value - for both 1 and 2 it performs cudnnFind call, but the default value of 1 makes it reject the algos that require more workspace than a threshold (for convolutions default threshold is 1 GB). Setting it to 2 makes it always choose the fastest algo: ``` DMLC_DECLARE_FIELD(cudnn_tune) .add_enum("off", conv::kOff) .add_enum("limited_workspace", conv::kLimited) .add_enum("fastest", conv::kFastest) .set_default(dmlc::optional<int>()) .describe("Whether to pick convolution algo by running performance test."); ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
