[GitHub] DickJC123 commented on issue #13142: [WIP] cudnnFind** API does not return deterministic algorithms
DickJC123 commented on issue #13142: [WIP] cudnnFind** API does not return deterministic algorithms URL: https://github.com/apache/incubator-mxnet/pull/13142#issuecomment-458775835 I have a possible explanation. When cudnnFind measures the runtime of algos that have similar runtimes, the selection of which is fastest may differ from run to run. Filtering the algorithms based on MXNET_ENFORCE_DETERMINISM=1 just throws out the non-deterministic algos. However, the results from run to run may differ based on the non-deterministic choice of which deterministic algo to run. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] DickJC123 commented on issue #13142: [WIP] cudnnFind** API does not return deterministic algorithms
DickJC123 commented on issue #13142: [WIP] cudnnFind** API does not return deterministic algorithms URL: https://github.com/apache/incubator-mxnet/pull/13142#issuecomment-450596124 This PR addresses a different type of determinism. My PR improved the framework's ability to select the same conv algorithm within a model run and from one run to the next based on improvements in the use of cudnnFind(). This PR deals with making that selection only within the domain of 'deterministic algorithms.' Some algos have unordered threads summing into a result value using atomic adds. These algos do not always produce the same result from the same input data (i.e. they are non-deterministic). This PR deals with limiting the cudnnFind() selection to only deterministic algos. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
