Zha0q1 edited a comment on issue #18926: URL: https://github.com/apache/incubator-mxnet/issues/18926#issuecomment-674333272
> Thanks Zhaoqi. This is applicable for tensors with greater than 2pow31 right? It doesn't have any implications for regular tensors? After more discussion with openblas people my understanding is that loss of precision will happen whenever two numbers to be added are certain magnitude apart from each other. (in fact loss of precision always happen but is especially bad under such condition) Direct quotes from them: > Expected precision is as of standard netlib blas with single addition per loop. If you need to sum vaues more than significand+1 apart you need bigger floating point variable with obviously bigger significand range. > What the naive loop reaches is limit of "significand" > https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754:_floating_point_in_modern_computers > i.e 2^24 (it rounds properly even near the end) i.e close to 1,6777216 ^ 10^7 **With that said much less error will be accumulated with regularly sized tensors.** Regular openblas is able to avoid this error accumulation by FMA/blocking so hence the discrepancy we saw between ilp64 openblas and vanilla openblas. My demo program calculates (1, 1, 1, ....) dot (1, 1, 1, ...).T. The error will start to accumulate when the sum reaches significand. Here are some result recaps **ILP64 openblas** With shape = (2^31, 1) ``` 2147483647 // 2^31 1.93274e+09 // ilp64 gemm answer 2.14748e+09 // accurate value ``` With shape = (2^29) ``` 536870911 // 2^29 5.36871e+08 // ilp64 openblas gemm answer 5.36871e+08 // accurate value ``` compared with **vanilla** openblas With shape = (2^31, 1) ``` 2147483647 // 2^31 2.14748e+09 // vanilla gemm answer 2.14748e+09 // accurate value ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
