Re: [cryptopp-users] Flipped bit in chacha encrypted file, avx vs sse
On Friday, September 17, 2021 at 9:19:33 PM UTC-4 austin@otoy.com wrote: > Great, thanks for the update. I'll follow the conversation over there. > ... > >> We're tracking this issue at >>> https://github.com/weidai11/cryptopp/issues/1069. >>> >> >> We disabled the AVX2 implementation at >> https://github.com/weidai11/cryptopp/commit/f0a32f0c6a51. >> > The issue was cleared at https://github.com/weidai11/cryptopp/commit/20962baf4440. Jeff -- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/53575c5c-d122-4dd8-b53d-8ff3777dc585n%40googlegroups.com.
Re: [cryptopp-users] Flipped bit in chacha encrypted file, avx vs sse
Great, thanks for the update. I'll follow the conversation over there. On Friday, September 17, 2021 at 5:42:17 PM UTC-7 Jeffrey Walton wrote: > On Friday, September 17, 2021 at 8:25:00 PM UTC-4 Jeffrey Walton wrote: > >> On Thu, Sep 16, 2021 at 8:42 PM austin clifton >> wrote: >> > >> > I have an issue which I believe may be a bug. I followed the >> instructions from the "Bug Report" page on the cryptopp wiki and all tests >> in cryptest.exe are passing, so figured I should post here first and make >> sure it isn't a build related issue. >> >> We're tracking this issue at >> https://github.com/weidai11/cryptopp/issues/1069. >> > > We disabled the AVX2 implementation at > https://github.com/weidai11/cryptopp/commit/f0a32f0c6a51. > > Jeff > -- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/fac31d30-5d7d-4576-94c2-9a45f6190ae9n%40googlegroups.com.
Re: [cryptopp-users] Flipped bit in chacha encrypted file, avx vs sse
On Friday, September 17, 2021 at 8:25:00 PM UTC-4 Jeffrey Walton wrote: > On Thu, Sep 16, 2021 at 8:42 PM austin clifton > wrote: > > > > I have an issue which I believe may be a bug. I followed the > instructions from the "Bug Report" page on the cryptopp wiki and all tests > in cryptest.exe are passing, so figured I should post here first and make > sure it isn't a build related issue. > > We're tracking this issue at > https://github.com/weidai11/cryptopp/issues/1069. > We disabled the AVX2 implementation at https://github.com/weidai11/cryptopp/commit/f0a32f0c6a51. Jeff -- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/163a502d-c641-49a8-8f5a-1441a891d74fn%40googlegroups.com.
Re: [cryptopp-users] Flipped bit in chacha encrypted file, avx vs sse
On Thu, Sep 16, 2021 at 8:42 PM austin clifton wrote: > > I have an issue which I believe may be a bug. I followed the instructions > from the "Bug Report" page on the cryptopp wiki and all tests in cryptest.exe > are passing, so figured I should post here first and make sure it isn't a > build related issue. We're tracking this issue at https://github.com/weidai11/cryptopp/issues/1069. Jeff -- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/CAH8yC8%3D_k4ZHt1E0Gj8d_D2-Kf3PU9AB5VXhUmRa1HJLXdsiHQ%40mail.gmail.com.
Re: [cryptopp-users] Flipped bit in chacha encrypted file, avx vs sse
On Fri, Sep 17, 2021 at 2:31 PM austin clifton wrote: > > Hey Jeff, thanks for the fast response! > > Yes, the encrypted sha from your AMD CPUs matches what I get on my Ryzen 7 > 3700X. > > The encrypted sha from the i7 is > 8F16077454F8477594CAD4304126B0A6F30C8C4D2536E2441FFFD320656E1DF1. That's also > the sha I get if I disable AVX on my Ryzen when compiling cryptopp. > > I'm not sure which sha is "correct" but we are seeing the same behavior > across the AMD CPUs. Would it make sense to disable assembly altogether to > get a reference encrypted sha256? Yes, that is what I usually do when I need a reference. I add -DCRYPTOPP_DISABLE_ASM to CXXFLAGS. That removes all inline assembly and intrinsics. In the case of ChaCha there is another option for a reference. You can use Bernstein's reference implementation. I use the author's implementation for test vectors, when available. I keep them stashed away at https://github.com/noloader/cryptopp-test. Note that Bernstein's ChaCah and the IETF's ChaCha are slightly different. Bernstein uses a 64-bit counter that starts at 0 and cannot wrap. IETF uses a 32-bit counter that can wrap. The IETF never specified what to do when wrap occurs. > We are seeing the same behavior from MSVC and GCC compilers. I can try master > with GCC here too but sounds like that fix is unrelated? Yeah, I think that is an issue with the CPU, not the compiler. > I'll try to glean more useful data from logs I have here. We distribute > encrypted assets to lots of machines (all running Windows) but I have to dig > around some. The guy to ask is Agner Fog, https://www.agner.org. I'm not sure whether he runs a mailing list. Jeff -- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/CAH8yC8%3DH%2BXwUP6XUyaVRGC%3DSC91PfBDwPncC33YvoySihUsXzQ%40mail.gmail.com.
Re: [cryptopp-users] Flipped bit in chacha encrypted file, avx vs sse
FWIW, I have two more test results from Windows machines with Intel CPUs: - i7 8750h - i9 10980KK Both are AVX-capable and produce the encrypted sha 6FBEE484EE64A2AB02235DDF29CA0B61EE3B811D227C2729836D3BD6161C9B18. On Friday, September 17, 2021 at 11:31:48 AM UTC-7 austin clifton wrote: > Hey Jeff, thanks for the fast response! > > Yes, the encrypted sha from your AMD CPUs matches what I get on my Ryzen 7 > 3700X. > > The encrypted sha from the i7 is > 8F16077454F8477594CAD4304126B0A6F30C8C4D2536E2441FFFD320656E1DF1. That's > also the sha I get if I disable AVX on my Ryzen when compiling cryptopp. > > I'm not sure which sha is "correct" but we are seeing the same behavior > across the AMD CPUs. Would it make sense to disable assembly altogether to > get a reference encrypted sha256? > > We are seeing the same behavior from MSVC and GCC compilers. I can try > master with GCC here too but sounds like that fix is unrelated? > > I'll try to glean more useful data from logs I have here. We distribute > encrypted assets to lots of machines (all running Windows) but I have to > dig around some. > > On Friday, September 17, 2021 at 12:23:55 AM UTC-7 Jeffrey Walton wrote: > >> On Thu, Sep 16, 2021 at 8:42 PM austin clifton >> wrote: >> > >> > I have an issue which I believe may be a bug. I followed the >> instructions from the "Bug Report" page on the cryptopp wiki and all tests >> in cryptest.exe are passing, so figured I should post here first and make >> sure it isn't a build related issue. >> > >> > I built cryptopp 8.5.0 as a static library, x64 multi-threaded debug >> (\MTd) using Visual Studio 2019 v16.10.0, on Windows 10 Pro v10.0.19043, >> using the .sln file provided with the cryptopp source code. >> > >> > I used two different machines for this test. One is the machine I built >> the cryptopp library with. Both are running the same version of Windows 10. >> One machine has a Ryzen 3700X, the other has an i7-990X. The Ryzen supports >> AVX, the i7 does not. >> > >> > I am VERY rarely finding that files encrypted with the chacha cipher by >> these two machines have differing sha256 hashes. If I do a hex diff against >> the files, there is a single flipped bit at 0x1EBB4 (0xD8 vs 0xD9). >> > >> > I am able to reproduce this consistently with a specific key, nonce, >> and input file, after test encrypting ~50GB of 1MB garbage files on each >> machine. >> > >> > If I disable AVX when building cryptlib by defining >> CRYPTOPP_DISABLE_AVX and CRYPTOPP_DISABLE_AVX2, the machine with the Ryzen >> will encrypt the file the same as the i7. >> > >> > Source code for a minimal reproducible example is here: >> > >> https://github.com/austin-clifton/cryptopp-chacha-asm-test/blob/main/src/main.cpp#L208 >> >> > >> > That repository includes a ready-to-build test case with Visual Studio, >> minus a built cryptlib.lib which should be added to libs/debug/ before >> building. >> >> Austin, >> >> I fetched main.cpp and run_459_file_76.bin from your GitHub. I >> compiled the library with: >> >> CXXFLAGS="-DNDEBUG -g2 -O3 -std=c++17" make >> >> I compiled/linked main.cpp with: >> >> g++ -DNDEBUG -g2 -O3 -fPIC -pthread -std=c++17 -I. main.cxx >> ./libcryptopp.a -o main.exe >> >> Running the program on a AMD Ryzen 3 3200U and AMD A6-9220e, this is the >> result. >> >> $ ./main.exe >> sha256: 0FC0FADCDF82770086C9DA8513A16FC785620D7B1C187CDD828E433EB0979847 >> encsha: 6FBEE484EE64A2AB02235DDF29CA0B61EE3B811D227C2729836D3BD6161C9B18 >> >> Is this correct? >> >> I have one Windows machine for testing. It is a Core i5. Sorry I don't >> have a good test environment. >> >> Jeff >> > -- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/e542e5a0-a842-4cad-b30b-699805cd8a81n%40googlegroups.com.
Re: [cryptopp-users] Flipped bit in chacha encrypted file, avx vs sse
Hey Jeff, thanks for the fast response! Yes, the encrypted sha from your AMD CPUs matches what I get on my Ryzen 7 3700X. The encrypted sha from the i7 is 8F16077454F8477594CAD4304126B0A6F30C8C4D2536E2441FFFD320656E1DF1. That's also the sha I get if I disable AVX on my Ryzen when compiling cryptopp. I'm not sure which sha is "correct" but we are seeing the same behavior across the AMD CPUs. Would it make sense to disable assembly altogether to get a reference encrypted sha256? We are seeing the same behavior from MSVC and GCC compilers. I can try master with GCC here too but sounds like that fix is unrelated? I'll try to glean more useful data from logs I have here. We distribute encrypted assets to lots of machines (all running Windows) but I have to dig around some. On Friday, September 17, 2021 at 12:23:55 AM UTC-7 Jeffrey Walton wrote: > On Thu, Sep 16, 2021 at 8:42 PM austin clifton > wrote: > > > > I have an issue which I believe may be a bug. I followed the > instructions from the "Bug Report" page on the cryptopp wiki and all tests > in cryptest.exe are passing, so figured I should post here first and make > sure it isn't a build related issue. > > > > I built cryptopp 8.5.0 as a static library, x64 multi-threaded debug > (\MTd) using Visual Studio 2019 v16.10.0, on Windows 10 Pro v10.0.19043, > using the .sln file provided with the cryptopp source code. > > > > I used two different machines for this test. One is the machine I built > the cryptopp library with. Both are running the same version of Windows 10. > One machine has a Ryzen 3700X, the other has an i7-990X. The Ryzen supports > AVX, the i7 does not. > > > > I am VERY rarely finding that files encrypted with the chacha cipher by > these two machines have differing sha256 hashes. If I do a hex diff against > the files, there is a single flipped bit at 0x1EBB4 (0xD8 vs 0xD9). > > > > I am able to reproduce this consistently with a specific key, nonce, and > input file, after test encrypting ~50GB of 1MB garbage files on each > machine. > > > > If I disable AVX when building cryptlib by defining CRYPTOPP_DISABLE_AVX > and CRYPTOPP_DISABLE_AVX2, the machine with the Ryzen will encrypt the file > the same as the i7. > > > > Source code for a minimal reproducible example is here: > > > https://github.com/austin-clifton/cryptopp-chacha-asm-test/blob/main/src/main.cpp#L208 > > > > That repository includes a ready-to-build test case with Visual Studio, > minus a built cryptlib.lib which should be added to libs/debug/ before > building. > > Austin, > > I fetched main.cpp and run_459_file_76.bin from your GitHub. I > compiled the library with: > > CXXFLAGS="-DNDEBUG -g2 -O3 -std=c++17" make > > I compiled/linked main.cpp with: > > g++ -DNDEBUG -g2 -O3 -fPIC -pthread -std=c++17 -I. main.cxx > ./libcryptopp.a -o main.exe > > Running the program on a AMD Ryzen 3 3200U and AMD A6-9220e, this is the > result. > > $ ./main.exe > sha256: 0FC0FADCDF82770086C9DA8513A16FC785620D7B1C187CDD828E433EB0979847 > encsha: 6FBEE484EE64A2AB02235DDF29CA0B61EE3B811D227C2729836D3BD6161C9B18 > > Is this correct? > > I have one Windows machine for testing. It is a Core i5. Sorry I don't > have a good test environment. > > Jeff > -- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/50ee9fd1-c7be-414d-a8b4-84b8fa7bac2an%40googlegroups.com.
Re: [cryptopp-users] Flipped bit in chacha encrypted file, avx vs sse
On Thu, Sep 16, 2021 at 8:42 PM austin clifton wrote: > > I have an issue which I believe may be a bug. I followed the instructions > from the "Bug Report" page on the cryptopp wiki and all tests in cryptest.exe > are passing, so figured I should post here first and make sure it isn't a > build related issue. > > I built cryptopp 8.5.0 as a static library, x64 multi-threaded debug (\MTd) > using Visual Studio 2019 v16.10.0, on Windows 10 Pro v10.0.19043, using the > .sln file provided with the cryptopp source code. > > I used two different machines for this test. One is the machine I built the > cryptopp library with. Both are running the same version of Windows 10. One > machine has a Ryzen 3700X, the other has an i7-990X. The Ryzen supports AVX, > the i7 does not. > > I am VERY rarely finding that files encrypted with the chacha cipher by these > two machines have differing sha256 hashes. If I do a hex diff against the > files, there is a single flipped bit at 0x1EBB4 (0xD8 vs 0xD9). > > I am able to reproduce this consistently with a specific key, nonce, and > input file, after test encrypting ~50GB of 1MB garbage files on each machine. > > If I disable AVX when building cryptlib by defining CRYPTOPP_DISABLE_AVX and > CRYPTOPP_DISABLE_AVX2, the machine with the Ryzen will encrypt the file the > same as the i7. > > Source code for a minimal reproducible example is here: > https://github.com/austin-clifton/cryptopp-chacha-asm-test/blob/main/src/main.cpp#L208 > > That repository includes a ready-to-build test case with Visual Studio, minus > a built cryptlib.lib which should be added to libs/debug/ before building. Austin, I fetched main.cpp and run_459_file_76.bin from your GitHub. I compiled the library with: CXXFLAGS="-DNDEBUG -g2 -O3 -std=c++17" make I compiled/linked main.cpp with: g++ -DNDEBUG -g2 -O3 -fPIC -pthread -std=c++17 -I. main.cxx ./libcryptopp.a -o main.exe Running the program on a AMD Ryzen 3 3200U and AMD A6-9220e, this is the result. $ ./main.exe sha256: 0FC0FADCDF82770086C9DA8513A16FC785620D7B1C187CDD828E433EB0979847 encsha: 6FBEE484EE64A2AB02235DDF29CA0B61EE3B811D227C2729836D3BD6161C9B18 Is this correct? I have one Windows machine for testing. It is a Core i5. Sorry I don't have a good test environment. Jeff -- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/CAH8yC8k5HNWTq7%2BOx6d27gNh5YwKyi0kxxwDAvMf6QczPcsqhQ%40mail.gmail.com.
Re: [cryptopp-users] Flipped bit in chacha encrypted file, avx vs sse
On Thu, Sep 16, 2021 at 8:42 PM austin clifton wrote: > > Hi all, > > I have an issue which I believe may be a bug. I followed the instructions > from the "Bug Report" page on the cryptopp wiki and all tests in cryptest.exe > are passing, so figured I should post here first and make sure it isn't a > build related issue. > > I built cryptopp 8.5.0 as a static library, x64 multi-threaded debug (\MTd) > using Visual Studio 2019 v16.10.0, on Windows 10 Pro v10.0.19043, using the > .sln file provided with the cryptopp source code. > > I used two different machines for this test. One is the machine I built the > cryptopp library with. Both are running the same version of Windows 10. One > machine has a Ryzen 3700X, the other has an i7-990X. The Ryzen supports AVX, > the i7 does not. > > I am VERY rarely finding that files encrypted with the chacha cipher by these > two machines have differing sha256 hashes. If I do a hex diff against the > files, there is a single flipped bit at 0x1EBB4 (0xD8 vs 0xD9). > > I am able to reproduce this consistently with a specific key, nonce, and > input file, after test encrypting ~50GB of 1MB garbage files on each machine. > > If I disable AVX when building cryptlib by defining CRYPTOPP_DISABLE_AVX and > CRYPTOPP_DISABLE_AVX2, the machine with the Ryzen will encrypt the file the > same as the i7. > > Source code for a minimal reproducible example is here: > https://github.com/austin-clifton/cryptopp-chacha-asm-test/blob/main/src/main.cpp#L208 > > That repository includes a ready-to-build test case with Visual Studio, minus > a built cryptlib.lib which should be added to libs/debug/ before building. Ugh, I dislike these reports. I've seen a similar problem on a Ryzen with SHA. It is GCC67, which is Ryzen 5 2400G, at https://cfarm.tetaneutral.net/machines/list/. One self test failed but no others. The one test that failed is one with a very long input string. I believe it is/was the last test at https://github.com/weidai11/cryptopp/blob/master/TestVectors/sha2_256_fips_180.txt. I am thinking the problem is with either GCC's AMD target code or the Ryzen CPU itself. The problem eventually cleared itself. With that said, can you give Master a try? We cleared a bug at https://github.com/weidai11/cryptopp/issues/1010 that may affect GCC's code generation. It was a bug related to GCC alias violations, and GCC would remove code on occasion. Something else to try... Update the firmware on the machine in case a firmware or microcode update addressed the issue. Jeff -- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/CAH8yC8nz5iMi5JbVVDBZQM%3D-uYZe_C8EEq7S%2B62V2bSqQg3dwQ%40mail.gmail.com.
[cryptopp-users] Flipped bit in chacha encrypted file, avx vs sse
Hi all, I have an issue which I believe may be a bug. I followed the instructions from the "Bug Report" page on the cryptopp wiki and all tests in cryptest.exe are passing, so figured I should post here first and make sure it isn't a build related issue. I built cryptopp 8.5.0 as a static library, x64 multi-threaded debug (\MTd) using Visual Studio 2019 v16.10.0, on Windows 10 Pro v10.0.19043, using the .sln file provided with the cryptopp source code. I used two different machines for this test. One is the machine I built the cryptopp library with. Both are running the same version of Windows 10. One machine has a Ryzen 3700X, the other has an i7-990X. The Ryzen supports AVX, the i7 does not. I am VERY rarely finding that files encrypted with the chacha cipher by these two machines have differing sha256 hashes. If I do a hex diff against the files, there is a single flipped bit at 0x1EBB4 (0xD8 vs 0xD9). I am able to reproduce this consistently with a specific key, nonce, and input file, after test encrypting ~50GB of 1MB garbage files on each machine. If I disable AVX when building cryptlib by defining CRYPTOPP_DISABLE_AVX and CRYPTOPP_DISABLE_AVX2, the machine with the Ryzen will encrypt the file the same as the i7. Source code for a minimal reproducible example is here: https://github.com/austin-clifton/cryptopp-chacha-asm-test/blob/main/src/main.cpp#L208 That repository includes a ready-to-build test case with Visual Studio, minus a built cryptlib.lib which should be added to libs/debug/ before building. Thanks, Austin -- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/bf3e9626-ac2b-4821-b158-7c9786ffb17dn%40googlegroups.com.
