[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
This bug was fixed in the package golang-google-grpc - 1.6.0-3ubuntu0.19.04.2 --- golang-google-grpc (1.6.0-3ubuntu0.19.04.2) disco; urgency=medium * Fix FTBFS caused by golang-goprotobuf-dev >= 1.2.0 (LP: #1828230) - Add Invoke and NewStream wrappers to ClientConn. - Use keyed fields for struct initializers (thanks to Doug Fawley). golang-google-grpc (1.6.0-3ubuntu0.19.04.1) disco; urgency=medium * Backport upstream fixes for intermittent panic due to a race condition when sending RPC status (LP: #1819936) - transport/handler_server.go: wrap status sending function in a mutex - transport/handler_server_test.go: add test cases for the races - Thanks to Gyuho Lee for the patches. -- Tiit Pikma Thu, 16 May 2019 10:54:50 + ** Changed in: golang-google-grpc (Ubuntu Disco) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
Hello again, all, I performed the same SRU verification steps as I described in comment #7 on disco using 1.6.0-3ubuntu0.19.04.2. The tests were successful and I changed the verification-needed tags to verification-done. The autopkgtest failure was caused by a temporary network issue and has been resolved since. ** Tags removed: verification-needed verification-needed-disco ** Tags added: verification-done verification-done-disco -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
This bug was fixed in the package golang-google-grpc - 1.6.0-3ubuntu0.18.04.1 --- golang-google-grpc (1.6.0-3ubuntu0.18.04.1) bionic; urgency=medium * Backport upstream fixes for intermittent panic due to a race condition when sending RPC status (LP: #1819936) - transport/handler_server.go: wrap status sending function in a mutex - transport/handler_server_test.go: add test cases for the races - Thanks to Gyuho Lee for the patches. -- Tiit Pikma Wed, 13 Mar 2019 15:32:22 + ** Changed in: golang-google-grpc (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
Thank you for your verification. The disco failure is indeed quite a shame, but since you are actively working on all the issues, I'll release the bionic and cosmic parts regardless. Let's just make sure that the FTBFS is resolved with some follow-up upload for disco as soon as possible. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
This bug was fixed in the package golang-google-grpc - 1.6.0-3ubuntu0.18.10.1 --- golang-google-grpc (1.6.0-3ubuntu0.18.10.1) cosmic; urgency=medium * Backport upstream fixes for intermittent panic due to a race condition when sending RPC status (LP: #1819936) - transport/handler_server.go: wrap status sending function in a mutex - transport/handler_server_test.go: add test cases for the races - Thanks to Gyuho Lee for the patches. -- Tiit Pikma Wed, 13 Mar 2019 15:32:22 + ** Changed in: golang-google-grpc (Ubuntu Cosmic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
I reported the disco FTBFS at https://bugs.launchpad.net/ubuntu/+source /golang-google-grpc/+bug/1828230. Hopefully this unrelated disco bug will not block the fixed bionic and cosmic packages from moving into -updates. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
First up, thank you Simon for sponsoring this patch and Steve for accepting it into proposed. The test case described in the bug report passed for bionic and cosmic when the packages were built successfully. I performed additional testing by building etcd 3.2.17+dfsg-1 using golang-google-grpc 1.6.0-3ubuntu0.18.04.1 on bionic and etcd 3.2.18+dfsg-1 using golang- google-grpc 1.6.0-3ubuntu0.18.10.1 on cosmic. I ran overnight stress tests on both and neither crashed, indicating that the issue seems to be resolved. As for disco, the package failed to build from source (https://launchpad.net/ubuntu/+source/golang-google- grpc/1.6.0-3ubuntu0.19.04.1/+build/16729837). However when I tried building golang-google-grpc 1.6.0-3 locally on a clean install of disco, then it failed with the same error WITHOUT the patch. So golang-google- grpc seems to FTBFS on disco at the moment. I will investigate a bit to see if I can find the cause for this. I am hesitant to mark this as verification-failed-disco, since the failure was not the patch's fault. ** Tags removed: verification-needed-bionic verification-needed-cosmic ** Tags added: verification-done-bionic verification-done-cosmic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
Hello Tiit, or anyone else affected, Accepted golang-google-grpc into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/golang- google-grpc/1.6.0-3ubuntu0.19.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Description changed: [Impact] The version of golang-google-grpc packaged in bionic and cosmic (1.6.0-3) crashes intermittently due to a race condition. Upstream bug report and pull requests: https://github.com/grpc/grpc-go/issues/ https://github.com/grpc/grpc-go/pull/1546 https://github.com/grpc/grpc-go/pull/1687 This can cause all packages Built-Using: golang-google-grpc=1.6.0-3 to crash as well (see https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1800973). The attached debdiff contains two upstream fixes squashed together: https://github.com/grpc/grpc-go/commit/22c3f92f5faea8db492fb0f5ae4daf0d2752b19e https://github.com/grpc/grpc-go/commit/c6b46087ab923e9f453ec433f99174cdd45b9b89 These fix the race by synchronizing the entire function. [Test Case] + The patch includes patches to test cases to exercise the race condition. One test case would fail sometimes without the fix; the second test case would always fail without the fix. - 1. Download the source: apt source golang-google-grpc && cd golang-google-grpc-1.6.0 - 2. Patch in the attached test cases: patch -p1 < test-cases-only.patch - - Theses are part of the upstream patch. Only apply manually to verify problem pre-patch. - 3. Build the package, running the test cases: dpkg-buildpackage --target clean,build - - Since this is a race condition, the first test case is not guaranteed to fail. - - The second test case should always fail. - - With the fixes applied there is no need for step 2 (the test cases are - part of the patch) and step 3 succeeds every time without failure. + If the package builds successfully with the patch, then the test cases + have passed. [Regression Potential] Regressions are unlikely due to the size and scope of the patch. The fix has been applied upstream since 30 Nov 2017 and is still in use in the latest release: https://github.com/grpc/grpc- go/blob/v1.19.0/internal/transport/handler_server.go#L193 A possible source of regressions would be the squashing of the two commits into one. Luckily they were also applied upstream immediately one after the other so it should work without problems. Another source would be if these patches depended on some other changes made since 1.6.0-3. However testing has not revealed this to be the case. Issues might also arise from my inexperience in updating Ubuntu packages and submitting patches for Stable Release Updates. [Other Info] As golang-google-grpc is a Go library it is linked into other packages statically. This means that simply updating this package is not enough and dependents need to be rebuilt to benefit from these fixes. If this patch is accepted then I plan to open another SRU request to set the fixed version as a minimal build dependency for etcd to have it also rebuilt and resolve https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1800973. Release: Ubuntu 18.04.2 LTS Package: golang-google-grpc 1.6.0-3 ** Changed in: golang-google-grpc (Ubuntu Disco) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-disco ** Changed in: golang-google-grpc (Ubuntu Cosmic) Status: In Progress => Fix Committed ** Tags added: verification-needed-cosmic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
I have uploaded the debdiff to Bionic, Cosmic, and Disco. Thank you for your contribution to Ubuntu, Tiit! ** Also affects: golang-google-grpc (Ubuntu Cosmic) Importance: Undecided Status: New ** Also affects: golang-google-grpc (Ubuntu Disco) Importance: Undecided Status: New ** Also affects: golang-google-grpc (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: golang-google-grpc (Ubuntu Bionic) Importance: Undecided => Medium ** Changed in: golang-google-grpc (Ubuntu Cosmic) Importance: Undecided => Medium ** Changed in: golang-google-grpc (Ubuntu Disco) Importance: Undecided => Medium ** Changed in: golang-google-grpc (Ubuntu) Status: New => Fix Committed ** Changed in: golang-google-grpc (Ubuntu Bionic) Status: New => In Progress ** Changed in: golang-google-grpc (Ubuntu Cosmic) Status: New => In Progress ** Changed in: golang-google-grpc (Ubuntu Disco) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
** No longer affects: etcd (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
** Tags added: patch patch-accepted-upstream ** Changed in: etcd (Ubuntu) Importance: Undecided => Medium ** Changed in: golang-google-grpc (Ubuntu) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
** Description changed: [Impact] The version of golang-google-grpc packaged in bionic and cosmic (1.6.0-3) crashes intermittently due to a race condition. Upstream bug report and pull requests: https://github.com/grpc/grpc-go/issues/ https://github.com/grpc/grpc-go/pull/1546 https://github.com/grpc/grpc-go/pull/1687 This can cause all packages Built-Using: golang-google-grpc=1.6.0-3 to crash as well (see https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1800973). The attached debdiff contains two upstream fixes squashed together: https://github.com/grpc/grpc-go/commit/22c3f92f5faea8db492fb0f5ae4daf0d2752b19e https://github.com/grpc/grpc-go/commit/c6b46087ab923e9f453ec433f99174cdd45b9b89 These fix the race by synchronizing the entire function. [Test Case] 1. Download the source: apt source golang-google-grpc && cd golang-google-grpc-1.6.0 2. Patch in the attached test cases: patch -p1 < test-cases-only.patch - Theses are part of the upstream patch. Only apply manually to verify problem pre-patch. 3. Build the package, running the test cases: dpkg-buildpackage --target clean,build - Since this is a race condition, the first test case is not guaranteed to fail. - The second test case should always fail. With the fixes applied there is no need for step 2 (the test cases are part of the patch) and step 3 succeeds every time without failure. [Regression Potential] - Any regressions are extremely unlikely due to the size and scope of the - patch. It removes a guard flag and instead synchronizes access to the - entire method with a mutex. The fix has been applied upstream since 30 - Nov 2017 and is still in use in the latest release: + Regressions are unlikely due to the size and scope of the patch. The fix + has been applied upstream since 30 Nov 2017 and is still in use in the + latest release: https://github.com/grpc/grpc- go/blob/v1.19.0/internal/transport/handler_server.go#L193 A possible source of regressions would be the squashing of the two commits into one. Luckily they were also applied upstream immediately one after the other so it should work without problems. + + Another source would be if these patches depended on some other changes + made since 1.6.0-3. However testing has not revealed this to be the + case. Issues might also arise from my inexperience in updating Ubuntu packages and submitting patches for Stable Release Updates. [Other Info] As golang-google-grpc is a Go library it is linked into other packages statically. This means that simply updating this package is not enough and dependents need to be rebuilt to benefit from these fixes. If this patch is accepted then I plan to open another SRU request to set the fixed version as a minimal build dependency for etcd to have it also rebuilt and resolve https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1800973. Release: Ubuntu 18.04.2 LTS Package: golang-google-grpc 1.6.0-3 ** Patch added: "fix-race-sending-rpc-status.patch" https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+attachment/5245886/+files/fix-race-sending-rpc-status.patch ** Also affects: etcd (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819936] Re: intermittent panic in 1.6.0-3 due to race in WriteStatus
** Description changed: [Impact] The version of golang-google-grpc packaged in bionic and cosmic (1.6.0-3) crashes intermittently due to a race condition. Upstream bug report and pull requests: https://github.com/grpc/grpc-go/issues/ https://github.com/grpc/grpc-go/pull/1546 https://github.com/grpc/grpc-go/pull/1687 This can cause all packages Built-Using: golang-google-grpc=1.6.0-3 to crash as well (see https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1800973). The attached debdiff contains two upstream fixes squashed together: https://github.com/grpc/grpc-go/commit/22c3f92f5faea8db492fb0f5ae4daf0d2752b19e https://github.com/grpc/grpc-go/commit/c6b46087ab923e9f453ec433f99174cdd45b9b89 These fix the race by synchronizing the entire function. [Test Case] 1. Download the source: apt source golang-google-grpc && cd golang-google-grpc-1.6.0 2. Patch in the attached test cases: patch -p1 < test-cases-only.patch -- Theses are part of the upstream patch. Only apply manually to verify problem pre-patch. + - Theses are part of the upstream patch. Only apply manually to verify problem pre-patch. 3. Build the package, running the test cases: dpkg-buildpackage --target clean,build -- Since this is a race condition, the first test case is not guaranteed to fail. -- The second test case should always fail. + - Since this is a race condition, the first test case is not guaranteed to fail. + - The second test case should always fail. With the fixes applied there is no need for step 2 (the test cases are part of the patch) and step 3 succeeds every time without failure. - [Regression potential] + [Regression Potential] Any regressions are extremely unlikely due to the size and scope of the patch. It removes a guard flag and instead synchronizes access to the entire method with a mutex. The fix has been applied upstream since 30 Nov 2017 and is still in use in the latest release: https://github.com/grpc/grpc- go/blob/v1.19.0/internal/transport/handler_server.go#L193 A possible source of regressions would be the squashing of the two commits into one. Luckily they were also applied upstream immediately one after the other so it should work without problems. Issues might also arise from my inexperience in updating Ubuntu packages and submitting patches for Stable Release Updates. [Other Info] As golang-google-grpc is a Go library it is linked into other packages statically. This means that simply updating this package is not enough and dependents need to be rebuilt to benefit from these fixes. If this patch is accepted then I plan to open another SRU request to set the fixed version as a minimal build dependency for etcd to have it also rebuilt and resolve https://bugs.launchpad.net/ubuntu/+source/etcd/+bug/1800973. Release: Ubuntu 18.04.2 LTS Package: golang-google-grpc 1.6.0-3 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819936 Title: intermittent panic in 1.6.0-3 due to race in WriteStatus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/golang-google-grpc/+bug/1819936/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
