[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

"AMD64 FreeBSD CURRENT Shared 2.7 #159 is complete: Success"

https://buildbot.python.org/all/#/builders/169/builds/159

The FreeBSD 2.7 buildbot is back to green, I close the issue.

Thanks everyone was helped to get these issues to be fixed!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:


New changeset c3642219090f2564c1790330cbf0ba31f19dcaf4 by Victor Stinner in 
branch '2.7':
bpo-34521: Fix FD transfer in multiprocessing on FreeBSD (GH-15422)
https://github.com/python/cpython/commit/c3642219090f2564c1790330cbf0ba31f19dcaf4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

Note: I also wrote PR 15421 to remove temporary files created by these 2 tests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

I am able to reproduce test_multiprocessing.test_fd_transfer() failure on the 
FreeBSD CURRENT buildbot:

CURRENT-amd64% ./python -m test -m test_fd_transfer test_multiprocessing 
(...)
Tests result: FAILURE

I converted my msg330654 patch into PR 15422: I confirm that this PR fix 
test_fd_transfer() and test_large_fd_transfer() of test_multiprocessing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +15121
pull_request: https://github.com/python/cpython/pull/15422

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[Kubilay Kocak]

Kubilay Kocak  added the comment:

Access to the buildbot has been restored (a while ago), and this issue being 
resolved (merge to 2.7) is the only test failing to get 2.7 back to green, 
which has been failing for months.

Can we test with Victors proposed patch in msg330654 ? I don't believe the 
change/fix/merge will be difficult, given the very small change scope of the 
master commits.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

I did not backport this still because the code is very different and it 
required more time to understand how it works on 2.7.

I do not have access currently to koobs buildbot (the IP has changed) so I 
cannot check if that works :(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

> Is test_socket of Python 2.7 affected by the issue? If not, I suggest to 
> remove 2.7 from this issue and close it.

Some "FD transfer" tests of test_multiprocessing still fail on AMD64 FreeBSD 
CURRENT Shared 2.7:

https://buildbot.python.org/all/#/builders/169/builds/36

Re-running test 'test_multiprocessing' in verbose mode
FAIL: test_fd_transfer (test.test_multiprocessing.WithProcessesTestConnection)
FAIL: test_large_fd_transfer 
(test.test_multiprocessing.WithProcessesTestConnection)

Modules/_multiprocessing/multiprocessing.c uses CMSG_SPACE() *and* CMSG_LEN(). 
Are they used properly? I don't understand the code, but I propose:

diff --git a/Modules/_multiprocessing/multiprocessing.c 
b/Modules/_multiprocessing/multiprocessing.c
index d192a074ba..eecace887e 100644
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -167,7 +167,7 @@ multiprocessing_recvfd(PyObject *self, PyObject *args)
 cmsg = CMSG_FIRSTHDR();
 cmsg->cmsg_level = SOL_SOCKET;
 cmsg->cmsg_type = SCM_RIGHTS;
-cmsg->cmsg_len = CMSG_LEN(sizeof(int));
+cmsg->cmsg_len = CMSG_SPACE(sizeof(int));
 msg.msg_controllen = cmsg->cmsg_len;
 
 Py_BEGIN_ALLOW_THREADS

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[Ned Deily]

Ned Deily  added the comment:

Thanks, Pablo, I cherry-picked the NEWS entries into 3.7.1 and 3.6.7.  So I 
guess the only remaining question here is what to do about 2.7.  I'll also 
leave that for you!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[Ned Deily]

Ned Deily  added the comment:


New changeset ce3b5a80cc6ee4f9aff7b673f457294d7e054349 by Ned Deily (Miss 
Islington (bot)) in branch '3.7':
bpo-34521: Add NEWS entry for changes in GH-9613 (GH-9850)
https://github.com/python/cpython/commit/ce3b5a80cc6ee4f9aff7b673f457294d7e054349


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[Ned Deily]

Ned Deily  added the comment:


New changeset d404ffa8adf252d49731fbd09b740360577274c8 by Ned Deily (Miss 
Islington (bot)) in branch '3.6':
bpo-34521: Add NEWS entry for changes in GH-9613 (GH-9850)
https://github.com/python/cpython/commit/d404ffa8adf252d49731fbd09b740360577274c8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[miss-islington]

miss-islington  added the comment:


New changeset ae011e00189d9083dd84c357718264e24fe77314 by Miss Islington (bot) 
in branch '3.6':
bpo-34521: Add NEWS entry for changes in GH-9613 (GH-9850)
https://github.com/python/cpython/commit/ae011e00189d9083dd84c357718264e24fe77314


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[miss-islington]

miss-islington  added the comment:


New changeset d991ede16b34399c5ea9bd30e9a5c520bed6bea8 by Miss Islington (bot) 
in branch '3.7':
bpo-34521: Add NEWS entry for changes in GH-9613 (GH-9850)
https://github.com/python/cpython/commit/d991ede16b34399c5ea9bd30e9a5c520bed6bea8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[miss-islington]

Change by miss-islington :


--
pull_requests: +9251

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[miss-islington]

Change by miss-islington :


--
pull_requests: +9250

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset bd036d3d15fc1310ccc32a43a3296b8c157ac221 by Pablo Galindo in 
branch 'master':
bpo-34521: Add NEWS entry for changes in GH-9613 (GH-9850)
https://github.com/python/cpython/commit/bd036d3d15fc1310ccc32a43a3296b8c157ac221


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +9220

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[Ned Deily]

Ned Deily  added the comment:

Shouldn't there be a NEWS entry for this change since the change seems to be to 
Lib/multiprocessing and not just its tests?

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:

Pablo: I merged your PR on master, and backport it to 3.6 and 3.7. But I let 
you handle 2.7, since I expect a conflict :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 007fda436f707ac95c2fa8f8886efd9e09d5b630 by Victor Stinner (Miss 
Islington (bot)) in branch '3.7':
bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE 
(GH-9613) (GH-9619)
https://github.com/python/cpython/commit/007fda436f707ac95c2fa8f8886efd9e09d5b630


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[miss-islington]

miss-islington  added the comment:


New changeset 5d33ee1595767d68800fda13bc68b7b01dba5117 by Miss Islington (bot) 
in branch '3.6':
bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE 
(GH-9613)
https://github.com/python/cpython/commit/5d33ee1595767d68800fda13bc68b7b01dba5117


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[miss-islington]

Change by miss-islington :


--
pull_requests: +9018

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[miss-islington]

Change by miss-islington :


--
pull_requests: +9017

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 077061a7b24917aaf31057885c69919c5a553c88 by Victor Stinner (Pablo 
Galindo) in branch 'master':
bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE 
(GH-9613)
https://github.com/python/cpython/commit/077061a7b24917aaf31057885c69919c5a553c88


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34521] Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

[Kubilay Kocak]

Kubilay Kocak  added the comment:

To clarify affected tests/versions:

Python 3.x - test_socket test_multiprocessing_spawn
Python 2.7 - test_multiprocessing_spawn

All of the issues are related to incorrect recvmsg(2) buffer length use, so 
I've amended the issue summary to describe the complete root problem.

The be explicit,

- test_socket fix requires backporting to all supported 3.x branches
- test_multiprocessing_* test fixes (via multiprocessing.reduction fix) 
requires backporting to all supported 3.x branches *and* 2.7

--
title: test_socket.Recvmsg[Into]SCMRightsStreamTest tests fail on AMD64 FreeBSD 
CURRENT -> Multiple tests (test_socket, test_multiprocessing_*) fail due to 
incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com