Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package python-gevent for openSUSE:Factory checked in at 2023-09-22 21:46:56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/python-gevent (Old) and /work/SRC/openSUSE:Factory/.python-gevent.new.1770 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-gevent" Fri Sep 22 21:46:56 2023 rev:45 rq:1112068 version:23.9.0 Changes: -------- --- /work/SRC/openSUSE:Factory/python-gevent/python-gevent.changes 2023-08-15 16:39:19.982711585 +0200 +++ /work/SRC/openSUSE:Factory/.python-gevent.new.1770/python-gevent.changes 2023-09-22 21:47:28.925008396 +0200 @@ -1,0 +2,40 @@ +Mon Sep 18 19:07:56 UTC 2023 - Dirk Müller <[email protected]> + +- update to 23.9.0 (CVE-2023-41419): + * Make ``gevent.select.select`` accept arbitrary iterables, not + just sequences. That is, you can now pass in a generator of file + descriptors instead of a realized list. Internally, arbitrary + iterables are copied into lists. This better matches what the + standard library does. + * On Python 3.11 and newer, opt out of Cython's fast exception + manipulation, which *may* be causing problems in certain + circumstances when combined with greenlets. + * On all versions of Python, adjust some error handling in the + default * -based loop. This fixes several assertion failures + on debug versions of CPython. Hopefully it has a positive + impact under real conditions. + * Make ``gevent.pywsgi`` comply more closely with the HTTP + specification for chunked transfer encoding. In particular, + we are much stricter about trailers, and trailers that are + invalid (too long or featuring disallowed characters) forcibly + close the connection to the client *after* the results have + been sent. + * Trailers otherwise continue to be ignored and are not + available to the WSGI application. + Previously, carefully crafted invalid trailers in chunked + requests on keep-alive connections might appear as two + requests to ``gevent.pywsgi``. Because this was handled + exactly as a normal keep-alive connection with two requests, + the WSGI application should handle it normally. However, if + you were counting on some upstream server to filter incoming + requests based on paths or header fields, and the upstream + server simply passed trailers through without + validating them, then this embedded second request would + bypass those checks. + (If the upstream server validated that the trailers + meet the* HTTP specification, this could not occur, + because characters that are required in an HTTP request, + like a space, are not allowed in trailers.) CVE-2023-41419 + was reserved for this. + +------------------------------------------------------------------- Old: ---- gevent-23.7.0.tar.gz New: ---- gevent-23.9.0.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ python-gevent.spec ++++++ --- /var/tmp/diff_new_pack.bLbV5Z/_old 2023-09-22 21:47:30.821077229 +0200 +++ /var/tmp/diff_new_pack.bLbV5Z/_new 2023-09-22 21:47:30.821077229 +0200 @@ -25,7 +25,7 @@ %endif %{?sle15_python_module_pythons} Name: python-gevent -Version: 23.7.0 +Version: 23.9.0 Release: 0 Summary: Python network library that uses greenlet and libevent License: MIT ++++++ gevent-23.7.0.tar.gz -> gevent-23.9.0.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/.github/workflows/ci.yml new/gevent-23.9.0/.github/workflows/ci.yml --- old/gevent-23.7.0/.github/workflows/ci.yml 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/.github/workflows/ci.yml 2023-09-02 01:22:30.000000000 +0200 @@ -83,18 +83,19 @@ # 3.10+ needs more work: dnspython for example doesn't work # with it. That means for the bulk of our testing we need to # stick to 3.9. - python-version: ["pypy-3.10", "3.12-dev", 3.8, 3.9, '3.10', '3.11'] + python-version: ["3.12-dev", "pypy-3.10", 3.8, 3.9, '3.10', '3.11'] os: [macos-latest, ubuntu-latest] exclude: # The bulk of the testing is on Linux and Windows (appveyor). # Experience shows that it's sufficient to only test the latest - # version on macOS. - - os: macos-latest - python-version: 3.8 - - os: macos-latest - python-version: 3.9 - - os: macos-latest - python-version: 3.10 + # version on macOS. However, that does mean you need to + # manually upload macOS wheels for those versions. + # - os: macos-latest + # python-version: 3.8 + # - os: macos-latest + # python-version: 3.9 + # - os: macos-latest + # python-version: 3.10 - os: macos-latest python-version: "pypy-3.10" @@ -169,8 +170,10 @@ pip install -U pip pip install -U -q setuptools wheel twine pip install -q -U 'cffi;platform_python_implementation=="CPython"' - pip install -q -U 'cython>=3.0b3; python_version < "3.12"' "Cython @ https://github.com/cython/cython/archive/37f4dcdc04547875e2836fda076f5707ec50e579.zip; python_version >= '3.12'" - pip install 'greenlet>=2.0.0 ;platform_python_implementation=="CPython"' 'greenlet >= 3.0a1; python_version >="3.12"' + pip install -q -U 'cython>=3.0.2' + # Use a debug version of greenlet to help catch any errors earlier. + CFLAGS="$CFLAGS -Og -g -UNDEBUG" pip install -v --no-binary :all: 'greenlet>=2.0.0;platform_python_implementation=="CPython" and python_version < "3.12"' + CFLAGS="$CFLAGS -Og -g -UNDEBUG" pip install -v --no-binary :all: 'greenlet>=3.0rc1;platform_python_implementation=="CPython" and python_version >= "3.12"' - name: Build gevent (non-Mac) if: ${{ ! startsWith(runner.os, 'Mac') }} @@ -193,6 +196,13 @@ # output (pip install uses a random temporary directory, making this difficult) python setup.py build_ext -i python setup.py bdist_wheel + # Something in the build system isn't detecting that we're building for both, + # so we're getting tagged with just x86_64. Force the universal2 tag. + # (I've verified that the .so files are in fact universal, with both architectures.) + #wheel tags --abi-tag universal2 dist/*whl + # XXX: That can produce invalid filenames, for some reason. 3.11 came up with + # gevent-23.7.1.dev0-cp311-universal2-macosx_10_9_universal2.whl, which is not valid. + # It's not clear why, because greenlet didn't do that. Maybe because it was already universal? env: # Unlike the above, we are actually distributing these # wheels, so they need to be built for production use. @@ -202,7 +212,7 @@ - name: Check gevent build run: | ls -l dist - twine check dist/* + twine check dist/*whl - name: Upload gevent wheel uses: actions/upload-artifact@v2 with: @@ -240,6 +250,7 @@ # it's sufficient to run the full suite on the current version # and oldest version. - name: "Tests: subproccess and FileObjectThread" + if: startsWith(runner.os, 'Linux') || (startsWith(runner.os, 'Mac') && matrix.python-version == '3.12-dev') # Now, the non-default threaded file object. # In the past, we included all test files that had a reference to 'subprocess'' somewhere in their # text. The monkey-patched stdlib tests were specifically included here. @@ -378,7 +389,7 @@ pip install -U pip pip install -U -q setuptools wheel twine pip install -q -U 'cffi;platform_python_implementation=="CPython"' - pip install -q -U 'cython>=3.0b3; python_version < "3.12"' "Cython @ https://github.com/cython/cython/archive/37f4dcdc04547875e2836fda076f5707ec50e579.zip; python_version >= '3.12'" + pip install -q -U 'cython>=3.0' pip install 'greenlet>=2.0.0; platform_python_implementation=="CPython"' - name: build libs and gevent @@ -435,7 +446,7 @@ - manylinux2014_aarch64 - manylinux2014_ppc64le - manylinux2014_s390x - - manylinux2014_x86_64 + - manylinux_2_28_x86_64 - musllinux_1_1_x86_64 - musllinux_1_1_aarch64 name: ${{ matrix.image }} diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/CHANGES.rst new/gevent-23.9.0/CHANGES.rst --- old/gevent-23.7.0/CHANGES.rst 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/CHANGES.rst 2023-09-02 01:22:30.000000000 +0200 @@ -6,6 +6,59 @@ .. towncrier release notes start +23.9.0 (2023-09-01) +=================== + + +Bugfixes +-------- + +- Make ``gevent.select.select`` accept arbitrary iterables, not just + sequences. That is, you can now pass in a generator of file + descriptors instead of a realized list. Internally, arbitrary + iterables are copied into lists. This better matches what the standard + library does. Thanks to David Salvisberg. + See :issue:`1979`. +- On Python 3.11 and newer, opt out of Cython's fast exception + manipulation, which *may* be causing problems in certain circumstances + when combined with greenlets. + + On all versions of Python, adjust some error handling in the default + C-based loop. This fixes several assertion failures on debug versions + of CPython. Hopefully it has a positive impact under real conditions. + See :issue:`1985`. +- Make ``gevent.pywsgi`` comply more closely with the HTTP specification + for chunked transfer encoding. In particular, we are much stricter + about trailers, and trailers that are invalid (too long or featuring + disallowed characters) forcibly close the connection to the client + *after* the results have been sent. + + Trailers otherwise continue to be ignored and are not available to the + WSGI application. + + Previously, carefully crafted invalid trailers in chunked requests on + keep-alive connections might appear as two requests to + ``gevent.pywsgi``. Because this was handled exactly as a normal + keep-alive connection with two requests, the WSGI application should + handle it normally. However, if you were counting on some upstream + server to filter incoming requests based on paths or header fields, + and the upstream server simply passed trailers through without + validating them, then this embedded second request would bypass those + checks. (If the upstream server validated that the trailers meet the + HTTP specification, this could not occur, because characters that are + required in an HTTP request, like a space, are not allowed in + trailers.) CVE-2023-41419 was reserved for this. + + Our thanks to the original reporters, Keran Mu + ([email protected]) and Jianjun Chen + ([email protected]), from Tsinghua University and Zhongguancun + Laboratory. + See :issue:`1989`. + + +---- + + 23.7.0 (2023-07-11) =================== diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/_setuputils.py new/gevent-23.9.0/_setuputils.py --- old/gevent-23.7.0/_setuputils.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/_setuputils.py 2023-09-02 01:22:30.000000000 +0200 @@ -23,6 +23,9 @@ PYPY = hasattr(sys, 'pypy_version_info') WIN = sys.platform.startswith('win') +PY311 = sys.version_info[:2] >= (3, 11) +PY312 = sys.version_info[:2] >= (3, 12) + RUNNING_ON_TRAVIS = os.environ.get('TRAVIS') RUNNING_ON_APPVEYOR = os.environ.get('APPVEYOR') @@ -233,6 +236,15 @@ # This is for generated include files; see below. '.', ] + if PY311: + # The "fast" code is Cython for manipulating + # exceptions is, unfortunately, broken, at least in 3.0.2. + # The implementation of __Pyx__GetException() doesn't properly set + # tstate->current_exception when it normalizes exceptions, + # causing assertion errors. + # This definitely seems to be a problem on 3.12, and MAY + # be a problem on 3.11 (#1985) + ext.define_macros.append(('CYTHON_FAST_THREAD_STATE', '0')) try: new_ext = cythonize( [ext], diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/appveyor.yml new/gevent-23.9.0/appveyor.yml --- old/gevent-23.7.0/appveyor.yml 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/appveyor.yml 2023-09-02 01:22:30.000000000 +0200 @@ -44,12 +44,12 @@ # Pre-installed Python versions, which Appveyor may upgrade to # a later point release. - # XXX: Cython 3.0b3 won't build us on 3.12 yet. - # - PYTHON: "C:\\Python312-x64" - # PYTHON_VERSION: "3.12.0b3" - # PYTHON_ARCH: "64" - # PYTHON_EXE: python - # APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022 + + - PYTHON: "C:\\Python312-x64" + PYTHON_VERSION: "3.12.0b4" + PYTHON_ARCH: "64" + PYTHON_EXE: python + APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022 # 64-bit - PYTHON: "C:\\Python311-x64" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/docs/servers.rst new/gevent-23.9.0/docs/servers.rst --- old/gevent-23.7.0/docs/servers.rst 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/docs/servers.rst 2023-09-02 01:22:30.000000000 +0200 @@ -43,6 +43,13 @@ :class:`WSGI server <gevent.pywsgi.WSGIServer>`. In addition, gunicorn_ is a stand-alone server that supports gevent. +.. important:: + + The provided server implementations are intended primarily for + development and testing, or internal usage, and otherwise only + generally "safe" scenarios. They have not been security audited. + Expose them to the public Internet at your own risk. + API Reference ============= diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/pyproject.toml new/gevent-23.9.0/pyproject.toml --- old/gevent-23.7.0/pyproject.toml 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/pyproject.toml 2023-09-02 01:22:30.000000000 +0200 @@ -17,18 +17,8 @@ # This was fixed in 3.0a5 (https://github.com/cython/cython/issues/3578) # 3.0a6 fixes an issue cythonizing source on 32-bit platforms. # 3.0a9 is needed for Python 3.10. - # Python 3.12 requires a snapshot at this writing (2023-06-29). This is just before - # 3.0rc1 - # "Cython @ https://github.com/cython/cython/archive/37f4dcdc04547875e2836fda076f5707ec50e579.zip";, - # Unfortunately, markers like python_version do not work for remote installs, - # so we always install that, even if we then provide a different requirement where - # markers DO work. This is a problem beacuse the snapshot cannot be compiled on Appveyor. - # That's OK, because our tests.yml and make-manylinux install this stuff manually where markers - # DO work. It just means that we cannot be built from a sdist on Python 3.12 without - # --no-build-isolation or using setup.py directly; but as soon as the Cython version is released, - # we can be. - - "Cython >= 3.0b3", + # Python 3.12 requires at least 3.0rc2. + "Cython >= 3.0.2", # See version requirements in setup.py "cffi >= 1.12.3 ; platform_python_implementation == 'CPython'", # Python 3.7 requires at least 0.4.14, which is ABI incompatible with earlier @@ -40,7 +30,7 @@ # 3.0 is ABI compatible with earlier releases, so we can switch back and # forth between 2 and 3 without recompiling. 3.0 is required for # Python 3.12 - "greenlet >= 3.0a1 ; platform_python_implementation == 'CPython'", + "greenlet >= 3.0rc1 ; platform_python_implementation == 'CPython'", ] [tool.towncrier] diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/scripts/releases/geventrel.sh new/gevent-23.9.0/scripts/releases/geventrel.sh --- old/gevent-23.7.0/scripts/releases/geventrel.sh 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/scripts/releases/geventrel.sh 2023-09-02 01:22:30.000000000 +0200 @@ -19,7 +19,8 @@ # because it's not available on earlier releases and leads to # segfaults because the symbol clock_gettime is NULL. # See https://github.com/gevent/gevent/issues/916 -export CPPFLAGS="-D_DARWIN_FEATURE_CLOCK_GETTIME=0" +#export CPPFLAGS="-D_DARWIN_FEATURE_CLOCK_GETTIME=0" +export ARCHFLAGS="-arch arm64 -arch x86_64" BASE=`pwd`/../../ BASE=`greadlink -f $BASE` diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/scripts/releases/geventreleases.sh new/gevent-23.9.0/scripts/releases/geventreleases.sh --- old/gevent-23.7.0/scripts/releases/geventreleases.sh 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/scripts/releases/geventreleases.sh 2023-09-02 01:22:30.000000000 +0200 @@ -7,22 +7,10 @@ mkdir /tmp/gevent/ -# 2.7 is a python.org build, builds a 10_6_intel wheel -./geventrel.sh /usr/local/bin/python2.7 - -# 3.5 is a python.org build, builds a 10_6_intel wheel -./geventrel.sh /usr/local/bin/python3.5 - -# 3.6 is a python.org build, builds a 10_6_intel wheel -./geventrel.sh /usr/local/bin/python3.6 - -# 3.7 is a python.org build, builds a 10_6_intel wheel -./geventrel.sh /usr/local/bin/python3.7 ./geventrel.sh /usr/local/bin/python3.8 ./geventrel.sh /usr/local/bin/python3.9 - - -# PyPy 4.0 -./geventrel.sh `which pypy` +./geventrel.sh /usr/local/bin/python3.10 +./geventrel.sh /usr/local/bin/python3.11 +./geventrel.sh /usr/local/bin/python3.12 wait diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/scripts/releases/make-manylinux new/gevent-23.9.0/scripts/releases/make-manylinux --- old/gevent-23.7.0/scripts/releases/make-manylinux 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/scripts/releases/make-manylinux 2023-09-02 01:22:30.000000000 +0200 @@ -155,7 +155,7 @@ # Start echoing commands (doing it earlier is too much) set -x - for variant in /opt/python/cp{312,38,39,310,311,312}*; do + for variant in /opt/python/cp{312,38,39,310,311}*; do echo $SEP export PATH="$variant/bin:$OPATH" if [ -n "$SLOW_BUILD" ]; then @@ -172,8 +172,7 @@ # The downside is that we must install dependencies manually. # NOTE: We can't upgrade ``wheel`` because ``auditwheel`` depends on # it, and auditwheel is installed in one of these environments. - time python -m pip install -U 'cython>=3.0b3; python_version < "3.12"' - time python -m pip install -v -U "Cython @ https://github.com/cython/cython/archive/37f4dcdc04547875e2836fda076f5707ec50e579.zip; python_version >= '3.12'" + time python -m pip install -U 'cython>=3.0' time python -mpip install -U cffi 'greenlet >= 2.0.0; python_version < "3.12"' 'greenlet >= 3.0a1; python_version >= "3.12"' setuptools echo "$variant: Building wheel" time (python setup.py bdist_wheel) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/setup.py new/gevent-23.9.0/setup.py --- old/gevent-23.7.0/setup.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/setup.py 2023-09-02 01:22:30.000000000 +0200 @@ -211,7 +211,7 @@ # 3.0 is ABI compatible and adds support for Python 3.12 (but right # now it's alpha because of Cython, so we only require it on 3.12) 'greenlet >= 2.0.0 ; platform_python_implementation=="CPython" and python_version < "3.12"', - 'greenlet >= 3.0a1 ; platform_python_implementation=="CPython" and python_version >= "3.12"', + 'greenlet >= 3.0rc1 ; platform_python_implementation=="CPython" and python_version >= "3.12"', ] # Note that we don't add cffi to install_requires, it's diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/__init__.py new/gevent-23.9.0/src/gevent/__init__.py --- old/gevent-23.7.0/src/gevent/__init__.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/__init__.py 2023-09-02 01:22:30.000000000 +0200 @@ -27,7 +27,7 @@ #: Use ``pkg_resources.parse_version(__version__)`` or #: ``packaging.version.Version(__version__)`` to get a machine-usable #: value. -__version__ = '23.7.0' +__version__ = '23.9.0' __all__ = [ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/_compat.h new/gevent-23.9.0/src/gevent/_compat.h --- old/gevent-23.7.0/src/gevent/_compat.h 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/_compat.h 2023-09-02 01:22:30.000000000 +0200 @@ -24,13 +24,9 @@ # define _PyCFrame CFrame #endif -/* FrameType and CodeType changed a lot in 3.11. */ -#if GREENLET_PY311 - /* _PyInterpreterFrame moved to the internal C API in Python 3.11 */ -# include <internal/pycore_frame.h> -#else #include <frameobject.h> -#if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION < 9) + +#if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION < 9 /* these were added in 3.9, though they officially became stable in 3.10 */ /* the official versions of these functions return strong references, so we need to increment the refcount before returning, not just to match the @@ -51,7 +47,7 @@ return result; } #endif /* support 3.8 and below. */ -#endif + /** Unlike PyFrame_GetBack, which can return NULL, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/_compat.py new/gevent-23.9.0/src/gevent/_compat.py --- old/gevent-23.7.0/src/gevent/_compat.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/_compat.py 2023-09-02 01:22:30.000000000 +0200 @@ -14,6 +14,7 @@ PY39 = sys.version_info[:2] >= (3, 9) PY311 = sys.version_info[:2] >= (3, 11) +PY312 = sys.version_info[:2] >= (3, 11) PYPY = hasattr(sys, 'pypy_version_info') WIN = sys.platform.startswith("win") LINUX = sys.platform.startswith('linux') diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/libev/callbacks.c new/gevent-23.9.0/src/gevent/libev/callbacks.c --- old/gevent-23.7.0/src/gevent/libev/callbacks.c 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/libev/callbacks.c 2023-09-02 01:22:30.000000000 +0200 @@ -39,13 +39,16 @@ #define GGIL_RELEASE PyGILState_Release(___save); -static CYTHON_INLINE void gevent_check_signals(struct PyGeventLoopObject* loop) { +static CYTHON_INLINE void gevent_check_signals(struct PyGeventLoopObject* loop) +{ if (!ev_is_default_loop(loop->_ptr)) { /* only reporting signals on the default loop */ return; } PyErr_CheckSignals(); - if (PyErr_Occurred()) gevent_handle_error(loop, Py_None); + if (PyErr_Occurred()) { + gevent_handle_error(loop, Py_None); + } } #define GET_OBJECT(PY_TYPE, EV_PTR, MEMBER) \ @@ -54,7 +57,8 @@ void gevent_noop(struct ev_loop* loop, void* watcher, int revents) {} -static void gevent_stop(PyObject* watcher, struct PyGeventLoopObject* loop) { +static void gevent_stop(PyObject* watcher, struct PyGeventLoopObject* loop) +{ PyObject *result, *method; int error; error = 1; @@ -68,12 +72,14 @@ Py_DECREF(method); } if (error) { + assert(PyErr_Occurred()); gevent_handle_error(loop, watcher); } } -static void gevent_callback(struct PyGeventLoopObject* loop, PyObject* callback, PyObject* args, PyObject* watcher, void *c_watcher, int revents) { +static void gevent_callback(struct PyGeventLoopObject* loop, PyObject* callback, PyObject* args, PyObject* watcher, void *c_watcher, int revents) +{ GGIL_DECLARE; PyObject *result, *py_events; long length; @@ -89,6 +95,8 @@ } length = PyTuple_Size(args); if (length < 0) { + /* returns -1 and sets an error if args isn't a tuple. */ + assert(PyErr_Occurred()); gevent_handle_error(loop, watcher); goto end; } @@ -108,6 +116,7 @@ Py_DECREF(result); } else { + assert(PyErr_Occurred()); gevent_handle_error(loop, watcher); if (revents & (EV_READ|EV_WRITE)) { /* io watcher: not stopping it may cause the failing callback to be called repeatedly */ @@ -134,7 +143,8 @@ } -void gevent_call(struct PyGeventLoopObject* loop, struct PyGeventCallbackObject* cb) { +void gevent_call(struct PyGeventLoopObject* loop, struct PyGeventCallbackObject* cb) +{ /* no need for GIL here because it is only called from run_callbacks which already has GIL */ PyObject *result, *callback, *args; if (!loop || !cb) @@ -152,12 +162,29 @@ Py_INCREF(Py_None); Py_DECREF(cb->callback); cb->callback = Py_None; - + /* + you are not allowed to use PyObject_Call() with an error pending; + debug builds crash with an assertion. + How do we get here with an error pending? good question. It's been + seen in 3.12 before we stopped using CYTHON_FAST_THREAD_STATE. + Hopefully changing that makes this dead code. + */ + if (PyErr_Occurred()) { + /* by now, cb->callback is None, so using it as the context doesn't + produce a useful output. The callable object is more helpful. + Writing unraisable clears the error, unless it gets an error of + its own. + */ + PyErr_WriteUnraisable(callback); + PyErr_Clear(); + } + assert(!PyErr_Occurred()); result = PyObject_Call(callback, args, NULL); if (result) { Py_DECREF(result); } else { + assert(PyErr_Occurred()); gevent_handle_error(loop, (PyObject*)cb); } @@ -200,6 +227,14 @@ Py_DECREF(result); } else { + /* + For reasons that are unclear, PyErr_WriteUnraisable, which invokes + sys.unraisablehook, isn't safe to call here, at least in some cases. + test__pool.TestCoroutinePool.test_stderr_raising fails its timeout + if we use that. Instead, we use PyErr_Print, which doesn't have any + context, but doesn't hang either. It calls sys.excepthook. + */ + PyErr_Print(); PyErr_Clear(); } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/libev/corecext.pyx new/gevent-23.9.0/src/gevent/libev/corecext.pyx --- old/gevent-23.7.0/src/gevent/libev/corecext.pyx 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/libev/corecext.pyx 2023-09-02 01:22:30.000000000 +0200 @@ -34,6 +34,10 @@ from cpython.exc cimport PyErr_NormalizeException from cpython.exc cimport PyErr_WriteUnraisable from libc.errno cimport errno +from cpython cimport PyErr_Fetch +from cpython cimport PyErr_Occurred +from cpython cimport PyObject +from cpython cimport PyErr_Clear cdef extern from "Python.h": int Py_ReprEnter(object) @@ -199,10 +203,9 @@ return result -if sys.version_info[0] >= 3: - basestring = (bytes, str) -else: - basestring = __builtins__.basestring + +basestring = (bytes, str) + cpdef unsigned int _flags_to_int(object flags) except? -1: @@ -483,19 +486,29 @@ self.starting_timer_may_update_loop_time = True cdef libev.ev_tstamp now = libev.ev_now(self._ptr) cdef libev.ev_tstamp expiration = now + <libev.ev_tstamp>getswitchinterval() - + cdef object cb_callable # for printing later + assert not PyErr_Occurred() try: libev.ev_timer_stop(self._ptr, &self._timer0) while self._callbacks.head is not None: cb = self._callbacks.popleft() - libev.ev_unref(self._ptr) # On entry, this will set cb.callback to None, # changing cb.pending from True to False; on exit, # this will set cb.args to None, changing bool(cb) # from True to False. # XXX: Why is this a C callback, not cython? + cb_callable = cb.callback gevent_call(self, cb) + if PyErr_Occurred(): + # Exceptions should not escape gevent_call, + # but just in case... + # note we don't use gevent_handle_error here, between + # running callbacks is a fairly fragile state and + # that directs back up to the hub and user code. + PyErr_WriteUnraisable(cb_callable) + PyErr_Clear() + cb_callable = None count -= 1 if count == 0 and self._callbacks.head is not None: @@ -1377,8 +1390,7 @@ # Things used in callbacks.c -from cpython cimport PyErr_Fetch -from cpython cimport PyObject +from traceback import print_exception cdef public void gevent_handle_error(loop loop, object context): cdef PyObject* typep @@ -1415,9 +1427,7 @@ traceback = <object>tracebackp Py_DECREF(traceback) - # If this method fails by raising an exception, - # cython will print it for us because we don't return a - # Python object and we don't declare an `except` clause. + # Prior to Cython 3.<something>, we relied on Cython printing an # uncaught exception here (because we don't return a Python object, and # we have no except clause). It seems that as-of 3.0b3 at least, @@ -1426,7 +1436,17 @@ try: loop.handle_error(context, type, value, traceback) except: - PyErr_WriteUnraisable(context) + # In an except: block, PyErr_Occurred() is actually false. + # Cython has captured the exception and moved it around. The + # exc_info is available at the python level, but + # the C level APIs aren't going to work. In debug builds, + # PyErr_WriteUnraisable will crash with an assertion. + # + # It would be nice to call ``sys.unraisablehook``, but the default + # implementation of that requires that the argument be of + # a specific private type we cannot construct. + print_exception(*sys.exc_info()) + cdef public tuple _empty_tuple = () diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/pywsgi.py new/gevent-23.9.0/src/gevent/pywsgi.py --- old/gevent-23.7.0/src/gevent/pywsgi.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/pywsgi.py 2023-09-02 01:22:30.000000000 +0200 @@ -1,13 +1,32 @@ # Copyright (c) 2005-2009, eventlet contributors # Copyright (c) 2009-2018, gevent contributors """ -A pure-Python, gevent-friendly WSGI server. +A pure-Python, gevent-friendly WSGI server implementing HTTP/1.1. The server is provided in :class:`WSGIServer`, but most of the actual WSGI work is handled by :class:`WSGIHandler` --- a new instance is created for each request. The server can be customized to use different subclasses of :class:`WSGIHandler`. +.. important:: + + This server is intended primarily for development and testing, and + secondarily for other "safe" scenarios where it will not be exposed to + potentially malicious input. The code has not been security audited, + and is not intended for direct exposure to the public Internet. For production + usage on the Internet, either choose a production-strength server such as + gunicorn, or put a reverse proxy between gevent and the Internet. + +.. versionchanged:: 23.9.0 + + Complies more closely with the HTTP specification for chunked transfer encoding. + In particular, we are much stricter about trailers, and trailers that + are invalid (too long or featuring disallowed characters) forcibly close + the connection to the client *after* the results have been sent. + + Trailers otherwise continue to be ignored and are not available to the + WSGI application. + """ from __future__ import absolute_import @@ -22,10 +41,7 @@ import traceback from datetime import datetime -try: - from urllib import unquote -except ImportError: - from urllib.parse import unquote # python 2 pylint:disable=import-error,no-name-in-module +from urllib.parse import unquote from gevent import socket import gevent @@ -50,29 +66,52 @@ MAX_REQUEST_LINE = 8192 # Weekday and month names for HTTP date/time formatting; always English! -_WEEKDAYNAME = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"] -_MONTHNAME = [None, # Dummy so we can use 1-based month numbers +_WEEKDAYNAME = ("Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun") +_MONTHNAME = (None, # Dummy so we can use 1-based month numbers "Jan", "Feb", "Mar", "Apr", "May", "Jun", - "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"] + "Jul", "Aug", "Sep", "Oct", "Nov", "Dec") # The contents of the "HEX" grammar rule for HTTP, upper and lowercase A-F plus digits, # in byte form for comparing to the network. _HEX = string.hexdigits.encode('ascii') +# The characters allowed in "token" rules. + +# token = 1*tchar +# tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" +# / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~" +# / DIGIT / ALPHA +# ; any VCHAR, except delimiters +# ALPHA = %x41-5A / %x61-7A ; A-Z / a-z +_ALLOWED_TOKEN_CHARS = frozenset( + # Remember we have to be careful because bytestrings + # inexplicably iterate as integers, which are not equal to bytes. + + # explicit chars then DIGIT + (c.encode('ascii') for c in "!#$%&'*+-.^_`|~0123456789") + # Then we add ALPHA +) | {c.encode('ascii') for c in string.ascii_letters} +assert b'A' in _ALLOWED_TOKEN_CHARS + + # Errors _ERRORS = {} _INTERNAL_ERROR_STATUS = '500 Internal Server Error' _INTERNAL_ERROR_BODY = b'Internal Server Error' -_INTERNAL_ERROR_HEADERS = [('Content-Type', 'text/plain'), - ('Connection', 'close'), - ('Content-Length', str(len(_INTERNAL_ERROR_BODY)))] +_INTERNAL_ERROR_HEADERS = ( + ('Content-Type', 'text/plain'), + ('Connection', 'close'), + ('Content-Length', str(len(_INTERNAL_ERROR_BODY))) +) _ERRORS[500] = (_INTERNAL_ERROR_STATUS, _INTERNAL_ERROR_HEADERS, _INTERNAL_ERROR_BODY) _BAD_REQUEST_STATUS = '400 Bad Request' _BAD_REQUEST_BODY = '' -_BAD_REQUEST_HEADERS = [('Content-Type', 'text/plain'), - ('Connection', 'close'), - ('Content-Length', str(len(_BAD_REQUEST_BODY)))] +_BAD_REQUEST_HEADERS = ( + ('Content-Type', 'text/plain'), + ('Connection', 'close'), + ('Content-Length', str(len(_BAD_REQUEST_BODY))) +) _ERRORS[400] = (_BAD_REQUEST_STATUS, _BAD_REQUEST_HEADERS, _BAD_REQUEST_BODY) _REQUEST_TOO_LONG_RESPONSE = b"HTTP/1.1 414 Request URI Too Long\r\nConnection: close\r\nContent-length: 0\r\n\r\n" @@ -200,23 +239,37 @@ # Read and return the next integer chunk length. If no # chunk length can be read, raises _InvalidClientInput. - # Here's the production for a chunk: - # (http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) - # chunk = chunk-size [ chunk-extension ] CRLF - # chunk-data CRLF - # chunk-size = 1*HEX - # chunk-extension= *( ";" chunk-ext-name [ "=" chunk-ext-val ] ) - # chunk-ext-name = token - # chunk-ext-val = token | quoted-string - - # To cope with malicious or broken clients that fail to send valid - # chunk lines, the strategy is to read character by character until we either reach - # a ; or newline. If at any time we read a non-HEX digit, we bail. If we hit a - # ;, indicating an chunk-extension, we'll read up to the next - # MAX_REQUEST_LINE characters - # looking for the CRLF, and if we don't find it, we bail. If we read more than 16 hex characters, - # (the number needed to represent a 64-bit chunk size), we bail (this protects us from - # a client that sends an infinite stream of `F`, for example). + # Here's the production for a chunk (actually the whole body): + # (https://www.rfc-editor.org/rfc/rfc7230#section-4.1) + + # chunked-body = *chunk + # last-chunk + # trailer-part + # CRLF + # + # chunk = chunk-size [ chunk-ext ] CRLF + # chunk-data CRLF + # chunk-size = 1*HEXDIG + # last-chunk = 1*("0") [ chunk-ext ] CRLF + # trailer-part = *( header-field CRLF ) + # chunk-data = 1*OCTET ; a sequence of chunk-size octets + # + # chunk-ext = *( ";" chunk-ext-name [ "=" chunk-ext-val ] ) + # + # chunk-ext-name = token + # chunk-ext-val = token / quoted-string + + # To cope with malicious or broken clients that fail to send + # valid chunk lines, the strategy is to read character by + # character until we either reach a ; or newline. If at any + # time we read a non-HEX digit, we bail. If we hit a ;, + # indicating an chunk-extension, we'll read up to the next + # MAX_REQUEST_LINE characters ("A server ought to limit the + # total length of chunk extensions received") looking for the + # CRLF, and if we don't find it, we bail. If we read more than + # 16 hex characters, (the number needed to represent a 64-bit + # chunk size), we bail (this protects us from a client that + # sends an infinite stream of `F`, for example). buf = BytesIO() while 1: @@ -224,16 +277,20 @@ if not char: self._chunked_input_error = True raise _InvalidClientInput("EOF before chunk end reached") - if char == b'\r': - break - if char == b';': + + if char in ( + b'\r', # Beginning EOL + b';', # Beginning extension + ): break - if char not in _HEX: + if char not in _HEX: # Invalid data. self._chunked_input_error = True raise _InvalidClientInput("Non-hex data", char) + buf.write(char) - if buf.tell() > 16: + + if buf.tell() > 16: # Too many hex bytes self._chunked_input_error = True raise _InvalidClientInput("Chunk-size too large.") @@ -253,11 +310,72 @@ if char == b'\r': # We either got here from the main loop or from the # end of an extension + self.__read_chunk_size_crlf(rfile, newline_only=True) + result = int(buf.getvalue(), 16) + if result == 0: + # The only time a chunk size of zero is allowed is the final + # chunk. It is either followed by another \r\n, or some trailers + # which are then followed by \r\n. + while self.__read_chunk_trailer(rfile): + pass + return result + + # Trailers have the following production (they are a header-field followed by CRLF) + # See above for the definition of "token". + # + # header-field = field-name ":" OWS field-value OWS + # field-name = token + # field-value = *( field-content / obs-fold ) + # field-content = field-vchar [ 1*( SP / HTAB ) field-vchar ] + # field-vchar = VCHAR / obs-text + # obs-fold = CRLF 1*( SP / HTAB ) + # ; obsolete line folding + # ; see Section 3.2.4 + + + def __read_chunk_trailer(self, rfile, ): + # With rfile positioned just after a \r\n, read a trailer line. + # Return a true value if a non-empty trailer was read, and + # return false if an empty trailer was read (meaning the trailers are + # done). + # If a single line exceeds the MAX_REQUEST_LINE, raise an exception. + # If the field-name portion contains invalid characters, raise an exception. + + i = 0 + empty = True + seen_field_name = False + while i < MAX_REQUEST_LINE: char = rfile.read(1) - if char != b'\n': + if char == b'\r': + # Either read the next \n or raise an error. + self.__read_chunk_size_crlf(rfile, newline_only=True) + break + # Not a \r, so we are NOT an empty chunk. + empty = False + if char == b':' and i > 0: + # We're ending the field-name part; stop validating characters. + # Unless : was the first character... + seen_field_name = True + if not seen_field_name and char not in _ALLOWED_TOKEN_CHARS: + raise _InvalidClientInput('Invalid token character: %r' % (char,)) + i += 1 + else: + # We read too much + self._chunked_input_error = True + raise _InvalidClientInput("Too large chunk trailer") + return not empty + + def __read_chunk_size_crlf(self, rfile, newline_only=False): + # Also for safety, correctly verify that we get \r\n when expected. + if not newline_only: + char = rfile.read(1) + if char != b'\r': self._chunked_input_error = True - raise _InvalidClientInput("Line didn't end in CRLF") - return int(buf.getvalue(), 16) + raise _InvalidClientInput("Line didn't end in CRLF: %r" % (char,)) + char = rfile.read(1) + if char != b'\n': + self._chunked_input_error = True + raise _InvalidClientInput("Line didn't end in LF: %r" % (char,)) def _chunked_read(self, length=None, use_readline=False): # pylint:disable=too-many-branches @@ -290,7 +408,7 @@ self.position += datalen if self.chunk_length == self.position: - rfile.readline() + self.__read_chunk_size_crlf(rfile) if length is not None: length -= datalen @@ -303,9 +421,9 @@ # determine the next size to read self.chunk_length = self.__read_chunk_length(rfile) self.position = 0 - if self.chunk_length == 0: - # Last chunk. Terminates with a CRLF. - rfile.readline() + # If chunk_length was 0, we already read any trailers and + # validated that we have ended with \r\n\r\n. + return b''.join(response) def read(self, length=None): @@ -524,7 +642,8 @@ elif len(words) == 2: self.command, self.path = words if self.command != "GET": - raise _InvalidClientRequest('Expected GET method: %r' % (raw_requestline,)) + raise _InvalidClientRequest('Expected GET method; Got command=%r; path=%r; raw=%r' % ( + self.command, self.path, raw_requestline,)) self.request_version = "HTTP/0.9" # QQQ I'm pretty sure we can drop support for HTTP/0.9 else: @@ -989,14 +1108,28 @@ finally: try: self.wsgi_input._discard() + except _InvalidClientInput: + # This one is deliberately raised to the outer + # scope, because, with the incoming stream in some bad state, + # we can't be sure we can synchronize and properly parse the next + # request. + raise except socket.error: - # Don't let exceptions during discarding + # Don't let socket exceptions during discarding # input override any exception that may have been # raised by the application, such as our own _InvalidClientInput. # In the general case, these aren't even worth logging (see the comment # just below) pass - except _InvalidClientInput: + except _InvalidClientInput as ex: + # DO log this one because: + # - Some of the data may have been read and acted on by the + # application; + # - The response may or may not have been sent; + # - It's likely that the client is bad, or malicious, and + # users might wish to take steps to block the client. + self._handle_client_error(ex) + self.close_connection = True self._send_error_response_if_possible(400) except socket.error as ex: if ex.args[0] in self.ignored_socket_errors: @@ -1039,17 +1172,22 @@ def _handle_client_error(self, ex): # Called for invalid client input # Returns the appropriate error response. - if not isinstance(ex, ValueError): + if not isinstance(ex, (ValueError, _InvalidClientInput)): # XXX: Why not self._log_error to send it through the loop's # handle_error method? + # _InvalidClientRequest is a ValueError; _InvalidClientInput is an IOError. traceback.print_exc() if isinstance(ex, _InvalidClientRequest): # No formatting needed, that's already been handled. In fact, because the # formatted message contains user input, it might have a % in it, and attempting # to format that with no arguments would be an error. - self.log_error(ex.formatted_message) + # However, the error messages do not include the requesting IP + # necessarily, so we do add that. + self.log_error('(from %s) %s', self.client_address, ex.formatted_message) else: - self.log_error('Invalid request: %s', str(ex) or ex.__class__.__name__) + self.log_error('Invalid request (from %s): %s', + self.client_address, + str(ex) or ex.__class__.__name__) return ('400', _BAD_REQUEST_RESPONSE) def _headers(self): diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/select.py new/gevent-23.9.0/src/gevent/select.py --- old/gevent-23.7.0/src/gevent/select.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/select.py 2023-09-02 01:22:30.000000000 +0200 @@ -159,6 +159,13 @@ # forward compatible raise ValueError("timeout must be non-negative") + # since rlist and wlist can be any iterable we will have to first + # copy them into a list, so we can use them in both _original_select + # and in SelectResult.select. We don't need to do it for xlist, since + # that one will only be passed into _original_select + rlist = rlist if isinstance(rlist, (list, tuple)) else list(rlist) + wlist = wlist if isinstance(wlist, (list, tuple)) else list(wlist) + # First, do a poll with the original select system call. This is # the most efficient way to check to see if any of the file # descriptors have previously been closed and raise the correct diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/subprocess.py new/gevent-23.9.0/src/gevent/subprocess.py --- old/gevent-23.7.0/src/gevent/subprocess.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/subprocess.py 2023-09-02 01:22:30.000000000 +0200 @@ -360,10 +360,11 @@ To capture standard error in the result, use ``stderr=STDOUT``:: - >>> print(check_output(["/bin/sh", "-c", + >>> output = check_output(["/bin/sh", "-c", ... "ls -l non_existent_file ; exit 0"], - ... stderr=STDOUT).decode('ascii').strip()) - ls: non_existent_file: No such file or directory + ... stderr=STDOUT).decode('ascii').strip() + >>> print(output.rsplit(':', 1)[1].strip()) + No such file or directory There is an additional optional argument, "input", allowing you to pass a string to the subprocess's stdin. If you use this argument diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/testing/__init__.py new/gevent-23.9.0/src/gevent/testing/__init__.py --- old/gevent-23.7.0/src/gevent/testing/__init__.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/testing/__init__.py 2023-09-02 01:22:30.000000000 +0200 @@ -112,6 +112,7 @@ from .skipping import skipOnLibuvOnTravisOnCPython27 from .skipping import skipOnPy37 from .skipping import skipOnPy310 +from .skipping import skipOnPy312 from .skipping import skipOnPy3 from .skipping import skipWithoutResource from .skipping import skipWithoutExternalNetwork diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/testing/patched_tests_setup.py new/gevent-23.9.0/src/gevent/testing/patched_tests_setup.py --- old/gevent-23.7.0/src/gevent/testing/patched_tests_setup.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/testing/patched_tests_setup.py 2023-09-02 01:22:30.000000000 +0200 @@ -1221,6 +1221,15 @@ 'test_threading.ThreadTests.test_gettrace_all_threads', ] + if WIN: + disabled_tests += [ + # These three are looking for an error string that matches, + # and ours differs very slightly + 'test_socket.BasicHyperVTest.testCreateHyperVSocketAddrNotTupleFailure', + 'test_socket.BasicHyperVTest.testCreateHyperVSocketAddrServiceIdNotValidUUIDFailure', + 'test_socket.BasicHyperVTest.testCreateHyperVSocketAddrVmIdNotValidUUIDFailure', + ] + if TRAVIS: disabled_tests += [ # These tests frequently break when we try to use newer Travis CI images, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/testing/skipping.py new/gevent-23.9.0/src/gevent/testing/skipping.py --- old/gevent-23.7.0/src/gevent/testing/skipping.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/testing/skipping.py 2023-09-02 01:22:30.000000000 +0200 @@ -48,6 +48,7 @@ skipOnPy3 = unittest.skip if sysinfo.PY3 else _do_not_skip skipOnPy37 = unittest.skip if sysinfo.PY37 else _do_not_skip skipOnPy310 = unittest.skip if sysinfo.PY310 else _do_not_skip +skipOnPy312 = unittest.skip if sysinfo.PY312 else _do_not_skip skipOnPurePython = unittest.skip if sysinfo.PURE_PYTHON else _do_not_skip skipWithCExtensions = unittest.skip if not sysinfo.PURE_PYTHON else _do_not_skip diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/testing/testcase.py new/gevent-23.9.0/src/gevent/testing/testcase.py --- old/gevent-23.7.0/src/gevent/testing/testcase.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/testing/testcase.py 2023-09-02 01:22:30.000000000 +0200 @@ -151,7 +151,7 @@ gevent.Timeout.__init__( self, timeout, - '%r: test timed out\n' % (method,), + '%r: test timed out (set class __timeout__ to increase)\n' % (method,), ref=False ) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/tests/known_failures.py new/gevent-23.9.0/src/gevent/tests/known_failures.py --- old/gevent-23.7.0/src/gevent/tests/known_failures.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/tests/known_failures.py 2023-09-02 01:22:30.000000000 +0200 @@ -92,6 +92,7 @@ BIT_64 = ConstantCondition(struct.calcsize('P') * 8 == 64, 'BIT_64') PY380_EXACTLY = ConstantCondition(sys.version_info[:3] == (3, 8, 0), 'PY380_EXACTLY') PY312B3_EXACTLY = ConstantCondition(sys.version_info == (3, 12, 0, 'beta', 3)) +PY312B4_EXACTLY = ConstantCondition(sys.version_info == (3, 12, 0, 'beta', 4)) class _Definition(object): __slots__ = ( @@ -213,7 +214,7 @@ So far, this is only seen on one version, in CI environment. """, - when=(CI & PY312B3_EXACTLY) + when=(CI & (PY312B3_EXACTLY | PY312B4_EXACTLY)) ) test__issue6 = Flaky( diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/tests/test__pool.py new/gevent-23.9.0/src/gevent/tests/test__pool.py --- old/gevent-23.7.0/src/gevent/tests/test__pool.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/tests/test__pool.py 2023-09-02 01:22:30.000000000 +0200 @@ -108,6 +108,11 @@ sys.stderr = FakeFile() waiter = pool.spawn(crash) with gevent.Timeout(2): + # Without the timeout, we can get caught...doing something? + # If we call PyErr_WriteUnraisable at a certain point, + # we appear to switch back to the hub and do nothing, + # meaning we sit forever. The timeout at least keeps us from + # doing that and fails the test if we mess up error handling. self.assertRaises(RuntimeError, waiter.get) # the pool should have something free at this point since the # waiter returned @@ -126,13 +131,13 @@ def crash(*_args, **_kw): - raise RuntimeError("Whoa") + raise RuntimeError("Raising an error from the crash() function") class FakeFile(object): def write(self, *_args): - raise RuntimeError('Whaaa') + raise RuntimeError('Writing to the file failed') class PoolBasicTests(greentest.TestCase): diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/tests/test__pywsgi.py new/gevent-23.9.0/src/gevent/tests/test__pywsgi.py --- old/gevent-23.7.0/src/gevent/tests/test__pywsgi.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/tests/test__pywsgi.py 2023-09-02 01:22:30.000000000 +0200 @@ -25,21 +25,11 @@ monkey.patch_all() from contextlib import contextmanager -try: - from urllib.parse import parse_qs -except ImportError: - # Python 2 - from urlparse import parse_qs +from urllib.parse import parse_qs import os import sys -try: - # On Python 2, we want the C-optimized version if - # available; it has different corner-case behaviour than - # the Python implementation, and it used by socket.makefile - # by default. - from cStringIO import StringIO -except ImportError: - from io import BytesIO as StringIO +from io import BytesIO as StringIO + import weakref import unittest from wsgiref.validate import validator @@ -156,6 +146,10 @@ @classmethod def read(cls, fd, code=200, reason='default', version='1.1', body=None, chunks=None, content_length=None): + """ + Read an HTTP response, optionally perform assertions, + and return the Response object. + """ # pylint:disable=too-many-branches _status_line, headers = read_headers(fd) self = cls(_status_line, headers) @@ -716,7 +710,14 @@ class TestChunkedPost(TestCase): + calls = 0 + + def setUp(self): + super().setUp() + self.calls = 0 + def application(self, env, start_response): + self.calls += 1 self.assertTrue(env.get('wsgi.input_terminated')) start_response('200 OK', [('Content-Type', 'text/plain')]) if env['PATH_INFO'] == '/a': @@ -730,6 +731,8 @@ if env['PATH_INFO'] == '/c': return list(iter(lambda: env['wsgi.input'].read(1), b'')) + return [b'We should not get here', env['PATH_INFO'].encode('ascii')] + def test_014_chunked_post(self): data = (b'POST /a HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n' b'Transfer-Encoding: chunked\r\n\r\n' @@ -797,6 +800,170 @@ fd.write(data) read_http(fd, code=400) + def test_trailers_keepalive_ignored(self): + # Trailers after a chunk are ignored. + data1 = ( + b'POST /a HTTP/1.1\r\n' + b'Host: localhost\r\n' + b'Connection: keep-alive\r\n' + b'Transfer-Encoding: chunked\r\n' + b'\r\n' + b'2\r\noh\r\n' + b'4\r\n hai\r\n' + b'0\r\n' # last-chunk + # Normally the final CRLF would go here, but if you put in a + # trailer, it doesn't. + b'trailer1: value1\r\n' + b'trailer2: value2\r\n' + b'\r\n' # Really terminate the chunk. + ) + data2 = ( + b'POST /a HTTP/1.1\r\n' + b'Host: localhost\r\n' + b'Connection: close\r\n' + b'Transfer-Encoding: chunked\r\n' + b'\r\n' + b'2\r\noh\r\n' + b'4\r\n bye\r\n' + b'0\r\n' # last-chunk + ) + with self.makefile() as fd: + fd.write(data1) + read_http(fd, body='oh hai') + fd.write(data2) + read_http(fd, body='oh bye') + + self.assertEqual(self.calls, 2) + + def test_trailers_close_ignored(self): + data = ( + b'POST /a HTTP/1.1\r\n' + b'Host: localhost\r\n' + b'Connection: close\r\n' + b'Transfer-Encoding: chunked\r\n' + b'\r\n' + b'2\r\noh\r\n' + b'4\r\n hai\r\n' + b'0\r\n' # last-chunk + # Normally the final CRLF would go here, but if you put in a + # trailer, it doesn't. + # b'\r\n' + b'GETpath2a:123 HTTP/1.1\r\n' + b'Host: a.com\r\n' + b'Connection: close\r\n' + b'\r\n' + ) + with self.makefile() as fd: + fd.write(data) + read_http(fd, body='oh hai') + with self.assertRaises(ConnectionClosed): + read_http(fd) + + def test_trailers_too_long(self): + # Trailers after a chunk are ignored. + data = ( + b'POST /a HTTP/1.1\r\n' + b'Host: localhost\r\n' + b'Connection: keep-alive\r\n' + b'Transfer-Encoding: chunked\r\n' + b'\r\n' + b'2\r\noh\r\n' + b'4\r\n hai\r\n' + b'0\r\n' # last-chunk + # Normally the final CRLF would go here, but if you put in a + # trailer, it doesn't. + b'trailer2: value2' # note lack of \r\n + ) + data += b't' * pywsgi.MAX_REQUEST_LINE + # No termination, because we detect the trailer as being too + # long and abort the connection. + with self.makefile() as fd: + fd.write(data) + read_http(fd, body='oh hai') + with self.assertRaises(ConnectionClosed): + read_http(fd, body='oh bye') + + def test_trailers_request_smuggling_missing_last_chunk_keep_alive(self): + # When something that looks like a request line comes in the trailer + # as the first line, immediately after an invalid last chunk. + # We detect this and abort the connection, because the + # whitespace in the GET line isn't a legal part of a trailer. + # If we didn't abort the connection, then, because we specified + # keep-alive, the server would be hanging around waiting for more input. + data = ( + b'POST /a HTTP/1.1\r\n' + b'Host: localhost\r\n' + b'Connection: keep-alive\r\n' + b'Transfer-Encoding: chunked\r\n' + b'\r\n' + b'2\r\noh\r\n' + b'4\r\n hai\r\n' + b'0' # last-chunk, but missing the \r\n + # Normally the final CRLF would go here, but if you put in a + # trailer, it doesn't. + # b'\r\n' + b'GET /path2?a=:123 HTTP/1.1\r\n' + b'Host: a.com\r\n' + b'Connection: close\r\n' + b'\r\n' + ) + with self.makefile() as fd: + fd.write(data) + read_http(fd, body='oh hai') + with self.assertRaises(ConnectionClosed): + read_http(fd) + + self.assertEqual(self.calls, 1) + + def test_trailers_request_smuggling_header_first(self): + # When something that looks like a header comes in the first line. + data = ( + b'POST /a HTTP/1.1\r\n' + b'Host: localhost\r\n' + b'Connection: keep-alive\r\n' + b'Transfer-Encoding: chunked\r\n' + b'\r\n' + b'2\r\noh\r\n' + b'4\r\n hai\r\n' + b'0\r\n' # last-chunk, but only one CRLF + b'Header: value\r\n' + b'GET /path2?a=:123 HTTP/1.1\r\n' + b'Host: a.com\r\n' + b'Connection: close\r\n' + b'\r\n' + ) + with self.makefile() as fd: + fd.write(data) + read_http(fd, body='oh hai') + with self.assertRaises(ConnectionClosed): + read_http(fd, code=400) + + self.assertEqual(self.calls, 1) + + def test_trailers_request_smuggling_request_terminates_then_header(self): + data = ( + b'POST /a HTTP/1.1\r\n' + b'Host: localhost\r\n' + b'Connection: keep-alive\r\n' + b'Transfer-Encoding: chunked\r\n' + b'\r\n' + b'2\r\noh\r\n' + b'4\r\n hai\r\n' + b'0\r\n' # last-chunk + b'\r\n' + b'Header: value' + b'GET /path2?a=:123 HTTP/1.1\r\n' + b'Host: a.com\r\n' + b'Connection: close\r\n' + b'\r\n' + ) + with self.makefile() as fd: + fd.write(data) + read_http(fd, body='oh hai') + read_http(fd, code=400) + + self.assertEqual(self.calls, 1) + class TestUseWrite(TestCase): diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/tests/test__select.py new/gevent-23.9.0/src/gevent/tests/test__select.py --- old/gevent-23.7.0/src/gevent/tests/test__select.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/tests/test__select.py 2023-09-02 01:22:30.000000000 +0200 @@ -106,6 +106,17 @@ finally: sock.close() + def test_iterable(self): + sock = socket.socket() + + def fileno_iter(): + yield int(sock.fileno()) + + try: + select.select(fileno_iter(), [], [], 0.001) + finally: + sock.close() + def test_string(self): self.switch_expected = False self.assertRaises(TypeError, select.select, ['hello'], [], [], 0.001) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/tests/test__util.py new/gevent-23.9.0/src/gevent/tests/test__util.py --- old/gevent-23.7.0/src/gevent/tests/test__util.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/tests/test__util.py 2023-09-02 01:22:30.000000000 +0200 @@ -51,6 +51,7 @@ io = NativeStrIO() g = gevent.spawn(util.print_run_info, file=io) g.join() + return io.getvalue() g = gevent.spawn(root) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/gevent-23.7.0/src/gevent/util.py new/gevent-23.9.0/src/gevent/util.py --- old/gevent-23.7.0/src/gevent/util.py 2023-07-11 17:34:08.000000000 +0200 +++ new/gevent-23.9.0/src/gevent/util.py 2023-09-02 01:22:30.000000000 +0200 @@ -371,46 +371,17 @@ def __str__(self): return self.format(False) - @staticmethod - def __render_tb(tree, label, frame, limit): + # Prior to greenlet 3.0rc1, getting tracebacks of inactive + # greenlets could crash on Python 3.12. So we added a version-based + # setting here to disable it. That's now fixed, but leave the + # hook just in case. + _SUPPORTS_TRACEBACK = True + + @classmethod + def __render_tb(cls, tree, label, frame, limit): tree.child_data(label) - # XXX: Issues with tblib? Seen with tblib 1.3 and 2.0. - # More likely, it's something wrong in greenlet and the way it's - # keeping track of the frames? - # - # In a test like this: - # - # g = gevent.spawn(util.print_run_info, file=io) - # g.join() (test__util.py, line 53) - # - # 3.12b3 is crashing walking the stack on macOS: - # It's a simple segfault on line 340, ``f = f.f_back``. - # I have confirmed that the object is a real frame object. - # - # It seems unlikely to be a greenlet thing though, because the frame we're - # crashing on is the root frame: - # - # <frame at 0x.., file '/gevent/tests/test__util.py', line 53, code root> - # - # Interestingly, we see the test case dump the stack of the greenlet (successfully), - # then dump the stack of the main thread (successfully) --- this ends in line 53 --, - # and then get _another_ frame for line 53, and this is where it crashes. - # The difference? The successful dump does not list it as a root frame, - # where the failed one does. - # - # - # on Linux CI (not sure what frame), it is failing with a nice attribute error - # (which watches where the macOS is failing, inside a call to - # Py_GetAttr): - # - # File "//python3.12/traceback.py", line 339, in walk_stack - # yield f, f.f_lineno - # AttributeError: 'dict' object has no attribute 'f_lineno' - # - # A workaround on macOS is to not dump the root frame, but that only fixes - # test__util. test__threadpool:test_greenlet_class crashes similarly, but - # not 100% of the time. - if sys.version_info != (3, 12, 0, 'beta', 3): + + if cls._SUPPORTS_TRACEBACK: tb = ''.join(traceback.format_stack(frame, limit)) else: tb = ''
