[issue47123] ZipFile.writestr should respect SOURCE_DATE_EPOCH
New submission from ghost43 :
Currently `ZipFile.writestr` writes the local time into the ZipFile.
(depends on both current time and local timezone)
See
https://github.com/python/cpython/blob/20e6e5636a06fe5e1472062918d0a302d82a71c3/Lib/zipfile.py#L1816-L1817
This makes pip installing a package generate non-reproducible build artifacts.
Specifically, `Scripts/*.exe` files (created for packages that define
entry_points/console_scripts) are not reproducible on Windows when installed by
pip. This also leaks into the `*.dist-info/RECORD` files.
For example, after running `pip install wheel` or `pip install pyinstaller`,
in `wheel-0.37.1.dist-info/RECORD`, I have this line:
```
../../Scripts/wheel.exe,sha256=u9TbPw2XNs_F9uy7y2zwumuzAZDbOSB7BXjLHZ0tTHg,97103
```
in `pyinstaller-4.10.dist-info/RECORD`, I have these lines:
```
../../Scripts/pyi-archive_viewer.exe,sha256=nC-9muPlIhUC1qvFkXHpyKJyRQqXISXxbUPXQ1XVOiM,97133
../../Scripts/pyi-bindepend.exe,sha256=udFHiAdndPpSwaIqmhmLEy36IUs1cNNoNQznSEnLJQQ,97128
../../Scripts/pyi-grab_version.exe,sha256=3ET9E841tFWujFL99aG4frzgwlP9f9pAkMgE0k2UGK0,97131
../../Scripts/pyi-makespec.exe,sha256=dJkfmITdLJhyPngmqziqqj5tH9qqfeQc5BTubeoXWUs,97127
../../Scripts/pyi-set_version.exe,sha256=sWmcOVS93fUY-wbdoz6ixBCvjy1tC4Aaw30DMmrmo-0,97130
../../Scripts/pyinstaller.exe,sha256=haInbhH0pImJn24cW4v917oUZmzXZj8OE89KFh4MO2Y,97112
```
Upon comparing multiple `Scripts/wheel.exe` files, I've found that the only
difference is due to the above-mentioned timestamp embedded inside the exe (or
rather, same timestamp embedded twice).
The `exe` files get created by `distlib` (vendored by pip).
Here is a traceback with an artificial exception to illustrate the codepath:
```
(env) PS C:\tmp> pip install --no-build-isolation pyinstaller
Collecting pyinstaller
Using cached pyinstaller-4.10-py3-none-win_amd64.whl (2.0 MB)
Requirement already satisfied: setuptools in c:\tmp\env\lib\site-packages (from
pyinstaller) (61.0.0)
Requirement already satisfied: pyinstaller-hooks-contrib>=2020.6 in
c:\tmp\env\lib\site-packages (from pyinstaller) (2022.3)
Requirement already satisfied: altgraph in c:\tmp\env\lib\site-packages (from
pyinstaller) (0.17.2)
Requirement already satisfied: pefile>=2017.8.1 in c:\tmp\env\lib\site-packages
(from pyinstaller) (2021.9.3)
Requirement already satisfied: pywin32-ctypes>=0.2.0 in
c:\tmp\env\lib\site-packages (from pyinstaller) (0.2.0)
Requirement already satisfied: future in c:\tmp\env\lib\site-packages (from
pefile>=2017.8.1->pyinstaller) (0.18.2)
Installing collected packages: pyinstaller
ERROR: Exception:
Traceback (most recent call last):
File "C:\tmp\env\lib\site-packages\pip\_internal\cli\base_command.py", line
167, in exc_logging_wrapper
status = run_func(*args)
File "C:\tmp\env\lib\site-packages\pip\_internal\cli\req_command.py", line
205, in wrapper
return func(self, options, args)
File "C:\tmp\env\lib\site-packages\pip\_internal\commands\install.py", line
405, in run
installed = install_given_reqs(
File "C:\tmp\env\lib\site-packages\pip\_internal\req\__init__.py", line 73,
in install_given_reqs
requirement.install(
File "C:\tmp\env\lib\site-packages\pip\_internal\req\req_install.py", line
769, in install
install_wheel(
File
"C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line
729, in install_wheel
_install_wheel(
File
"C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line
646, in _install_wheel
generated_console_scripts = maker.make_multiple(scripts_to_generate)
File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 440,
in make_multiple
filenames.extend(self.make(specification, options))
File
"C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line
427, in make
return super().make(specification, options)
File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 429,
in make
self._make_script(entry, filenames, options=options)
File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 329,
in _make_script
self._write_script(scriptnames, shebang, script, filenames, ext)
File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 263,
in _write_script
raise Exception(f"heyheyhey2. {sha256(launcher)=}. {sha256(shebang)=}.
{sha256(zip_data)=}. " +
Exception: heyheyhey2. sha256(launcher)='a00a877acefc'.
sha256(shebang)='58628e924f22'. sha256(zip_data)='a423496a0482'.
('SOURCE_DATE_EPOCH' in os.environ)=True
```
The interesting code is here:
https://github.com/pypa/distlib/blob/d0e3f49df5d1aeb9daeaaabf0391c9e13e4a6562/distlib/scripts.py#L251-L252
This calls into the cpython standard library, where `time.time()` gets written
into the file:
https://github.com/python/cpython/b
[issue44036] asyncio SSL server can be DOSed, event loop gets blocked: busy loops and uses 100% CPU
New submission from ghost43 : This is about a potential DOS vector that can get an asyncio server serving SSL connections to enter a busy loop and hang. To recover the server (python process) needs to be restarted. See downstream report at https://github.com/spesmilo/electrumx/issues/92 ElectrumX is server software that serves JSON-RPC requests either directly over plaintext TCP or over SSL/TLS (no HTTP involved). The server code is written using asyncio. The most popular client is also written using python+asyncio. However, there are other server implementations (e.g. in Rust), and other clients (e.g. in javascript, scala, java). The servers are part of a federated network and peer with each other. In the last 1+ year, server operators have started reporting that the python process (of ElectrumX) sporadically hangs: it uses 100% CPU and stops serving clients or accepting new connections. The only way we could find to recover is restarting the server (the python process); waiting for even days did not help, python was stuck. The hang seemed to mostly happen on busy servers that serve several thousands of concurrent sessions, and even on those only happened e.g. once a month. So the hang was difficult to reproduce. Nevertheless we observed that the hang only happens if it is the asyncio server that handles SSL, i.e. if the server operator put nginx in front of the python process and handles SSL termination in nginx, they would be unaffected. One server operator whose server at one point hanged multiple times a day confirmed this, and reported that nginx subsequently started logging lines like this: ``` 2021/01/11 02:28:30 [crit] 21#21: *437620 SSL_shutdown() failed (SSL: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify) while proxying connection, client: REDACTED, server: 0.0.0.0:50002, upstream: "127.0.0.1:50213", bytes from/to client:81/205, bytes from/to upstream:205/0 ``` Over these last months, the hang has been reproduced on many different python versions by different people, e.g. 3.7.1, 3.7.5, 3.8.5, 3.9.1, 3.9.4. A few days ago, many servers hanged almost simultaneously, and when restarted, they would soon hang again at most a few hours later. Presumably someone either flooded the network with buggy clients, or it might have been an attack. Anyway, this allowed me to look into this better. I set up gdb and waited. This was on ubuntu 20.04 with python 3.8.5 and openssl=1.1.1f-1ubuntu2.3. It seems the event loop thread is stuck in a busy loop. The deepest common stack frame looks to be at https://github.com/python/cpython/blob/v3.8.5/Lib/asyncio/sslproto.py#L675 ``` (gdb) py-bt Traceback (most recent call first): File "/usr/lib/python3.8/asyncio/sslproto.py", line 535, in feed_appdata File "/usr/lib/python3.8/asyncio/sslproto.py", line 675, in _process_write_backlog ssldata, offset = self._sslpipe.feed_appdata(data, offset) File "/usr/lib/python3.8/asyncio/sslproto.py", line 599, in _write_appdata self._process_write_backlog() File "/usr/lib/python3.8/asyncio/sslproto.py", line 387, in write self._ssl_protocol._write_appdata(data) File "/usr/local/lib/python3.8/dist-packages/aiorpcx/rawsocket.py", line 118, in write self._asyncio_transport.write(framed_message) File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 153, in _send_message await self.transport.write(message) File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 496, in _throttled_request await self._send_message(message) File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run self._context.run(self._callback, *self._args) File "/usr/lib/python3.8/asyncio/base_events.py", line 2627, in _run_once --Type for more, q to quit, c to continue without paging-- File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever None, getaddr_func, host, port, family, type, proto, flags) File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete self.run_forever() File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run File "/usr/local/bin/electrumx_server", line 35, in main asyncio.run(controller.run()) File "/usr/local/bin/electrumx_server", line 43, in main() ``` ``` (gdb) py-list 530except (SystemExit, KeyboardInterrupt): 531raise 532except BaseException as e: 533self._fatal_error(e, 'SSL error in data received') 534return >535 536for chunk in ssldata: 537self._transport.write(chunk) 538 539for chunk in appdata: 540if chunk: ``` ``` (gdb) py-locals self = <_SSLPipe(_context=, _server_side=True, _server_hostname=None, _sta
[issue40963] distutils make_zipfile uses random order
New submission from ghost43 : I am trying to generate .zip sdists for a project in a reproducible manner, using setuptoools. The generated zips differ in the order of packed files. The root cause of the non-determinicity is using os.walk() in make_zipfile here: https://github.com/python/cpython/blob/0d3350daa8123a3e16d4a534b6e873eb12c10d7c/Lib/distutils/archive_util.py#L174 For a potential fix, see https://github.com/pypa/setuptools/commit/29688821b381268a0d59c0d26317d88ad518f966 I guess https://bugs.python.org/issue30693 is sort of related. The change made there is necessary, and was sufficient to make the tars reproducible but not the zips. (sidenote: Is it acceptable to sign the PSF CLA with a pseudonym?) -- components: Distutils messages: 371400 nosy: dstufft, eric.araujo, ghost43 priority: normal severity: normal status: open title: distutils make_zipfile uses random order type: behavior versions: Python 3.10, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue40963> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
