[issue42471] Slowdown in socket tests noticed since Py3.8
Bert JW Regeer added the comment: Ned: You are correct, I hadn't even considered that as a potential issue. Added a context to grab the fork multiprocessing context and we are back to the speed it was before. This slowdown is pretty huge for just changing the way the process is forked. -- ___ Python tracker <https://bugs.python.org/issue42471> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42471] Slowdown in socket tests noticed since Py3.8
New submission from Bert JW Regeer : I am being fairly vague here, but it is mainly because I don't know the best way to reduce the test cases down to pinpoint the problem. I maintain Waitress, a pure Python HTTP server, and ever since Python 3.8 there's been a marked slowdown of the functional tests. Python 3.6 on my MacBook Pro M1 Rosetta: 15.91s to run the entire test suite, similar slow tests as Py37. So not included. Python 3.7 on my MacBook Pro M1 Rosetta: 15.57s to run the entire test suite Slowest tests on Py37 is: 3.10s call tests/test_functional.py::SleepyThreadTests::test_it 1.01s call tests/test_channel.py::TestHTTPChannelLookahead::test_client_disconnect 0.41s call tests/test_wasyncore.py::DispatcherWithSendTests::test_send 0.17s call tests/test_wasyncore.py::TestAPI_UseIPv6Poll::test_handle_close_after_conn_broken 0.11s call tests/test_wasyncore.py::TestAPI_UseIPv4Select::test_handle_read 0.11s call tests/test_wasyncore.py::TestAPI_UseIPv4Select::test_handle_close_after_conn_broken 0.11s call tests/test_wasyncore.py::TestAPI_UseIPv4Poll::test_handle_close_after_conn_broken 0.11s call tests/test_wasyncore.py::TestAPI_UseUnixSocketsPoll::test_handle_close_after_conn_broken 0.11s call tests/test_wasyncore.py::TestAPI_UseIPv6Select::test_handle_close_after_conn_broken 0.11s call tests/test_wasyncore.py::TestAPI_UseIPv4Select::test_handle_close Python 3.8 on my MacBook Pro M1 Rosetta: 44.33 seconds to run the entire test suite 3.34s call tests/test_functional.py::SleepyThreadTests::test_it 1.01s call tests/test_channel.py::TestHTTPChannelLookahead::test_client_disconnect 0.41s call tests/test_wasyncore.py::DispatcherWithSendTests::test_send 0.40s call tests/test_functional.py::TcpInternalServerErrorTests::test_before_start_response_http_11_close 0.36s call tests/test_functional.py::TcpInternalServerErrorTests::test_before_start_response_http_11 0.35s call tests/test_functional.py::UnixWriteCallbackTests::test_no_content_length 0.34s call tests/test_functional.py::TcpInternalServerErrorTests::test_after_write_cb 0.33s call tests/test_functional.py::TcpInternalServerErrorTests::test_after_start_response_http11_close 0.32s call tests/test_functional.py::TcpFileWrapperTests::test_filelike_nocl_http10 0.32s call tests/test_functional.py::TcpFileWrapperTests::test_filelike_http10 Python 3.9 on my MacBook Pro M1 Rosetta: 41.93 seconds 3.28s call tests/test_functional.py::SleepyThreadTests::test_it 1.01s call tests/test_channel.py::TestHTTPChannelLookahead::test_client_disconnect 0.41s call tests/test_wasyncore.py::DispatcherWithSendTests::test_send 0.36s call tests/test_functional.py::TcpWriteCallbackTests::test_short_body 0.33s call tests/test_functional.py::TcpFileWrapperTests::test_notfilelike_nocl_http10 0.32s call tests/test_functional.py::TcpEchoTests::test_chunking_request_without_content 0.32s call tests/test_functional.py::TcpEchoTests::test_date_and_server 0.31s call tests/test_functional.py::TcpBadContentLengthTests::test_long_body 0.30s call tests/test_functional.py::TcpEchoTests::test_broken_chunked_encoding_missing_chunk_end 0.30s call tests/test_functional.py::TcpTooLargeTests::test_request_body_too_large_with_wrong_cl_http10 This is very visible too in the test times on Github actions: https://github.com/Pylons/waitress/runs/1335343659?check_suite_focus=true (Python 3.7, running tox takes 40 seconds) https://github.com/Pylons/waitress/runs/1335343690?check_suite_focus=true (Python 3.8, running tox takes 3m and 42seconds) Although GitHub actions are not always reliable since it is shared infrastructure and you never know who your neighbors are, but the same issue holds across all of the Mac devices I own (2017 MacBook Pro i7, 2012 Mac Mini i7). Just as an extra data point, my 2017 MacBook Pro runs the py39 tests in 67 seconds, and the py37 tests in 26 seconds. The problem is that I don't know where to begin attempting figure out what is going on. So I am shooting my shot here in the bug tracker. How can I help debug this/track this down so that we can get rid of the regression and speed things back up? Open source project: https://github.com/Pylons/waitress Uses pytest to run tests, tox is the easiest way to accomplish this. -- messages: 381891 nosy: X-Istence priority: normal severity: normal status: open title: Slowdown in socket tests noticed since Py3.8 versions: Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue42471> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30717] Add unicode grapheme cluster break algorithm
Change by Bert JW Regeer : -- nosy: +Bert JW Regeer ___ Python tracker <https://bugs.python.org/issue30717> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27777] cgi.FieldStorage can't parse simple body with Content-Length and no Content-Disposition
Bert JW Regeer added the comment: I'll take a look and see if I can get the other fixes from WebOb and add them to a patch, and create a follow-up PR. If I can stop carrying a monkey patch for the standard library I am all for it! Thanks for running with this! -- nosy: +Bert JW Regeer ___ Python tracker <https://bugs.python.org/issue2> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4963] mimetypes.guess_extension result changes after mimetypes.init()
Change by Bert JW Regeer <ber...@regeer.org>: -- nosy: +X-Istence ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue4963> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32097] doctest does not consider \r\n a
New submission from Bert JW Regeer <ber...@regeer.org>: doctest fails to consider `\r\n` as a blank line. -- components: Library (Lib) files: test.py messages: 306595 nosy: X-Istence priority: normal severity: normal status: open title: doctest does not consider \r\n a versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 Added file: https://bugs.python.org/file47280/test.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32097> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27777] cgi.FieldStorage can't parse simple body with Content-Length and no Content-Disposition
Bert JW Regeer added the comment: Unfortunately I need to spin another patch, the one I created didn't solve the issue for one of WebOb's users: https://github.com/Pylons/webob/pull/300 (Thanks Julien Meyer!) I have his permission to grab his test/patch and update this patch, I will get this done later today. That being said, this is a real issue, and WebOb will be shipping a fix for Python less than 3.6 anyway, so whether this gets released in 3.6 or not doesn't matter to me. I'd prefer this to be fixed in the standard library for all users, rather than just for WebOb users. Even if this were released for 3.6.1, WebOb will have to carry the fix for the foreseeable future. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue2> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24764] cgi.FieldStorage can't parse multipart part headers with Content-Length and no filename in Content-Disposition
Bert JW Regeer added the comment: I've uploaded a patchset to bug #2 that fixes this issue by fixing make_file, and doesn't cause Python to throw out the content-length information. It also fixes FieldStorage for when you provide it a non-multipart form submission and there is no list in FS. Please see http://bugs.python.org/issue2 -- nosy: +X-Istence ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24764> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27777] cgi.FieldStorage can't parse simple body with Content-Length and no Content-Disposition
Bert JW Regeer added the comment: @berker.peksag: Attached is a patch with a test case that exercises this issue. Code path is that read_single() checks if the length is greater than 0, and then it reads binary, otherwise it reads it as a single line. This fixes make_file so that if self.length is greater than or equal to 0, it opens the file in binary mode, this matches the checks that write stuff as binary. This also undoes the change that was made in https://bugs.python.org/issue24764. Fixing this issue fixed that one as well, and arguably throwing data away doesn't seem like a good idea. -- keywords: +patch Added file: http://bugs.python.org/file45527/bug2.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue2> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28719] zipfile increase in size
Bert JW Regeer added the comment: Here's a dump from Python 3.6: b'PK\x03\x04\x14\x00\x08\x00\x00\x00\xc0~pI\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00zinfo_or_arcnamefoo!es\x8c\x03\x00\x00\x00\x03\x00\x00\x00PK\x01\x02\x14\x03\x14\x00\x08\x00\x00\x00\xc0~pI!es\x8c\x03\x00\x00\x00\x03\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\x01\x00\x00\x00\x00zinfo_or_arcnamePK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x00>\x00\x00\x00=\x00\x00\x00\x00\x00' You are correct that ResponseBodyFile does not have a seek() method and is not seekable. Adding seek() to ResponseBodyFile might be a little more complicated... -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28719> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28719] zipfile increase in size
Bert JW Regeer added the comment:
It's literally the string written:
writer.writestr('zinfo_or_arcname', b'foo')
rbo in this case is a simple file like object.
I can get dumps from Python 3.5 and Python 3.6 if necessary.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28719>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28719] zipfile increase in size
New submission from Bert JW Regeer:
I am the current maintainer of WebOb, and noticed that on Python 3.6 and 3.7 I
noticed that a test started failing.
Granted, the test is checking the size of the file created and it is not the
brightest idea in a test, but it's been stable since Python 2.5...
https://travis-ci.org/Pylons/webob/jobs/176505096#L224
shows the failure.
_ test_response_file_body_tell _
def test_response_file_body_tell():
import zipfile
from webob.response import ResponseBodyFile
rbo = ResponseBodyFile(Response())
assert rbo.tell() == 0
writer = zipfile.ZipFile(rbo, 'w')
writer.writestr('zinfo_or_arcname', b'foo')
writer.close()
> assert rbo.tell() == 133
E assert 145 == 133
E+ where 145 = >>()
E+where >> = >.tell
tests/test_response.py:608: AssertionError
I am not sure that this is necessarily a bug, but it would be good to know why
files are no longer generated the same way.
--
messages: 280992
nosy: X-Istence
priority: normal
severity: normal
status: open
title: zipfile increase in size
type: resource usage
versions: Python 3.6, Python 3.7
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28719>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27777] cgi.FieldStorage can't parse simple body with Content-Length and no Content-Disposition
Bert JW Regeer added the comment: Updated versions this applies to. -- versions: +Python 3.3, Python 3.4, Python 3.6, Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue2> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27777] cgi.FieldStorage can't parse simple body with Content-Length and no Content-Disposition
Changes by Bert JW Regeer <ber...@regeer.org>: -- nosy: +berker.peksag ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue2> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27777] cgi.FieldStorage can't parse simple body with Content-Length and no Content-Disposition
Bert JW Regeer added the comment: On line #890 in self.make_file() the check for _binary_file should be changed to also check for self.length >= 0. https://github.com/python/cpython/blob/3.4/Lib/cgi.py#L890 becomes: if self._binary_file or self.length >= 0: _binary_file is only ever set if there is a content disposition, which there is not in the test case provided. In the case of no content disposition we can use the content-length as a hint that we have a file that has been uploaded. All files uploaded should be treated as binary if they are not a text type. This is a duplicate of #27308, however the patch in that report is incorrect IMHO. -- nosy: +X-Istence, haypo ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue2> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27308] Inconsistency in cgi.FieldStorage() causes unicode/byte TypeError.
Bert JW Regeer added the comment: This is not a duplicate of https://bugs.python.org/issue24764 -- nosy: +X-Istence ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21878] wsgi.simple_server's wsgi.input read/readline waits forever in certain circumstances
Bert JW Regeer added the comment: This is still very much an issue, and makes it more difficult to write generic python request/response libraries because we can't assume that a read() will return, and relying on the Content-Length being set is not always possible unfortunately. -- nosy: +X-Istence versions: +Python 2.7, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue21878> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25379] Changes in traceback broke existing code (Python 3.5)
New submission from Bert JW Regeer: One of the changes in Python 3.5's traceback functionality broke existing code compared to Python 3.4 by injecting an extra stack frame into the list when using traceback.extract_stack: What this looks like on Python 3.5: pyramid/tests/test_config/test_init.py 1108 includeme2 config.add_view(view2) pyramid/util.py 526 wrapper return wrapped(self, *arg, **defaults) pyramid/util.py 557 wrapper f = traceback.extract_stack(limit=4) /Users/xistence/.pyenv/versions/3.5.0/lib/python3.5/traceback.py 201 extract_stack stack = StackSummary.extract(walk_stack(f), limit=limit) What it used to look like on Python 3.4: pyramid/config/__init__.py 798 include c(configurator) pyramid/tests/test_config/test_init.py 1108 includeme2 config.add_view(view2) pyramid/util.py 526 wrapper return wrapped(self, *arg, **defaults) pyramid/util.py 557 wrapper f = traceback.extract_stack(limit=4) Notice that in the Python 3.5 version: /Users/xistence/.pyenv/versions/3.5.0/lib/python3.5/traceback.py 201 extract_stack stack = StackSummary.extract(walk_stack(f), limit=limit) Is added. We are tracking this issue here: https://github.com/Pylons/pyramid/issues/1973 on our side. -- messages: 252841 nosy: X-Istence priority: normal severity: normal status: open title: Changes in traceback broke existing code (Python 3.5) type: behavior versions: Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25379> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25379] Changes in traceback broke existing code (Python 3.5)
Bert JW Regeer added the comment: Looks like this is a dup of: https://bugs.python.org/issue25108 -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25379> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10910] pyport.h FreeBSD/Mac OS X fix causes errors in C++ compilation
Bert JW Regeer ber...@regeer.org added the comment: In my first comment on this bug post I posted what project has issues with this, Botan with Boost.Python on FreeBSD and Mac OS X. If required I will post how to reproduce this error using that project. If you would prefer a simplified test case, unfortunately I don't have the time to dig into that and find one. One way to solve it is by including the Python header before any other file, however this is not an option for most 3rd party software that never sees this bug because those functions are not redefined for Linux... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10910 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10910] pyport.h FreeBSD/Mac OS X fix causes errors in C++ compilation
New submission from Bert JW Regeer ber...@regeer.org: I was recently attempting to get Botan (http://botan.randombit.net) working with Python 2.6.6 on FreeBSD when it failed to compile, I filled a bug with Botan (http://bugs.randombit.net/show_bug.cgi?id=135) and first thought it was a compiler issue, and then thought it might be a Boost.Python issue. However after further digging, see comment 3 (http://bugs.randombit.net/show_bug.cgi?id=135#c3) I figured out it had to do with the fix that went in to pyport.h for issues with isspace/toupper/et al. on FreeBSD's libc and also Mac OS X. As can be seen in http://svn.python.org/view/python/trunk/Include/pyport.h?r1=36519r2=36793 a change was added to override the original definitions with a new one that used the wide character support. It was modified again in http://svn.python.org/view/python/trunk/Include/pyport.h?r1=77585r2=80178 to also include Mac OS X for http://bugs.python.org/issue7072. The issue here is that if this file is included in any C++ code that then also includes locale or any of other functions that might pull in localefwd.h the defines will cause actual C++ code functions to be overwritten with invalid data, and the C++ compiler will throw errors such as: /usr/include/c++/4.2/bits/localefwd.h:58:34: error: macro isspace passed 2 arguments, but takes just 1 Putting an #if 0, #endif around that block of code allows Botan's python module to cleanly compile without issues. I do apologise that I don't have a simple C++ program that reproduces the problem. -- assignee: ronaldoussoren components: Build, Extension Modules, Library (Lib), Macintosh, Unicode, ctypes messages: 126299 nosy: X-Istence, ronaldoussoren priority: normal severity: normal status: open title: pyport.h FreeBSD/Mac OS X fix causes errors in C++ compilation type: compile error versions: Python 2.6, Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10910 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
