[issue28147] Unbounded memory growth resizing split-table dicts

Xiang Zhang added the comment: Ahh, I see. > If there are any integer a such as ESTIMATE_SIZE(a) == n and n == 2**m and > USABLE_FRACTION(n) == a - 1. There are, such as 11, 43... > a items cannot be inserted into dict after dictresize(d, ESTIMATE_SIZE(a)) It can but needs another resize in

[issue28197] range.index mismatch with documentation

Vedran Čačić added the comment: Yes, I agree these are useless _if you know you're dealing with range_. However, if you have a Sequence, it would be very useful not to have range be a special case. Of course, one solution is to have a default .index implementation in the Sequence ABC itself,

[issue28202] Python 3.5.1 C API, the global available available is not destroyed when delete the module

New submission from Jack Liu: 0 down vote favorite I have a app loading python35.dll. Use python API PyImport_AddModule to run a py file. And use PyDict_DelItemString to delete the module. There is a global vailable in the py file. The global variable is not destroyed when calling

[issue28199] Compact dict resizing is doing too much work

Serhiy Storchaka added the comment: > (After OrderedDict implementation is improved, functools.lru_cache can use > OrderedDict and remove doubly linked list too.) functools.lru_cache can use just ordered dict. But simple implementation is 1.5 times slower. I'm working on this. I think that

[issue28201] dict: perturb shift should be done when first conflict

Raymond Hettinger added the comment: I agree and my opinion counts even more because I long ago made this change for setobjects ;-) -- ___ Python tracker

[issue28189] dictitems_contains swallows compare errors

Xiang Zhang added the comment: Thanks for the merge Raymond. :) -- stage: patch review -> resolved ___ Python tracker ___

[issue28189] dictitems_contains swallows compare errors

Raymond Hettinger added the comment: Thanks for the patch. -- resolution: -> fixed status: open -> closed ___ Python tracker ___

[issue28189] dictitems_contains swallows compare errors

Roundup Robot added the comment: New changeset 2a9e0e869ca7 by Raymond Hettinger in branch '3.5': Issue #28189: dictitems_contains no longer swallows compare errors. https://hg.python.org/cpython/rev/2a9e0e869ca7 -- nosy: +python-dev ___ Python

[issue28200] Windows: path_converter() leaks memory for Unicode filenames

Serhiy Storchaka added the comment: Deprecated functions and types of the C API --- The :c:type:`Py_UNICODE` has been deprecated by :pep:`393` and will be removed in Python 4. All functions using this type are deprecated: Unicode functions and methods

[issue28199] Compact dict resizing is doing too much work

INADA Naoki added the comment: > We can still clean this up for Python 3.6. We're in feature freeze, not > development freeze. Does it mean there is a chance to improve OrderedDict to use new dict implementation, if it seems safe enough? Is new implementation a feature? (After OrderedDict

[issue28201] dict: perturb shift should be done when first conflict

Tim Peters added the comment: Good catch! I agree - and I wrote this code to begin with, so my opinion should count ;-) -- nosy: +tim.peters ___ Python tracker

[issue28201] dict: perturb shift should be done when first conflict

Changes by Serhiy Storchaka : -- nosy: +rhettinger, serhiy.storchaka ___ Python tracker ___

[issue28201] dict: perturb shift should be done when first conflict

New submission from INADA Naoki: Current perturb shift code is like following: for (perturb = hash; ; perturb >>= PERTURB_SHIFT) { i = mask & ((i << 2) + i + perturb + 1); This loop is start after first conflict. It means perturb == hash for first conflict. The purpose of perturb

[issue28200] Windows: path_converter() leaks memory for Unicode filenames

Steve Dower added the comment: It's not clear to me that Py_UNICODE is guaranteed to be wchar_t for all time, that's all. If it is, go ahead. Otherwise the path_t object has the ability to clean up after itself, so perhaps it should be used here? --

[issue28199] Compact dict resizing is doing too much work

INADA Naoki added the comment: Current compact ordered dict implementation is bit different from yours. When there was removed item, there are NULL entries in ma_entries, instead of swapping last item and deleted item. It's important to keep insertion order. But it's easy to detect clean dict.

[issue28189] dictitems_contains swallows compare errors

Changes by Raymond Hettinger : -- assignee: -> rhettinger nosy: +rhettinger ___ Python tracker ___

[issue28197] range.index mismatch with documentation

Raymond Hettinger added the comment: > These broke some concrete Sequence implementations. Poor choice of words. The concrete implementations didn't change at all. Perhaps the concrete implementation need to be brought more in-sync with the ABC. That would be reasonable; afteralll the goal

[issue28193] Consider using lru_cache for the re.py caches

Roundup Robot added the comment: New changeset 88110cfbf4dc by Raymond Hettinger in branch '3.6': Issue #28193: Use lru_cache in the re module. https://hg.python.org/cpython/rev/88110cfbf4dc -- nosy: +python-dev ___ Python tracker

[issue28193] Consider using lru_cache for the re.py caches

Changes by Raymond Hettinger : -- resolution: -> fixed status: open -> closed ___ Python tracker ___

[issue28199] Compact dict resizing is doing too much work

Xiang Zhang added the comment: Then how about entries(key, value pairs)? The size of entries does not match the available hash slots. For example, an empty dict allocates a hash array with 5 available slots and 5 entries. Now we resize the hash array to size 16 and it can afford 10 entries

[issue28197] range.index mismatch with documentation

Vedran Čačić added the comment: Yes, that's the precise reason I caught this. I was implementing some tools to do with Sequences (seqtools, like itertools but having non-ephemeral output given non-ephemeral input), and the Sequence that was easiest to test quickly was range. In `positions`

[issue27961] remove support for platforms without "long long"

Roundup Robot added the comment: New changeset cf6e9968ebb7 by Benjamin Peterson in branch '3.6': always define HAVE_LONG_LONG (#27961) https://hg.python.org/cpython/rev/cf6e9968ebb7 -- ___ Python tracker

[issue28193] Consider using lru_cache for the re.py caches

Raymond Hettinger added the comment: I think I'll just take the low hanging fruit in _compile_repl and call it a day. -- ___ Python tracker ___

[issue28199] Compact dict resizing is doing too much work

Raymond Hettinger added the comment: Just before the re-insertion, we should also do a compaction-in-place for the keys/values/hashes array if it has a significant number of holes for previously deleted entries. -- ___ Python tracker

[issue28200] Windows: path_converter() leaks memory for Unicode filenames

STINNER Victor added the comment: I didn't push the fix myself, because Steve Dower maybe made the change for a specific reason? -- ___ Python tracker

[issue28195] test_huntrleaks_fd_leak fails on Windows

STINNER Victor added the comment: Ah, I found a leak: see issue #28200. -- ___ Python tracker ___ ___

[issue28200] Windows: path_converter() leaks memory for Unicode filenames

New submission from STINNER Victor: Memory leak spotted by the issue #28195: path_converter() calls PyUnicode_AsWideCharString() which allocates a new buffer at each call, but this buffer is never released. On Python 3.5, PyUnicode_AsWideCharString() was used which handles internally the

[issue28195] test_huntrleaks_fd_leak fails on Windows

STINNER Victor added the comment: I modified the test_regrtest unit test because the test is too fragile, it expects a line on a specific line number. But there is still a real memory leak detected by this unit test. -- ___ Python tracker

[issue28183] Clean up and speed up dict iteration

Raymond Hettinger added the comment: Moved dictresize() discussion to http://bugs.python.org/issue28199 -- ___ Python tracker ___

[issue28195] test_huntrleaks_fd_leak fails on Windows

Roundup Robot added the comment: New changeset 91285b24f1c2 by Victor Stinner in branch '3.6': Fix test_huntrleaks_fd_leak() of test_regrtest https://hg.python.org/cpython/rev/91285b24f1c2 -- nosy: +python-dev ___ Python tracker

[issue28199] Compact dict resizing is doing too much work

New submission from Raymond Hettinger: The dictresize() method unnecessarily copies the keys, values, and hashes as well as making insertions in the index table. Only the latter step is necessary or desirable. Here in the pure Python code for resizing taking from the original

[issue28198] heap-buffer-overflow in tok_nextc (Parser/tokenizer.c:954)

Changes by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___

[issue28183] Clean up and speed up dict iteration

Serhiy Storchaka added the comment: Raymond, I think this is different issue. -- ___ Python tracker ___ ___

[issue28193] Consider using lru_cache for the re.py caches

Serhiy Storchaka added the comment: Yes, raising an exception with a result as a payload is one option. Other option is to check a result. Something like: def _compile_is_valid(value): p, loc = value return loc is None or loc == _locale.setlocale(_locale.LC_CTYPE) def

[issue28198] heap-buffer-overflow in tok_nextc (Parser/tokenizer.c:954)

New submission from geeknik: As per the security team, I'm submitting this here as it doesn't appear exploitable. Fuzzing Python 2.7.12 with AFL, ASAN and libdislocator.so on Debian 8.5 x64. (This might also affect Python 3.5) The attached 1-byte testcase triggers a heap-buffer-overflow:

[issue27929] asyncio.AbstractEventLoop.sock_connect broken for AF_BLUETOOTH

Robert Jordens added the comment: It is still in cpython master e6e9ddd. import asyncio import socket sock = socket.socket(family=socket.AF_BLUETOOTH, type=socket.SOCK_STREAM, proto=socket.BTPROTO_RFCOMM) sock.setblocking(False) addr =

[issue28183] Clean up and speed up dict iteration

Raymond Hettinger added the comment: Here is a rough (very rough and not compileable) sketch of the direction that resizing should take: static void insert_indices_clean(PyDictObject *mp, Py_hash_t hash) { size_t i, perturb; PyDictKeysObject *k = mp->ma_keys; size_t mask =

[issue28182] Expose OpenSSL verification results in SSLError

Christian Heimes added the comment: You don't have to be concerned about additional arguments. fill_and_set_sslerror() is an internal helper function. In fact it's a helper function for two other helper functions. Let's postpone the discussion until the argument sizes grows out of proportion.

[issue28193] Consider using lru_cache for the re.py caches

Raymond Hettinger added the comment: Yes, I saw that. If a function could raise a NoCache exception, re._compile() could take advantage of it. But I don't feel good about going down that path (adding coupling between the caching decorator and the cached function). It would be better to

[issue28197] range.index mismatch with documentation

Serhiy Storchaka added the comment: Parameters start and stop were added to the Sequence.index() mixin method in issue23086. These broke some concrete Sequence implementations. -- nosy: +Devin Jeanpierre, rhettinger, serhiy.storchaka ___ Python

[issue28193] Consider using lru_cache for the re.py caches

Serhiy Storchaka added the comment: lru_cache can be used for re._compile() if add the ability to bypass the cache and to validate cached value. -- ___ Python tracker

[issue28197] range.index mismatch with documentation

Changes by SilentGhost : -- versions: +Python 3.7 ___ Python tracker ___ ___

[issue27961] remove support for platforms without "long long"

STINNER Victor added the comment: It seems fair to keep the define for backwrad compatibility. -- ___ Python tracker ___

[issue28151] testPythonOrg() of test_robotparser fails on validating python.org HTTPS certificate

STINNER Victor added the comment: Thank you Berker! I'm quite sure that the issue is very old, like 1 year old or more. And this fix is welcomed! -- ___ Python tracker

[issue28197] range.index mismatch with documentation

New submission from Vedran Čačić: Look at this: >>> from collections.abc import Sequence >>> help(Sequence.index) index(self, value, start=0, stop=None) S.index(value, [start, [stop]]) -> integer -- return first index of value. Raises ValueError if the value is not

[issue23591] enum: Add Flags and IntFlags

Roundup Robot added the comment: New changeset b56290a80ff7 by Ethan Furman in branch '3.6': issue23591: fix flag decomposition and repr https://hg.python.org/cpython/rev/b56290a80ff7 New changeset 7372c042e9a1 by Ethan Furman in branch 'default': issue23591: fix flag decomposition and repr

[issue28182] Expose OpenSSL verification results in SSLError

Chi Hsuan Yen added the comment: That looks much better. I should have create a subclass of SSLError. Here's a minor concern: fill_and_set_sslerror adds a new argument for verification errors. If someone else wants to support more errors, this function would have more arguments, which sounds

[issue27806] 2.7 32-bit builds fail on macOS 10.12 Sierra due to dependency on deleted header file QuickTime/QuickTime.h

Roundup Robot added the comment: New changeset a8771f230c06 by Ned Deily in branch '2.7': Issue #27806: add Aleks to Misc/ACKS. https://hg.python.org/cpython/rev/a8771f230c06 -- ___ Python tracker

[issue28182] Expose OpenSSL verification results in SSLError

Christian Heimes added the comment: Good work! I completely forgot that the SSL object holds the last verification error in its struct. This allows the ssl module to print some information when cert verification fails. It's still not perfect, because it is missing information about the the

[issue28183] Clean up and speed up dict iteration

Serhiy Storchaka added the comment: Actually most of optimization is not specific for new dict implementation. -- ___ Python tracker ___

[issue28183] Clean up and speed up dict iteration

Raymond Hettinger added the comment: We can still clean this up for Python 3.6. We're in feature freeze, not development freeze. The compact dict patch was very rough when it landed and is expected to continue to be polished so that the expected benefits are realized. -- versions:

[issue28183] Clean up and speed up dict iteration

Changes by Serhiy Storchaka : -- assignee: -> serhiy.storchaka versions: -Python 3.6 ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: xiang: dictresize(d, n) may choose keysize==n (when n == 2**m) with this patch. If there are any integer a such as ESTIMATE_SIZE(a) == n and n == 2**m and USABLE_FRACTION(n) == a - 1, a items cannot be inserted into dict after dictresize(d, ESTIMATE_SIZE(a))

[issue25400] robotparser doesn't return crawl delay for default entry

Changes by Berker Peksag : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue25400] robotparser doesn't return crawl delay for default entry

Roundup Robot added the comment: New changeset d5d910cfd288 by Berker Peksag in branch '3.6': Issue #25400: RobotFileParser now correctly returns default values for crawl_delay and request_rate https://hg.python.org/cpython/rev/d5d910cfd288 New changeset 911070065e38 by Berker Peksag in branch

[issue28183] Clean up and speed up dict iteration

Raymond Hettinger added the comment: > Raymond: With such implementation keys, values and hashes are all > organized together and there seems no _resize operation can only > adjust hashes without breaking the entire layout. None of those needs to change during a resize. Only indices array

[issue28182] Expose OpenSSL verification results in SSLError

Chi Hsuan Yen added the comment: Here's a quick try. I didn't add tests and update docs as it's my first serious patch to CPython and I'm not sure whether my approach is OK or not. -- keywords: +patch Added file: http://bugs.python.org/file44741/expose-x509-verify-result.patch

[issue25400] robotparser doesn't return crawl delay for default entry

Changes by Berker Peksag : Added file: http://bugs.python.org/file44740/issue25400_v3.diff ___ Python tracker ___

[issue25400] robotparser doesn't return crawl delay for default entry

Berker Peksag added the comment: Here's an updated patch. -- versions: +Python 3.7 Added file: http://bugs.python.org/file44739/issue25400_v2.diff ___ Python tracker

[issue24363] httplib fails to handle semivalid HTTP headers

R. David Murray added the comment: I will try to review this in the not too distant future. You can ping me if I don't get to it by next Saturday. I think I'll probably prefer to call the flag something like _greedy_header_parsing, to reflect a change from assuming we've got body on a

[issue28147] Unbounded memory growth resizing split-table dicts

Xiang Zhang added the comment: LGTM. But why this change? -#define ESTIMATE_SIZE(n) (((n)*3) >> 1) +#define ESTIMATE_SIZE(n) (((n)*3+1) >> 1) -- ___ Python tracker

[issue28194] Clean up some checks in dict implementation

Xiang Zhang added the comment: Apply methane's comment, preserving the comment. -- Added file: http://bugs.python.org/file44738/dict_clean_up_v2.patch ___ Python tracker

[issue28115] Use argparse for the zipfile module

SilentGhost added the comment: Neither of those fit under definition of "permissible" modules. So, I would then suggest to close this issue and any further issues against CLI of those modules. -- ___ Python tracker

[issue28194] Clean up some checks in dict implementation

Serhiy Storchaka added the comment: LGTM if address methane's comment. -- ___ Python tracker ___ ___

[issue28115] Use argparse for the zipfile module

Serhiy Storchaka added the comment: They should be added. Otherwise we can't know that the code works as expected. You can use test_calendar.py or test_tarfile.py as a guide. -- ___ Python tracker

[issue6087] distutils.sysconfig.get_python_lib gives surprising result when used with a Python build

Vinay Sajip added the comment: Updated patch to cover tests, and changed the implementation slightly. Tests pass on Linux and Windows (I'm not able to build on OS X). The patch is against the default branch but the diff should apply equally well to 3.6. -- Added file:

[issue28183] Clean up and speed up dict iteration

INADA Naoki added the comment: LGTM, thanks. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue26661] python fails to locate system libffi

Christian Heimes added the comment: Thanks Robert, I have been running into the same issue for a while. -- resolution: -> fixed stage: -> resolved status: open -> closed ___ Python tracker

[issue26661] python fails to locate system libffi

Roundup Robot added the comment: New changeset c4cec8f7c727 by Christian Heimes in branch '3.5': Issue #26661: setup.py now detects system libffi with multiarch wrapper. https://hg.python.org/cpython/rev/c4cec8f7c727 New changeset 18825546acbc by Christian Heimes in branch '3.6': Issue #26661:

[issue28176] Fix callbacks race in asyncio.SelectorLoop.sock_connect

Berker Peksag added the comment: For some reason, some tests fail randomly on FreeBSD buildbots. See also issue27784. I'm planning to apply the attached patch. -- keywords: +patch nosy: +koobs Added file: http://bugs.python.org/file44736/issue28176_skip_tests.diff

[issue26661] python fails to locate system libffi

Christian Heimes added the comment: This affects Python 3.7 because our own copy of libffi was removed from 'default'. -- assignee: -> christian.heimes nosy: +christian.heimes priority: normal -> high versions: +Python 2.7, Python 3.5, Python 3.6, Python 3.7

[issue26610] test_venv.test_with_pip() fails when ctypes is missing

Roundup Robot added the comment: New changeset 82bd70f83796 by Berker Peksag in branch '3.5': Issue #26610: Skip test_with_pip if _ctypes is not available in OpenIndiana https://hg.python.org/cpython/rev/82bd70f83796 -- ___ Python tracker

[issue28195] test_huntrleaks_fd_leak fails on Windows

Berker Peksag added the comment: It looks like http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/11581 is first time that the test was failed. Revisions (without documentation changes) in that build: * 378e000a68785fdb3d74b3a930bc4ac5f6a04ce5 *

[issue28196] ssl.match_hostname() should check for SRV-ID and URI-ID

Changes by Christian Heimes : -- dependencies: +Support RFC4985 SRVName in SAN name ___ Python tracker ___

[issue28196] ssl.match_hostname() should check for SRV-ID and URI-ID

New submission from Christian Heimes: The ssl.match_hostname() function does not conform to RFC 6125 because it can fall back to Subject CN when a cert has no dNSName SAN (subject alternative name) but a SRVName otherName SAN or URI SAN. --- https://tools.ietf.org/search/rfc6125#section-6.4.4

[issue27961] remove support for platforms without "long long"

Stefan Behnel added the comment: Removing HAVE_LONG_LONG entirely causes breakage of third party code that uses this macro to enable PY_LONG_LONG support. Could you please always define it instead of removing it? -- nosy: +scoder ___ Python tracker

[issue28151] testPythonOrg() of test_robotparser fails on validating python.org HTTPS certificate

Berker Peksag added the comment: test_robotparser is now green. I've opened issue 28195 for test_huntrleaks_fd_leak. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue28134] socket.socket(fileno=fd) does not work as documented

Christian Heimes added the comment: I'm well aware that it does not work in all cases. However it works good enough in most cases. Right now the fileno argument must be considered broken because it leads to wrong results. It is a problem and possible security issue for a couple of use cases,

[issue28195] test_huntrleaks_fd_leak fails on Windows

New submission from Berker Peksag: >From >http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/11654/steps/test/logs/stdio == FAIL: test_huntrleaks_fd_leak (test.test_regrtest.ArgsTestCase)

[issue28075] os.stat fails when access is denied

Roundup Robot added the comment: New changeset 20c4ad866620 by Berker Peksag in branch '3.5': Issue #28075: Fix test_access_denied in Python 3.5 https://hg.python.org/cpython/rev/20c4ad866620 -- ___ Python tracker

[issue28115] Use argparse for the zipfile module

SilentGhost added the comment: Given that at present zipfile CLI has no test or docs, and along with Guido's aversion for such tools in general, I'm not at all sure they should be added. In any case, here is the patch for the main part of conversion. -- keywords: +patch stage: needs

[issue9004] datetime.utctimetuple() should not set tm_isdst flag to 0

Changes by Mark Dickinson : -- nosy: -mark.dickinson ___ Python tracker ___ ___

[issue5516] equality not symmetric for subclasses of datetime.date and datetime.datetime

Changes by Mark Dickinson : -- nosy: -mark.dickinson ___ Python tracker ___ ___

[issue28184] Trailing whitespace in C source code

Changes by Mark Dickinson : -- nosy: -mark.dickinson ___ Python tracker ___ ___

[issue28185] Tabs in C source code

Changes by Mark Dickinson : -- nosy: -mark.dickinson ___ Python tracker ___ ___

[issue28194] Clean up some checks in dict implementation

New submission from Xiang Zhang: The proposed patch cleans up some unnecessary parts in dict implementation especially NULL checks in lookup functions. There are four states a DictKeyEntry can be. Only in unused(empty) and dummy states me_key can be NULL. So NULL checks in used and pending

[issue28151] testPythonOrg() of test_robotparser fails on validating python.org HTTPS certificate

Roundup Robot added the comment: New changeset 83bca958adc9 by Berker Peksag in branch '3.6': Issue #28151: Use pythontest.net in test_robotparser https://hg.python.org/cpython/rev/83bca958adc9 New changeset 4bfd91a45c81 by Berker Peksag in branch 'default': Issue #28151: Merge from 3.6

[issue28183] Clean up and speed up dict iteration

Xiang Zhang added the comment: Serhiy: Patch LGTM except two trivial comments on Rietveld. Raymond: With such implementation keys, values and hashes are all organized together and there seems no _resize operation can only adjust hashes without breaking the entire layout. --

[issue10109] itertools.product with infinite iterator cause MemoryError.

Terry J. Reedy added the comment: Recipes for handling an infinite first iterator for product, or an infinite iterator for combinations (they could be similar), that use and build on the current functions, without the output order constraint, might be candidates for the recipe section. I

[issue28143] ASDL compatibility with Python 3 system interpreter

Malthe Borch added the comment: I forgot to add "from __future__ import print_function" to the beginning of "asdl.py" and "spark.py". It should then work on Python 2. That is, with the imported print-function, the incompatibilities that Martin pointed out are no longer there. As for Windows,