[issue46337] urllib.parse: Allow more flexibility in schemes and URL resolution behavior
karl added the comment: Just to note that there is a maintained list of officially accepted schemes at IANA. https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml In addition there is a list of unofficial schemes on wikipedia https://en.wikipedia.org/wiki/List_of_URI_schemes#Unofficial_but_common_URI_schemes -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue46337> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44423] copy2 / sendfile fails on linux with large file
karl added the comment: could not reproduce the error -- stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue44423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44423] copy2 / sendfile fails on linux with large file
New submission from karl : by copy a large file e.g. -rwxrwxr-x 1 1002 1001 5359338160 Feb 9 2019 xxx_file_xxx.mdx copy2 / sendfile / fastcopy fails with: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) File "/usr/local/lib/python3.8/dist-packages/pybcpy/diff_bak_copy.py", line 212, in _init_copy_single shutil.copy2(f, dest_path) File "/usr/lib/python3.8/shutil.py", line 432, in copy2 copyfile(src, dst, follow_symlinks=follow_symlinks) File "/usr/lib/python3.8/shutil.py", line 272, in copyfile _fastcopy_sendfile(fsrc, fdst) File "/usr/lib/python3.8/shutil.py", line 169, in _fastcopy_sendfile raise err File "/usr/lib/python3.8/shutil.py", line 149, in _fastcopy_sendfile sent = os.sendfile(outfd, infd, offset, blocksize) OSError: [Errno 75] Value too large for defined data type: 'xxx_file_xxx.mdx' -> 'dest/xxx_file_xxx.mdx' """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.8/dist-packages/pybcpy/__main__.py", line 433, in main_func() File "/usr/local/lib/python3.8/dist-packages/pybcpy/__main__.py", line 425, in main_func args.func(args) File "/usr/local/lib/python3.8/dist-packages/pybcpy/__main__.py", line 75, in cmd_init ) = dbak.init_backup_repo(tarmode=args.tar) File "/usr/local/lib/python3.8/dist-packages/pybcpy/diff_bak_copy.py", line 231, in init_backup_repo files = p.map(self._init_copy_single, ifiles) File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get raise self._value OSError: [Errno 75] Value too large for defined data type: 'xxx_file_xxx.mdx' -> 'dest/xxx_file_xxx.mdx' reference to code: https://github.com/kr-g/pybcpy/blob/master/pybcpy/diff_bak_copy.py -- messages: 395862 nosy: kr-g priority: normal severity: normal status: open title: copy2 / sendfile fails on linux with large file type: crash versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue44423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44209] urllib.robotparser fail on Disallow: /? from google.com
New submission from Karl Y. Pradene : In robotparser.py On line 222 path = urllib.parse.urlunparse(urllib.parse.urlparse(path)) tranform the entry Disallow: /? in the google.com/robots.txt in : Disallow: / making every can_fetch request return False -- components: Library (Lib) messages: 394144 nosy: karl.pradene priority: normal severity: normal status: open title: urllib.robotparser fail on Disallow: /? from google.com type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue44209> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41748] HTMLParser: comma in attribute values with/without space
karl added the comment: Status: The PR should be ready and completed https://github.com/python/cpython/pull/24072 and eventually be merged at a point. Thanks to ezio.melotti for the wonderful guidance. -- ___ Python tracker <https://bugs.python.org/issue41748> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19683] test_minidom has many empty tests
Change by karl : -- pull_requests: +22980 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/24152 ___ Python tracker <https://bugs.python.org/issue19683> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19683] test_minidom has many empty tests
karl added the comment: Ah no. They ARE used through defproperty and minicompat.py get = getattr(klass, ("_get_" + name)) -- ___ Python tracker <https://bugs.python.org/issue19683> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19683] test_minidom has many empty tests
karl added the comment: These methods are not used anywhere in the code. https://github.com/python/cpython/blob/5c30145afb6053998e3518befff638d207047f00/Lib/xml/dom/minidom.py#L71-L80 What was the purpose when they were created… hmm maybe blame would give clue. Ah they were added a long time ago https://github.com/python/cpython/commit/73678dac48e5858e40cba6d526970cba7e7c769c#diff-365c30899ded02b18a2d8f92de47af6ca213eefe7883064c8723598da600ea42R83-R88 but never used? or was it in the spirit to reserve the keyword for future use? https://developer.mozilla.org/en-US/docs/Web/API/Node/firstChild -- ___ Python tracker <https://bugs.python.org/issue19683> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19683] test_minidom has many empty tests
karl added the comment: @zach.ware @r.david.murray I'm going through the source currently. I see that the test file is using: class MinidomTest(unittest.TestCase): def confirm(self, test, testname = "Test"): self.assertTrue(test, testname) Is there a specific reason to use this form instead of just directly using self.assertEqual or similar forms for new tests or reorganizing some of the tests. I see that it is used for example for giving a testname but def testAAA(self): dom = parseString("") el = dom.documentElement el.setAttribute("spam", "jam2") self.confirm(el.toxml() == '', "testAAA") testAAA is not specifically helping. :) -- ___ Python tracker <https://bugs.python.org/issue19683> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25258] HtmlParser doesn't handle void element tags correctly
karl added the comment: I wonder if the confusion comes from the name. The HTMLParser is kind of a tokenizer more than a full HTML parser, but that's probably a detail. It doesn't create a DOM Tree which you can access, but could help you to build a DOM Tree (!= DOM Document object) https://html.spec.whatwg.org/multipage/parsing.html#overview-of-the-parsing-model > Implementations that do not support scripting do not have to actually create > a DOM Document object, but the DOM tree in such cases is still used as the > model for the rest of the specification. -- ___ Python tracker <https://bugs.python.org/issue25258> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25258] HtmlParser doesn't handle void element tags correctly
karl added the comment: The parsing rules for tokenization of html are at https://html.spec.whatwg.org/multipage/parsing.html#tokenization In the stack of open elements, there are specific rules for certain elements. https://html.spec.whatwg.org/multipage/parsing.html#special from a DOM point of view, there is indeed no difference in between https://software.hixie.ch/utilities/js/live-dom-viewer/?%3C!DOCTYPE%20html%3E%0A%3Cimg%20src%3D%22somewhere%22%3E%3Cimg%20src%3D%22somewhere%22%2F%3E -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue25258> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42821] HTMLParser: subsequent duplicate attributes should be ignored
New submission from karl : This comes up while working on issue 41748 browser input data:text/html,text browser output text Actual HTMLParser output see https://github.com/python/cpython/pull/24072#discussion_r551158342 ('starttag', 'div', [('class', 'bar'), ('class', 'foo')])] Expected HTMLParser output ('starttag', 'div', [('class', 'bar')])] -- components: Library (Lib) messages: 384308 nosy: karlcow priority: normal severity: normal status: open title: HTMLParser: subsequent duplicate attributes should be ignored versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue42821> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28937] str.split(): allow removing empty strings (when sep is not None)
Change by karl : -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue28937> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41748] HTMLParser: comma in attribute values with/without space
Change by karl : -- keywords: +patch pull_requests: +22904 stage: test needed -> patch review pull_request: https://github.com/python/cpython/pull/24072 ___ Python tracker <https://bugs.python.org/issue41748> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41748] HTMLParser: comma in attribute values with/without space
karl added the comment: Ah! This is fixing it diff --git a/Lib/html/parser.py b/Lib/html/parser.py index 6083077981..790666 100644 --- a/Lib/html/parser.py +++ b/Lib/html/parser.py @@ -44,7 +44,7 @@ (?:\s*=+\s*# value indicator (?:'[^']*' # LITA-enclosed value |"[^"]*" # LIT-enclosed value - |(?!['"])[^>\s]* # bare value + |(?!['"])[^>]* # bare value ) (?:\s*,)* # possibly followed by a comma )?(?:\s|/(?!>))* Ran 48 tests in 0.175s OK == Tests result: SUCCESS == -- ___ Python tracker <https://bugs.python.org/issue41748> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41748] HTMLParser: comma in attribute values with/without space
Change by karl : -- title: HTMLParser: parsing error -> HTMLParser: comma in attribute values with/without space ___ Python tracker <https://bugs.python.org/issue41748> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41748] HTMLParser: parsing error
karl added the comment: Ezio, TL,DR: Testing in browsers and adding two tests for this issue. Should I create a PR just for the tests? https://github.com/python/cpython/blame/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/test/test_htmlparser.py#L479-L485 A: comma without spaces --- Tests for browsers: data:text/html,text Serializations: * Firefox, Gecko (86.0a1 (2020-12-28) (64-bit)) * Edge, Blink (Version 89.0.752.0 (Version officielle) Canary (64 bits)) * Safari, WebKit (Release 117 (Safari 14.1, WebKit 16611.1.7.2)) Same serialization in these 3 rendering engines text Adding: def test_comma_between_unquoted_attributes(self): # bpo 41748 self._run_check('', [('starttag', 'div', [('class', 'bar,baz=asd')])]) ❯ ./python.exe -m test -v test_htmlparser … test_comma_between_unquoted_attributes (test.test_htmlparser.HTMLParserTestCase) ... ok … Ran 47 tests in 0.168s OK == Tests result: SUCCESS == 1 test OK. Total duration: 369 ms Tests result: SUCCESS So this is working as expected for the first test. B: comma with spaces Tests for browsers: data:text/html,text Serializations: * Firefox, Gecko (86.0a1 (2020-12-28) (64-bit)) * Edge, Blink (Version 89.0.752.0 (Version officielle) Canary (64 bits)) * Safari, WebKit (Release 117 (Safari 14.1, WebKit 16611.1.7.2)) Same serialization in these 3 rendering engines text Adding def test_comma_with_space_between_unquoted_attributes(self): # bpo 41748 self._run_check('', [('starttag', 'div', [ ('class', 'bar'), (',baz', 'asd')])]) ❯ ./python.exe -m test -v test_htmlparser This is failing. == FAIL: test_comma_with_space_between_unquoted_attributes (test.test_htmlparser.HTMLParserTestCase) -- Traceback (most recent call last): File "/Users/karl/code/cpython/Lib/test/test_htmlparser.py", line 493, in test_comma_with_space_between_unquoted_attributes self._run_check('', File "/Users/karl/code/cpython/Lib/test/test_htmlparser.py", line 95, in _run_check self.fail("received events did not match expected events" + AssertionError: received events did not match expected events Source: '' Expected: [('starttag', 'div', [('class', 'bar'), (',baz', 'asd')])] Received: [('data', '')] -- I started to look into the code of parser.py which I'm not familiar (yet) with. https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/html/parser.py#L42-L52 Do you have a suggestion to fix it? -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue41748> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4643] cgitb.html fails if getattr call raises exception
karl added the comment: > The getattr call here has a default value, so it should not raise > AttributeError. It should also not raise any other exception because a valid > implementation of __getattr__ should raise only AttributeError: but probably the intent of the patch here is to surface a meaningful error with regards to the script, more than an error on the cgitb python lib itself as this is a traceback tool. A bit like an HTML validator which continue to process things to give some kind of meaning. Diving into previous issues about scanvars https://bugs.python.org/issue966992 Similar https://bugs.python.org/issue1047397 The last comment is basically this issue here. There is a patch which is lot better for this bug and which addresses the issues here. https://github.com/python/cpython/pull/15094 It has not been merged yet. Not sure why or if there is anything missing. Probably this bug could be closed as a duplicate of https://bugs.python.org/issue966992 And someone needs to push to push the latest bits for https://github.com/python/cpython/pull/15094 What do you think @iritkatriel ? I will close my PR. -- ___ Python tracker <https://bugs.python.org/issue4643> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4643] cgitb.html fails if getattr call raises exception
karl added the comment: Converted into GitHub PR https://github.com/python/cpython/pull/24038 -- ___ Python tracker <https://bugs.python.org/issue4643> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4643] cgitb.html fails if getattr call raises exception
Change by karl : -- nosy: +karlcow nosy_count: 4.0 -> 5.0 pull_requests: +22878 stage: test needed -> patch review pull_request: https://github.com/python/cpython/pull/24038 ___ Python tracker <https://bugs.python.org/issue4643> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25479] Increase unit test coverage for abc.py
karl added the comment: @iritkatriel Github PR done. https://github.com/python/cpython/pull/24034 -- ___ Python tracker <https://bugs.python.org/issue25479> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25479] Increase unit test coverage for abc.py
Change by karl : -- keywords: +patch nosy: +karlcow nosy_count: 2.0 -> 3.0 pull_requests: +22875 pull_request: https://github.com/python/cpython/pull/24034 ___ Python tracker <https://bugs.python.org/issue25479> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42731] Enhancement request for proxying PyString
New submission from Karl Nelson : When developing with JPype, the largest hole currently is that Java returns a string type which cannot be represented as a str. Java strings are string like and immutable and can be pulled to Python when needed, but it is best if they remain in Java until Python requests it as pulling all string values through the API and pushing them back can result in serious overhead. Thus they need to be represented as a Proxy to a string, which can be accessed as a string at anytime. Throughout the Python API str is treated as a concrete type (though it is somewhat polymorphic due to different storage for code points sizes.) There is also handling for an "unready" string which needs additional treatment before it can be accessed. Unfortunately this does not appear to be suitable for creating a proxy object which can be pulled from another source to create a string on demand. Having a "__str__()" method is insufficient as that merely makes an object able to become a string rather than considered to be a string by the rest of the API. Would it be possible to generalize the concept of an unready string so that when Ready is called it fetches the actually string contents, creates a piece of memory to store the string contents (outside of the object itself), and sets the access flags for so that the code points can be interpreted? Is this already possible in the API? Are there any other plans to make the str type able to operate as a proxy? -- components: Extension Modules messages: 383701 nosy: Thrameos priority: normal severity: normal status: open title: Enhancement request for proxying PyString versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue42731> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30988] Exception parsing invalid email address headers starting or ending with dot
Karl Fogel added the comment: I can also confirm this bug, with Python 3.9.1 on Debian GNU/Linux ('testing' distro up-to-date as of 2020-12-21). 1) Create a parser `p` with `p = email.parser.HeaderParser(policy=email.policy.default)`. 2) Parse a single problematic (as described below) message: `msg_headers = p.parsestr(msg_str)` 3) Try to get the To field: `msg_headers.get_all('to', [ ])` and get an exception raised: `AttributeError: 'str' object has no attribute 'token_type'` (with the same stack trace as OP shows). Here is a minimal problematic message you can reproduce this with (i.e., just make `msg_str` have this string value): ``` >From nobody Mon Dec 21 12:00:00 2020 From: sen...@example.com To: . Date: Mon, 21 Dec 2020 12:00:00 - Message-ID: <87ab5rvds7@example.com> Subject: This is the Subject header. Here is the body of the message. ``` Note that *any* number of dots for the recipient's name would also result in an error. The above example uses just ".", but it could be "..", "...", ".", etc. -- nosy: +kfogel ___ Python tracker <https://bugs.python.org/issue30988> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Okay, well at least now googling Python + "A dynamic link library (DLL) initialization routine failed." give something which could point a user may be able to identify the issue. It wasn't obvious to me that imports did not hold the GIL, but it is clear in retrospect that it should have been. Thanks very much and sorry for the wild goose chase. -- resolution: -> third party stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I found it. The change that hit JPype is https://bugs.python.org/issue33895 Thanks, Eryk Sun for figuring this out. I would never have gotten myself. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Drat I missed that one when I was scanning for statics because I was focusing C++ resource rather than looking for Python resources. I also wouldn't have expected this to happen on only one platform, but that was my bad. Is it possible to improve the error reporting for this? This was a horrible error to trace down at least for a non-windows programmer. At least a mention in the doc as to the symptoms so someone else can google it. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Gave another series of shots at the problem. So the reason that attaching the debugger fixes the issue is that it calls LoadLibraryExW for a bunch libraries when it attaches. I was also able to unconditionally trigger the bug even when loading without the pyc. It appears that calling "import _ssh" will cause whatever conflict is happening to trigger immediately. Other than than progress was minimal. Attaching the debugger causes so many changes to the control flow that I can't get the bad path to trigger and I can't get the debugger to attach once you start down the path. I tried creating a segfault but again I can't get the debugger to attach because Detours is interfering with the attachment process. I also tried disabling some of the flags that JPype was adding to the python build process to see if any are related. "/DEBUG" on the linker does not do anything to the problem. The other options were '/Zi', '/EHsc', '/std:c++14'. Nothing here is out of the ordinary. I do not see anything in the link line that shows evidence of static linkage or likely to cause a conflict. The exact same line is being used with Python 3.8. link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\nelson85\AppData\Local\Programs\Python\Python39\libs /LIBPATH:C:\Users\nelson85\AppData\Local\Programs\Python\Python39\PCbuild\amd64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\ATLMFC\lib\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\ATLMFC\lib\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\ lib\10.0.18362.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64 /EXPORT:PyInit__jpype $(OBJECTS) /OUT:build\lib.win-amd64-3.9\_jpype.cp39-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.9\Release\native\common\_jpype.cp39-win_amd64.lib So I remain at a loss how to proceed forward. Nothing has changed in JPype or its symbols since Python 3.8 so this is clearly taking place in Python or at the OS level. Is it possible that Python shipped with bad dlls for the 3.9.0 build? If so then looking at the source code may be the wrong approach. After all depending on the order of libraries I trigger or don't trigger the bug. I should note that the difference between loading successfully py and pyc seems to have been triggered in PC/getpathc.c could it be the change to remove loading api-ms-win-core-path? ``` static void join(wchar_t *buffer, const wchar_t *stuff) { -if (_PathCchCombineEx_Initialized == 0) { -HMODULE pathapi = LoadLibraryExW(L"api-ms-win-core-path-l1-1-0.dll", NULL, - LOAD_LIBRARY_SEARCH_SYSTEM32); -if (pathapi) { -_PathCchCombineEx = (PPathCchCombineEx)GetProcAddress(pathapi, "PathCchCombineEx"); -} -else { -_PathCchCombineEx = NULL; -} -_PathCchCombineEx_Initialized = 1; -} - -if (_PathCchCombineEx) { -if (FAILED(_PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0))) { -Py_FatalError("buffer overflow in getpathp.c's join()"); -} -} else { -if (!PathCombineW(buffer, buffer, stuff)) { -Py_FatalError("buffer overflow in getpathp.c's join()"); -} +if (FAILED(PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0))) { +Py_FatalError("buffer overflow in getpathp.c's join()"); } } ``` Also given JPype is really a generic C++ CPython module (with no special options other than /Zi /EHsc why hasn't anyone else triggered this? -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I will look through the list of samples in Detours to see if there is something that can give us better stacktrace information. If we really need a stack trace I can make Detours bust by creating an intentional segfault and see if we can attach the debugger after the source of the problem has been triggered. Unfortunately as my windows programming knowledge is all from 1994 I may be missing something obvious. In the meantime I am upgrading the sample with some more hooks for any other potential LoadLibrary variants and FlsGetValue. Though given the GetProcAddress calls I think that it must be something in the chain leading up so the variants are likely not envolved. The other thing I can do is look at instrumenting SetErrorMode so that I can give myself a marker of when we are entering and leaving the dyload code. ``` /* Don't display a message box when Python can't load a DLL */ old_mode = SetErrorMode(SEM_FAILCRITICALERRORS); /* bpo-36085: We use LoadLibraryEx with restricted search paths to avoid DLL preloading attacks and enable use of the AddDllDirectory function. We add SEARCH_DLL_LOAD_DIR to ensure DLLs adjacent to the PYD are preferred. */ Py_BEGIN_ALLOW_THREADS hDLL = LoadLibraryExW(wpathname, NULL, LOAD_LIBRARY_SEARCH_DEFAULT_DIRS | LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR); Py_END_ALLOW_THREADS #if !USE_UNICODE_WCHAR_CACHE PyMem_Free(wpathname); #endif /* USE_UNICODE_WCHAR_CACHE */ /* restore old error mode settings */ SetErrorMode(old_mode); ``` I will post back once I get some results. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I looked more at the logs to see if I can find out what to instrument next. The log files unfortunately don't diff well because every line contains a timestamp so I can't a proper alignment as well as all the real addresses. So I wrote a short program to convert all the numbers into an "X". After doing so I find there is a single difference in the process between loading from a py file and a pyc file. # import _jpype ## Only if loaded from a Py file trclnk64: 001 GetProcAddress(7ffc4c71,FlsGetValue) X X X trclnk64: 001 GetProcAddress(,) -> X ## All X X X trclnk64: 001 LoadLibraryExW(C:\Users\nelson85\AppData\Local\Programs\Python\Python39\python3,0,1000) X X X trclnk64: 001 LoadLibraryExW(,,) -> X X X X trclnk64: ### X C:\Users\nelson85\AppData\Local\Programs\Python\Python39\python3.DLL 000140bc So here is the call that fails X X X trclnk64: 001 LoadLibraryExW(c:\users\nelson85\documents\devel\open\jpype\_jpype.cp39-win_amd64.pyd,0,1100) X X X trclnk64: 001 LoadLibraryExW(api-ms-win-core-synch-l1-2-0,0,800) X X X trclnk64: 001 LoadLibraryExW(,,) -> X X X X trclnk64: ### X C:\WINDOWS\System32\KERNELBASE.dll 002b306b X X X trclnk64: ntdll.dll [7ffc4c974138 X So the problem is something that happened before the LoadLibraryExW call when python3.dll was accessed. The correct path called FlsGetValue while the incorrect one skipped it which lead to a failure. Thus my conclusion is that the vehicle left the edge of the cliff just prior to LoadLibraryExW(_jpype) and hit the ocean when C++ runtime library was requested. Does that point to anything we can use to solve this? -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Eryk, Unfortunately, this particular bug is resistant to attaching the debugger. Whenever the debugger is attached the bug disappears. Fortunately the suggestion to use Detours by WildCard65 appears to offer a way to diagnose the issue. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: The last libraries loaded prior to the failure were... ``` 20201218192451066 20440 50.60: trclnk64: api-ms-win-eventing-provider-l1-1-0.dll [7ffc4c974108 7ffc4c8b7808] 20201218192451066 20440 50.60: trclnk64: EventUnregister0 7ffc4eab37a0 20201218192451066 20440 50.60: trclnk64: EventRegister0 7ffc4ea6a640 20201218192451066 20440 50.60: trclnk64: EventActivityIdControl0 7ffc4eac6190 20201218192451066 20440 50.60: trclnk64: EventWriteTransfer0 7ffc4eab2cf0 20201218192451066 20440 50.60: trclnk64: EventSetInformation0 7ffc4ea6a3e0 20201218192451066 20440 50.60: trclnk64: api-ms-win-core-apiquery-l1-1-0.dll [7ffc4c9740e8 7ffc4c8b77e8] 20201218192451066 20440 50.60: trclnk64: ApiSetQueryApiSetPresence0 7ffc4ead5030 20201218192451066 20440 50.60: trclnk64: api-ms-win-core-apiquery-l1-1-1.dll [7ffc4c9740f8 7ffc4c8b77f8] 20201218192451066 20440 50.60: trclnk64: ApiSetQueryApiSetPresenceEx0 7ffc4eb35730 ``` And as far as I can tell that was successful. Perhaps there is another library at this point, but my reading of the log is that LoadLibraryExW was called run though all required elements to load, got the end section before it started to load dependencies of _jpype, started fetching things that have to do with locale and then hit a failure point inside of LoadLibrary. If I look at the successful trace then I see the next action is to start linking in C++ symbols. But perhaps I am misreading... Looking farther back in the log, the last LoadLibrary to get called before the failure is. ``` 20201218192451054 20440 50.60: trclnk64: 001 LoadLibraryExW(api-ms-win-core-fibers-l1-1-1,0,800) 20201218192451054 20440 50.60: trclnk64: 001 LoadLibraryExW(,,) -> 7ffc4c71 20201218192451054 20440 50.60: trclnk64: ### 4c71: C:\WINDOWS\System32\KERNELBASE.dll 002b306b 20201218192451054 20440 50.60: trclnk64: ntdll.dll [7ffc4c974138 7ffc4c8b7838] ``` Fibers? I have no clue at this point. Is this a Python bug or a windows bug? -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I used Detours with trclnk64 to get the following log: ``` 20201218193836960 4332 50.60: trclnk64: 001 GetProcAddress(,) -> 7ffc4ccebef0 20201218193836960 4332 50.60: trclnk64: 001 GetProcAddress(7ffc4ccd,LCMapStringEx) 20201218193836960 4332 50.60: trclnk64: 001 GetProcAddress(,) -> 7ffc4cce6640 <= This is the point where it differs from a clean run. 20201218193838163 4332 50.60: trclnk64: 001 GetProcAddress(7ffc4c71,FlsFree) 20201218193838163 4332 50.60: trclnk64: 001 GetProcAddress(,) -> 7ffc4c78b000 20201218193838163 4332 50.60: trclnk64: 001 FAIL LoadLibraryExW(c:\users\nelson85\documents\devel\open\jpype\_jpype.cp39-win_amd64.pyd,0,1100) 20201218193838164 4332 50.60: trclnk64: 001 LoadLibraryExW(,,) -> 0 ``` It really is nothing to go on... Everything succeeds, then we get an extra successful request for FlsFree then it immediately returns with a fail. In the successful run (without the cache) we get... ``` 20201218192451068 20440 50.60: trclnk64: 001 GetProcAddress(7ffc4ccd,LCMapStringEx) 20201218192451068 20440 50.60: trclnk64: 001 GetProcAddress(,) -> 7ffc4cce6640 20201218192451068 20440 50.60: trclnk64: 001 LoadLibraryExW(,,) -> 7ffc349f 20201218192451068 20440 50.60: trclnk64: ### 349f: c:\users\nelson85\documents\devel\open\jpype\_jpype.cp39-win_amd64.pyd 20201218192451068 20440 50.60: trclnk64: MSVCP140.dll [7ffc34a7af48 7ffc34a5a0c8] 20201218192451068 20440 50.60: trclnk64: ?_Lock@?$basic_streambuf@DU?$char_traits@D@std@@@std@@UEAAXXZ0 7ffc36f22350 20201218192451068 20440 50.60: trclnk64: ?_Unlock@?$basic_streambuf@DU?$char_traits@D@std@@@std@@UEAAXXZ0 7ffc36f22350 ``` So the failure appears to be internal to the call before we load MSVCP140.dll. Unfortunately, I am way outside my element here as a Linux/Java/CPython programmer. My next guess would be to instrument the set system error and if we are within LoadLibrary then execute "int *i=0; *i=0;" and hope the debugger would get me the point of failure so that I can examine the stack trace to find out where we are. But if you have another direction I would be happy to hear it. Is there some function I should add to trclnk64 to that may have gone wrong? It must be something that fetched the address of LCMapStringEx, but I have no clue what that may be. This really does not look like a JPype bug. Yes, two minutes in the future from where the car ran over the pedestrian, he was going to run into the bank and demand money (monkey patch and other bad things). But at the point in time the car hit, the pedestrian was just minding his own business. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Can you be so kind as pointing me to a LoadLibraryExW detour example for Python? I have shimmed a DLL before to capture transaction logs in the past, but not with a Python or a system library. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Just for reference here are all the dependencies that _jpype has ``` MSVCP140.dll python39.dll KERNEL32.dll VCRUNTIME140_1.dll VCRUNTIME140.dll api-ms-win-crt-runtime-l1-1-0.dll api-ms-win-crt-string-l1-1-0.dll api-ms-win-crt-heap-l1-1-0.dll ``` So it isn't like there are a whole lot of things that I am doing that are likely to trigger a dependency problem. As far as I can see the only items that are not already on python39.dll dependency list are VCRUNTIME140_1.dll and MSVCP140.dll. But if that were the case then surely other projects would also be hitting this bug. So this error message makes no sense at all. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I attempted another deep dive on this one. 1) Removed manual library add platform_specific['libraries'] = ['Advapi32'] No change. 2) Compared symbol wise imports Two imports changed PyIndex_Check PyObject_CheckBuffer plus one additional "fflush" dependency in api-ms-win-crt-stdio-l1-1-0.dll 3) Compared symbol wise exports No changes detected. 4) Disassembled both binaries Other than a addressing that is about a 0x1000 change in the relocation tables. Guessing that this is just a change in the number of pages. There were functional changes likely related to Python macros differences. 5) Removed all references to printf and flush to remove api-ms-win-crt-stdio-l1-1-0.dll dependency. No change. 6) Scanned the changelog and Code changes for the past year in Python/dynload_win.c. Nothing appears likely to trigger this change. 7) Deleted the _jpype module entirely to verify that the error message is related to the copy that I am working on and not some other copy on system. Error message changes to "ModuleNotFoundError: No module named '_jpype'" 8) Scanned the source code for static variables. None detected thus far. But it is a many hours long process. Structurally these should not exist based on the module design. Statics found thus far. 2 jobject and jmethodID pointer types One C++ class called PLATFORM_ADAPTER which is the required routine determining whether to load jvm.dll or jvm.so using LoadLibrary or dlopen or shload. The global pointer to the context C++ class; a bunch of statics related to logging when enabled. Python C style structure definitions. Everything was a primitive or pointer with the exception of one class. All pointers I found were initialized to constant values. I rewrote the PLATFORM_ADAPTER to use a pointer rather than be initialized at load time. No joy. (after scanning through 17k lines of code take break and ponder life) It is certainly possible I missed something here but it looks pretty unlikely this is the source of the error. 9) Verified that the behavior does not occur on Python 3.8.5 with the current source code. Pass. This only happens on Python 3.9.0 10) Scanned the linker line for differences. One major difference. In Python 3.8 the linker line contains quotes around all the arguments while in Python 3.9 the linker line is missing the quotes on all /LIBPATH statements. However this must be a logging difference as the statement without the quotes would not work. 11) Manually ran linker command. Nothing changed, same behavior. So link command is a red herring. 12) Profit??? Okay I have once again exhausted everything that I can think of on the JPype side. So that leaves something in the build process that does not appear in the logs, a symbol conflict that so how only triggers on one path, or Python3.9.0 has a corrupt build in the wild. Steve is there anything else that I can try here? -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Well that gives me a place to search from. Unfortunately statics are not likely the source of the issue. I scrubbed all C++ structures from the project into a big structure which has only a single global pointer which I initialize in the init method. It is possible that one crept back in at some point, but it is a possibility. The use of actual C++ in this project is nearly zero. Other than exceptions, strings and the vector on the buried in some class, the rest has been purged already. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Without the pyc everything goes fine... ``` ... _bootlocale' # <_frozen_importlib_external.SourceFileLoader object at 0x0222E9FD5A30> import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0222E9F88F40> Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. # code object from c:\users\nelson85\documents\devel\open\jpype\jpype\__init__.py # created 'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\jpype\\__pycache__\\__init__.cpython-39.pyc' # extension module '_jpype' loaded from 'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\_jpype.cp39-win_amd64.pyd' # extension module '_jpype' executed from 'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\_jpype.cp39-win_amd64.pyd' import '_jpype' # <_frozen_importlib_external.ExtensionFileLoader object at 0x0222EA4C6250> # c:\users\nelson85\documents\devel\open\jpype\jpype\__pycache__\_jinit.cpython-39.pyc matches c:\users\nelson85\documents\devel\open\jpype\jpype\_jinit.py # code o ... ``` With the pyc file, failure... ``` ... _bootlocale' # <_frozen_importlib_external.SourceFileLoader object at 0x015278445A30> import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x0152783F8F40> Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. # c:\users\nelson85\documents\devel\open\jpype\jpype\__pycache__\__init__.cpython-39.pyc matches c:\users\nelson85\documents\devel\open\jpype\jpype\__init__.py # code object from 'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\jpype\\__pycache__\\__init__.cpython-39.pyc' Traceback (most recent call last): File "C:\Users\nelson85\Documents\test.py", line 1, in import jpype File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "", line 790, in exec_module File "", line 228, in _call_with_frames_removed File "c:\users\nelson85\documents\devel\open\jpype\jpype\__init__.py", line 18, in import _jpype File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 666, in _load_unlocked File "", line 565, in module_from_spec File "", line 1108, in create_module File "", line 228, in _call_with_frames_removed ImportError: DLL load failed while importing _jpype: A dynamic link library (DLL) initialization routine failed. # clear builtins._ # clear sys.path # clear sys.argv # clear sys.ps1 # clear sys.ps2 # clear sys.last_type # clear sys.last_value # clear sys.last_traceback # destroy jpype # clear sys.path_hooks # clear sys.path_importer_cache # clear sys.meta_path # clear sys.__interactivehook__ # restore sys.stdin # restore sys.stdout # restore sys.stderr ... ``` I see the codecs all get loaded earlier in the trace, so that can't be the issue. There were no differences (other than memory addresses) up to this point. So it must be going wrong at bootstrap_external:1108. So guessing from the message we failed in. ``` def exec_module(self, module): """Initialize an extension module""" _bootstrap._call_with_frames_removed(_imp.exec_dynamic, module) _bootstrap._verbose_message('extension module {!r} executed from {!r}', self.name, self.path) ``` After that point it gets really fuzzy. I would guess we are going to import.c#_imp_create_dynamic_impl which takes us to _PyImport_LoadDynamicModuleWithSpec. But I will need some time to read code to see if this guess is correct. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Thanks, I will see if I can get additional diagnostics today. Otherwise I will have to start recompiling Python with diagnostic hooks. I have never had to rebuild python on windows so it may take a while to figure out how to make progress. The monkey patch is less scary than it looks. Since I only need these slots of specific basic types that I designate (not user selected), the extra types all extend without altering the underlaying type, and the class layout is controlled by a meta class so that only Java can create these type objects, it is reasonably robust. However, yes changes in Python can break it. It is basically an attempt to create a mixin for concrete classes without the benefit of a dictoffset type slot. It took about a month do develop something that safely added the slot without overrunning, leaking or creating other memory issues (and played nice when Python added its own dict slot). Unfortunately, as I do many 100s on JSlot tests during method resolution using a dict based approach was unacceptably slow. I would love if there was a formal method to create multiple inheritance mixins for Python basic types, but that is another issue. ``` /** * Allocate a new Python object with a slot for Java. * * We need extra space to store our values, but because there * is no way to do so without disturbing the object layout. * Fortunately, Python already handles this for dict and weakref. * Python aligns the ends of the structure and increases the * base type size to add additional slots to a standard object. * * We will use the same trick to add an additional slot for Java * after the end of the object outside of where Python is looking. * As the memory is aligned this is safe to do. We will use * the alloc and finalize slot to recognize which objects have this * extra slot appended. */ PyObject* PyJPValue_alloc(PyTypeObject* type, Py_ssize_t nitems ) { // Modification from Python to add size elements const size_t size = _PyObject_VAR_SIZE(type, nitems + 1) + sizeof (JPValue); PyObject *obj = (PyType_IS_GC(type)) ? _PyObject_GC_Malloc(size) : (PyObject *) PyObject_MALLOC(size); if (obj == NULL) return PyErr_NoMemory(); // GCOVR_EXCL_LINE memset(obj, 0, size); Py_ssize_t refcnt = ((PyObject*) type)->ob_refcnt; if (type->tp_itemsize == 0) PyObject_Init(obj, type); else PyObject_InitVar((PyVarObject *) obj, type, nitems); // This line is required to deal with Python bug (GH-11661) // Some versions of Python fail to increment the reference counter of // heap types properly. if (refcnt == ((PyObject*) type)->ob_refcnt) Py_INCREF(type); // GCOVR_EXCL_LINE if (PyType_IS_GC(type)) { PyObject_GC_Track(obj); } return obj; } bool PyJPValue_hasJavaSlot(PyTypeObject* type) { if (type == NULL || type->tp_alloc != (allocfunc) PyJPValue_alloc || type->tp_finalize != (destructor) PyJPValue_finalize) return false; // GCOVR_EXCL_LINE return true; } // Check for a Java slot (must work on all object types) Py_ssize_t PyJPValue_getJavaSlotOffset(PyObject* self) { PyTypeObject *type = Py_TYPE(self); if (type == NULL || type->tp_alloc != (allocfunc) PyJPValue_alloc || type->tp_finalize != (destructor) PyJPValue_finalize) return 0; Py_ssize_t offset; Py_ssize_t sz = Py_SIZE(self); // I have no clue what negative sizes mean if (sz < 0) sz = -sz; if (type->tp_itemsize == 0) offset = _PyObject_VAR_SIZE(type, 1); else offset = _PyObject_VAR_SIZE(type, sz + 1); return offset; } JPValue* PyJPValue_getJavaSlot(PyObject* self) { Py_ssize_t offset = PyJPValue_getJavaSlotOffset(self); if (offset == 0) return NULL; JPValue* value = (JPValue*) (((char*) self) + offset); if (value->getClass() == NULL) return NULL; return value; } void PyJPValue_free(void* obj) { // Normally finalize is not run on simple classes. PyTypeObject *type = Py_TYPE(obj); if (type->tp_finalize != NULL) type->tp_finalize((PyObject*) obj); if (type->tp_flags & Py_TPFLAGS_HAVE_GC) PyObject_GC_Del(obj); else PyObject_Free(obj); // GCOVR_EXCL_LINE } void PyJPValue_finalize(void* obj) { JPValue* value = PyJPValue_getJavaSlot((PyObject*) obj); ... destroy the objects hidden slot if it exists } ``` -- ___ Python t
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I am fairly sure this is a Python "bug" in the sense that there was some change in undocumented change in requirements and the distutils pattern for building a module no longer reflects that requirement. That said very likely JPype is the only module to be affected and thus I will have to manually adjust to account for the new requirement. Unfortunately as far as developers, I am it so fixing it (with your assistance) is going to have to fall on me. So lets do a run down of how this all working so you can point me where to look. 1) JPype gets built as an ordinary setup.py CPython module. So it is possibly a bug in the build pattern of the customizers in JPype that was exposed by Python 3.9. I am just going to cut and paste so that it is easy to follow along. jpype/setup.py ``` ... from setuptools import Extension ... import setupext ... jpypeLib = Extension(name='_jpype', **setupext.platform.Platform( include_dirs=[Path('native', 'common', 'include'), Path('native', 'python', 'include'), Path('native', 'embedded', 'include')], sources=[Path('native', 'common', '*.cpp'), Path('native', 'python', '*.cpp'), Path('native', 'embedded', '*.cpp')], platform=platform, )) ``` We have two sets of customization in setup.py. Both are included from a module called jpype/setupext/ The key one is the jpype/setupext/platform.py which defines the compiler flags. There are two sections of interest... jpype/setupext/platform.py contains these modifications for win32. (So there is the Advapi32, not sure why it appears there because this section is all before my time as this was started in 2001 and I joined in 2018) ``` static = True if platform == 'win32': distutils.log.info("Add windows settings") platform_specific['libraries'] = ['Advapi32'] platform_specific['define_macros'] = [('WIN32', 1)] if sys.version > '3': platform_specific['extra_compile_args'] = [ '/Zi', '/EHsc', '/std:c++14'] else: platform_specific['extra_compile_args'] = ['/Zi', '/EHsc'] platform_specific['extra_link_args'] = ['/DEBUG'] jni_md_platform = 'win32' ``` The second section is currently commented out. Though it is relevant because JPype is exotic in one way. It is a module which is loaded in three ways. When imported from Python it is an ordinary library (1) which will later pull in Java which will then load library a second time (2) to bind Java native methods. It can also be used to launch Python if Java starts the session (3). In that case, it needs libpython.dll to be bound to module so that the Java equivalent to LoadLibrary can work. When it does Java first manually loads libpython.dll then loads the module and calls the hook to get Python started. ``` # This code is used to include python library in the build when starting Python from # within Java. It will be used in the future, but is not currently required. #if static and sysconfig.get_config_var('BLDLIBRARY') is not None: # platform_specific['extra_link_args'].append(sysconfig.get_config_var('BLDLIBRARY')) ``` The actual buildext has a few minor patches so that Java libraries can run through the normal process. But nothing seems like a good candidate We have one section tweeking some of the build options. ``` def initialize_options(self, *args): """omit -Wstrict-prototypes from CFLAGS since its only valid for C code.""" self.android = False self.makefile = False self.jar = False import distutils.sysconfig cfg_vars = distutils.sysconfig.get_config_vars() replacement = { '-Wstrict-prototypes': '', '-Wimplicit-function-declaration': '', } tracing = self.distribution.enable_tracing # Arguments to remove so we set debugging and optimization level remove_args = ['-O0', '-O1', '-O2', '-O3', '-g'] for k, v in cfg_vars.items(): if not isinstance(v, str): continue if not k == "OPT" and not "FLAGS" in k: continue args = v.split() for r in remove_args: args = list(filter((r).__ne__, args)) cfg_vars[k] = " ".join(args) super().initiali
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Any progress on this item? I am seeing additional reports of this error in the conda stream tracker and elsewhere. As it only occurs when the debugger is not hooked up I don't have much traction to make progress myself. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42617] Enhancement request for PyType_FromSpecWIthBases add option for meta class
Karl Nelson added the comment: Perhaps just having PyType_InitHeapFromSpec which have just heap type linkage and slot copy section would help with the issue. The major problem I have is maintaining the slot code which may change at some point in the future. There would still be some risk of missing a portion of the init procedure but it would be a bit safer some of the process was in Python. -- title: Enhancement request for PyType_FromSpecWIthBases add option for meta class -> Enhancement request for PyType_FromSpecWIthBases add option for meta class ___ Python tracker <https://bugs.python.org/issue42617> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: Oddly that was the only exception that I got. When I hit continue it proceeded without issue and the dll was loaded correctly. However, when I try without the debugger attached the error reappears. So this is one of those Schrodinger errors. I know the error is there because I see it when we run on the CI and on a local machine, but all attempts to isolate it say that is should not exist. There can't really be a path that is different when compiling then loading from a pyc unless this is some type of uninitialized variable issues where we just happened to have been unfortunate enough to hit it. -- ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42618] Enhancement request for importing stacktraces from foreign sources
New submission from Karl Nelson : In JPype, I am transfer stack information from Java into Python for diagnostics and display purposed. Unfortunately, as the exception system is directly accessing traceback structure elements this cannot be replicated without creating traceback structures in C. I have thus been forced to create new methods that create this using internal Python structures. It would be much better if there were C API method to allow importing a foreign stack trace into Python. Here is an example of the code used in JPype for reference. ``` // Transfer list of filenames, functions and lines to Python. PyObject* PyTrace_FromJPStackTrace(JPStackTrace& trace) { PyTracebackObject *last_traceback = NULL; PyObject *dict = PyModule_GetDict(PyJPModule); for (JPStackTrace::iterator iter = trace.begin(); iter != trace.end(); ++iter) { last_traceback = tb_create(last_traceback, dict, iter->getFile(), iter->getFunction(), iter->getLine()); } if (last_traceback == NULL) Py_RETURN_NONE; return (PyObject*) last_traceback; } PyTracebackObject *tb_create( PyTracebackObject *last_traceback, PyObject *dict, const char* filename, const char* funcname, int linenum) { // Create a code for this frame. PyCodeObject *code = PyCode_NewEmpty(filename, funcname, linenum); // Create a frame for the traceback. PyFrameObject *frame = (PyFrameObject*) PyFrame_Type.tp_alloc(&PyFrame_Type, 0); frame->f_back = NULL; if (last_traceback != NULL) { frame->f_back = last_traceback->tb_frame; Py_INCREF(frame->f_back); } frame->f_builtins = dict; Py_INCREF(frame->f_builtins); frame->f_code = (PyCodeObject*) code; frame->f_executing = 0; frame->f_gen = NULL; frame->f_globals = dict; Py_INCREF(frame->f_globals); frame->f_iblock = 0; frame->f_lasti = 0; frame->f_lineno = 0; frame->f_locals = NULL; frame->f_localsplus[0] = 0; frame->f_stacktop = NULL; frame->f_trace = NULL; frame->f_valuestack = 0; #if PY_VERSION_HEX>=0x0307 frame->f_trace_lines = 0; frame->f_trace_opcodes = 0; #endif // Create a traceback PyTracebackObject *traceback = (PyTracebackObject*) PyTraceBack_Type.tp_alloc(&PyTraceBack_Type, 0); traceback->tb_frame = frame; traceback->tb_lasti = frame->f_lasti; traceback->tb_lineno = linenum; traceback->tb_next = last_traceback; return traceback; } ``` -- components: C API messages: 382852 nosy: Thrameos priority: normal severity: normal status: open title: Enhancement request for importing stacktraces from foreign sources type: enhancement ___ Python tracker <https://bugs.python.org/issue42618> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42617] Enhancement request for PyType_FromSpecWIthBases add option for meta class
New submission from Karl Nelson : PyType_FromSpecWithBases is missing an argument for taking a meta class. As a result it is necessary to replicate a large portion of Python code when I need to create a new heap type with a specified meta class. This is a maintenance issue as replicating Python code is likely to get broken in future. I have replicated to code from JPype so that it is clear how the meta class is coming into place. PyJPClass_Type is derived from a PyHeapType with additional slots for Java to use and overrides allocation slots to that it can add extra slots (needs true multiple inheritance as it needed to add Java slots to types with mismatching memory layouts object, long, float, exception, ...). This code could be made much safer if there were a PyType_FromSpecWithBasesMeta which used the meta class to allocate the memory (then I could safely override the slots after creation). ``` PyObject* PyJPClass_FromSpecWithBases(PyType_Spec *spec, PyObject *bases) { JP_PY_TRY("PyJPClass_FromSpecWithBases"); // Python lacks a FromSpecWithMeta so we are going to have to fake it here. PyTypeObject* type = (PyTypeObject*) PyJPClass_Type->tp_alloc(PyJPClass_Type, 0); // <== we need to use the meta class here PyHeapTypeObject* heap = (PyHeapTypeObject*) type; type->tp_flags = spec->flags | Py_TPFLAGS_HEAPTYPE | Py_TPFLAGS_HAVE_GC; type->tp_name = spec->name; const char *s = strrchr(spec->name, '.'); if (s == NULL) s = spec->name; else s++; heap->ht_qualname = PyUnicode_FromString(s); heap->ht_name = heap->ht_qualname; Py_INCREF(heap->ht_name); if (bases == NULL) type->tp_bases = PyTuple_Pack(1, (PyObject*) & PyBaseObject_Type); else { type->tp_bases = bases; Py_INCREF(bases); } type->tp_base = (PyTypeObject*) PyTuple_GetItem(type->tp_bases, 0); Py_INCREF(type->tp_base); type->tp_as_async = &heap->as_async; type->tp_as_buffer = &heap->as_buffer; type->tp_as_mapping = &heap->as_mapping; type->tp_as_number = &heap->as_number; type->tp_as_sequence = &heap->as_sequence; type->tp_basicsize = spec->basicsize; if (spec->basicsize == 0) type->tp_basicsize = type->tp_base->tp_basicsize; type->tp_itemsize = spec->itemsize; if (spec->itemsize == 0) type->tp_itemsize = type->tp_base->tp_itemsize; // <=== Replicated code from the meta class type->tp_alloc = PyJPValue_alloc; type->tp_free = PyJPValue_free; type->tp_finalize = (destructor) PyJPValue_finalize; // <= Replicated code from Python for (PyType_Slot* slot = spec->slots; slot->slot; slot++) { switch (slot->slot) { case Py_tp_free: type->tp_free = (freefunc) slot->pfunc; break; case Py_tp_new: type->tp_new = (newfunc) slot->pfunc; break; case Py_tp_init: type->tp_init = (initproc) slot->pfunc; break; case Py_tp_getattro: type->tp_getattro = (getattrofunc) slot->pfunc; break; case Py_tp_setattro: type->tp_setattro = (setattrofunc) slot->pfunc; break; case Py_tp_dealloc: type->tp_dealloc = (destructor) slot->pfunc; break; case Py_tp_str: type->tp_str = (reprfunc) slot->pfunc; break; case Py_tp_repr: type->tp_repr = (reprfunc) slot->pfunc; break; case Py_tp_methods: type->tp_methods = (PyMethodDef*) slot->pfunc; break; case Py_sq_item: heap->as_sequence.sq_item = (ssizeargfunc) slot->pfunc; break; case Py_sq_length: heap->as_sequence.sq_length = (lenfunc) slot->pfunc; break; case Py_mp_ass_subscrip
[issue42529] CPython DLL initialization routine failed from PYC cache file
Karl Nelson added the comment: I managed to get the debugger attached (unfortunately I am not a windows programmer so I don't use these tools). It appears when loading from a pyc, it is attempting to open the directory as a Zip file which is throwing an exception resulting in a failure to load the internal _jpype module. Unfortunately this is outside my area as nothing in jpype nor _jpype is calling the zipimport module. So it must be something internal to Python. It is a major issue as I can't release to anaconda on windows until I can resolve the problem. Can you direct me how to proceed? Details: zipimport.ZipImportError Message=not a Zip file StackTrace: :88 in __init__ Replicating the problem is simple. Download jpype from git, use "python setup.py develop" to create the module, run python -c "import jpype", and repeat python -c "import jpype" Screen shot of the visual studio error is shown. -- Added file: https://bugs.python.org/file49665/Capture.PNG ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42529] CPython DLL initialization routine failed from PYC cache file
New submission from Karl Nelson : While trying to use JPype on Windows Python 3.9.0, we are running into an bizarre issue with loading the internal module which is written in C. When running a python script the first time the internal module loads correctly. However, the second time that script is run the internal module reports "A dynamic link library (DLL) initialization routine failed." If you then erase the pyc cache file that is importing the internal module then it works again. This only occurs on Windows and was not present using the same source prior versions of Python 3.8. We investigate the byte codes from both version and they are doing the same series of actions so the problem appears to be calling the same opcode to execute an import. I make sure all required symbols were found in the libraries and only only copy of the internal DLL was the same with and without loading from a pyc. It may be a change in the requirements of module initialization, but I don't know how to proceed. There was one deprecation warning but correcting that did not alter the outcome. It appears that the execute path for importing a CPython module takes a different path when the script was imported from something compiled on the fly as opposed to loading from a pyc. ``` import: 'jpype' Traceback (most recent call last): File "D:\bld\jpype1_1605785280189\test_tmp\run_test.py", line 2, in import jpype File "D:\bld\jpype1_1605785280189\_test_env\lib\site-packages\jpype\__init__.py", line 18, in import _jpype ImportError: DLL load failed while importing _jpype: A dynamic link library (DLL) initialization routine failed. ``` -- components: Windows messages: 382275 nosy: Thrameos, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: CPython DLL initialization routine failed from PYC cache file type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue42529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39416] Document default numeric string formats
Change by Karl O. Pinc : -- pull_requests: +21809 pull_request: https://github.com/python/cpython/pull/22867 ___ Python tracker <https://bugs.python.org/issue39416> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42104] xml.etree should support contains() function
New submission from karl : In XPath 1.0 The function contains() is available > Function: boolean contains(string, string) > The contains function returns true if the first argument string contains the > second argument string, and otherwise returns false. In https://www.w3.org/TR/1999/REC-xpath-19991116/#function-contains ``` One attribute: doc Two Attributes: doc test One Attribute: test ``` Currently, we can do this ``` >>> from lxml import etree >>> root = etree.fromstring(""" ...One attribute ...Two Attributes: doc test ...Two Attributes: doc2 test ... ... """) >>> elts = root.xpath("//p[@class='doc']") >>> elts, etree.tostring(elts[0]) ([], b'One attribute\n ') ``` One way of extracting the list of 2 elements which contains the attribute doc with XPath is: ``` >>> root.xpath("//p[contains(@class, 'doc')]") [, ] >>> [etree.tostring(elt) for elt in root.xpath("//p[contains(@class, 'doc')]")] [b'One attribute: doc\n ', b'Two Attributes: doc test\n '] ``` There is no easy way to extract all elements containing a "doc" value in a multi-values attribute in python 3.10 with xml.etree, which is quite common in html. ``` >>> import xml.etree.ElementTree as ET >>> root = ET.fromstring(""" ...One attribute: doc ...Two Attributes: doc test ...One Attribute: test ... """ ... ) >>> root.xpath("//p[contains(@class, 'doc')]") Traceback (most recent call last): File "", line 1, in AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'xpath' ``` -- components: Library (Lib) messages: 379185 nosy: karlcow priority: normal severity: normal status: open title: xml.etree should support contains() function type: enhancement versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue42104> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40236] datetime.datetime.strptime get day error
karl added the comment: Also this. >>> import datetime >>> d0 = datetime.datetime.strptime("2024-0-3 00:00:00", "%Y-%W-%w %H:%M:%S") >>> d0.strftime("%Y-%W-%w %H:%M:%S") '2024-01-3 00:00:00' >>> d1 = datetime.datetime.strptime("2024-1-3 00:00:00", "%Y-%W-%w %H:%M:%S") >>> d1.strftime("%Y-%W-%w %H:%M:%S") '2024-01-3 00:00:00' >>> d2301 = datetime.datetime.strptime("2023-0-1 00:00:00", "%Y-%W-%w %H:%M:%S") >>> d2311 = datetime.datetime.strptime("2023-1-1 00:00:00", "%Y-%W-%w %H:%M:%S") >>> d2301 datetime.datetime(2022, 12, 26, 0, 0) >>> d2311 datetime.datetime(2023, 1, 2, 0, 0) >>> d2311.strftime("%Y-%W-%w %H:%M:%S") '2023-01-1 00:00:00' >>> d2301.strftime("%Y-%W-%w %H:%M:%S") '2022-52-1 00:00:00' Week 0 2023 became Week 52 2022 (which is correct but might lead to surprises) -- ___ Python tracker <https://bugs.python.org/issue40236> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40236] datetime.datetime.strptime get day error
karl added the comment: Same on macOS 10.15.6 (19G73) Python 3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) [Clang 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import datetime >>> datetime.datetime.strptime("2024-0-3 00:00:00", "%Y-%W-%w %H:%M:%S") datetime.datetime(2024, 1, 3, 0, 0) >>> datetime.datetime.strptime("2024-1-3 00:00:00", "%Y-%W-%w %H:%M:%S") datetime.datetime(2024, 1, 3, 0, 0) Also https://pubs.opengroup.org/onlinepubs/007908799/xsh/strptime.html note that iso8601 doesn't have this issue. %V - ISO 8601 week of the year as a decimal number [01, 53]. https://en.wikipedia.org/wiki/ISO_week_date -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue40236> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36661] Missing dataclass decorator import in dataclasses module docs
karl added the comment: This should be closed. The PR has been merged and the doc is now up to date. -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue36661> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40297] test_socket.CANTest is broken at HEAD on master
Karl Ding added the comment: Related: I've started a thread on Discourse [0] looking into why the tests don't seem to be run on the Buildbot cluster (and what it would take to enable them). Hopefully it gains some traction. [0] https://discuss.python.org/t/what-would-it-take-to-run-socketcan-tests-on-a-buildbot/4137 -- ___ Python tracker <https://bugs.python.org/issue40297> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40291] socket library support for CAN_J1939
Karl Ding added the comment: Should this be added to the What's New for 3.9? I see a smaller change that exposes the CAN_RAW_JOIN_FILTERS constant mentioned. I've created https://github.com/python/cpython/pull/20248 if this is needed. -- ___ Python tracker <https://bugs.python.org/issue40291> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40291] socket library support for CAN_J1939
Change by Karl Ding : -- pull_requests: +19536 pull_request: https://github.com/python/cpython/pull/20248 ___ Python tracker <https://bugs.python.org/issue40291> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40297] test_socket.CANTest is broken at HEAD on master
Change by Karl Ding : -- keywords: +patch pull_requests: +18894 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19548 ___ Python tracker <https://bugs.python.org/issue40297> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40297] test_socket.CANTest is broken at HEAD on master
New submission from Karl Ding : While working on https://bugs.python.org/issue40291, I was trying to run the SocketCAN tests to ensure that my changes weren't causing any regressions. However, I was seeing test failures at HEAD. I'm running the tests like so: # Kernel version uname -r # 5.4.23-1-MANJARO # Load SocketCAN vcan kernel module sudo modprobe vcan # Start and set up a virtual SocketCAN interface sudo ip link add dev vcan0 type vcan sudo ip link set up vcan0 # Run the socket tests using locally built python ./python -m unittest -v test.test_socket.CANTest After bisecting, I discovered that the test started failing back in 2017, with the introduction of the ISOTP protocol. https://github.com/python/cpython/pull/2956/files#diff-a47fd74731aeb547ad780900bb8e6953L1376-R1391 Here, the address family (the second tuple item) is explicitly removed: diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index bf8d19fe2f..beadecfad5 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1373,9 +1373,22 @@ makesockaddr(SOCKET_T sockfd, struct sockaddr *addr, size_t addrlen, int proto) ifname = ifr.ifr_name; } -return Py_BuildValue("O&h", PyUnicode_DecodeFSDefault, -ifname, -a->can_family); +switch (proto) { +#ifdef CAN_ISOTP + case CAN_ISOTP: + { + return Py_BuildValue("O&kk", PyUnicode_DecodeFSDefault, + ifname, + a->can_addr.tp.rx_id, + a->can_addr.tp.tx_id); + } +#endif + default: + { + return Py_BuildValue("O&", PyUnicode_DecodeFSDefault, +ifname); + } +} } #endif This seems to be an intentional breakage, since the code in getsockaddrarg also operates on just the interface name, ignoring the address family. if (!PyArg_ParseTuple(args, "O&;AF_CAN address must be a tuple " "(interface, )", PyUnicode_FSConverter, &interfaceName)) As such, I believe the test should have been updated, but was missed during the ISOTP changes. Can someone (a core member?) confirm that this is the correct approach? Note: However, this also implies that the CANTest test was never being run (either via CI on PRs or on the Buildbot cluster) and instead was always skipped, since I would've expected this failure to show up right away... -- components: Library (Lib) messages: 366576 nosy: karlding priority: normal severity: normal status: open title: test_socket.CANTest is broken at HEAD on master type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue40297> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40291] socket library support for CAN_J1939
Change by Karl Ding : -- keywords: +patch pull_requests: +18886 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19538 ___ Python tracker <https://bugs.python.org/issue40291> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40291] socket library support for CAN_J1939
New submission from Karl Ding : It would be nice to have support J1939 sockets. Support for J1939 landed as part of the SAE J1939 SocketCAN patches, which are available after the Linux 5.4rc1 kernel. This is another protocol family for SocketCAN, much like the existing support for ISOTP and BCM. Effectively, supporting this would hand off as much to the kernel as possible, which gives Python programs the ability deal with J1939 without having to implement the logic existing in the kernel in userspace. This is provided to userspace [0] via the header. I'd be happy to work on this and provide a PR. [0] https://www.kernel.org/doc/html/latest/networking/j1939.html -- components: Library (Lib) messages: 366487 nosy: karlding priority: normal severity: normal status: open title: socket library support for CAN_J1939 type: enhancement versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue40291> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39416] Document default numeric string formats
Karl O. Pinc added the comment: On Wed, 22 Jan 2020 06:09:41 + "Eric V. Smith" wrote: > Eric V. Smith added the comment: > > Is the lack of this documentation causing some confusion somewhere? > This isn't rhetorical, I'm genuinely curious what problem you're > trying to solve. I'd say no, there's no real confusion anywhere. This started with a search for documentation on Python's default string representation for float. This accidentally wound up on the python-dev mailing list and became a more general discussion during which I suggested that some fundamentals are clear. This is pretty much just a follow-up to that discussion. > Is there any mainstream programming language where the basics of what > you've laid out aren't true? It all seems pretty obvious to me. For > example, I can't see anyone looking at this and saying "Ah, base 10. > That's why it's producing the output I'm seeing." Well, I just got done looking at YAML which has a lot of base-related syntax including base 60. :) The problem that would be solved is that it's easy to rely on the default Numeric output formats. Most output probably does not go through a formatter, although this may change now that f-strings are so easy. Anyway, this would guarantee "normal output" for numbers, even when switching between Python implementations. Likewise, immutability of value when round-tripping through a string is also obvious. But is still an important property. So if these properties are important then include them in the spec. That's what specs are for. (I could probably find some non-mainstream languages (scheme perhaps) that have specs which include similar documentation. You pretty have to have a formatter in compiled languages, so those won't specify a default. Anyway, it shouldn't matter what other language specs do.) I understand if nobody sees this as a real problem. And I don't want to stand up as a big proponent. I'll point out the argument here and let others decide. Regards, Karl Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein -- ___ Python tracker <https://bugs.python.org/issue39416> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39416] Document default numeric string formats
Change by Karl O. Pinc : -- keywords: +patch pull_requests: +17498 stage: -> patch review pull_request: https://github.com/python/cpython/pull/18111 ___ Python tracker <https://bugs.python.org/issue39416> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39416] Document default numeric string formats
New submission from Karl O. Pinc : Seems sane to put _some_ restrictions on the string representations of the Numeric classes. This would be a change to the Python language specification. Suggestions made in a pull request. See the email thread: Subject: Documenting Python's float.__str__() https://mail.python.org/archives/list/python-...@python.org/thread/FV22TKT3S2Q3P7PNN6MCXI6IX3HRRNAL/ -- assignee: docs@python components: Documentation messages: 360442 nosy: docs@python, kop priority: normal severity: normal status: open title: Document default numeric string formats ___ Python tracker <https://bugs.python.org/issue39416> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1375011] http.cookies, Cookie.py: Improper handling of duplicate cookies
karl added the comment: Relevant spec https://tools.ietf.org/html/rfc6265 -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue1375011> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9004] datetime.utctimetuple() should not set tm_isdst flag to 0
karl added the comment: @gaurav The pull request https://github.com/python/cpython/pull/10870 has been closed in favor of https://github.com/python/cpython/pull/15773 which has already been merged. So we can probably close here. -- message_count: 7.0 -> 8.0 nosy: +karlcow nosy_count: 7.0 -> 8.0 pull_requests: +16145 pull_request: https://github.com/python/cpython/pull/15773 ___ Python tracker <https://bugs.python.org/issue9004> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19683] test_minidom has many empty tests
karl added the comment: err… Errata on my previous comment. """ Simple implementation of the Level 1 DOM. Namespaces and other minor Level 2 features are also supported. """ https://github.com/python/cpython/blob/c65119d5bfded03f80a9805889391b66fa7bf551/Lib/xml/dom/minidom.py#L1-L3 https://www.w3.org/TR/REC-DOM-Level-1/ -- ___ Python tracker <https://bugs.python.org/issue19683> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19683] test_minidom has many empty tests
karl added the comment: @zach.ware @r.david.murray So I was looking at that issue. There is a lot of work. I had a couple of questions, because there are different categories # Empty tests for existing functions. This seems to be straightforward as they would complete the module. Example: ```python def testGetAttributeNode(self): pass ``` https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/test/test_minidom.py#L412 which refers to: `GetAttributeNode` https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/xml/dom/minidom.py#L765-L768 https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/test/test_minidom.py#L285-L294 # Tests without any logical reference in the module. This is puzzling because I'm not sure which DOM feature they should be testing. For example: ``` def testGetAttrList(self): pass ``` https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/test/test_minidom.py#L383-L384 Or maybe this is just supposed to test Element.attributes returning a list of attributes, such as `NamedNodeMap [ def="ghi", jkl="mno"]` returned by a browser. ``` >>> import xml.dom.minidom >>> from xml.dom.minidom import parse, Node, Document, parseString >>> from xml.dom.minidom import getDOMImplementation >>> dom = parseString("") >>> el = dom.documentElement >>> el.setAttribute("def", "ghi") >>> el.setAttribute("jkl", "mno") >>> el.attributes ``` or is it supposed to test something like ``` >>> el.attributes.items() [('def', 'ghi'), ('jkl', 'mno')] ``` This is slightly confusing. And the missing docstrings are not making it easier. # Tests which do not really test the module(?) I think for example about this, which is testing that `del` is working, but it doesn't have anything to do with the DOM. ``` def testDeleteAttr(self): dom = Document() child = dom.appendChild(dom.createElement("abc")) self.confirm(len(child.attributes) == 0) child.setAttribute("def", "ghi") self.confirm(len(child.attributes) == 1) del child.attributes["def"] self.confirm(len(child.attributes) == 0) dom.unlink() ``` https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/test/test_minidom.py#L285-L294 Specifically when there is a function for it: `removeAttribute` https://www.w3.org/TR/2000/REC-DOM-Level-2-Core-20001113/core.html#ID-6D6AC0F9 which is tested just below that test. https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/test/test_minidom.py#L296-L305 so I guess these should be removed or do I miss something in the testing logic? # Missing docstrings. Both the testing module and the module lack a lot of docstrings. Would it be good to fix this too, probably in a separate commit. # DOM Level 2 So the module intent is to implement DOM Level 2. but does that make sense in the light of https://dom.spec.whatwg.org/ Should minidom tries to follow the current DOM spec? -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue19683> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38352] In typing docs, note explicit import needed for IO and Pattern/Match
New submission from Karl Kornel : Hello! In https://github.com/python/cpython/blob/master/Lib/typing.py#L115-L117, there is a note about the io and re classes not being included in typing.__all__. I am a relatively new user of typing, and I did `from typing import *` in my code. I ran the code through mypy first, which reported no problems, but then running Python 3.6 failed with a NameError (name 'IO' is not defined). Reading through the typing source, it's clear that this was an intentional decision. So, instead of reporting a bug, I'd like to request a documentation enhancement! The docs for typing make no mention of typing.io or typing.re. So, my request is: In the sections for the IO/TextIO/BinaryIO and Pattern/Match classes, include text warning the user that these types are not imported when you do `from typing import *`. -- assignee: docs@python components: Documentation messages: 353763 nosy: Karl Kornel, docs@python priority: normal severity: normal status: open title: In typing docs, note explicit import needed for IO and Pattern/Match type: behavior versions: Python 3.6 ___ Python tracker <https://bugs.python.org/issue38352> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22377] %Z in strptime doesn't match EST and others
karl added the comment: I created a PR following the recommendations of p-ganssle https://github.com/python/cpython/pull/16507 -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue22377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18182] xml.dom.createElement() does not take implicit namespaces into account
karl added the comment: The current specification as of today documents https://dom.spec.whatwg.org/#dom-document-createelementns If you run this in the browser console, var nsdoc = 'http://foo.bar/zoo'; var xmldoc = document.implementation.createDocument(nsdoc, 'Zoo', null); var cpd = document.createElementNS(nsdoc, 'Compound'); var chimp = document.createElementNS(nsdoc, 'Chimp'); cpd.appendChild(chimp) xmldoc.documentElement.appendChild(cpd); /* serializing */ var docserializer = new XMLSerializer(); var flatxml = docserializer.serializeToString(xmldoc); flatxml you get: http://foo.bar/zoo";> but if you run this in the browser console, var nsdoc = 'http://foo.bar/zoo'; var xmldoc = document.implementation.createDocument(nsdoc, 'Zoo', null); var cpd = document.createElement('Compound'); var chimp = document.createElement('Chimp'); cpd.appendChild(chimp) xmldoc.documentElement.appendChild(cpd); /* serializing */ var docserializer = new XMLSerializer(); var flatxml = docserializer.serializeToString(xmldoc); flatxml you get: http://foo.bar/zoo";> http://www.w3.org/1999/xhtml";> which is a complete different beast. I don't think there is an issue here. And we can close this bug safely. -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue18182> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35638] Introduce fixed point locale aware format type for floating point numbers
Change by Karl Ove Hufthammer : -- nosy: +huftis ___ Python tracker <https://bugs.python.org/issue35638> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23312] google thinks the docs are mobile unfriendly
karl added the comment: I created https://github.com/python/python-docs-theme/issues/30 -- ___ Python tracker <https://bugs.python.org/issue23312> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23312] google thinks the docs are mobile unfriendly
karl added the comment: This issue should probably be addressed now on https://github.com/python/python-docs-theme -- nosy: +karlcow ___ Python tracker <https://bugs.python.org/issue23312> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24772] Smaller viewport shifts the "expand left menu" character into the text
karl added the comment: So I had time to look at it today. And it would probably be better to solve https://bugs.python.org/issue23312 which would make this one here useless and would actually provide a solution for many people. -- ___ Python tracker <https://bugs.python.org/issue24772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24772] Smaller viewport shifts the "expand left menu" character into the text
karl added the comment: I'm at Mozilla All Hands this week. I'll check if my solution still makes sense next week and will make a pull request and/or propose another solution. Thanks for the reminder. adding to my calendar. -- ___ Python tracker <https://bugs.python.org/issue24772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29753] Ctypes Packing Bitfields Incorrectly - Linux
Karl Ding added the comment: I believe the example can be simplified to the following: class MyStructure(ctypes.Structure): _pack_ = 1 _fields_ = [('A', ctypes.c_uint32, 8)] assert ctypes.sizeof(MyStructure) == 1 Here, ctypes.sizeof returns 4 on my machine (64-bit Ubuntu 18.04 LTS), while I would expect it to return 1 like GCC. This is using a tree checked out at 33ce3f012f249782507df896824b045b34f765aa, if it makes a difference. If we compile the equivalent C code (on 64-bit Ubuntu 18.04 LTS): #include #include #include struct my_structure_uint32 { uint32_t a : 8; } __attribute__((packed)); int main(int argc, char *argv[]) { printf("sizeof(struct my_structure_uint32): %d\n", sizeof(struct my_structure_uint32)); printf("alignof(struct my_structure_uint32): %d\n", alignof(struct my_structure_uint32)); return 0; } Using the following GCC invocation: gcc temp.c -o test && ./test We get the following result: sizeof(struct my_structure_uint32): 1 alignof(struct my_structure_uint32): 1 However, if I compile with MSVC bitfields: gcc -mms-bitfields test.c -o test && ./test We get the following result: sizeof(struct my_structure_uint32): 4 alignof(struct my_structure_uint32): 1 Similarly, adding a second and third 8-bit wide bitfield member fails and returns 4, instead of the expected 2 and 3. This is not just c_uint32, c_uint16 and c_int also exhibit the same behaviour: class MyStructure(ctypes.Structure): _pack_ = 1 _fields_ = [('A', ctypes.c_uint16, 8)] assert ctypes.sizeof(MyStructure) == 1 -- nosy: +karlding ___ Python tracker <https://bugs.python.org/issue29753> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37085] Expose additional socket constants for CAN_BCM flags
Change by Karl Ding : -- keywords: +patch pull_requests: +13541 stage: -> patch review pull_request: https://github.com/python/cpython/pull/13646 ___ Python tracker <https://bugs.python.org/issue37085> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37085] Expose additional socket constants for CAN_BCM flags
New submission from Karl Ding : When reading through the values exposed via the socket library, I noticed that currently, only the SocketCAN BCM opcode enums are exposed via the socket constants. These correspond to the following from the Linux headers: enum { TX_SETUP = 1, /* create (cyclic) transmission task */ TX_DELETE, /* remove (cyclic) transmission task */ TX_READ,/* read properties of (cyclic) transmission task */ TX_SEND,/* send one CAN frame */ RX_SETUP, /* create RX content filter subscription */ RX_DELETE, /* remove RX content filter subscription */ RX_READ,/* read properties of RX content filter subscription */ TX_STATUS, /* reply to TX_READ request */ TX_EXPIRED, /* notification on performed transmissions (count=0) */ RX_STATUS, /* reply to RX_READ request */ RX_TIMEOUT, /* cyclic message is absent */ RX_CHANGED /* updated CAN frame (detected content change) */ }; It would be nice to expose the BCM flags #defines as well. #define SETTIMER0x0001 #define STARTTIMER 0x0002 #define TX_COUNTEVT 0x0004 #define TX_ANNOUNCE 0x0008 #define TX_CP_CAN_ID0x0010 #define RX_FILTER_ID0x0020 #define RX_CHECK_DLC0x0040 #define RX_NO_AUTOTIMER 0x0080 #define RX_ANNOUNCE_RESUME 0x0100 #define TX_RESET_MULTI_IDX 0x0200 #define RX_RTR_FRAME0x0400 #define CAN_FD_FRAME0x0800 These BCM flags are used as part of the BCM header that has the aforementioned opcodes. The flags are set on the bcm_msg_head struct: struct bcm_msg_head { __u32 opcode; __u32 flags; __u32 count; struct bcm_timeval ival1, ival2; canid_t can_id; __u32 nframes; struct can_frame frames[0]; }; The existing documentation for the BCM constants (https://docs.python.org/3/library/socket.html#socket.CAN_BCM) seems to imply that these constants should already be included, but they are not. See the Linux headers for more details: https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/can/bcm.h -- components: Library (Lib) messages: 343865 nosy: karlding priority: normal severity: normal status: open title: Expose additional socket constants for CAN_BCM flags type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue37085> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8136] urllib.unquote decodes percent-escapes with Latin-1
karl added the comment: #8143 was fixed. Python 2.7.10 (default, Feb 7 2017, 00:08:15) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import urllib >>> urllib.unquote(u"%CE%A3") u'\xce\xa3' What should become of this one? -- nosy: +karlcow ___ Python tracker <http://bugs.python.org/issue8136> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24712] Docs page's sidebar vibrates on mouse wheel scroll on Chrome.
karl added the comment: @ezio.melotti What is the next step for getting the patch accepted. -- nosy: +karlcow ___ Python tracker <http://bugs.python.org/issue24712> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2943] Distutils should generate a better error message when the SDK is not installed
Changes by Karl Stahl : -- nosy: +Karl Stahl ___ Python tracker <http://bugs.python.org/issue2943> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24772] Smaller viewport shifts the "expand left menu" character into the text
karl added the comment: Where should I propose a patch to help resolve this issue? A pointer to the code in the repo and I will do it. -- ___ Python tracker <http://bugs.python.org/issue24772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15873] datetime: add ability to parse RFC 3339 dates and times
karl added the comment: About > Actually, I realized that the best implementation of parsing rfc 3339 > is in django dateparse utils. To me it's the finest, the most > elegant, and no other one can claim to be more robust since it's > probably the #1 iso parsing functions used in python. Have a look at > https://docs.djangoproject.com/en/1.9/_modules/django/utils/dateparse/#parse_datetime How does it parse this date: 2016-02-15T11:59:46.16588638674+09:00 -- ___ Python tracker <http://bugs.python.org/issue15873> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15873] datetime: add ability to parse RFC 3339 dates and times
karl added the comment: >From https://www.djangoproject.com/foundation/cla/faq/ > Am I giving away the copyright to my contributions? > > No. This is a pure license agreement, not a copyright assignment. You > still maintain the full copyright for your contributions. You are > only providing a license to the DSF to distribute your code without > further restrictions. This is not the case for all CLA's, but it is > the case for the one we are using. -- ___ Python tracker <http://bugs.python.org/issue15873> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26011] Document necesities for cmp argument of sorted
New submission from Karl Richter: The docstring of `sorted` doesn't explain what can be passed to the `cmp` and `key` argument of `sorted`. -- assignee: docs@python components: Documentation messages: 257505 nosy: docs@python, krichter priority: normal severity: normal status: open title: Document necesities for cmp argument of sorted type: enhancement versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue26011> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25946] configure should pick /usr/bin/g++ automatically if present
New submission from Karl Richter: `./configure` both prints `checking for g++... no` and WARNING: By default, distutils will build C++ extension modules with "g++". If this is not intended, then set CXX on the configure command line. if `/usr/bin/g++` is present and executable which doesn't seem to be constructive because it's quite common that one wants to use `/usr/bin/g++` as CXX compiler if available. In case incompatibilities exists with other C++ compilers there should a check and more detailed error message. Furthermore the error message doesn't explain if a part of distutils won't be build because the message sounds like the C++ extension is built, but does it work without a C++ compiler? Specifying `CXX` environment variable or `--with-cxx-main=/usr/bin/g++` `configure` option works fine. experienced with 8a2e735 (on branch 2.7) -- components: Build messages: 256977 nosy: krichter priority: normal severity: normal status: open title: configure should pick /usr/bin/g++ automatically if present versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue25946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25787] Add an explanation what happens with subprocess parent and child processes when signals are sent
Karl Richter added the comment: Please also explain how to deal with process replacement in child processes (assuming that http://stackoverflow.com/questions/34059576/how-to-register-a-signal-handler-for-a-subprocess/34065587#34065587 is correct). -- ___ Python tracker <http://bugs.python.org/issue25787> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25787] Add an explanation what happens with subprocess parent and child processes when signals are sent
New submission from Karl Richter: The [documentation of subprocess](https://docs.python.org/3.6/library/subprocess.html) doesn't contain a substantial statement how signals are handled which are send to the python interpreter. After reading the referenced docs it should be clear * whether a signal is passed to both the parent and the child (If yes in which order? What happens if the child process spawns a process which isn't controlled by python?) * whether signal handlers are inherited (judging from the `restore_signals` parameter some are overwritten -> what's the purpose of this?). Are changes of a signal handler in the parent reflected in the child? -- assignee: docs@python components: Documentation messages: 255802 nosy: docs@python, krichter priority: normal severity: normal status: open title: Add an explanation what happens with subprocess parent and child processes when signals are sent versions: Python 3.6 ___ Python tracker <http://bugs.python.org/issue25787> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25377] Mention octal format of mode argument of os.chmod
New submission from Karl Richter: `help(os.chmod)` doesn't explain that it expects an octal number for its `mode` argument. This is very confusing for beginners and people who are not involved with filesystems and permissions every day. It doesn't need to be explained, just mentioned, maybe with a link to an extended explanation. Example for confusion: http://stackoverflow.com/questions/15607903/python-module-os-chmodfile-664-does-not-change-the-permission-to-rw-rw-r-bu -- assignee: docs@python components: Documentation messages: 252817 nosy: docs@python, krichter priority: normal severity: normal status: open title: Mention octal format of mode argument of os.chmod type: enhancement versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue25377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24772] Smaller viewport shifts the "expand left menu" character into the text
New submission from karl: Adding the following to basic.css: ```css dl { margin-bottom: 15px; word-wrap: break-word; } ``` will solve the issue. See https://github.com/webcompat/web-bugs/issues/1479 -- assignee: docs@python components: Documentation messages: 247803 nosy: docs@python, karlcow priority: normal severity: normal status: open title: Smaller viewport shifts the "expand left menu" character into the text ___ Python tracker <http://bugs.python.org/issue24772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24506] make fails with gcc 4.9 due to fatal warning of unused variable and empty macro in Parser/pgen.c
Karl Richter added the comment: It's a fatal warning of `gcc 4.9.2`, not an error (my bad) for `int i;` in `Parser/pgen.c` line 227. It might be ignored as well, but I think my approach is more elegant and deals with issues sooner than later. -- ___ Python tracker <http://bugs.python.org/issue24506> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24506] make fails with gcc 4.9 due to fatal warning of unused variable and empty macro in Parser/pgen.c
New submission from Karl Richter: `gcc` 4.9 is more restictive and recognizes that the empty definition of the `REQN` macro doesn't use some variables. It's more suitable to wrap the usage of the macro in the same preprocessor conditionals like the macro definition. experienced with 41fbc222af8c503e1659250a36f4e293d864a92b Patch attached. Tests pass (see https://semaphoreci.com/krichter/cpython/branches/empty_macro/builds/1 for details). -- components: Build files: 0001-removed-empty-definition-of-REQN-preprocessor-macro-.patch keywords: patch messages: 245781 nosy: krichter priority: normal severity: normal status: open title: make fails with gcc 4.9 due to fatal warning of unused variable and empty macro in Parser/pgen.c versions: Python 2.7 Added file: http://bugs.python.org/file39806/0001-removed-empty-definition-of-REQN-preprocessor-macro-.patch ___ Python tracker <http://bugs.python.org/issue24506> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24331] *** Error in `/usr/bin/python': double free or corruption (!prev): 0x0000000000f5c760 *** when cloning hg repository into directory on cifs
New submission from Karl Richter: I experience the error in the title exclusive when invoking `hg clone` (e.g. `hg clone https://bitbucket.org/Coin3D/coin` or `hg clone http://hg.netbeans.org/main-golden/ netbeans-main-golden`) when the target directory is on a cifs mount. `gdb` backtrace: #0 0x7761c267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x7761deca in __GI_abort () at abort.c:89 #2 0x7765fc53 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x777781a8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x77667c69 in malloc_printerr (ptr=, str=0x777782d8 "double free or corruption (!prev)", action=1) at malloc.c:4965 #4 _int_free (av=, p=, have_lock=0) at malloc.c:3834 #5 0x7766b89c in __GI___libc_free (mem=) at malloc.c:2950 #6 0x7130ce11 in lfree (a=a@entry=0xecde10) at mercurial/mpatch.c:67 #7 0x7130d2e7 in decode (len=196, bin=0x1037ee8 "") at mercurial/mpatch.c:228 #8 fold (bins=[], start=start@entry=0, end=end@entry=1) at mercurial/mpatch.c:296 #9 0x7130d3d3 in patches (self=, args=) at mercurial/mpatch.c:327 #10 0x004ccd05 in call_function (oparg=, pp_stack=) at ../Python/ceval.c:4035 #11 PyEval_EvalFrameEx () at ../Python/ceval.c:2681 ---Type to continue, or q to quit--- #12 0x004cd4e2 in fast_function (nk=, na=, n=, pp_stack=, func=) at ../Python/ceval.c:4121 #13 call_function (oparg=, pp_stack=) at ../Python/ceval.c:4056 #14 PyEval_EvalFrameEx () at ../Python/ceval.c:2681 #15 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267 #16 0x004ce7d3 in fast_function (nk=, na=, n=, pp_stack=, func=) at ../Python/ceval.c:4131 #17 call_function (oparg=, pp_stack=) at ../Python/ceval.c:4056 #18 PyEval_EvalFrameEx () at ../Python/ceval.c:2681 #19 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267 #20 0x004ce7d3 in fast_function (nk=, na=, n=, pp_stack=, func=) at ../Python/ceval.c:4131 #21 call_function (oparg=, pp_stack=) at ../Python/ceval.c:4056 #22 PyEval_EvalFrameEx () at ../Python/ceval.c:2681 #23 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267 #24 0x004ce7d3 in fast_function (nk=, na=, n=, pp_stack=, ---Type to continue, or q to quit--- func=) at ../Python/ceval.c:4131 #25 call_function (oparg=, pp_stack=) at ../Python/ceval.c:4056 #26 PyEval_EvalFrameEx () at ../Python/ceval.c:2681 #27 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267 #28 0x004cd217 in fast_function (nk=, na=, n=, pp_stack=, func=) at ../Python/ceval.c:4131 #29 call_function (oparg=, pp_stack=) at ../Python/ceval.c:4056 #30 PyEval_EvalFrameEx () at ../Python/ceval.c:2681 #31 0x004cd4e2 in fast_function (nk=, na=, n=, pp_stack=, func=) at ../Python/ceval.c:4121 #32 call_function (oparg=, pp_stack=) at ../Python/ceval.c:4056 #33 PyEval_EvalFrameEx () at ../Python/ceval.c:2681 #34 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267 #35 0x004cd217 in fast_function (nk=, na=, n=, pp_stack=, func=) at ../Python/ceval.c:4131 #36 call_function (oparg=, pp_stack=) at ../Python/ceval.c:4056 ---Type to continue, or q to quit--- #37 PyEval_EvalFrameEx () at ../Python/ceval.c:2681 #38 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267 #39 0x004cd217 in fast_function (nk=, na=, n=, pp_stack=, func=) at ../Python/ceval.c:4131 #40 call_function (oparg=, pp_stack=) at ../Python/ceval.c:4056 #41 PyEval_EvalFrameEx () at ../Python/ceval.c:2681 #42 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267 #43 0x004cd217 in fast_function (nk=, na=, n=, pp_stack=, func=) at ../Python/ceval.c:4131 #44 call_function (oparg=, pp_stack=) at ../Python/ceval.c:4056 #45 PyEval_EvalFrameEx () at ../Python/ceval.c:2681 #46 0x004e7cc8 in PyEval_EvalCodeEx (closure=, defcount=, defs=, kwcount=, kws=, argcount=, args=, locals=, globals=, co=) at ../Python/ceval.c:3267 #47 function_call.lto_priv () at ../Objects/funcobject.c:526 #48 0x004cf239 in PyObject_Call (kw=, arg=, func=) at ../Objects/abstract.c:2529 ---Type to continue, or q to quit--- #49 ext_do_call (nk=, na=, flags=, pp_stack=, func=) at ../Python/ceval.c:4348 #50 PyEval_EvalFrameEx (
[issue24143] Makefile in tarball don't provide make uninstall target
Changes by Karl Richter : -- components: Build nosy: krichter priority: normal severity: normal status: open title: Makefile in tarball don't provide make uninstall target versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue24143> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24060] Clearify necessities for logging with timestamps
New submission from Karl Richter: The `Formatter` section of the `logging` module at https://docs.python.org/2/library/logging.html#formatter-objects reads like it's sufficient to create an instance of `Formatter` with default arguments (and set it as formatter of the `Handler` of a `Logger`) to have add timestamps to logging output. -- assignee: docs@python components: Documentation messages: 242024 nosy: docs@python, krichter priority: normal severity: normal status: open title: Clearify necessities for logging with timestamps versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue24060> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24021] document urllib.urlretrieve
Karl Richter added the comment: > I suspect the complaint might be about the lack of doc string Exactly. It'd be helpful to figure out the return value and the means of the function arguments in a more compact form than the referenced website docs and to have it available in the interpreter (with `help`), i.e. with docstring documentation. -- ___ Python tracker <http://bugs.python.org/issue24021> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24021] document urllib.urlretrieve
Changes by Karl Richter : -- assignee: docs@python components: Documentation nosy: docs@python, krichter priority: normal severity: normal status: open title: document urllib.urlretrieve versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue24021> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23814] argparse: Parser level defaults do not always override argument level defaults
New submission from Karl O. Pinc: In the argparse library parser library, contrary to the documentation, parser-level defaults do not always override argument-level defaults. https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.set_defaults says "Note that parser-level defaults always override argument-level defaults:" (And so does the python 3.3 docs.) The docs then provide this example: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', default='bar') >>> parser.set_defaults(foo='spam') >>> parser.parse_args([]) Namespace(foo='spam') But it is only true that parser-level defaults override argument-level defaults when they are established after the argument is added. The output below shows an argument level default overrideing a parser level default. $ python3 Python 3.3.2 (default, Jun 4 2014, 11:36:37) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import argparse >>> parser = argparse.ArgumentParser() >>> parser.set_defaults(foo='spam') >>> parser.add_argument('--foo', default='bar') _StoreAction(option_strings=['--foo'], dest='foo', nargs=None, const=None, default='bar', type=None, choices=None, help=None, metavar=None) >>> parser.parse_args([]) Namespace(foo='bar') It seems that whichever default is set last is the one which is used. Or perhaps there are not argument level defaults and parser level defaults, there are just defaults, period. (It might, possibly, be nice if there _were_ both argument and parser level defaults and parser level defaults had priority. Then this would not be a documentation bug.) -- assignee: docs@python components: Documentation messages: 239632 nosy: docs@python, kop priority: normal severity: normal status: open title: argparse: Parser level defaults do not always override argument level defaults type: behavior versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue23814> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23174] shelve.open fails with error "anydbm.error: db type could not be determined"
Karl Richter added the comment: After checking the code, I think that it'd make more sense to document `whichdb.py`. It needs to be enhanced with references to criteria for the determination of the database type. Currently there're only function comments and the fact that some variables are named magic speaks for itself, i.e. it's ok for the implementation of the module to be a black box, but the requirements for the input, i.e. the potential database file, which is per se not part of the black box, needs to be specified. Then just link that in the `shelve` docs stating that `shelve.open` is basically a wrapper around `whichdb`. -- ___ Python tracker <http://bugs.python.org/issue23174> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23174] shelve.open fails with error "anydbm.error: db type could not be determined"
Karl Richter added the comment: For example, it should be clear why `shelve.open(tempfile.mkstemp()[1])` fails with the mentioned exception and `shelve.open("/tmp/bla")` fails. I still haven't figured out the constraints to create a working `shelve.Shelve` at all. It should be clear why `shelve.open("/tmp/tmphTTQLd")` fails and `shelve.open("/tmp/tmphTTQLda")` succeeds. There has to be something unrelated to extensions. -- ___ Python tracker <http://bugs.python.org/issue23174> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23163] pdb docs need to contain a statement on threads/multithreaded debugging
Karl Richter added the comment: Sorry, I mean #!/usr/bin/python import threading def debugging(): def __a_thread__(): print("2") a_thread = threading.Thread(target=__a_thread__) a_thread.start() a_thread.join() print("1") if __name__ == "__main__": debugging() It's very uncommon to set the current debugging thread in the source. -- ___ Python tracker <http://bugs.python.org/issue23163> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com