[issue46337] urllib.parse: Allow more flexibility in schemes and URL resolution behavior

[karl]

karl  added the comment:

Just to note that there is a maintained list of officially accepted schemes at 
IANA.
https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml

In addition there is a list of unofficial schemes on wikipedia 
https://en.wikipedia.org/wiki/List_of_URI_schemes#Unofficial_but_common_URI_schemes

--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue46337>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44423] copy2 / sendfile fails on linux with large file

[karl]

karl  added the comment:

could not reproduce the error

--
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue44423>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44423] copy2 / sendfile fails on linux with large file

[karl]

New submission from karl :

by copy a large file e.g.

-rwxrwxr-x 1 1002 1001 5359338160 Feb  9  2019 xxx_file_xxx.mdx

copy2 / sendfile / fastcopy fails with:


Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
  File "/usr/local/lib/python3.8/dist-packages/pybcpy/diff_bak_copy.py", line 
212, in _init_copy_single
shutil.copy2(f, dest_path)
  File "/usr/lib/python3.8/shutil.py", line 432, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.8/shutil.py", line 272, in copyfile
_fastcopy_sendfile(fsrc, fdst)
  File "/usr/lib/python3.8/shutil.py", line 169, in _fastcopy_sendfile
raise err
  File "/usr/lib/python3.8/shutil.py", line 149, in _fastcopy_sendfile
sent = os.sendfile(outfd, infd, offset, blocksize)
OSError: [Errno 75] Value too large for defined data type: 'xxx_file_xxx.mdx' 
-> 'dest/xxx_file_xxx.mdx'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/pybcpy/__main__.py", line 433, 
in 
main_func()
  File "/usr/local/lib/python3.8/dist-packages/pybcpy/__main__.py", line 425, 
in main_func
args.func(args)
  File "/usr/local/lib/python3.8/dist-packages/pybcpy/__main__.py", line 75, in 
cmd_init
) = dbak.init_backup_repo(tarmode=args.tar)
  File "/usr/local/lib/python3.8/dist-packages/pybcpy/diff_bak_copy.py", line 
231, in init_backup_repo
files = p.map(self._init_copy_single, ifiles)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
OSError: [Errno 75] Value too large for defined data type: 'xxx_file_xxx.mdx' 
-> 'dest/xxx_file_xxx.mdx'


reference to code:
https://github.com/kr-g/pybcpy/blob/master/pybcpy/diff_bak_copy.py

--
messages: 395862
nosy: kr-g
priority: normal
severity: normal
status: open
title: copy2 / sendfile fails on linux with large file
type: crash
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue44423>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44209] urllib.robotparser fail on Disallow: /? from google.com

[Karl Y. Pradene]

New submission from Karl Y. Pradene :

In robotparser.py
On line 222
path = urllib.parse.urlunparse(urllib.parse.urlparse(path))
tranform the entry Disallow: /?
in the google.com/robots.txt
in : Disallow: /
making every can_fetch request return False

--
components: Library (Lib)
messages: 394144
nosy: karl.pradene
priority: normal
severity: normal
status: open
title: urllib.robotparser fail on Disallow: /? from google.com
type: behavior
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue44209>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41748] HTMLParser: comma in attribute values with/without space

[karl]

karl  added the comment:

Status: The PR should be ready and completed
https://github.com/python/cpython/pull/24072
and eventually be merged at a point. 
Thanks to ezio.melotti for the wonderful guidance.

--

___
Python tracker 
<https://bugs.python.org/issue41748>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19683] test_minidom has many empty tests

[karl]

Change by karl :


--
pull_requests: +22980
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/24152

___
Python tracker 
<https://bugs.python.org/issue19683>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19683] test_minidom has many empty tests

[karl]

karl  added the comment:

Ah no. They ARE used

through defproperty and minicompat.py

get = getattr(klass, ("_get_" + name))

--

___
Python tracker 
<https://bugs.python.org/issue19683>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19683] test_minidom has many empty tests

[karl]

karl  added the comment:

These methods are not used anywhere in the code. 

https://github.com/python/cpython/blob/5c30145afb6053998e3518befff638d207047f00/Lib/xml/dom/minidom.py#L71-L80

What was the purpose when they were created… hmm maybe blame would give clue. 
Ah they were added a long time ago

https://github.com/python/cpython/commit/73678dac48e5858e40cba6d526970cba7e7c769c#diff-365c30899ded02b18a2d8f92de47af6ca213eefe7883064c8723598da600ea42R83-R88

but never used? or was it in the spirit to reserve the keyword for future use?
https://developer.mozilla.org/en-US/docs/Web/API/Node/firstChild

--

___
Python tracker 
<https://bugs.python.org/issue19683>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19683] test_minidom has many empty tests

[karl]

karl  added the comment:

@zach.ware 
@r.david.murray

I'm going through the source currently. 

I see that the test file is using:

class MinidomTest(unittest.TestCase):
def confirm(self, test, testname = "Test"):
self.assertTrue(test, testname)


Is there a specific reason to use this form instead of just directly using 
self.assertEqual or similar forms for new tests or reorganizing some of the 
tests. 


I see that it is used for example for giving a testname but

def testAAA(self):
dom = parseString("")
el = dom.documentElement
el.setAttribute("spam", "jam2")
self.confirm(el.toxml() == '', "testAAA")


testAAA is not specifically helping. :)

--

___
Python tracker 
<https://bugs.python.org/issue19683>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25258] HtmlParser doesn't handle void element tags correctly

[karl]

karl  added the comment:

I wonder if the confusion comes from the name. The HTMLParser is kind of a 
tokenizer more than a full HTML parser, but that's probably a detail. It 
doesn't create a DOM Tree which you can access, but could help you to build a 
DOM Tree (!= DOM Document object)

https://html.spec.whatwg.org/multipage/parsing.html#overview-of-the-parsing-model

> Implementations that do not support scripting do not have to actually create 
> a DOM Document object, but the DOM tree in such cases is still used as the 
> model for the rest of the specification.

--

___
Python tracker 
<https://bugs.python.org/issue25258>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25258] HtmlParser doesn't handle void element tags correctly

[karl]

karl  added the comment:

The parsing rules for tokenization of html are at 
https://html.spec.whatwg.org/multipage/parsing.html#tokenization

In the stack of open elements, there are specific rules for certain elements. 
https://html.spec.whatwg.org/multipage/parsing.html#special

from a DOM point of view, there is indeed no difference in between 


https://software.hixie.ch/utilities/js/live-dom-viewer/?%3C!DOCTYPE%20html%3E%0A%3Cimg%20src%3D%22somewhere%22%3E%3Cimg%20src%3D%22somewhere%22%2F%3E

--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue25258>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42821] HTMLParser: subsequent duplicate attributes should be ignored

[karl]

New submission from karl :

This comes up while working on issue 41748


browser input 
data:text/html,text

browser output
text

Actual HTMLParser output

see https://github.com/python/cpython/pull/24072#discussion_r551158342
('starttag', 'div', [('class', 'bar'), ('class', 'foo')])]

Expected HTMLParser output
('starttag', 'div', [('class', 'bar')])]

--
components: Library (Lib)
messages: 384308
nosy: karlcow
priority: normal
severity: normal
status: open
title: HTMLParser: subsequent duplicate attributes should be ignored
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue42821>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28937] str.split(): allow removing empty strings (when sep is not None)

[karl]

Change by karl :


--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue28937>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41748] HTMLParser: comma in attribute values with/without space

[karl]

Change by karl :


--
keywords: +patch
pull_requests: +22904
stage: test needed -> patch review
pull_request: https://github.com/python/cpython/pull/24072

___
Python tracker 
<https://bugs.python.org/issue41748>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41748] HTMLParser: comma in attribute values with/without space

[karl]

karl  added the comment:

Ah!

This is fixing it

diff --git a/Lib/html/parser.py b/Lib/html/parser.py
index 6083077981..790666 100644
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -44,7 +44,7 @@
   (?:\s*=+\s*# value indicator
 (?:'[^']*'   # LITA-enclosed value
   |"[^"]*"   # LIT-enclosed value
-  |(?!['"])[^>\s]*   # bare value
+  |(?!['"])[^>]*   # bare value
  )
  (?:\s*,)*   # possibly followed by a comma
)?(?:\s|/(?!>))*




Ran 48 tests in 0.175s

OK

== Tests result: SUCCESS ==

--

___
Python tracker 
<https://bugs.python.org/issue41748>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41748] HTMLParser: comma in attribute values with/without space

[karl]

Change by karl :


--
title: HTMLParser: parsing error -> HTMLParser: comma in attribute values 
with/without space

___
Python tracker 
<https://bugs.python.org/issue41748>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41748] HTMLParser: parsing error

[karl]

karl  added the comment:

Ezio,

TL,DR: Testing in browsers and adding two tests for this issue. 
   Should I create a PR just for the tests?

https://github.com/python/cpython/blame/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/test/test_htmlparser.py#L479-L485


A: comma without spaces
---


Tests for browsers:
data:text/html,text

Serializations:
* Firefox, Gecko (86.0a1 (2020-12-28) (64-bit)) 
* Edge, Blink (Version 89.0.752.0 (Version officielle) Canary (64 bits))
* Safari, WebKit (Release 117 (Safari 14.1, WebKit 16611.1.7.2))

Same serialization in these 3 rendering engines
text


Adding:

def test_comma_between_unquoted_attributes(self):
# bpo 41748
self._run_check('',
[('starttag', 'div', [('class', 'bar,baz=asd')])])


❯ ./python.exe -m test -v test_htmlparser

…
test_comma_between_unquoted_attributes 
(test.test_htmlparser.HTMLParserTestCase) ... ok
…

Ran 47 tests in 0.168s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 369 ms
Tests result: SUCCESS


So this is working as expected for the first test.


B: comma with spaces


Tests for browsers:
data:text/html,text

Serializations:
* Firefox, Gecko (86.0a1 (2020-12-28) (64-bit)) 
* Edge, Blink (Version 89.0.752.0 (Version officielle) Canary (64 bits))
* Safari, WebKit (Release 117 (Safari 14.1, WebKit 16611.1.7.2))

Same serialization in these 3 rendering engines
text


Adding
def test_comma_with_space_between_unquoted_attributes(self):
# bpo 41748
self._run_check('',
[('starttag', 'div', [
('class', 'bar'),
(',baz', 'asd')])])


❯ ./python.exe -m test -v test_htmlparser


This is failing.

==
FAIL: test_comma_with_space_between_unquoted_attributes 
(test.test_htmlparser.HTMLParserTestCase)
--
Traceback (most recent call last):
  File "/Users/karl/code/cpython/Lib/test/test_htmlparser.py", line 493, in 
test_comma_with_space_between_unquoted_attributes
self._run_check('',
  File "/Users/karl/code/cpython/Lib/test/test_htmlparser.py", line 95, in 
_run_check
self.fail("received events did not match expected events" +
AssertionError: received events did not match expected events
Source:
''
Expected:
[('starttag', 'div', [('class', 'bar'), (',baz', 'asd')])]
Received:
[('data', '')]

--


I started to look into the code of parser.py which I'm not familiar (yet) with.

https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/html/parser.py#L42-L52

Do you have a suggestion to fix it?

--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue41748>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4643] cgitb.html fails if getattr call raises exception

[karl]

karl  added the comment:

> The getattr call here has a default value, so it should not raise 
> AttributeError. It should also not raise any other exception because a valid 
> implementation of __getattr__ should raise only AttributeError:


but probably the intent of the patch here is to surface a meaningful error with 
regards to the script, more than an error on the cgitb python lib itself as 
this is a traceback tool. A bit like an HTML validator which continue to 
process things to give some kind of meaning.

Diving into previous issues about scanvars
https://bugs.python.org/issue966992

Similar
https://bugs.python.org/issue1047397 The last comment is basically this issue 
here. 

There is a patch which is lot better for this bug and which addresses the 
issues here. 
https://github.com/python/cpython/pull/15094

It has not been merged yet. Not sure why or if there is anything missing. 

Probably this bug could be closed as a duplicate of 
https://bugs.python.org/issue966992


And someone needs to push to push the latest bits for 
https://github.com/python/cpython/pull/15094


What do you think @iritkatriel ?

I will close my PR.

--

___
Python tracker 
<https://bugs.python.org/issue4643>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4643] cgitb.html fails if getattr call raises exception

[karl]

karl  added the comment:

Converted into GitHub PR https://github.com/python/cpython/pull/24038

--

___
Python tracker 
<https://bugs.python.org/issue4643>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4643] cgitb.html fails if getattr call raises exception

[karl]

Change by karl :


--
nosy: +karlcow
nosy_count: 4.0 -> 5.0
pull_requests: +22878
stage: test needed -> patch review
pull_request: https://github.com/python/cpython/pull/24038

___
Python tracker 
<https://bugs.python.org/issue4643>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25479] Increase unit test coverage for abc.py

[karl]

karl  added the comment:

@iritkatriel Github PR done. 
https://github.com/python/cpython/pull/24034

--

___
Python tracker 
<https://bugs.python.org/issue25479>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25479] Increase unit test coverage for abc.py

[karl]

Change by karl :


--
keywords: +patch
nosy: +karlcow
nosy_count: 2.0 -> 3.0
pull_requests: +22875
pull_request: https://github.com/python/cpython/pull/24034

___
Python tracker 
<https://bugs.python.org/issue25479>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42731] Enhancement request for proxying PyString

[Karl Nelson]

New submission from Karl Nelson :

When developing with JPype, the largest hole currently is that Java returns a 
string type which cannot be represented as a str.  Java strings are string like 
and immutable and can be pulled to Python when needed, but it is best if they 
remain in Java until Python requests it as pulling all string values through 
the API and pushing them back can result in serious overhead.  Thus they need 
to be represented as a Proxy to a string, which can be accessed as a string at 
anytime.

Throughout the Python API str is treated as a concrete type (though it is 
somewhat polymorphic due to different storage for code points sizes.)  There is 
also handling for an "unready" string which needs additional treatment before 
it can be accessed.  Unfortunately this does not appear to be suitable for 
creating a proxy object which can be pulled from another source to create a 
string on demand.   Having a "__str__()" method is insufficient as that merely 
makes an object able to become a string rather than considered to be a string 
by the rest of the API.

Would it be possible to generalize the concept of an unready string so that 
when Ready is called it fetches the actually string contents, creates a piece 
of memory to store the string contents (outside of the object itself), and sets 
the access flags for so that the code points can be interpreted?   Is this 
already possible in the API?  Are there any other plans to make the str type 
able to operate as a proxy?

--
components: Extension Modules
messages: 383701
nosy: Thrameos
priority: normal
severity: normal
status: open
title: Enhancement request for proxying PyString
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue42731>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30988] Exception parsing invalid email address headers starting or ending with dot

[Karl Fogel]

Karl Fogel  added the comment:

I can also confirm this bug, with Python 3.9.1 on Debian GNU/Linux ('testing' 
distro up-to-date as of 2020-12-21).

1) Create a parser `p` with `p = 
email.parser.HeaderParser(policy=email.policy.default)`.

2) Parse a single problematic (as described below) message: `msg_headers = 
p.parsestr(msg_str)`

3) Try to get the To field: `msg_headers.get_all('to', [ ])` and get an 
exception raised: `AttributeError: 'str' object has no attribute 'token_type'` 
(with the same stack trace as OP shows).

Here is a minimal problematic message you can reproduce this with (i.e., just 
make `msg_str` have this string value):

```
>From nobody Mon Dec 21 12:00:00  2020
From: sen...@example.com
To: . 
Date: Mon, 21 Dec 2020 12:00:00 -
Message-ID: <87ab5rvds7@example.com>
Subject: This is the Subject header.

Here is the body of the message.
```

Note that *any* number of dots for the recipient's name would also result in an 
error.  The above example uses just ".", but it could be "..", "...", 
".", etc.

--
nosy: +kfogel

___
Python tracker 
<https://bugs.python.org/issue30988>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Okay, well at least now googling Python + "A dynamic link library (DLL) 
initialization routine failed." give something which could point a user may be 
able to identify the issue.  It wasn't obvious to me that imports did not hold 
the GIL, but it is clear in retrospect that it should have been.

Thanks very much and sorry for the wild goose chase.

--
resolution:  -> third party
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

I found it.  The change that hit JPype is https://bugs.python.org/issue33895

Thanks, Eryk Sun for figuring this out.  I would never have gotten myself.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Drat I missed that one when I was scanning for statics because I was focusing 
C++ resource rather than looking for Python resources.  I also wouldn't have 
expected this to happen on only one platform, but that was my bad.  

Is it possible to improve the error reporting for this?  This was a horrible 
error to trace down at least for a non-windows programmer.  At least a mention 
in the doc as to the symptoms so someone else can google it.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Gave another series of shots at the problem.  So the reason that attaching the 
debugger fixes the issue is that it calls LoadLibraryExW for a bunch libraries 
when it attaches.  

I was also able to unconditionally trigger the bug even when loading without 
the pyc.   It appears that calling "import _ssh"  will cause whatever conflict 
is happening to trigger immediately.

Other than than progress was minimal.  Attaching the debugger causes so many 
changes to the control flow that I can't get the bad path to trigger and I 
can't get the debugger to attach once you start down the path.   I tried 
creating a segfault but again I can't get the debugger to attach because 
Detours is interfering with the attachment process.

I also tried disabling some of the flags that JPype was adding to the python 
build process to see if any are related.   "/DEBUG" on the linker does not do 
anything to the problem.  The other options were '/Zi', '/EHsc', '/std:c++14'.  
 Nothing here is out of the ordinary.  

I do not see anything in the link line that shows evidence of static linkage or 
likely to cause a conflict.  The exact same line is being used with Python 3.8.

link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 
/MANIFESTUAC:NO 
/LIBPATH:C:\Users\nelson85\AppData\Local\Programs\Python\Python39\libs 
/LIBPATH:C:\Users\nelson85\AppData\Local\Programs\Python\Python39\PCbuild\amd64 
/LIBPATH:C:\Program Files (x86)\Microsoft Visual 
Studio\2019\Community\VC\Tools\MSVC\14.28.29333\ATLMFC\lib\x64 
/LIBPATH:C:\Program Files (x86)\Microsoft Visual 
Studio\2019\Community\VC\Tools\MSVC\14.28.29333\lib\x64 /LIBPATH:C:\Program 
Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files 
(x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64 /LIBPATH:C:\Program Files 
(x86)\Windows Kits\10\lib\10.0.18362.0\um\x64 /LIBPATH:C:\Program Files 
(x86)\Microsoft Visual 
Studio\2019\Community\VC\Tools\MSVC\14.28.29333\ATLMFC\lib\x64 
/LIBPATH:C:\Program Files (x86)\Microsoft Visual 
Studio\2019\Community\VC\Tools\MSVC\14.28.29333\lib\x64 /LIBPATH:C:\Program 
Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files 
(x86)\Windows Kits\10\
 lib\10.0.18362.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows 
Kits\10\lib\10.0.18362.0\um\x64 /EXPORT:PyInit__jpype 
$(OBJECTS) /OUT:build\lib.win-amd64-3.9\_jpype.cp39-win_amd64.pyd 
/IMPLIB:build\temp.win-amd64-3.9\Release\native\common\_jpype.cp39-win_amd64.lib

So I remain at a loss how to proceed forward.  Nothing has changed in JPype or 
its symbols since Python 3.8 so this is clearly taking place in Python or at 
the OS level.  Is it possible that Python shipped with bad dlls for the 3.9.0 
build?  If so then looking at the source code may be the wrong approach.   
After all depending on the order of libraries I trigger or don't trigger the 
bug.

I should note that the difference between loading successfully py and pyc seems 
to have been triggered in PC/getpathc.c  could it be the change to remove 
loading api-ms-win-core-path?


```
 static void
 join(wchar_t *buffer, const wchar_t *stuff)
 {
-if (_PathCchCombineEx_Initialized == 0) {
-HMODULE pathapi = LoadLibraryExW(L"api-ms-win-core-path-l1-1-0.dll", 
NULL,
- LOAD_LIBRARY_SEARCH_SYSTEM32);
-if (pathapi) {
-_PathCchCombineEx = (PPathCchCombineEx)GetProcAddress(pathapi, 
"PathCchCombineEx");
-}
-else {
-_PathCchCombineEx = NULL;
-}
-_PathCchCombineEx_Initialized = 1;
-}
-
-if (_PathCchCombineEx) {
-if (FAILED(_PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0))) 
{
-Py_FatalError("buffer overflow in getpathp.c's join()");
-}
-} else {
-if (!PathCombineW(buffer, buffer, stuff)) {
-Py_FatalError("buffer overflow in getpathp.c's join()");
-}
+if (FAILED(PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0))) {
+Py_FatalError("buffer overflow in getpathp.c's join()");
 }
 }
```

Also given JPype is really a generic C++ CPython module (with no special 
options other than /Zi /EHsc why hasn't anyone else triggered this?

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

I will look through the list of samples in Detours to see if there is something 
that can give us better stacktrace information.   If we really need a stack 
trace I can make Detours bust by creating an intentional segfault and see if we 
can attach the debugger after the source of the problem has been triggered.  
Unfortunately as my windows programming knowledge is all from 1994 I may be 
missing something obvious.

In the meantime I am upgrading the sample with some more hooks for any other 
potential LoadLibrary variants and FlsGetValue.  Though given the 
GetProcAddress calls I think that it must be something in the chain leading up 
so the variants are likely not envolved.

The other thing I can do is look at instrumenting SetErrorMode so that I can 
give myself a marker of when we are entering and leaving the dyload code.

```

/* Don't display a message box when Python can't load a DLL */
old_mode = SetErrorMode(SEM_FAILCRITICALERRORS);

/* bpo-36085: We use LoadLibraryEx with restricted search paths
   to avoid DLL preloading attacks and enable use of the
   AddDllDirectory function. We add SEARCH_DLL_LOAD_DIR to
   ensure DLLs adjacent to the PYD are preferred. */
Py_BEGIN_ALLOW_THREADS
hDLL = LoadLibraryExW(wpathname, NULL,
  LOAD_LIBRARY_SEARCH_DEFAULT_DIRS |
  LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR);
Py_END_ALLOW_THREADS
#if !USE_UNICODE_WCHAR_CACHE
PyMem_Free(wpathname);
#endif /* USE_UNICODE_WCHAR_CACHE */

/* restore old error mode settings */
SetErrorMode(old_mode);
```

I will post back once I get some results.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

I looked more at the logs to see if I can find out what to instrument next.   
The log files unfortunately don't diff well because every line contains a 
timestamp so I can't a proper alignment as well as all the real addresses.  So 
I wrote a short program to convert all the numbers into an "X".

After doing so I find there is a single difference in the process between 
loading from a py file and a pyc file.


# import _jpype
## Only if loaded from a Py file
trclnk64: 001 GetProcAddress(7ffc4c71,FlsGetValue)
X X X trclnk64: 001 GetProcAddress(,) -> X
## All
X X X trclnk64: 001 
LoadLibraryExW(C:\Users\nelson85\AppData\Local\Programs\Python\Python39\python3,0,1000)
X X X trclnk64: 001 LoadLibraryExW(,,) -> X
X X X trclnk64: ### X 
C:\Users\nelson85\AppData\Local\Programs\Python\Python39\python3.DLL 000140bc
  So here is the call that fails
X X X trclnk64: 001 
LoadLibraryExW(c:\users\nelson85\documents\devel\open\jpype\_jpype.cp39-win_amd64.pyd,0,1100)
X X X trclnk64: 001 LoadLibraryExW(api-ms-win-core-synch-l1-2-0,0,800)
X X X trclnk64: 001 LoadLibraryExW(,,) -> X
X X X trclnk64: ### X C:\WINDOWS\System32\KERNELBASE.dll 002b306b
X X X trclnk64: ntdll.dll [7ffc4c974138 X

So the problem is something that happened before the LoadLibraryExW call when 
python3.dll was accessed.  The correct path called FlsGetValue  while the 
incorrect one skipped it which lead to a failure.

Thus my conclusion is that the vehicle left the edge of the cliff just prior to 
LoadLibraryExW(_jpype) and hit the ocean when C++ runtime library was requested.

Does that point to anything we can use to solve this?

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Eryk,

Unfortunately, this particular bug is resistant to attaching the debugger.  
Whenever the debugger is attached the bug disappears.   Fortunately the 
suggestion to use Detours by WildCard65 appears to offer a way to diagnose the 
issue.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

The last libraries loaded prior to the failure were...

```
20201218192451066 20440 50.60: trclnk64: 
api-ms-win-eventing-provider-l1-1-0.dll [7ffc4c974108 7ffc4c8b7808]
20201218192451066 20440 50.60: trclnk64:   EventUnregister0 7ffc4eab37a0
20201218192451066 20440 50.60: trclnk64:   EventRegister0 7ffc4ea6a640
20201218192451066 20440 50.60: trclnk64:   EventActivityIdControl0 
7ffc4eac6190
20201218192451066 20440 50.60: trclnk64:   EventWriteTransfer0 7ffc4eab2cf0
20201218192451066 20440 50.60: trclnk64:   EventSetInformation0 7ffc4ea6a3e0
20201218192451066 20440 50.60: trclnk64: api-ms-win-core-apiquery-l1-1-0.dll 
[7ffc4c9740e8 7ffc4c8b77e8]
20201218192451066 20440 50.60: trclnk64:   ApiSetQueryApiSetPresence0 
7ffc4ead5030
20201218192451066 20440 50.60: trclnk64: api-ms-win-core-apiquery-l1-1-1.dll 
[7ffc4c9740f8 7ffc4c8b77f8]
20201218192451066 20440 50.60: trclnk64:   ApiSetQueryApiSetPresenceEx0 
7ffc4eb35730
```

And as far as I can tell that was successful.  Perhaps there is another library 
at this point, but my reading of the log is that LoadLibraryExW was called run 
though all required elements to load, got the end section before it started to 
load dependencies of _jpype, started fetching things that have to do with 
locale and then hit a failure point inside of LoadLibrary.   If I look at the 
successful trace then I see the next action is to start linking in C++ symbols. 
  

But perhaps I am misreading...   Looking farther back in the log, the last 
LoadLibrary to get called before the failure is.

```
20201218192451054 20440 50.60: trclnk64: 001   
LoadLibraryExW(api-ms-win-core-fibers-l1-1-1,0,800)
20201218192451054 20440 50.60: trclnk64: 001   LoadLibraryExW(,,) -> 
7ffc4c71
20201218192451054 20440 50.60: trclnk64: ### 4c71: 
C:\WINDOWS\System32\KERNELBASE.dll 002b306b
20201218192451054 20440 50.60: trclnk64: ntdll.dll [7ffc4c974138 7ffc4c8b7838]
```

Fibers?  I have no clue at this point.   Is this a Python bug or a windows bug?

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

I used Detours with trclnk64 to get the following log:

```
20201218193836960 4332 50.60: trclnk64: 001   GetProcAddress(,) -> 7ffc4ccebef0
20201218193836960 4332 50.60: trclnk64: 001   
GetProcAddress(7ffc4ccd,LCMapStringEx)
20201218193836960 4332 50.60: trclnk64: 001   GetProcAddress(,) -> 7ffc4cce6640
 <= This is the point where it differs from a clean run.
20201218193838163 4332 50.60: trclnk64: 001   
GetProcAddress(7ffc4c71,FlsFree)
20201218193838163 4332 50.60: trclnk64: 001   GetProcAddress(,) -> 7ffc4c78b000
20201218193838163 4332 50.60: trclnk64: 001   FAIL 
LoadLibraryExW(c:\users\nelson85\documents\devel\open\jpype\_jpype.cp39-win_amd64.pyd,0,1100)
20201218193838164 4332 50.60: trclnk64: 001 LoadLibraryExW(,,) -> 0
```

It really is nothing to go on...  Everything succeeds, then we get an extra 
successful request for FlsFree then it immediately returns with a fail.

In the successful run (without the cache) we get...
```
20201218192451068 20440 50.60: trclnk64: 001   
GetProcAddress(7ffc4ccd,LCMapStringEx)
20201218192451068 20440 50.60: trclnk64: 001   GetProcAddress(,) -> 7ffc4cce6640
20201218192451068 20440 50.60: trclnk64: 001 LoadLibraryExW(,,) -> 7ffc349f
20201218192451068 20440 50.60: trclnk64: ### 349f: 
c:\users\nelson85\documents\devel\open\jpype\_jpype.cp39-win_amd64.pyd 
20201218192451068 20440 50.60: trclnk64: MSVCP140.dll [7ffc34a7af48 
7ffc34a5a0c8]
20201218192451068 20440 50.60: trclnk64:   
?_Lock@?$basic_streambuf@DU?$char_traits@D@std@@@std@@UEAAXXZ0 7ffc36f22350
20201218192451068 20440 50.60: trclnk64:   
?_Unlock@?$basic_streambuf@DU?$char_traits@D@std@@@std@@UEAAXXZ0 
7ffc36f22350
```

So the failure appears to be internal to the call before we load MSVCP140.dll.  
 Unfortunately, I am way outside my element here as a Linux/Java/CPython 
programmer.   My next guess would be to instrument the set system error and if 
we are within LoadLibrary then execute "int *i=0; *i=0;"  and hope the debugger 
would get me the point of failure so that I can examine the stack trace to find 
out where we are.  But if you have another direction I would be happy to hear 
it.   Is there some function I should add to trclnk64 to that may have gone 
wrong?  It must be something that fetched the address of LCMapStringEx, but I 
have no clue what that may be.

This really does not look like a JPype bug.  Yes, two minutes in the future 
from where the car ran over the pedestrian, he was going to run into the bank 
and demand money (monkey patch and other bad things).  But at the point in time 
the car hit, the pedestrian was just minding his own business.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Can you be so kind as pointing me to a LoadLibraryExW detour example for 
Python?   I have shimmed a DLL before to capture transaction logs in the past, 
but not with a Python or a system library.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Just for reference here are all the dependencies that _jpype has

```
MSVCP140.dll

python39.dll

KERNEL32.dll

VCRUNTIME140_1.dll  

VCRUNTIME140.dll


api-ms-win-crt-runtime-l1-1-0.dll   
  
   api-ms-win-crt-string-l1-1-0.dll 

   api-ms-win-crt-heap-l1-1-0.dll
```

So it isn't like there are a whole lot of things that I am doing that are 
likely to trigger a dependency problem.

As far as I can see the only items that are not already on python39.dll 
dependency list are VCRUNTIME140_1.dll and MSVCP140.dll.  But if that were the 
case then surely other projects would also be hitting this bug.  So this error 
message makes no sense at all.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

I attempted another deep dive on this one.

1) Removed manual library add
   platform_specific['libraries'] = ['Advapi32']
No change.

2) Compared symbol wise imports
Two imports changed
PyIndex_Check
PyObject_CheckBuffer
plus one additional "fflush" dependency in api-ms-win-crt-stdio-l1-1-0.dll 

3) Compared symbol wise exports 
No changes detected.

4) Disassembled both binaries
Other than a addressing that is about a 0x1000 change in the relocation tables. 
 Guessing that this is just a change in the number of pages.  There were 
functional changes likely related to Python macros differences.

5) Removed all references to printf and flush to remove 
api-ms-win-crt-stdio-l1-1-0.dll dependency.
No change.

6) Scanned the changelog and Code changes for the past year in  
Python/dynload_win.c.  
Nothing appears likely to trigger this change.

7) Deleted the _jpype module entirely to verify that the error message is 
related to the copy that I am working on and not some other copy on system.
Error message changes to "ModuleNotFoundError: No module named '_jpype'"

8) Scanned the source code for static variables.
None detected thus far.  But it is a many hours long process.  Structurally 
these should not exist based on the module design.

Statics found thus far.
2 jobject and jmethodID pointer types
One C++ class called PLATFORM_ADAPTER which is the required routine determining 
whether to load jvm.dll or jvm.so using LoadLibrary or dlopen or shload.
The global pointer to the context C++ class;
a bunch of statics related to logging when enabled.
Python C style structure definitions.

Everything was a primitive or pointer with the exception of one class.  All 
pointers I found were initialized to constant values.

I rewrote the PLATFORM_ADAPTER to use a pointer rather than be initialized at 
load time.  No joy.

(after scanning through 17k lines of code take break and ponder life)

It is certainly possible I missed something here but it looks pretty unlikely 
this is the source of the error.


9) Verified that the behavior does not occur on Python 3.8.5 with the current 
source code.  
Pass.  This only happens on Python 3.9.0

10) Scanned the linker line for differences.
One major difference.  In Python 3.8 the linker line contains quotes around all 
the arguments while in Python 3.9 the linker line is missing the quotes on all 
/LIBPATH statements.  However this must be a logging difference as the 
statement without the quotes would not work.

11) Manually ran linker command.  
Nothing changed, same behavior.  So link command is a red herring.

12) Profit???

Okay I have once again exhausted everything that I can think of on the JPype 
side.   So that leaves something in the build process that does not appear in 
the logs, a symbol conflict that so how only triggers on one path, or 
Python3.9.0 has a corrupt build in the wild.   

Steve is there anything else that I can try here?

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Well that gives me a place to search from.   Unfortunately statics are not 
likely the source of the issue.   I scrubbed all C++ structures from the 
project into a big structure which has only a single global pointer which I 
initialize in the init method.   It is possible that one crept back in at some 
point, but it is a possibility.   The use of actual C++ in this project is 
nearly zero.  Other than exceptions, strings and the vector on the buried in 
some class, the rest has been purged already.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Without the pyc everything goes fine...

```
...
_bootlocale' # <_frozen_importlib_external.SourceFileLoader object at 
0x0222E9FD5A30>
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 
0x0222E9F88F40>
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# code object from 
c:\users\nelson85\documents\devel\open\jpype\jpype\__init__.py
# created 
'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\jpype\\__pycache__\\__init__.cpython-39.pyc'
# extension module '_jpype' loaded from 
'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\_jpype.cp39-win_amd64.pyd'
# extension module '_jpype' executed from 
'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\_jpype.cp39-win_amd64.pyd'
import '_jpype' # <_frozen_importlib_external.ExtensionFileLoader object at 
0x0222EA4C6250>
# 
c:\users\nelson85\documents\devel\open\jpype\jpype\__pycache__\_jinit.cpython-39.pyc
 matches c:\users\nelson85\documents\devel\open\jpype\jpype\_jinit.py
# code o
...
```

With the pyc file, failure... 

```
...
_bootlocale' # <_frozen_importlib_external.SourceFileLoader object at 
0x015278445A30>
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 
0x0152783F8F40>
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# 
c:\users\nelson85\documents\devel\open\jpype\jpype\__pycache__\__init__.cpython-39.pyc
 matches c:\users\nelson85\documents\devel\open\jpype\jpype\__init__.py
# code object from 
'c:\\users\\nelson85\\documents\\devel\\open\\jpype\\jpype\\__pycache__\\__init__.cpython-39.pyc'
Traceback (most recent call last):
  File "C:\Users\nelson85\Documents\test.py", line 1, in 
import jpype
  File "", line 1007, in _find_and_load
  File "", line 986, in _find_and_load_unlocked
  File "", line 680, in _load_unlocked
  File "", line 790, in exec_module
  File "", line 228, in _call_with_frames_removed
  File "c:\users\nelson85\documents\devel\open\jpype\jpype\__init__.py", line 
18, in 
import _jpype
  File "", line 1007, in _find_and_load
  File "", line 986, in _find_and_load_unlocked
  File "", line 666, in _load_unlocked
  File "", line 565, in module_from_spec
  File "", line 1108, in create_module
  File "", line 228, in _call_with_frames_removed
ImportError: DLL load failed while importing _jpype: A dynamic link library 
(DLL) initialization routine failed.
# clear builtins._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# destroy jpype
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.__interactivehook__
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
...
```

I see the codecs all get loaded earlier in the trace, so that can't be the 
issue.  There were no differences (other than memory addresses) up to this 
point.  So it must be going wrong at bootstrap_external:1108.

So guessing from the message we failed in.

```
def exec_module(self, module):
"""Initialize an extension module"""
_bootstrap._call_with_frames_removed(_imp.exec_dynamic, module)
_bootstrap._verbose_message('extension module {!r} executed from {!r}',
 self.name, self.path)
```

After that point it gets really fuzzy.  I would guess we are going to 
import.c#_imp_create_dynamic_impl which takes us to 
_PyImport_LoadDynamicModuleWithSpec.  But I will need some time to read code to 
see if this guess is correct.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Thanks, I will see if I can get additional diagnostics today.  Otherwise I will 
have to start recompiling Python with diagnostic hooks.  I have never had to 
rebuild python on windows so it may take a while to figure out how to make 
progress.

The monkey patch is less scary than it looks.   Since I only need these slots 
of specific basic types that I designate (not user selected), the extra types 
all extend without altering the underlaying type, and the class layout is 
controlled by a meta class so that only Java can create these type objects, it 
is reasonably robust.   However, yes changes in Python can break it.  It is 
basically an attempt to create a mixin for concrete classes without the benefit 
of a dictoffset type slot.  It took about a month do develop something that 
safely added the slot without overrunning, leaking or creating other memory 
issues (and played nice when Python added its own dict slot). 

Unfortunately, as I do many 100s on JSlot tests during method resolution using 
a dict based approach was unacceptably slow.  I would love if there was a 
formal method to create multiple inheritance mixins for Python basic types, but 
that is another issue.


```
/**
 * Allocate a new Python object with a slot for Java.
 *
 * We need extra space to store our values, but because there
 * is no way to do so without disturbing the object layout.
 * Fortunately, Python already handles this for dict and weakref.
 * Python aligns the ends of the structure and increases the
 * base type size to add additional slots to a standard object.
 *
 * We will use the same trick to add an additional slot for Java
 * after the end of the object outside of where Python is looking.
 * As the memory is aligned this is safe to do.  We will use
 * the alloc and finalize slot to recognize which objects have this
 * extra slot appended.
 */
PyObject* PyJPValue_alloc(PyTypeObject* type, Py_ssize_t nitems )
{
// Modification from Python to add size elements
const size_t size = _PyObject_VAR_SIZE(type, nitems + 1) + sizeof 
(JPValue);
PyObject *obj = (PyType_IS_GC(type)) ? _PyObject_GC_Malloc(size)
: (PyObject *) PyObject_MALLOC(size);
if (obj == NULL)
return PyErr_NoMemory(); // GCOVR_EXCL_LINE
memset(obj, 0, size);

Py_ssize_t refcnt = ((PyObject*) type)->ob_refcnt;
if (type->tp_itemsize == 0)
PyObject_Init(obj, type);
else
PyObject_InitVar((PyVarObject *) obj, type, nitems);

// This line is required to deal with Python bug (GH-11661)
// Some versions of Python fail to increment the reference counter of
// heap types properly.
if (refcnt == ((PyObject*) type)->ob_refcnt)
Py_INCREF(type);  // GCOVR_EXCL_LINE

if (PyType_IS_GC(type))
{
PyObject_GC_Track(obj);
}
return obj;
}

bool PyJPValue_hasJavaSlot(PyTypeObject* type)
{
if (type == NULL
|| type->tp_alloc != (allocfunc) PyJPValue_alloc
|| type->tp_finalize != (destructor) PyJPValue_finalize)
return false;  // GCOVR_EXCL_LINE
return true;
}

// Check for a Java slot (must work on all object types)
Py_ssize_t PyJPValue_getJavaSlotOffset(PyObject* self)
{
PyTypeObject *type = Py_TYPE(self);
if (type == NULL
|| type->tp_alloc != (allocfunc) PyJPValue_alloc
|| type->tp_finalize != (destructor) PyJPValue_finalize)
return 0;
Py_ssize_t offset;
Py_ssize_t sz = Py_SIZE(self);
// I have no clue what negative sizes mean
if (sz < 0)
sz = -sz;
if (type->tp_itemsize == 0)
offset = _PyObject_VAR_SIZE(type, 1);
else
offset = _PyObject_VAR_SIZE(type, sz + 1);
return offset;
}

JPValue* PyJPValue_getJavaSlot(PyObject* self)
{
Py_ssize_t offset = PyJPValue_getJavaSlotOffset(self);
if (offset == 0)
return NULL;
JPValue* value = (JPValue*) (((char*) self) + offset);
if (value->getClass() == NULL)
return NULL;
return value;
}

void PyJPValue_free(void* obj)
{
// Normally finalize is not run on simple classes.
PyTypeObject *type = Py_TYPE(obj);
if (type->tp_finalize != NULL)
type->tp_finalize((PyObject*) obj);
if (type->tp_flags & Py_TPFLAGS_HAVE_GC)
PyObject_GC_Del(obj);
else
PyObject_Free(obj);  // GCOVR_EXCL_LINE
}

void PyJPValue_finalize(void* obj)
{
JPValue* value = PyJPValue_getJavaSlot((PyObject*) obj);
... destroy the objects hidden slot if it exists
}

```

--

___
Python t

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

I am fairly sure this is a Python "bug" in the sense that there was some change 
in undocumented change in requirements and the distutils pattern for building a 
module no longer reflects that requirement.   That said very likely JPype is 
the only module to be affected and thus I will have to manually adjust to 
account for the new requirement.

Unfortunately as far as developers, I am it so fixing it (with your assistance) 
is going to have to fall on me.  So lets do a run down of how this all working 
so you can point me where to look.

1) JPype gets built as an ordinary setup.py CPython module.  So it is possibly 
a bug in the build pattern of the customizers in JPype that was exposed by 
Python 3.9.   I am just going to cut and paste so that it is easy to follow 
along.

jpype/setup.py
```
...
from setuptools import Extension
...
import setupext
...
jpypeLib = Extension(name='_jpype', **setupext.platform.Platform(
include_dirs=[Path('native', 'common', 'include'),
  Path('native', 'python', 'include'),
  Path('native', 'embedded', 'include')],
sources=[Path('native', 'common', '*.cpp'),
 Path('native', 'python', '*.cpp'),
 Path('native', 'embedded', '*.cpp')], platform=platform,
))
```

We have two sets of customization in setup.py.  Both are included from a module 
called jpype/setupext/

The key one is the jpype/setupext/platform.py which defines the compiler flags. 
 There are two sections of interest...

jpype/setupext/platform.py contains these modifications for win32.   
(So there is the Advapi32, not sure why it appears there because this section 
is all before my time as this was started in 2001 and I joined in 2018)
```
static = True
if platform == 'win32':
distutils.log.info("Add windows settings")
platform_specific['libraries'] = ['Advapi32']
platform_specific['define_macros'] = [('WIN32', 1)]
if sys.version > '3':
platform_specific['extra_compile_args'] = [
'/Zi', '/EHsc', '/std:c++14']
else:
platform_specific['extra_compile_args'] = ['/Zi', '/EHsc']
platform_specific['extra_link_args'] = ['/DEBUG']
jni_md_platform = 'win32'
```

The second section is currently commented out.  Though it is relevant because 
JPype is exotic in one way.  It is a module which is loaded in three ways.   
When imported from Python it is an ordinary library (1) which will later pull 
in Java which will then load library a second time (2) to bind Java native 
methods.   It can also be used to launch Python if Java starts the session (3). 
  In that case, it needs libpython.dll to be bound to module so that the Java 
equivalent to LoadLibrary can work.  When it does Java first manually loads 
libpython.dll then loads the module and calls the hook to get Python started. 
```
# This code is used to include python library in the build when starting Python 
from
# within Java.  It will be used in the future, but is not currently required.
#if static and sysconfig.get_config_var('BLDLIBRARY') is not None:
#
platform_specific['extra_link_args'].append(sysconfig.get_config_var('BLDLIBRARY'))
```

The actual buildext has a few minor patches so that Java libraries can run 
through the normal process.  But nothing seems like a good candidate

We have one section tweeking some of the build options.
```
def initialize_options(self, *args):
"""omit -Wstrict-prototypes from CFLAGS since its only valid for C 
code."""
self.android = False
self.makefile = False
self.jar = False
import distutils.sysconfig
cfg_vars = distutils.sysconfig.get_config_vars()
replacement = {
'-Wstrict-prototypes': '',
'-Wimplicit-function-declaration': '',
}
tracing = self.distribution.enable_tracing

# Arguments to remove so we set debugging and optimization level
remove_args = ['-O0', '-O1', '-O2', '-O3', '-g']

for k, v in cfg_vars.items():
if not isinstance(v, str):
continue
if not k == "OPT" and not "FLAGS" in k:
continue

args = v.split()
for r in remove_args:
args = list(filter((r).__ne__, args))

cfg_vars[k] = " ".join(args)
super().initialize_options()
```

Then later we interrupt the build process for Java.
```
def build_extension(self, ext):
if ext.language == "java":
return self.build_java_ext(ext)
if self.jar:
return
print("Call build ext")
return super().build_extension(ext)
```

2) Next we have the module start.  I am guessing this is not the source of the 
error because adding a printf shows we never got to this poi

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Any progress on this item?  I am seeing additional reports of this error in the 
conda stream tracker and elsewhere.  As it only occurs when the debugger is not 
hooked up I don't have much traction to make progress myself.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42617] Enhancement request for PyType_FromSpecWIthBases add option for meta class

[Karl Nelson]

Karl Nelson  added the comment:

Perhaps just having PyType_InitHeapFromSpec which have just heap type linkage 
and slot copy section would help with the issue.   The major problem I have is 
maintaining the slot code which may change at some point in the future.   There 
would still be some risk of missing a portion of the init procedure but it 
would be a bit safer some of the process was in Python.

--
title: Enhancement request  for PyType_FromSpecWIthBases add option for meta 
class -> Enhancement request for PyType_FromSpecWIthBases add option for meta 
class

___
Python tracker 
<https://bugs.python.org/issue42617>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

Oddly that was the only exception that I got.  When I hit continue it proceeded 
without issue and the dll was loaded correctly.   However, when I try without 
the debugger attached the error reappears.   So this is one of those 
Schrodinger errors.   I know the error is there because I see it when we run on 
the CI and on a local machine,  but all attempts to isolate it say that is 
should not exist.  There can't really be a path that is different when 
compiling then loading from a pyc unless this is some type of uninitialized 
variable issues where we just happened to have been unfortunate enough to hit 
it.

--

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42618] Enhancement request for importing stacktraces from foreign sources

[Karl Nelson]

New submission from Karl Nelson :

In JPype, I am transfer stack information from Java into Python for diagnostics 
and display purposed.  Unfortunately, as the exception system is directly 
accessing traceback structure elements this cannot be replicated without 
creating traceback structures in C.   I have thus been forced to create new 
methods that create this using internal Python structures.  It would be much 
better if there were C API method to allow importing a foreign stack trace into 
Python.

Here is an example of the code used in JPype for reference.


```
// Transfer list of filenames, functions and lines to Python.
PyObject* PyTrace_FromJPStackTrace(JPStackTrace& trace)
{
PyTracebackObject *last_traceback = NULL;
PyObject *dict = PyModule_GetDict(PyJPModule);
for (JPStackTrace::iterator iter = trace.begin(); iter != trace.end(); 
++iter)
{
last_traceback = tb_create(last_traceback, dict,  
iter->getFile(),
iter->getFunction(), iter->getLine());
}
if (last_traceback == NULL)
Py_RETURN_NONE;
return (PyObject*) last_traceback;
}

PyTracebackObject *tb_create(   

PyTracebackObject *last_traceback,
PyObject *dict, 

const char* filename,
const char* funcname,
int linenum)
{
// Create a code for this frame.
PyCodeObject *code = PyCode_NewEmpty(filename, funcname, linenum);
// Create a frame for the traceback.
PyFrameObject *frame = (PyFrameObject*) 
PyFrame_Type.tp_alloc(_Type, 0);
frame->f_back = NULL;
if (last_traceback != NULL)
{
frame->f_back = last_traceback->tb_frame;
Py_INCREF(frame->f_back);
}
frame->f_builtins = dict;
Py_INCREF(frame->f_builtins);
frame->f_code = (PyCodeObject*) code;
frame->f_executing = 0;
frame->f_gen = NULL;
frame->f_globals = dict;
Py_INCREF(frame->f_globals);
frame->f_iblock = 0;
frame->f_lasti = 0;
frame->f_lineno = 0;
frame->f_locals = NULL;
frame->f_localsplus[0] = 0;
frame->f_stacktop = NULL;
frame->f_trace = NULL;
frame->f_valuestack = 0;
#if PY_VERSION_HEX>=0x0307
frame->f_trace_lines = 0;
frame->f_trace_opcodes = 0;
#endif
// Create a traceback
PyTracebackObject *traceback = (PyTracebackObject*)
PyTraceBack_Type.tp_alloc(_Type, 0);
traceback->tb_frame = frame;
traceback->tb_lasti = frame->f_lasti;
traceback->tb_lineno = linenum;
traceback->tb_next = last_traceback;
return traceback;
}

```

--
components: C API
messages: 382852
nosy: Thrameos
priority: normal
severity: normal
status: open
title: Enhancement request for importing stacktraces from foreign sources
type: enhancement

___
Python tracker 
<https://bugs.python.org/issue42618>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42617] Enhancement request for PyType_FromSpecWIthBases add option for meta class

[Karl Nelson]

New submission from Karl Nelson :

PyType_FromSpecWithBases is missing an argument for taking a meta class.  As a 
result it is necessary to replicate a large portion of Python code when I need 
to create a new heap type with a specified meta class.  This is a maintenance 
issue as replicating Python code is likely to get broken in future.

I have replicated to code from JPype so that it is clear how the meta class is 
coming into place.   PyJPClass_Type is derived from a 
PyHeapType with additional slots for Java to use and overrides allocation slots 
to that it can add extra slots (needs true multiple inheritance as it needed to 
add Java slots to types with mismatching memory layouts object, long, float, 
exception, ...).

This code could be made much safer if there were a PyType_FromSpecWithBasesMeta 
which used the meta class to allocate the memory (then I could safely override 
the slots after creation).
 

```
PyObject* PyJPClass_FromSpecWithBases(PyType_Spec *spec, PyObject *bases)
{
JP_PY_TRY("PyJPClass_FromSpecWithBases");
// Python lacks a FromSpecWithMeta so we are going to have to fake it 
here.
PyTypeObject* type = (PyTypeObject*) 
PyJPClass_Type->tp_alloc(PyJPClass_Type, 0); // <== we need to use the meta 
class here
PyHeapTypeObject* heap = (PyHeapTypeObject*) type;
type->tp_flags = spec->flags | Py_TPFLAGS_HEAPTYPE | Py_TPFLAGS_HAVE_GC;
type->tp_name = spec->name;
const char *s = strrchr(spec->name, '.');
if (s == NULL)
s = spec->name;
else
s++;
heap->ht_qualname = PyUnicode_FromString(s);
heap->ht_name = heap->ht_qualname;
Py_INCREF(heap->ht_name);
if (bases == NULL)
type->tp_bases = PyTuple_Pack(1, (PyObject*) & 
PyBaseObject_Type);
else
{
type->tp_bases = bases;
Py_INCREF(bases);
}
type->tp_base = (PyTypeObject*) PyTuple_GetItem(type->tp_bases, 0);
Py_INCREF(type->tp_base);
type->tp_as_async = >as_async;
type->tp_as_buffer = >as_buffer;
type->tp_as_mapping = >as_mapping;
type->tp_as_number = >as_number;
type->tp_as_sequence = >as_sequence;
type->tp_basicsize = spec->basicsize;
if (spec->basicsize == 0)
type->tp_basicsize = type->tp_base->tp_basicsize;
type->tp_itemsize = spec->itemsize;
if (spec->itemsize == 0)
type->tp_itemsize = type->tp_base->tp_itemsize;

// <=== Replicated code from the meta class 
type->tp_alloc = PyJPValue_alloc;
type->tp_free = PyJPValue_free;
type->tp_finalize = (destructor) PyJPValue_finalize;

// <= Replicated code from Python
for (PyType_Slot* slot = spec->slots; slot->slot; slot++)
{
   switch (slot->slot)
{
case Py_tp_free:
type->tp_free = (freefunc) slot->pfunc;
break;
case Py_tp_new:
type->tp_new = (newfunc) slot->pfunc;
break;
case Py_tp_init:
type->tp_init = (initproc) slot->pfunc;
break;
case Py_tp_getattro:
type->tp_getattro = (getattrofunc) slot->pfunc;
break;
case Py_tp_setattro:
type->tp_setattro = (setattrofunc) slot->pfunc;
break;
case Py_tp_dealloc:
type->tp_dealloc = (destructor) slot->pfunc;
break;
case Py_tp_str:
type->tp_str = (reprfunc) slot->pfunc;
break;
case Py_tp_repr:
type->tp_repr = (reprfunc) slot->pfunc;
break;
case Py_tp_methods:
type->tp_methods = (PyMethodDef*) slot->pfunc;
break;
case Py_sq_item:
heap->as_sequence.sq_item = (ssizeargfunc) 
slot->pfunc;
break;
case Py_sq_length:
heap->as_sequence.sq_length = (lenfunc) 
slot->pfunc;
break;
case Py_mp_ass_subscript:
heap->as_mapping.mp_ass_subscri

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

Karl Nelson  added the comment:

I managed to get the debugger attached (unfortunately I am not a windows 
programmer so I don't use these tools). 

It appears when loading from a pyc, it is attempting to open the directory as a 
Zip file which is throwing an exception resulting in a failure to load the 
internal _jpype module.   

Unfortunately this is outside my area as nothing in jpype nor _jpype is calling 
the zipimport module.  So it must be something internal to Python.  It is a 
major issue as I can't release to anaconda on windows until I can resolve the 
problem.   Can you direct me how to proceed?

Details:

zipimport.ZipImportError
  Message=not a Zip file
  StackTrace:
:88 in __init__

Replicating the problem is simple.   Download jpype from git, use "python 
setup.py develop" to create the module, run python -c  "import jpype", and 
repeat python -c "import jpype"

Screen shot of the visual studio error is shown.

--
Added file: https://bugs.python.org/file49665/Capture.PNG

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42529] CPython DLL initialization routine failed from PYC cache file

[Karl Nelson]

New submission from Karl Nelson :

While trying to use JPype on Windows Python 3.9.0, we are running into an 
bizarre issue with loading the internal module which is written in C.  When 
running a python script the first time the internal module loads correctly.  
However, the second time that script is run the internal module reports "A 
dynamic link library (DLL) initialization routine failed."  If you then erase 
the pyc cache file that is importing the internal module then it works again.  
This only occurs on Windows and was not present using the same source prior 
versions of Python 3.8.

We investigate the byte codes from both version and they are doing the same 
series of actions so the problem appears to be calling the same opcode to 
execute an import.

I make sure all required symbols were found in the libraries and only only copy 
of the internal DLL was the same with and without loading from a pyc.

It may be a change in the requirements of module initialization, but I don't 
know how to proceed.  There was one deprecation warning but correcting that did 
not alter the outcome.  

It appears that the execute path for importing a CPython module takes a 
different path when the script was imported from something compiled on the fly 
as opposed to loading from a pyc.
 

 ```
 import: 'jpype'
 Traceback (most recent call last):
File "D:\bld\jpype1_1605785280189\test_tmp\run_test.py", line 2, in 
  import jpype
File 
"D:\bld\jpype1_1605785280189\_test_env\lib\site-packages\jpype\__init__.py", 
line 18, in 
  import _jpype
 ImportError: DLL load failed while importing _jpype: A dynamic link library 
(DLL) initialization routine failed.
 ```

--
components: Windows
messages: 382275
nosy: Thrameos, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: CPython DLL initialization routine failed from PYC cache file
type: behavior
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue42529>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39416] Document default numeric string formats

[Karl O. Pinc]

Change by Karl O. Pinc :


--
pull_requests: +21809
pull_request: https://github.com/python/cpython/pull/22867

___
Python tracker 
<https://bugs.python.org/issue39416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42104] xml.etree should support contains() function

[karl]

New submission from karl :

In XPath 1.0 
The function contains() is available 

> Function: boolean contains(string, string)
> The contains function returns true if the first argument string contains the 
> second argument string, and otherwise returns false.

In https://www.w3.org/TR/1999/REC-xpath-19991116/#function-contains


```

   One attribute: doc
   Two Attributes: doc test
   One Attribute: test

```

Currently, we can do this

```
>>> from lxml import etree
>>> root = etree.fromstring("""
...One attribute
...Two Attributes: doc test
...Two Attributes: doc2 test
... 
... """)
>>> elts = root.xpath("//p[@class='doc']")
>>> elts, etree.tostring(elts[0])
([], b'One attribute\n   ')
```


One way of extracting the list of 2 elements which contains the attribute doc 
with XPath is:


```
>>> root.xpath("//p[contains(@class, 'doc')]")
[, ]
>>> [etree.tostring(elt) for elt in root.xpath("//p[contains(@class, 'doc')]")]
[b'One attribute: doc\n   ', b'Two 
Attributes: doc test\n   ']
```


There is no easy way to extract all elements containing a "doc" value in a 
multi-values attribute in python 3.10 with xml.etree, which is quite common in 
html. 


```
>>> import xml.etree.ElementTree as ET
>>> root = ET.fromstring("""
...One attribute: doc
...Two Attributes: doc test
...One Attribute: test
... """
... )
>>> root.xpath("//p[contains(@class, 'doc')]")
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'xpath'
```

--
components: Library (Lib)
messages: 379185
nosy: karlcow
priority: normal
severity: normal
status: open
title: xml.etree should support contains() function
type: enhancement
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue42104>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40236] datetime.datetime.strptime get day error

[karl]

karl  added the comment:

Also this.

>>> import datetime
>>> d0 = datetime.datetime.strptime("2024-0-3 00:00:00", "%Y-%W-%w %H:%M:%S")
>>> d0.strftime("%Y-%W-%w %H:%M:%S")
'2024-01-3 00:00:00'
>>> d1 = datetime.datetime.strptime("2024-1-3 00:00:00", "%Y-%W-%w %H:%M:%S")
>>> d1.strftime("%Y-%W-%w %H:%M:%S")
'2024-01-3 00:00:00'
>>> d2301 = datetime.datetime.strptime("2023-0-1 00:00:00", "%Y-%W-%w %H:%M:%S")
>>> d2311 = datetime.datetime.strptime("2023-1-1 00:00:00", "%Y-%W-%w %H:%M:%S")
>>> d2301
datetime.datetime(2022, 12, 26, 0, 0)
>>> d2311
datetime.datetime(2023, 1, 2, 0, 0)
>>> d2311.strftime("%Y-%W-%w %H:%M:%S")
'2023-01-1 00:00:00'
>>> d2301.strftime("%Y-%W-%w %H:%M:%S")
'2022-52-1 00:00:00'


Week 0 2023 became Week 52 2022 (which is correct but might lead to surprises)

--

___
Python tracker 
<https://bugs.python.org/issue40236>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40236] datetime.datetime.strptime get day error

[karl]

karl  added the comment:

Same on macOS 10.15.6 (19G73)


Python 3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> datetime.datetime.strptime("2024-0-3 00:00:00", "%Y-%W-%w %H:%M:%S")
datetime.datetime(2024, 1, 3, 0, 0)
>>> datetime.datetime.strptime("2024-1-3 00:00:00", "%Y-%W-%w %H:%M:%S")
datetime.datetime(2024, 1, 3, 0, 0)


Also 
https://pubs.opengroup.org/onlinepubs/007908799/xsh/strptime.html

note that iso8601 doesn't have this issue.
%V - ISO 8601 week of the year as a decimal number [01, 53].
https://en.wikipedia.org/wiki/ISO_week_date

--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue40236>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36661] Missing dataclass decorator import in dataclasses module docs

[karl]

karl  added the comment:

This should be closed. The PR has been merged and the doc is now up to date.

--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue36661>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40297] test_socket.CANTest is broken at HEAD on master

[Karl Ding]

Karl Ding  added the comment:

Related: I've started a thread on Discourse [0] looking into why the tests 
don't seem to be run on the Buildbot cluster (and what it would take to enable 
them). Hopefully it gains some traction.

[0] 
https://discuss.python.org/t/what-would-it-take-to-run-socketcan-tests-on-a-buildbot/4137

--

___
Python tracker 
<https://bugs.python.org/issue40297>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40291] socket library support for CAN_J1939

[Karl Ding]

Karl Ding  added the comment:

Should this be added to the What's New for 3.9? I see a smaller change that 
exposes the CAN_RAW_JOIN_FILTERS constant mentioned.

I've created https://github.com/python/cpython/pull/20248 if this is needed.

--

___
Python tracker 
<https://bugs.python.org/issue40291>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40291] socket library support for CAN_J1939

[Karl Ding]

Change by Karl Ding :


--
pull_requests: +19536
pull_request: https://github.com/python/cpython/pull/20248

___
Python tracker 
<https://bugs.python.org/issue40291>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40297] test_socket.CANTest is broken at HEAD on master

[Karl Ding]

Change by Karl Ding :


--
keywords: +patch
pull_requests: +18894
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19548

___
Python tracker 
<https://bugs.python.org/issue40297>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40297] test_socket.CANTest is broken at HEAD on master

[Karl Ding]

New submission from Karl Ding :

While working on https://bugs.python.org/issue40291, I was trying to run the 
SocketCAN tests to ensure that my changes weren't causing any regressions. 
However, I was seeing test failures at HEAD.

I'm running the tests like so:

# Kernel version
uname -r
# 5.4.23-1-MANJARO

# Load SocketCAN vcan kernel module
sudo modprobe vcan

# Start and set up a virtual SocketCAN interface
sudo ip link add dev vcan0 type vcan
sudo ip link set up vcan0

# Run the socket tests using locally built python
./python -m unittest -v test.test_socket.CANTest

After bisecting, I discovered that the test started failing back in 2017, with 
the introduction of the ISOTP protocol. 
https://github.com/python/cpython/pull/2956/files#diff-a47fd74731aeb547ad780900bb8e6953L1376-R1391

Here, the address family (the second tuple item) is explicitly removed:

diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
index bf8d19fe2f..beadecfad5 100644
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -1373,9 +1373,22 @@ makesockaddr(SOCKET_T sockfd, struct sockaddr *addr, 
size_t addrlen, int proto)
 ifname = ifr.ifr_name;
 }

-return Py_BuildValue("O", PyUnicode_DecodeFSDefault,
-ifname,
-a->can_family);
+switch (proto) {
+#ifdef CAN_ISOTP
+  case CAN_ISOTP:
+  {
+  return Py_BuildValue("O", PyUnicode_DecodeFSDefault,
+  ifname,
+  a->can_addr.tp.rx_id,
+  a->can_addr.tp.tx_id);
+  }
+#endif
+  default:
+  {
+  return Py_BuildValue("O&", PyUnicode_DecodeFSDefault,
+ifname);
+  }
+}
 }
 #endif

This seems to be an intentional breakage, since the code in getsockaddrarg also 
operates on just the interface name, ignoring the address family.

if (!PyArg_ParseTuple(args,
  "O&;AF_CAN address must be a tuple "
  "(interface, )",
  PyUnicode_FSConverter, ))

As such, I believe the test should have been updated, but was missed during the 
ISOTP changes. Can someone (a core member?) confirm that this is the correct 
approach?

Note: However, this also implies that the CANTest test was never being run 
(either via CI on PRs or on the Buildbot cluster) and instead was always 
skipped, since I would've expected this failure to show up right away...

--
components: Library (Lib)
messages: 366576
nosy: karlding
priority: normal
severity: normal
status: open
title: test_socket.CANTest is broken at HEAD on master
type: behavior
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue40297>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40291] socket library support for CAN_J1939

[Karl Ding]

Change by Karl Ding :


--
keywords: +patch
pull_requests: +18886
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19538

___
Python tracker 
<https://bugs.python.org/issue40291>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40291] socket library support for CAN_J1939

[Karl Ding]

New submission from Karl Ding :

It would be nice to have support J1939 sockets.

Support for J1939 landed as part of the SAE J1939 SocketCAN patches, which are 
available after the Linux 5.4rc1 kernel. This is another protocol family for 
SocketCAN, much like the existing support for ISOTP and BCM.

Effectively, supporting this would hand off as much to the kernel as possible, 
which gives Python programs the ability deal with J1939 without having to 
implement the logic existing in the kernel in userspace.

This is provided to userspace [0] via the  header.

I'd be happy to work on this and provide a PR.

[0] https://www.kernel.org/doc/html/latest/networking/j1939.html

--
components: Library (Lib)
messages: 366487
nosy: karlding
priority: normal
severity: normal
status: open
title: socket library support for CAN_J1939
type: enhancement
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue40291>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39416] Document default numeric string formats

[Karl O. Pinc]

Karl O. Pinc  added the comment:

On Wed, 22 Jan 2020 06:09:41 +
"Eric V. Smith"  wrote:

> Eric V. Smith  added the comment:
> 
> Is the lack of this documentation causing some confusion somewhere?
> This isn't rhetorical, I'm genuinely curious what problem you're
> trying to solve.

I'd say no, there's no real confusion anywhere.

This started with a search for documentation on Python's
default string representation for float.  This accidentally
wound up on the python-dev mailing list and became a
more general discussion during which I suggested that
some fundamentals are clear.  This is pretty much just
a follow-up to that discussion.

> Is there any mainstream programming language where the basics of what
> you've laid out aren't true? It all seems pretty obvious to me. For
> example, I can't see anyone looking at this and saying "Ah, base 10.
> That's why it's producing the output I'm seeing."

Well, I just got done looking at YAML which has a lot of base-related
syntax including base 60.  :)

The problem that would be solved is that it's easy to rely
on the default Numeric output formats.  Most output probably
does not go through a formatter, although this may change
now that f-strings are so easy.  Anyway, this would guarantee
"normal output" for numbers, even when switching between
Python implementations.

Likewise, immutability of value when round-tripping through a
string is also obvious.  But is still an important property.

So if these properties are important then include them in the spec.
That's what specs are for.  (I could probably find some non-mainstream
languages (scheme perhaps) that have specs which include
similar documentation.  You pretty have to have a formatter
in compiled languages, so those won't specify a default.
Anyway, it shouldn't matter what other language specs do.)

I understand if nobody sees this as a real problem.  And
I don't want to stand up as a big proponent.  I'll
point out the argument here and let others decide.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

--

___
Python tracker 
<https://bugs.python.org/issue39416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39416] Document default numeric string formats

[Karl O. Pinc]

Change by Karl O. Pinc :


--
keywords: +patch
pull_requests: +17498
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/18111

___
Python tracker 
<https://bugs.python.org/issue39416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39416] Document default numeric string formats

[Karl O. Pinc]

New submission from Karl O. Pinc :

Seems sane to put _some_ restrictions on the string representations of the 
Numeric classes.  This would be a change to the Python language
specification.

Suggestions made in a pull request.

See the email thread:
Subject: Documenting Python's float.__str__()
https://mail.python.org/archives/list/python-...@python.org/thread/FV22TKT3S2Q3P7PNN6MCXI6IX3HRRNAL/

--
assignee: docs@python
components: Documentation
messages: 360442
nosy: docs@python, kop
priority: normal
severity: normal
status: open
title: Document default numeric string formats

___
Python tracker 
<https://bugs.python.org/issue39416>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1375011] http.cookies, Cookie.py: Improper handling of duplicate cookies

[karl]

karl  added the comment:

Relevant spec
https://tools.ietf.org/html/rfc6265

--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue1375011>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9004] datetime.utctimetuple() should not set tm_isdst flag to 0

[karl]

karl  added the comment:

@gaurav The pull request 
https://github.com/python/cpython/pull/10870
has been closed in favor of 
https://github.com/python/cpython/pull/15773
which has already been merged.

So we can probably close here.

--
message_count: 7.0 -> 8.0
nosy: +karlcow
nosy_count: 7.0 -> 8.0
pull_requests: +16145
pull_request: https://github.com/python/cpython/pull/15773

___
Python tracker 
<https://bugs.python.org/issue9004>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19683] test_minidom has many empty tests

[karl]

karl  added the comment:

err… Errata on my previous comment.

"""
Simple implementation of the Level 1 DOM.

Namespaces and other minor Level 2 features are also supported.
"""
https://github.com/python/cpython/blob/c65119d5bfded03f80a9805889391b66fa7bf551/Lib/xml/dom/minidom.py#L1-L3


https://www.w3.org/TR/REC-DOM-Level-1/

--

___
Python tracker 
<https://bugs.python.org/issue19683>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19683] test_minidom has many empty tests

[karl]

karl  added the comment:

@zach.ware
@r.david.murray


So I was looking at that issue. There is a lot of work. 

I had a couple of questions, because there are different categories


# Empty tests for existing functions.

This seems to be straightforward as they would complete the module.

Example:
```python
def testGetAttributeNode(self): pass
```
https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/test/test_minidom.py#L412

which refers to: `GetAttributeNode`
https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/xml/dom/minidom.py#L765-L768
https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/test/test_minidom.py#L285-L294



# Tests without any logical reference in the module.

This is puzzling because I'm not sure which DOM feature they should be testing.

For example:

```
def testGetAttrList(self):
pass
```

https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/test/test_minidom.py#L383-L384


Or maybe this is just supposed to test Element.attributes returning a list of 
attributes, such as 
`NamedNodeMap [ def="ghi", jkl="mno"]` returned by a browser.


```
>>> import xml.dom.minidom
>>> from xml.dom.minidom import parse, Node, Document, parseString
>>> from xml.dom.minidom import getDOMImplementation
>>> dom = parseString("")
>>> el = dom.documentElement
>>> el.setAttribute("def", "ghi")
>>> el.setAttribute("jkl", "mno")
>>> el.attributes

```

or is it supposed to test something like 

```
>>> el.attributes.items()
[('def', 'ghi'), ('jkl', 'mno')]
```

This is slightly confusing. And the missing docstrings are not making it easier.



# Tests which do not really test the module(?)

I think for example about this, which is testing that `del` is working, but it 
doesn't have anything to do with the DOM. 

```
def testDeleteAttr(self):
dom = Document()
child = dom.appendChild(dom.createElement("abc"))

self.confirm(len(child.attributes) == 0)
child.setAttribute("def", "ghi")
self.confirm(len(child.attributes) == 1)
del child.attributes["def"]
self.confirm(len(child.attributes) == 0)
dom.unlink()
```

https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/test/test_minidom.py#L285-L294

Specifically when there is a function for it: `removeAttribute`
https://www.w3.org/TR/2000/REC-DOM-Level-2-Core-20001113/core.html#ID-6D6AC0F9 

which is tested just below that test.
https://github.com/python/cpython/blob/3e04cd268ee9a57f95dc78d8974b21a6fac3f666/Lib/test/test_minidom.py#L296-L305

so I guess these should be removed or do I miss something in the testing logic?




# Missing docstrings.

Both the testing module and the module lack a lot of docstrings.
Would it be good to fix this too, probably in a separate commit.



# DOM Level 2

So the module intent is to implement DOM Level 2.
but does that make sense in the light of 
https://dom.spec.whatwg.org/

Should minidom tries to follow the current DOM spec?

--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue19683>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38352] In typing docs, note explicit import needed for IO and Pattern/Match

[Karl Kornel]

New submission from Karl Kornel :

Hello!

In https://github.com/python/cpython/blob/master/Lib/typing.py#L115-L117, there 
is a note about the io and re classes not being included in typing.__all__.  I 
am a relatively new user of typing, and I did `from typing import *` in my 
code.  I ran the code through mypy first, which reported no problems, but then 
running Python 3.6 failed with a NameError (name 'IO' is not defined).

Reading through the typing source, it's clear that this was an intentional 
decision.  So, instead of reporting a bug, I'd like to request a documentation 
enhancement!

The docs for typing make no mention of typing.io or typing.re.  So, my request 
is: In the sections for the IO/TextIO/BinaryIO and Pattern/Match classes, 
include text warning the user that these types are not imported when you do 
`from typing import *`.

--
assignee: docs@python
components: Documentation
messages: 353763
nosy: Karl Kornel, docs@python
priority: normal
severity: normal
status: open
title: In typing docs, note explicit import needed for IO and Pattern/Match
type: behavior
versions: Python 3.6

___
Python tracker 
<https://bugs.python.org/issue38352>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22377] %Z in strptime doesn't match EST and others

[karl]

karl  added the comment:

I created a PR following the recommendations of p-ganssle
https://github.com/python/cpython/pull/16507

--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue22377>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18182] xml.dom.createElement() does not take implicit namespaces into account

[karl]

karl  added the comment:

The current specification as of today documents
https://dom.spec.whatwg.org/#dom-document-createelementns


If you run this in the browser console, 

var nsdoc = 'http://foo.bar/zoo';
var xmldoc = document.implementation.createDocument(nsdoc, 'Zoo', null);
var cpd = document.createElementNS(nsdoc, 'Compound');
var chimp = document.createElementNS(nsdoc, 'Chimp');
cpd.appendChild(chimp)
xmldoc.documentElement.appendChild(cpd);

/* serializing */
var docserializer = new XMLSerializer();
var flatxml = docserializer.serializeToString(xmldoc);
flatxml


you get:

http://foo.bar/zoo;>
  

  



but if you run this in the browser console,

var nsdoc = 'http://foo.bar/zoo';
var xmldoc = document.implementation.createDocument(nsdoc, 'Zoo', null);
var cpd = document.createElement('Compound');
var chimp = document.createElement('Chimp');
cpd.appendChild(chimp)
xmldoc.documentElement.appendChild(cpd);

/* serializing */
var docserializer = new XMLSerializer();
var flatxml = docserializer.serializeToString(xmldoc);
flatxml


you get:


http://foo.bar/zoo;>
  http://www.w3.org/1999/xhtml;>

  



which is a complete different beast.


I don't think there is an issue here. And we can close this bug safely.

--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue18182>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35638] Introduce fixed point locale aware format type for floating point numbers

[Karl Ove Hufthammer]

Change by Karl Ove Hufthammer :


--
nosy: +huftis

___
Python tracker 
<https://bugs.python.org/issue35638>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23312] google thinks the docs are mobile unfriendly

[karl]

karl  added the comment:

I created https://github.com/python/python-docs-theme/issues/30

--

___
Python tracker 
<https://bugs.python.org/issue23312>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23312] google thinks the docs are mobile unfriendly

[karl]

karl  added the comment:

This issue should probably be addressed now on 
https://github.com/python/python-docs-theme

--
nosy: +karlcow

___
Python tracker 
<https://bugs.python.org/issue23312>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24772] Smaller viewport shifts the "expand left menu" character into the text

[karl]

karl  added the comment:

So I had time to look at it today. 
And it would probably be better to solve 
https://bugs.python.org/issue23312

which would make this one here useless and would actually provide a solution 
for many people.

--

___
Python tracker 
<https://bugs.python.org/issue24772>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24772] Smaller viewport shifts the "expand left menu" character into the text

[karl]

karl  added the comment:

I'm at Mozilla All Hands this week. 
I'll check if my solution still makes sense next week and will make a pull 
request and/or propose another solution. 
Thanks for the reminder. adding to my calendar.

--

___
Python tracker 
<https://bugs.python.org/issue24772>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Karl Ding]

Karl Ding  added the comment:

I believe the example can be simplified to the following:

class MyStructure(ctypes.Structure):
_pack_ = 1
_fields_ = [('A', ctypes.c_uint32, 8)]

assert ctypes.sizeof(MyStructure) == 1

Here, ctypes.sizeof returns 4 on my machine (64-bit Ubuntu 18.04 LTS), while I 
would expect it to return 1 like GCC. This is using a tree checked out at 
33ce3f012f249782507df896824b045b34f765aa, if it makes a difference.

If we compile the equivalent C code (on 64-bit Ubuntu 18.04 LTS):

#include 
#include 
#include 

struct my_structure_uint32 {
  uint32_t a : 8;
} __attribute__((packed));

int main(int argc, char *argv[]) {
  printf("sizeof(struct my_structure_uint32): %d\n", sizeof(struct 
my_structure_uint32));
  printf("alignof(struct my_structure_uint32): %d\n", alignof(struct 
my_structure_uint32));
  return 0;
}

Using the following GCC invocation:

gcc temp.c -o test && ./test

We get the following result:

sizeof(struct my_structure_uint32): 1
alignof(struct my_structure_uint32): 1

However, if I compile with MSVC bitfields:

gcc -mms-bitfields test.c -o test && ./test

We get the following result:

sizeof(struct my_structure_uint32): 4
alignof(struct my_structure_uint32): 1

Similarly, adding a second and third 8-bit wide bitfield member fails and 
returns 4, instead of the expected 2 and 3.

This is not just c_uint32, c_uint16 and c_int also exhibit the same behaviour:

class MyStructure(ctypes.Structure):
_pack_ = 1
_fields_ = [('A', ctypes.c_uint16, 8)]

assert ctypes.sizeof(MyStructure) == 1

--
nosy: +karlding

___
Python tracker 
<https://bugs.python.org/issue29753>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37085] Expose additional socket constants for CAN_BCM flags

[Karl Ding]

Change by Karl Ding :


--
keywords: +patch
pull_requests: +13541
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/13646

___
Python tracker 
<https://bugs.python.org/issue37085>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37085] Expose additional socket constants for CAN_BCM flags

[Karl Ding]

New submission from Karl Ding :

When reading through the values exposed via the socket library, I noticed that 
currently, only the SocketCAN BCM opcode enums are exposed via the socket 
constants. These correspond to the following from the Linux headers:

enum {
TX_SETUP = 1,   /* create (cyclic) transmission task */
TX_DELETE,  /* remove (cyclic) transmission task */
TX_READ,/* read properties of (cyclic) transmission 
task */
TX_SEND,/* send one CAN frame */
RX_SETUP,   /* create RX content filter subscription */
RX_DELETE,  /* remove RX content filter subscription */
RX_READ,/* read properties of RX content filter 
subscription */
TX_STATUS,  /* reply to TX_READ request */
TX_EXPIRED, /* notification on performed transmissions 
(count=0) */
RX_STATUS,  /* reply to RX_READ request */
RX_TIMEOUT, /* cyclic message is absent */
RX_CHANGED  /* updated CAN frame (detected content change) 
*/
};

It would be nice to expose the BCM flags #defines as well.

#define SETTIMER0x0001
#define STARTTIMER  0x0002
#define TX_COUNTEVT 0x0004
#define TX_ANNOUNCE 0x0008
#define TX_CP_CAN_ID0x0010
#define RX_FILTER_ID0x0020
#define RX_CHECK_DLC0x0040
#define RX_NO_AUTOTIMER 0x0080
#define RX_ANNOUNCE_RESUME  0x0100
#define TX_RESET_MULTI_IDX  0x0200
#define RX_RTR_FRAME0x0400
#define CAN_FD_FRAME0x0800

These BCM flags are used as part of the BCM header that has the aforementioned 
opcodes. The flags are set on the bcm_msg_head struct:

struct bcm_msg_head {
__u32 opcode;
__u32 flags;
__u32 count;
struct bcm_timeval ival1, ival2;
canid_t can_id;
__u32 nframes;
struct can_frame frames[0];
};

The existing documentation for the BCM constants 
(https://docs.python.org/3/library/socket.html#socket.CAN_BCM) seems to imply 
that these constants should already be included, but they are not.

See the Linux headers for more details: 
https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/can/bcm.h

--
components: Library (Lib)
messages: 343865
nosy: karlding
priority: normal
severity: normal
status: open
title: Expose additional socket constants for CAN_BCM flags
type: enhancement
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue37085>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8136] urllib.unquote decodes percent-escapes with Latin-1

[karl]

karl added the comment:

#8143 was fixed.

Python 2.7.10 (default, Feb  7 2017, 00:08:15) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib
>>> urllib.unquote(u"%CE%A3")
u'\xce\xa3'

What should become of this one?

--
nosy: +karlcow

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue8136>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24712] Docs page's sidebar vibrates on mouse wheel scroll on Chrome.

[karl]

karl added the comment:

@ezio.melotti 
What is the next step for getting the patch accepted.

--
nosy: +karlcow

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue24712>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2943] Distutils should generate a better error message when the SDK is not installed

[Karl Stahl]

Changes by Karl Stahl <stahl.k...@gmail.com>:


--
nosy: +Karl Stahl

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue2943>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24772] Smaller viewport shifts the "expand left menu" character into the text

[karl]

karl added the comment:

Where should I propose a patch to help resolve this issue?
A pointer to the code in the repo and I will do it.

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue24772>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15873] datetime: add ability to parse RFC 3339 dates and times

[karl]

karl added the comment:

About

> Actually, I realized that the best implementation of parsing rfc 3339 
> is in django dateparse utils. To me it's the finest, the most 
> elegant, and no other one can claim to be more robust since it's 
> probably the #1 iso parsing functions used in python. Have a look at 
> https://docs.djangoproject.com/en/1.9/_modules/django/utils/dateparse/#parse_datetime

How does it parse this date: 

2016-02-15T11:59:46.16588638674+09:00

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue15873>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15873] datetime: add ability to parse RFC 3339 dates and times

[karl]

karl added the comment:

>From https://www.djangoproject.com/foundation/cla/faq/

> Am I giving away the copyright to my contributions?
> 
> No. This is a pure license agreement, not a copyright assignment. You 
> still maintain the full copyright for your contributions. You are 
> only providing a license to the DSF to distribute your code without 
> further restrictions. This is not the case for all CLA's, but it is 
> the case for the one we are using.

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue15873>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26011] Document necesities for cmp argument of sorted

[Karl Richter]

New submission from Karl Richter:

The docstring of `sorted` doesn't explain what can be passed to the `cmp` and 
`key` argument of `sorted`.

--
assignee: docs@python
components: Documentation
messages: 257505
nosy: docs@python, krichter
priority: normal
severity: normal
status: open
title: Document necesities for cmp argument of sorted
type: enhancement
versions: Python 2.7

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26011>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25946] configure should pick /usr/bin/g++ automatically if present

[Karl Richter]

New submission from Karl Richter:

`./configure` both prints `checking for g++... no` and 

WARNING:

  By default, distutils will build C++ extension modules with "g++".
  If this is not intended, then set CXX on the configure command line.

if `/usr/bin/g++` is present and executable which doesn't seem to be 
constructive because it's quite common that one wants to use `/usr/bin/g++` as 
CXX compiler if available. In case incompatibilities exists with other C++ 
compilers there should a check and more detailed error message.

Furthermore the error message doesn't explain if a part of distutils won't be 
build because the message sounds like the C++ extension is built, but does it 
work without a C++ compiler?

Specifying `CXX` environment variable or `--with-cxx-main=/usr/bin/g++` 
`configure` option works fine.

experienced with 8a2e735 (on branch 2.7)

--
components: Build
messages: 256977
nosy: krichter
priority: normal
severity: normal
status: open
title: configure should pick /usr/bin/g++ automatically if present
versions: Python 2.7

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue25946>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25787] Add an explanation what happens with subprocess parent and child processes when signals are sent

[Karl Richter]

Karl Richter added the comment:

Please also explain how to deal with process replacement in child processes 
(assuming that 
http://stackoverflow.com/questions/34059576/how-to-register-a-signal-handler-for-a-subprocess/34065587#34065587
 is correct).

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue25787>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25787] Add an explanation what happens with subprocess parent and child processes when signals are sent

[Karl Richter]

New submission from Karl Richter:

The [documentation of 
subprocess](https://docs.python.org/3.6/library/subprocess.html) doesn't 
contain a substantial statement how signals are handled which are send to the 
python interpreter. After reading the referenced docs it should be clear

  * whether a signal is passed to both the parent and the child (If yes in 
which order? What happens if the child process spawns a process which isn't 
controlled by python?)
  * whether signal handlers are inherited (judging from the `restore_signals` 
parameter some are overwritten -> what's the purpose of this?). Are changes of 
a signal handler in the parent reflected in the child?

--
assignee: docs@python
components: Documentation
messages: 255802
nosy: docs@python, krichter
priority: normal
severity: normal
status: open
title: Add an explanation what happens with subprocess parent and child 
processes when signals are sent
versions: Python 3.6

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue25787>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24772] Smaller viewport shifts the expand left menu character into the text

[karl]

New submission from karl:

Adding the following to basic.css:

```css
dl {
margin-bottom: 15px;
word-wrap: break-word;
}
```

will solve the issue.

See https://github.com/webcompat/web-bugs/issues/1479

--
assignee: docs@python
components: Documentation
messages: 247803
nosy: docs@python, karlcow
priority: normal
severity: normal
status: open
title: Smaller viewport shifts the expand left menu character into the text

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24772
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24506] make fails with gcc 4.9 due to fatal warning of unused variable and empty macro in Parser/pgen.c

[Karl Richter]

New submission from Karl Richter:

`gcc` 4.9 is more restictive and recognizes that the empty definition of the 
`REQN` macro doesn't use some variables. It's more suitable to wrap the usage 
of the macro in the same preprocessor conditionals like the macro definition.

experienced with 41fbc222af8c503e1659250a36f4e293d864a92b

Patch attached. Tests pass (see 
https://semaphoreci.com/krichter/cpython/branches/empty_macro/builds/1 for 
details).

--
components: Build
files: 0001-removed-empty-definition-of-REQN-preprocessor-macro-.patch
keywords: patch
messages: 245781
nosy: krichter
priority: normal
severity: normal
status: open
title: make fails with gcc 4.9 due to fatal warning of unused variable and 
empty macro in Parser/pgen.c
versions: Python 2.7
Added file: 
http://bugs.python.org/file39806/0001-removed-empty-definition-of-REQN-preprocessor-macro-.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24506
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24506] make fails with gcc 4.9 due to fatal warning of unused variable and empty macro in Parser/pgen.c

[Karl Richter]

Karl Richter added the comment:

It's a fatal warning of `gcc 4.9.2`, not an error (my bad) for `int i;` in 
`Parser/pgen.c` line 227. It might be ignored as well, but I think my approach 
is more elegant and deals with issues sooner than later.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24506
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24331] *** Error in `/usr/bin/python': double free or corruption (!prev): 0x0000000000f5c760 *** when cloning hg repository into directory on cifs

[Karl Richter]

New submission from Karl Richter:

I experience the error in the title exclusive when invoking `hg clone` (e.g. 
`hg clone https://bitbucket.org/Coin3D/coin` or `hg clone 
http://hg.netbeans.org/main-golden/ netbeans-main-golden`) when the target 
directory is on a cifs mount.

`gdb` backtrace:

#0  0x7761c267 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:55
#1  0x7761deca in __GI_abort () at abort.c:89
#2  0x7765fc53 in __libc_message (do_abort=do_abort@entry=1, 
fmt=fmt@entry=0x777781a8 *** Error in `%s': %s: 0x%s ***\n)
at ../sysdeps/posix/libc_fatal.c:175
#3  0x77667c69 in malloc_printerr (ptr=optimized out, 
str=0x777782d8 double free or corruption (!prev), action=1)
at malloc.c:4965
#4  _int_free (av=optimized out, p=optimized out, have_lock=0)
at malloc.c:3834
#5  0x7766b89c in __GI___libc_free (mem=optimized out)
at malloc.c:2950
#6  0x7130ce11 in lfree (a=a@entry=0xecde10) at 
mercurial/mpatch.c:67
#7  0x7130d2e7 in decode (len=196, bin=0x1037ee8 )
at mercurial/mpatch.c:228
#8  fold (bins=[buffer at remote 0x710d40b0], start=start@entry=0, 
end=end@entry=1) at mercurial/mpatch.c:296
#9  0x7130d3d3 in patches (self=optimized out, args=optimized 
out)
at mercurial/mpatch.c:327
#10 0x004ccd05 in call_function (oparg=optimized out, 
pp_stack=optimized out) at ../Python/ceval.c:4035
#11 PyEval_EvalFrameEx () at ../Python/ceval.c:2681
---Type return to continue, or q return to quit---
#12 0x004cd4e2 in fast_function (nk=optimized out, 
na=optimized out, n=optimized out, pp_stack=optimized out, 
func=optimized out) at ../Python/ceval.c:4121
#13 call_function (oparg=optimized out, pp_stack=optimized out)
at ../Python/ceval.c:4056
#14 PyEval_EvalFrameEx () at ../Python/ceval.c:2681
#15 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267
#16 0x004ce7d3 in fast_function (nk=optimized out, 
na=optimized out, n=optimized out, pp_stack=optimized out, 
func=optimized out) at ../Python/ceval.c:4131
#17 call_function (oparg=optimized out, pp_stack=optimized out)
at ../Python/ceval.c:4056
#18 PyEval_EvalFrameEx () at ../Python/ceval.c:2681
#19 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267
#20 0x004ce7d3 in fast_function (nk=optimized out, 
na=optimized out, n=optimized out, pp_stack=optimized out, 
func=optimized out) at ../Python/ceval.c:4131
#21 call_function (oparg=optimized out, pp_stack=optimized out)
at ../Python/ceval.c:4056
#22 PyEval_EvalFrameEx () at ../Python/ceval.c:2681
#23 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267
#24 0x004ce7d3 in fast_function (nk=optimized out, 
na=optimized out, n=optimized out, pp_stack=optimized out, 
---Type return to continue, or q return to quit---
func=optimized out) at ../Python/ceval.c:4131
#25 call_function (oparg=optimized out, pp_stack=optimized out)
at ../Python/ceval.c:4056
#26 PyEval_EvalFrameEx () at ../Python/ceval.c:2681
#27 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267
#28 0x004cd217 in fast_function (nk=optimized out, 
na=optimized out, n=optimized out, pp_stack=optimized out, 
func=optimized out) at ../Python/ceval.c:4131
#29 call_function (oparg=optimized out, pp_stack=optimized out)
at ../Python/ceval.c:4056
#30 PyEval_EvalFrameEx () at ../Python/ceval.c:2681
#31 0x004cd4e2 in fast_function (nk=optimized out, 
na=optimized out, n=optimized out, pp_stack=optimized out, 
func=optimized out) at ../Python/ceval.c:4121
#32 call_function (oparg=optimized out, pp_stack=optimized out)
at ../Python/ceval.c:4056
#33 PyEval_EvalFrameEx () at ../Python/ceval.c:2681
#34 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267
#35 0x004cd217 in fast_function (nk=optimized out, 
na=optimized out, n=optimized out, pp_stack=optimized out, 
func=optimized out) at ../Python/ceval.c:4131
#36 call_function (oparg=optimized out, pp_stack=optimized out)
at ../Python/ceval.c:4056
---Type return to continue, or q return to quit---
#37 PyEval_EvalFrameEx () at ../Python/ceval.c:2681
#38 0x004cb6b1 in PyEval_EvalCodeEx () at ../Python/ceval.c:3267
#39 0x004cd217 in fast_function (nk=optimized out, 
na=optimized out, n=optimized out, pp_stack=optimized out, 
func=optimized out) at ../Python/ceval.c:4131
#40 call_function (oparg=optimized out, pp_stack=optimized out)
at ../Python/ceval.c:4056
#41 PyEval_EvalFrameEx () at ../Python/ceval.c:2681
#42

[issue24143] Makefile in tarball don't provide make uninstall target

[Karl Richter]

Changes by Karl Richter krichter...@aol.de:


--
components: Build
nosy: krichter
priority: normal
severity: normal
status: open
title: Makefile in tarball don't provide make uninstall target
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24143
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24060] Clearify necessities for logging with timestamps

[Karl Richter]

New submission from Karl Richter:

The `Formatter` section of the `logging` module at 
https://docs.python.org/2/library/logging.html#formatter-objects reads like 
it's sufficient to create an instance of `Formatter` with default arguments 
(and set it as formatter of the `Handler` of a `Logger`) to have add timestamps 
to logging output.

--
assignee: docs@python
components: Documentation
messages: 242024
nosy: docs@python, krichter
priority: normal
severity: normal
status: open
title: Clearify necessities for logging with timestamps
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24060
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24021] document urllib.urlretrieve

[Karl Richter]

Karl Richter added the comment:

 I suspect the complaint might be about the lack of doc string
Exactly. It'd be helpful to figure out the return value and the means of the 
function arguments in a more compact form than the referenced website docs and 
to have it available in the interpreter (with `help`), i.e. with docstring 
documentation.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24021
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24021] document urllib.urlretrieve

[Karl Richter]

Changes by Karl Richter krichter...@aol.de:


--
assignee: docs@python
components: Documentation
nosy: docs@python, krichter
priority: normal
severity: normal
status: open
title: document urllib.urlretrieve
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24021
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23814] argparse: Parser level defaults do not always override argument level defaults

[Karl O. Pinc]

New submission from Karl O. Pinc:

In the argparse library parser library, contrary to the documentation,
parser-level defaults do not always override argument-level defaults.

https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.set_defaults

says Note that parser-level defaults always override argument-level defaults:

(And so does the python 3.3 docs.)

The docs then provide this example:

   parser = argparse.ArgumentParser()
   parser.add_argument('--foo', default='bar')
   parser.set_defaults(foo='spam')
   parser.parse_args([])
  Namespace(foo='spam')

But it is only true that parser-level defaults override argument-level
defaults when they are established after the argument is added.

The output below shows an argument level default overrideing
a parser level default.

$ python3
Python 3.3.2 (default, Jun  4 2014, 11:36:37) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux
Type help, copyright, credits or license for more information.
 import argparse
 parser = argparse.ArgumentParser()
 parser.set_defaults(foo='spam')
 parser.add_argument('--foo', default='bar')
_StoreAction(option_strings=['--foo'], dest='foo', nargs=None, const=None, 
default='bar', type=None, choices=None, help=None, metavar=None)
 parser.parse_args([])
Namespace(foo='bar')

It seems that whichever default is set last is the one which is used.
Or perhaps there are not argument level defaults and parser level
defaults, there are just defaults, period.  (It might, possibly,
be nice if there _were_ both argument and parser level defaults
and parser level defaults had priority.  Then this would not be
a documentation bug.)

--
assignee: docs@python
components: Documentation
messages: 239632
nosy: docs@python, kop
priority: normal
severity: normal
status: open
title: argparse: Parser level defaults do not always override argument level 
defaults
type: behavior
versions: Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23814
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23174] shelve.open fails with error anydbm.error: db type could not be determined

[Karl Richter]

Karl Richter added the comment:

For example, it should be clear why `shelve.open(tempfile.mkstemp()[1])` fails 
with the mentioned exception and `shelve.open(/tmp/bla)` fails. I still 
haven't figured out the constraints to create a working `shelve.Shelve` at all. 
It should be clear why `shelve.open(/tmp/tmphTTQLd)` fails and 
`shelve.open(/tmp/tmphTTQLda)` succeeds. There has to be something unrelated 
to extensions.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23174
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23174] shelve.open fails with error anydbm.error: db type could not be determined

[Karl Richter]

Karl Richter added the comment:

After checking the code, I think that it'd make more sense to document 
`whichdb.py`. It needs to be enhanced with references to criteria for the 
determination of the database type. Currently there're only function comments 
and the fact that some variables are named magic speaks for itself, i.e. it's 
ok for the implementation of the module to be a black box, but the requirements 
for the input, i.e. the potential database file, which is per se not part of 
the black box, needs to be specified. Then just link that in the `shelve` docs 
stating that `shelve.open` is basically a wrapper around `whichdb`.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23174
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23163] pdb docs need to contain a statement on threads/multithreaded debugging

[Karl Richter]

Karl Richter added the comment:

Sorry, I mean 

#!/usr/bin/python

import threading

def debugging():
def __a_thread__():
print(2)
a_thread = threading.Thread(target=__a_thread__)
a_thread.start()
a_thread.join()
print(1)

if __name__ == __main__:
debugging()

It's very uncommon to set the current debugging thread in the source.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23163
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23163] pdb docs need to contain a statement on threads/multithreaded debugging

[Karl Richter]

Karl Richter added the comment:

My initial description was imprecise. Clearification: The fact that in

#!/usr/bin/python

import threading

def debugging():
def __a_thread__():
print(2)
a_thread = threading.Thread(target=__a_thread__)
print(1)

if __name__ == __main__:
debugging()

when invoked with `python -m pdb` the breakpoint at line 7 is ignored, like in

$ python -m pdb multithreaded_debugging.py 
 
/mnt/DATA/richter/examples/python/multithreaded_debugging/multithreaded_debugging.py(3)module()
- import threading
(Pdb) break 7
Breakpoint 1 at 
/mnt/DATA/richter/examples/python/multithreaded_debugging/multithreaded_debugging.py:7
(Pdb) c
1
The program finished and will be restarted
 
/mnt/DATA/richter/examples/python/multithreaded_debugging/multithreaded_debugging.py(3)module()
- import threading
(Pdb)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23163
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com