date:20210418



Change by Christian Heimes :


--
pull_requests: +24195
pull_request: https://github.com/python/cpython/pull/25470

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43880] 3.10 SSL module deprecations



Christian Heimes  added the comment:


New changeset 2875c603b2a7691b55c2046aca54831c91efda8e by Christian Heimes in 
branch 'master':
bpo-43880: Show DeprecationWarnings for deprecated ssl module features 
(GH-25455)
https://github.com/python/cpython/commit/2875c603b2a7691b55c2046aca54831c91efda8e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43837] Operator precedence documentation could be more clear

2021-04-18 Thread Chris Jerdonek


Chris Jerdonek  added the comment:

> So maybe we should change the terminology while we’re at it.

When math is taught to elementary school students in the US, it's called "order 
of operations": https://en.wikipedia.org/wiki/Order_of_operations

Since this was raised in the context of newcomers to coding, it might be worth 
mentioning that parallel. Being able to connect to the familiar concepts of 
"first" and "last" might help people not familiar with precedence and binding.

--
nosy: +chris.jerdonek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42854] OpenSSL 1.1.1: use SSL_write_ex() and SSL_read_ex()



Change by Christian Heimes :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42854] OpenSSL 1.1.1: use SSL_write_ex() and SSL_read_ex()



Christian Heimes  added the comment:


New changeset 89d1550d14ba689af12eeb726e4ff8ce73cee7e1 by Christian Heimes in 
branch 'master':
bpo-42854: Use SSL_read/write_ex() (GH-25468)
https://github.com/python/cpython/commit/89d1550d14ba689af12eeb726e4ff8ce73cee7e1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43837] Operator precedence documentation could be more clear

2021-04-18 Thread Ammar Askar



Ammar Askar  added the comment:

Opened https://github.com/python/cpython/pull/25469 for the first suggestion of 
reversing the precedence table order since it seems like a good improvement. I 
think the tight binding to weak order is more natural and aside from K, the 
following other languages also follow the same ordering:

* Javascript - 
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_Precedence#table
* PHP - https://www.php.net/manual/en/language.operators.precedence.php
* Rust - 
https://doc.rust-lang.org/reference/expressions.html#expression-precedence

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43837] Operator precedence documentation could be more clear

2021-04-18 Thread Ammar Askar



Change by Ammar Askar :


--
keywords: +patch
nosy: +ammar2
nosy_count: 8.0 -> 9.0
pull_requests: +24194
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/25469

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43837] Operator precedence documentation could be more clear



Change by Raymond Hettinger :


--
assignee: rhettinger -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43837] Operator precedence documentation could be more clear



Raymond Hettinger  added the comment:

Let's reverse the table in accordance with the de facto standard.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39298] add BLAKE3 to hashlib



Christian Heimes  added the comment:

3.10 feature freeze is in two weeks (May 3). I don't feel comfortable to add so 
much new C code shortly before beta 1. If I understandly correctly the code is 
new and hasn't been published on PyPI yet. I also don't have much time to 
properly review the code. OpenSSL 3.0.0 and PEP 644 is keeping me busy.

I would prefer to postpone the inclusion of blake3. Could you please publish 
the C version on PyPI first and let people test it?

Apropos OpenSSL, do you have plans to submit the algorithm to OpenSSL for 
inclusion in 3.1.0?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42853] `OverflowError: signed integer is greater than maximum` in ssl.py for files larger than 2GB



Christian Heimes  added the comment:

Python 3.10 will use SSL_write_ex() and SSL_read_ex(), which support > 2 GB 
data.

--
versions:  -Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42854] OpenSSL 1.1.1: use SSL_write_ex() and SSL_read_ex()



Christian Heimes  added the comment:

SSL_write_ex() and SSL_read_ex() solve two issues:

* bpo-42853: SSLSocket no longer raises overflow error when sending or 
receiving more than 2 GB of data
* bpo-31711: empty send(b"") no longer fails with protocol violation exception

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31711] ssl.SSLSocket.send(b"") fails



Christian Heimes  added the comment:

Thanks to PEP 644 the issue will be fixed in 3.10 by using SSL_read_ex and 
SSL_write_ex() functions. I couldn't use the functions earlier because Python 
had to support older OpenSSL versions and LibreSSL. 

See https://github.com/python/cpython/pull/25468 and bpo-42854

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
superseder:  -> OpenSSL 1.1.1: use SSL_write_ex() and SSL_read_ex()
versions: +Python 3.10 -Python 2.7, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39298] add BLAKE3 to hashlib

2021-04-18 Thread Larry Hastings



Larry Hastings  added the comment:

I note that Python already ships with some #ifdefs around SSE and the like.  
So, yes, we already do this sort of thing, although I think this usually uses 
compiler intrinsics rather than actual assembly.  A quick grep shows zero .s 
files and only one .asm file (./Modules/_decimal/libmpdec/vcdiv64.asm) in the 
Python tree.  Therefore it wouldn't be completely novel for Python but it's 
unusual.

I assume there's a completely generic platform-agnostic C implementation, for 
build environments where the assembly won't work, yes?


Disclaimer: I've been corresponding with Jack sporadically over the past year 
regarding the BLAKE3 Python API.  I also think BLAKE3 is super duper cool 
neat-o, and I have uses for it.  So I'd love to see it in Python 3.10.

One note, just to draw attention to it: the "blake3-py" module, also published 
by Jack, is written using the Rust implementation, which I understand is even 
more performant.  Obviously there's no chance Python would ship that 
implementation.  But by maintaining exact API compatibility between "blake3-py" 
and the "blake3" added to hashlib, this means code can use the fast one when 
it's available, and the built-in one when it isn't, a la CStringIO:

try:
from blake3 import blake3
except ImportError:
from hashlib import blake3

--
versions: +Python 3.10 -Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42854] OpenSSL 1.1.1: use SSL_write_ex() and SSL_read_ex()



Change by Christian Heimes :


--
keywords: +patch
pull_requests: +24193
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/25468

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39298] add BLAKE3 to hashlib

2021-04-18 Thread Jack O'Connor



Jack O'Connor  added the comment:

An update a year later: I have a proof-of-concept branch that adds BLAKE3 
support to hashlib: https://github.com/oconnor663/cpython/tree/blake3. That 
branch is API compatible with the current master branch of 
https://github.com/oconnor663/blake3-py. Both that module and the upstream 
BLAKE3 repo are ready to be tagged 1.0, just waiting to see whether any 
integrations like this one end up requesting changes.

Would anyone be interested in moving ahead with this? One of the open questions 
would be whether CPython would vendor the BLAKE3 optimized assembly files, or 
whether we'd prefer to stick to C intrinsics.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43833] Unexpected Parsing of Numeric Literals Concatenated with Boolean Operators

2021-04-18 Thread Guido van Rossum



Guido van Rossum  added the comment:

Actually I believe a real case was reported on python-dev. I think it is not 
clean that the boundary between numbers and identifiers is so fluid.

--
nosy: +Guido.van.Rossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43602] Include Decimal's in numbers.Real



Raymond Hettinger  added the comment:

Considering Oscar's response, Mark's comments, and prior discussions, we should 
close this.  No strong use cases have emerged that would warrant overturning 
the long-standing prior decisions on this topic.

--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43883] Making urlparse WHATWG conformant

2021-04-18 Thread Karthikeyan Singaravelan



Change by Karthikeyan Singaravelan :


--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43882] urllib.parse should sanitize urls containing ASCII newline and tabs.

2021-04-18 Thread Karthikeyan Singaravelan



Karthikeyan Singaravelan  added the comment:

See also a related issue to sanitise newline on other helper functions 
https://bugs.python.org/issue30713

See also discussion and compatibility on disallowing control characters : 
https://bugs.python.org/issue30458

--
nosy: +gregory.p.smith, vstinner, xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43833] Unexpected Parsing of Numeric Literals Concatenated with Boolean Operators



Raymond Hettinger  added the comment:

I recommend just letting this be.  Aside from it allowing for a cute riddle, in 
the real world seems to be harmless and not worth breaking code.

There are lots of other harmless oddities such as the space-invader increment 
operator:

   x -=- 1

FWIW, a code reformatter such as Black will remove any weirdness.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43837] Operator precedence documentation could be more clear



Change by Raymond Hettinger :


--
assignee: docs@python -> rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43884] Cannot cleanly kill a subprocess using high-level asyncio APIs

2021-04-18 Thread Ronal Abraham



Ronal Abraham  added the comment:

Reproducing the program here:

import asyncio

async def test():
process = await asyncio.create_subprocess_shell(
"sleep 2 && echo done",
stdout=asyncio.subprocess.PIPE,
)
await asyncio.sleep(1)
process.kill()
await process.wait()
# process._transport.close()

asyncio.run(test())

Can I use the high-level API to kill a subprocess cleanly without having to 
access the protected member process._transport? Seems like an oversight perhaps?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43884] Cannot cleanly kill a subprocess using high-level asyncio APIs

2021-04-18 Thread Ronal Abraham



New submission from Ronal Abraham :

There doesn't appear to be a way to prematurely kill a subprocess using the 
high-level asyncio subprocess APIs 
(https://docs.python.org/3.9/library/asyncio-subprocess.html) without getting a 
traceback on exit.

On exit, the attached program writes the following to stderr:

$ python3.9 kill_subprocess.py

Exception ignored in: 
Traceback (most recent call last):
...
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

If I uncomment `# process._transport.close()` or comment `asyncio.sleep(1)`, 
the walkback disappears. (I get the same behavior in python 3.8. I haven't 
tried other python versions.)

--
components: asyncio
files: kill_subprocess.py
messages: 391349
nosy: asvetlov, rabraham, yselivanov
priority: normal
severity: normal
status: open
title: Cannot cleanly kill a subprocess using high-level asyncio APIs
type: behavior
versions: Python 3.8, Python 3.9
Added file: https://bugs.python.org/file49965/kill_subprocess.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Determine what the calling program is

2021-04-18 Thread Dan Stromberg

It's now at:
https://stromberg.dnsalias.org/~strombrg/just-one/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Determine what the calling program is

2021-04-18 Thread Dan Stromberg

I use this little program for shell-level locking.

It just checks for a pid file.  If the pid file does not exist, or the pid
no longer exists, it'll start the process, and write the new process' pid
to the pid file.

It's at:
https://stromberg.dnsalias.org/svn/just-one/

...and usage looks like:
$ ./just-one -h
below cmd output started 2021 Sun Apr 18 04:44:03 PM PDT
Usage: ./just-one --command command --string string

The "string" part needs to be a unique identifier for each process you want
only one of.  The command, naturally, is a shell command.

I just noticed that I don't have a web page describing it yet.  I'll
probably set one up a little later.

It does not use locking: advisory or mandatory.  Just a lock file
containing a pid.

HTH.


On Sun, Apr 18, 2021 at 6:47 AM Jason Friedman  wrote:

> I should state at the start that I have a solution to my problem. I am
> writing to see if there is a better solution.
>
> I have a program that runs via crontab every five minutes. It polls a
> Box.com folder for files and, if any are found, it copies them locally and
> performs a computation on them that can exceed five minutes. It pushes the
> results back up to Box. (Box.com ensures that only complete files are
> visible when I poll.) Files are dropped into this Box.com folder rarely,
> but to ensure a good customer experience I do not want to set my crontab to
> run less frequently. My hardware cannot support multiple simultaneous
> computations.
>
> I have written a piece of code to detect if more than 1 instance of my
> program is running, and I put this code into a separate module (support.py)
> so that other programs can use it.
>
> support.py contains:
>
> 
> import sys
> def check(calling_program):
> import psutil
> # some logic here to count
> # count = N
> if count > 1:
> print(f"I was called by {calling_program}.")
> sys.exit()
> if __name__ == "__main__":
> check()
>
> 
>
> actual-program.py contains:
>
> 
> import support.py
> support.check(__file__)
> # Poll, and if files download, perform expensive computations, push results
>
> 
>
> To me it would be more elegant to be able to do something like this:
>
> def check():
> # Something here that tells me the name of the calling program
> import psutil
> # ...
>
> And then the calling program just does:
> support.check()
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Determine what the calling program is

2021-04-18 Thread Cameron Simpson

On 18Apr2021 07:46, Jason Friedman  wrote:
>I should state at the start that I have a solution to my problem. I am
>writing to see if there is a better solution.
[...]
>I have written a piece of code to detect if more than 1 instance of my
>program is running, and I put this code into a separate module (support.py)
>so that other programs can use it.
[... sniff the process table ...]

Sniffing ps has always seemed unreliable to me.

It is usually better to use some kind of filesystem based lock, named to 
represent your task.

My personal preference is lock directories. Shell version goes like 
this:

if mkdir /my/lock/directory/name-of-task
then
   .. do task ..
   rmdir /my/lock/directory/name-of-task
else
  echo "lock /my/lock/directory/name-of-task already taken"
fi

Simple, reliable, even works over NFS if you care.

In Python this looks like (sketch, untested):

try:
os.mkdir('/my/lock/directory/name-of-task')
except FileExistsError:
error("lock taken")
else:
.. do task ..
os.rmdir('/my/lock/directory/name-of-task')

You can even put a pid file in there for added richness, identifying the 
pid of the competing process. Or whatever.

You can also make O_EXCL or O_CREAT/unwriteable files for locks:

# untested, check spelling etc
os.open('/my/lock/directory/name-of-task', O_CREAT|O_WRONLY, 0o000)

On a UNIX system this opens an unwriteable file for write (you get to 
open it for write because it is new, but its permissions are 
unwriteable, preventing anyone else from opening it for write).

These (mkdir, os.open) have the benefits of making a nice direct 
filesystem object rather than hoping to see your task in ps. And ps 
sniffing is racey, in addition to its other issues.

Cheers,
Cameron Simpson 
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue43883] Making urlparse WHATWG conformant



Serhiy Storchaka  added the comment:

It would be interesting to test also with the yarl module. It is based on 
urlparse and urljoin, but does extra normalization of %-encoding.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: TIME IN XARRAY

2021-04-18 Thread Gys


On 4/15/21 7:58 PM, Jorge Conforte wrote:



Hi,


I'm using xarray to read netcdf data and I had to time in my data the 
values:



xarray.DataArray 'time' (time: 507)>
array(['1979-01-01T00:00:00.0', '1979-02-01T00:00:00.0',
    '1979-03-01T00:00:00.0', ..., 
'2021-01-01T00:00:00.0',

    '2021-02-01T00:00:00.0', '2021-03-01T00:00:00.0'],
   dtype='datetime64[ns]')


Please, how can I get the years and months values from this array.


Thanks,


Conrado


Hi,
maybe this :

from datetime import datetime
import time
# Convert Event to a string
Event="1979-01-01T00:00:00.0"
strDate=time.strftime(Event)
print("Date string ",strDate)
# Get the Year from the string strDate
print("Year ",strDate[0:4])
# Get the month from the string strDate
print("Month ",strDate[5:7])
print()
#
# Convert Event to a datetime object
Event="1979-01-01T00:00:00.0"
dtmDate=datetime.strptime(Event,"%Y-%m-%dT%H:%M:%S.0")
print("datetime object",dtmDate)
# Get the Year from the datetime object
print("Year ",dtmDate.year)
# Get the month from the datetime object
print("Month ",dtmDate.month)

-hth
Gys

--
https://mail.python.org/mailman/listinfo/python-list

[issue43284] sys.getwindowsversion().platform_version is incorrect

2021-04-18 Thread Eryk Sun



Eryk Sun  added the comment:

> But kernel32.dll (since it's of a different version) isn't 
> accurate at all right? 

To clarify, CMD's VER command calls GetVersion(). It has nothing to do with the 
file version of any system DLL. Because CMD is a system component, the 
GetVersion() call returns the true OS version number and build number, directly 
from its PEB OSMajorVersion, OSMinorVersion, and OSBuildNumber values. When the 
PEB for a process is created, these values are copied from the kernel's 
NtMajorVersion, NtMinorVersion, and NtBuildNumber values. 

While the latter may be and most likely are the same as the 
"CurrentMajorVersionNumber", "CurrentMinorVersionNumber", and 
"CurrentBuildNumber" values in the registry, the kernel values are not based on 
the registry. They're compiled into the kernel image when it's built. At boot, 
the configuration manager sets the registry values to the current values, so 
that's all fine -- unless someone with admin or system access changes the 
values (but if a hacker has admin or system access; they own the OS, so such a 
prank would be the least of one's problems). 

My concern, if you read my previous comments, is that an engineer at Microsoft 
rejected someone's attempt to recommend using "CurrentBuildNumber" as a 
replacement for the old advice to use the file version of "kernel32.dll". 
That's enough to make me worry about what's planned for a future release. So 
I'd rather just parse the output from CMD.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43802] Seg fault on macOS using multiprocessing.JoinableQueue

2021-04-18 Thread Jacob Walls



Jacob Walls  added the comment:

Unfortunately, at the outset I should have tested this without multiprocessing. 
I can reproduce without multiprocessing[1], which meant I could more easily 
pinpoint the failure. There is an expensive O(nm) algorithm[2] in the music21 
library that is overflowing.

I appreciate your time looking into this. Closing.

Regards, Jacob


[1] in the provided script, after one call to lc.save() call 
lc.rebuildMetadataCache(useMultiprocessing=False)

[2] music21.analysis.discrete.Ambitus.getPitchRanges(), and I plan to do 
something about it.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Determine what the calling program is

2021-04-18 Thread dn via Python-list

On 19/04/2021 01.46, Jason Friedman wrote:
> I should state at the start that I have a solution to my problem. I am
> writing to see if there is a better solution.
> 
> I have a program that runs via crontab every five minutes. It polls a
> Box.com folder for files and, if any are found, it copies them locally and
> performs a computation on them that can exceed five minutes. It pushes the
> results back up to Box. (Box.com ensures that only complete files are
> visible when I poll.) Files are dropped into this Box.com folder rarely,
> but to ensure a good customer experience I do not want to set my crontab to
> run less frequently. My hardware cannot support multiple simultaneous
> computations.
> 
> I have written a piece of code to detect if more than 1 instance of my
> program is running, and I put this code into a separate module (support.py)
> so that other programs can use it.

In a similar situation, one of my teams used an (OpSys) environment
variable (available in both *nux and MS-Win).
- when the application starts, it checks for the variable
- if exists, stops running, else may proceed

During code review (when I noticed this tactic) I was slightly
surprised, because back when I was young (men were men, and knights were
bold, ...), we used file-stubs.

However, such systems face two complementary, potential-problems:
'single-instance' (which is being addressed), and 'blocking-instance'.

If there is a risk that the long-running computations may fail into a
never-ending loop, the system effectively dies (but silently!) and
source files (ie at Box.com) may accumulate without receiving attention.

Accordingly, the above-mentioned environment-variable was filled with a
time-stamp. Then a second step in the check-routine reviewed the time
since the 'blocking' instance started, in order to log or raise suitable
alerts if things went awry.

YMMV!

An alternative, if the system already uses a database, is to keep a
local record in the DB of all the files lodged at box.com. This can
include a note that each file has/not been processed (plus any other
stats or logging you may deem appropriate). A third state would be 'in
process'. Now, at start-up, the application can quickly check to see if
there is any file in that state...
-- 
Regards,
=dn
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue43883] Making urlparse WHATWG conformant

2021-04-18 Thread Senthil Kumaran

New submission from Senthil Kumaran :

Mike Lissner reported that a set test suites that exercise extreme conditions
with URLs, but in conformance with url.spec.whatwg.org
was maintained here:

https://github.com/web-platform-tests/wpt/tree/77da471a234e03e65a22ee6df8ceff7aaba391f8/url

These test cases were used against urlparse and urljoin method.

https://gist.github.com/mlissner/4d2110d7083d74cff3893e261a801515

Quoting verbatim

```
The basic idea is to iterate over the test cases and try joining and parsing
them. The script wound up messier than I wanted b/c there's a fair bit of
normalization you have to do (e.g., the test cases expect blank paths to be
'/', while urlparse returns an empty string), but you'll get the idea.

The bad news is that of the roughly 600 test cases fewer than half pass. Some
more normalization would fix some more of this, and I don't imagine all of
these have security concerns (I haven't thought through it, honestly, but there
are issues with domain parsing too that look meddlesome). For now I've taken it
as far as I can, and it should be a good start, I think.

The final numbers the script cranks out are:

Done. 231/586 successes. 1 skipped.
```

--
assignee: orsenthil
messages: 391344
nosy: orsenthil
priority: normal
severity: normal
stage: needs patch
status: open
title: Making urlparse WHATWG conformant
type: behavior

___
Python tracker

___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43882] urllib.parse should sanitize urls containing ASCII newline and tabs.

2021-04-18 Thread Senthil Kumaran



New submission from Senthil Kumaran :

A security issue was reported by Mike Lissner wherein an attacker was able to 
use `\r\n` in the url path, the urlparse method didn't sanitize and allowed 
those characters be present in the request.

> In [9]: from urllib.parse import urlsplit
> In [10]: urlsplit("java\nscript:alert('bad')")
> Out[10]: SplitResult(scheme='', netloc='', path="java\nscript:alert('bad')", 
> query='', fragment='')



Firefox and other browsers ignore newlines in the scheme. From
the browser console:

>> new URL("java\nscript:alert(bad)")
<< URL { href: "javascript:alert(bad)", origin: "null", protocol:
"javascript:", username: "", password: "", host: "", hostname: "", port: "", 
pathname: "alert(bad)", search: "" 

Mozilla Developers informed about the controlling specification for URLs is in 
fact defined by the "URL Spec"
from WHATWG which updates RFC 3986 and specifies that tabs and newlines
should be stripped from the scheme.

See: https://url.spec.whatwg.org/#concept-basic-url-parser

That link defines an automaton for URL parsing. From that link, steps 2 and 3 
of scheme parsing read:

If input contains any ASCII tab or newline, validation error.
3. Remove all ASCII tab or newline from input.

 


urlparse module behavior should be updated, and an ASCII tab or newline should 
be removed from the url (sanitized) before it is sent to the request, as WHATWG 
spec.

--
assignee: orsenthil
messages: 391343
nosy: orsenthil
priority: normal
severity: normal
stage: needs patch
status: open
title: urllib.parse should sanitize urls containing ASCII newline and tabs.
type: security
versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Comparing text strings

2021-04-18 Thread dn via Python-list

On 14/04/2021 04.05, Mats Wichmann wrote:
> On 4/12/21 5:11 PM, Rich Shepard wrote:
>> I'm running Slackware64-14.2 and keep a list of installed packages.
>> When a
>> package is upgraded I want to remove the earlier version, and I've not
>> before written a script like this. Could there be a module or tool that
>> already exists to do this? If not, which string function would be best
>> suited to the task?
>>
>> Here's an example:
>> atftp-0.7.2-x86_64-2_SBo.tgz
>> atftp-0.7.4-x86_64-1_SBo.tgz

The 'trick' here is to understand how the distro handles versioning (and
multiple architectures, etc) and then to split the long name into
components before comparison, simplified example:

only relevant comparison if x86_64 == x86_64, then
atftp ? atftp == same, and thus
0.7.4 ? 0.7.4 => version update
(perhaps)


>> and there are others like this. I want the python3 script to remove the
>> first one. Tools like like 'find' or 'sort -u' won't work because
>> while the
>> file name is the same the version or build numbers differ.
> 
> Yes, you've identified why this is hard: package versioning takes many
> forms.  As suggested elsewhere, for Linux distribution packages, the
> only reliable approach is to lean on the distro's packaging
> infrastructure in some way, because those version strings (plus package
> metadata which may have "replaces" or "obsoletes" or some similar
> information) all have a defined meaning to *it* - it's the intended
> audience.
> 
> Don't know if Slack exposes this information in some way, it may be hard
> to make a reliable script if not. I know Debian actually does what
> you're looking for as a feature of the packaging system (apt-get
> autoclean), and the Fedora/RedHat universe does not, so I've also looked
> for what you're looking for :)


Not a sand-box I've played in. However, dnf is at least partly-written
in Python (it may still employs the older rpm, even yum, code).

Maybe the OP could learn from, or even piggy-back off, the existing code?
(which may be at https://github.com/rpm-software-management/dnf)
-- 
Regards,
=dn
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue43804] Add more info about building C/C++ Extensions on Windows using MSVC



Shreyan Avigyan  added the comment:

Kindly have a review of my PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43833] Unexpected Parsing of Numeric Literals Concatenated with Boolean Operators

2021-04-18 Thread sco1



sco1  added the comment:

Makes sense, thanks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43833] Unexpected Parsing of Numeric Literals Concatenated with Boolean Operators



Serhiy Storchaka  added the comment:

There is no issues with lists and strings. "]" clearly ends the list display, 
and a quote ends a string literal. The problem with numeric literals is that 
they can contain letters, so it is not clear (for human reader) where the 
numeric literals ends and the keyword starts. Adding new numeric prefixes or 
suffixes or new keywords can break existing code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43874] argparse crashes on subparsers with no dest/metava

2021-04-18 Thread Terence Honles



Terence Honles  added the comment:

Thanks for the context Paul. I didn't think to look in the tracker for an 
existing issue, sorry!

I'm not sure if documenting the requirement is sufficient or something that a 
user would go towards with the error as it is.

I _might_ suggest throwing an error if dest/metavar isn't provided when 
required is true but that would probably only work on creating the sub parser 
and not if setting attributes as I saw some tests doing.

If an error is thrown I'd expect it to be where I placed the default name and 
it to basically say sub parser has no name (and possibly suggest setting dest 
or metavar).

I think both might be confusing to the end user and likely something that a 
user relying on argparse might not test but expect to work (I am in that camp 
except I tested it... I was only wondering what it said). That is why I went 
with filling out the default of "command" (I also checked only one sub parser 
was allowed so that wouldn't be too ambiguous). Initially I went with 
"subcommand" because "subparser" didn't seem to make sense for an end user, but 
I settled with command since that's what some tests were using for dest and I 
liked it. I had also thought of expanding to all the options as one of the 
comments had in the other issue, but required argument: {command1,command2,...} 
looked a little funny and was less obvious what it meant since it could also 
looked like N arguments were missing and being represented in a collapsed 
representation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Determine what the calling program is

2021-04-18 Thread Kushal Kumaran

On Sun, Apr 18 2021 at 07:46:53 AM, Jason Friedman  wrote:
> I should state at the start that I have a solution to my problem. I am
> writing to see if there is a better solution.
>
> I have a program that runs via crontab every five minutes. It polls a
> Box.com folder for files and, if any are found, it copies them locally and
> performs a computation on them that can exceed five minutes. It pushes the
> results back up to Box. (Box.com ensures that only complete files are
> visible when I poll.) Files are dropped into this Box.com folder rarely,
> but to ensure a good customer experience I do not want to set my crontab to
> run less frequently. My hardware cannot support multiple simultaneous
> computations.
>
> I have written a piece of code to detect if more than 1 instance of my
> program is running, and I put this code into a separate module (support.py)
> so that other programs can use it.
>
> support.py contains:
> 
> import sys
> def check(calling_program):
> import psutil
> # some logic here to count
> # count = N
> if count > 1:
> print(f"I was called by {calling_program}.")
> sys.exit()
> if __name__ == "__main__":
> check()
> 
>
> actual-program.py contains:
> 
> import support.py
> support.check(__file__)
> # Poll, and if files download, perform expensive computations, push results
> 
>
> To me it would be more elegant to be able to do something like this:
>
> def check():
> # Something here that tells me the name of the calling program
> import psutil
> # ...
>
> And then the calling program just does:
> support.check()

The standard library provides locking primitives in the Unix-specific
fcntl module.  You can use those to make sure only a single instance of
your process runs.  Use the non-blocking forms of the lock to ensure
that if you are unable to get the lock, you exit rather than wait.  If
your process waits for locks, the crontab will keep piling on waiters.

There are libraries[1][2] on pypi that wrap the platform-specific
locking primitives and provide terse APIs.

A simpler solution might be to use the flock(1) command, if you have it
available, directly in the crontab entry.

[1] https://pypi.org/project/fasteners/
[2] https://pypi.org/project/oslo.concurrency/

-- 
regards,
kushal
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue43284] sys.getwindowsversion().platform_version is incorrect



Shreyan Avigyan  added the comment:

But kernel32.dll (since it's of a different version) isn't accurate at all 
right? To find the accurate result we have to use the CurrentBuildNumber 
registry key. I think this key will not be removed so easily by Microsoft. This 
key have been in existence from NT 3.5 till today. I think it's the safest and 
reliable way to get the version number other than WMI query. I know the key can 
be removed in the future but if it that happens what about reimplementing the 
kernel32.dll then. We need a way to get the accurate version right?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18233] SSLSocket.getpeercertchain()



Change by Christian Heimes :


--
pull_requests: +24191
pull_request: https://github.com/python/cpython/pull/25467

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43878] ./configure fails on Apple Silicon

2021-04-18 Thread Keith Smiley



Keith Smiley  added the comment:

Thanks for taking a look. My limited understanding is also that these should be 
able to be updated separately from autoconf, and I feel slightly more confident 
knowing that in the past folks treated this update as trivial. It seems like 
the changes should be entirely additive, as in they support new triples but 
otherwise are API compatible.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Determine what the calling program is

2021-04-18 Thread Dieter Maurer

Jason Friedman wrote at 2021-4-18 07:46 -0600:
> ...
>I have a program that runs via crontab every five minutes. It polls a
>Box.com folder for files and, if any are found, it copies them locally and
>performs a computation on them that can exceed five minutes. It pushes the
>results back up to Box. (Box.com ensures that only complete files are
>visible when I poll.) Files are dropped into this Box.com folder rarely,
>but to ensure a good customer experience I do not want to set my crontab to
>run less frequently. My hardware cannot support multiple simultaneous
>computations.

Programs typically use some form of file locking to detect
attempts to run the same program multiple times.

The optimal form of locking depends on the operating system.
Under *nix, so called advisory locks seem promising.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue43833] Unexpected Parsing of Numeric Literals Concatenated with Boolean Operators

2021-04-18 Thread sco1



sco1  added the comment:

We can also see this kind of thing with other literals, would that be in scope 
here?

e.g.

```
Python 3.9.4 (default, Apr  5 2021, 12:33:45) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> "foo"in ["foo", "bar"]
True
>>> [1,]in [[1,]]
True
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Comparing text strings

2021-04-18 Thread Peter Pearson

On Sun, 18 Apr 2021 06:38:16 GMT, Gilmeh Serda wrote:
> On Mon, 12 Apr 2021 16:11:21 -0700, Rich Shepard wrote:
>
>> All suggestions welcome.
>
> Assuming you want to know which is the oldest version and that the same 
> scheme is used all the time, could this work?
>
 s1='atftp-0.7.2-x86_64-2_SBo.tgz'
 s2='atftp-0.7.4-x86_64-1_SBo.tgz'
 s1>s2
> False
 s2>s1
> True
[snip]

However, beware:

>>> s2='atftp-0.7.4-x86_64-1_SBo.tgz'
>>> s3='atftp-0.7.10-x86_64-1_SBo.tgz'
>>> s2>s3
True

-- 
To email me, substitute nowhere->runbox, invalid->com.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue43833] Unexpected Parsing of Numeric Literals Concatenated with Boolean Operators



Serhiy Storchaka  added the comment:

PR 25466 makes the tokenizer emitting a deprecation warning if the numeric 
literal is followed by one of keywords which are valid after numeric literals. 
In future releases it will be changed to syntax warning, and finally to syntax 
error.

It is breaking change, because it makes invalid currently allowed syntax like 
`0 in x` or `1or x` (but `0or x` is already error).

See also issue21642 which allowed parsing "1else" as "1 else". Not all were 
agreed with that fix.

Perhaps we need to rewrite also some paragraphs in the language specification.

--
components: +Interpreter Core
nosy: +Joshua.Landau, gvanrossum, steve.dower
versions:  -Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43833] Unexpected Parsing of Numeric Literals Concatenated with Boolean Operators



Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +24190
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/25466

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43284] sys.getwindowsversion().platform_version is incorrect

2021-04-18 Thread Eryk Sun



Eryk Sun  added the comment:

The "CurrentBuild" and "CurrentVersion" values go back to the first release of 
Windows NT 3.1 in 1993 (build 511, which was quickly replaced by build 528). In 
NT 3.1 (build 528), the "CurrentBuild" value was "1.528.1 () (July 1993)". In 
NT 3.5, this awkward string was replaced by the "CurrentBuildNumber" value, and 
up to NT 5.2 the old "CurrentBuild" value was set to the string "1.511.1 () 
(Obsolete data - do not use)". It was brought back as the build number starting 
with Windows Vista (NT 6.0). However, in Windows 10 the related 
"CurrentVersion" value is now obsolete and frozen at "6.3". The true value is 
now split into "CurrentMajorVersionNumber" and "CurrentMinorVersionNumber".

These registry values aren't a reliable source source of truth. The reliable 
values are the kernel global variables NtMajorVersion, NtMinorVersion, and 
NtBuildNumber, which are compiled into the kernel image. They get copied into 
the process environment block (PEB) of each process as OSMajorVersion, 
OSMinorVersion, and OSBuildNumber. If the application manifest supports the 
current OS version, then GetVersion() and GetVersionExW() will simply return 
these values from the PEB. That's why it was suggested to spawn an instance of 
the system CMD shell and parse the output of its VER command.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Determine what the calling program is

2021-04-18 Thread Jason Friedman

I should state at the start that I have a solution to my problem. I am
writing to see if there is a better solution.

I have a program that runs via crontab every five minutes. It polls a
Box.com folder for files and, if any are found, it copies them locally and
performs a computation on them that can exceed five minutes. It pushes the
results back up to Box. (Box.com ensures that only complete files are
visible when I poll.) Files are dropped into this Box.com folder rarely,
but to ensure a good customer experience I do not want to set my crontab to
run less frequently. My hardware cannot support multiple simultaneous
computations.

I have written a piece of code to detect if more than 1 instance of my
program is running, and I put this code into a separate module (support.py)
so that other programs can use it.

support.py contains:

import sys
def check(calling_program):
import psutil
# some logic here to count
# count = N
if count > 1:
print(f"I was called by {calling_program}.")
sys.exit()
if __name__ == "__main__":
check()


actual-program.py contains:

import support.py
support.check(__file__)
# Poll, and if files download, perform expensive computations, push results


To me it would be more elegant to be able to do something like this:

def check():
# Something here that tells me the name of the calling program
import psutil
# ...

And then the calling program just does:
support.check()
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue43284] sys.getwindowsversion().platform_version is incorrect



Shreyan Avigyan  added the comment:

I researched a little more and found that before Vista the winver command used 
the CurrentBuildNumber key instead of CurrentBuild key. In fact before Vista 
CurrentBuild was marked as obsolete by Microsoft. But that changed in Vista, 
when Microsoft started using CurrentBuild key instead of the CurrentBuildNumber 
key. Though still today CurrentBuildNumber key actually gives the exact version 
(for backward compatibilities maybe?). Therefore why not use 
CurrentBuildNumber? At least CurrentBuildNumber gives the right info. The only 
problem is that we have to only use that key to determine the windows version 
(Eg - Windows 10, 8.1, 8, 7, Vista, etc.) and to do that we have to use a very 
long switch or if-else statement. 

https://en.wikipedia.org/wiki/Comparison_of_Microsoft_Windows_versions#Windows_NT
 has the whole list with the column "RTM build" as the CurrentBuildNumber value.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36076] ssl.get_server_certificate should use SNI



Christian Heimes  added the comment:

Thanks!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.10 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36076] ssl.get_server_certificate should use SNI

2021-04-18 Thread miss-islington



miss-islington  added the comment:


New changeset 49fdf118aeda891401d638ac32296c7d55d54678 by juhovh in branch 
'master':
bpo-36076: Add SNI support to ssl.get_server_certificate. (GH-16820)
https://github.com/python/cpython/commit/49fdf118aeda891401d638ac32296c7d55d54678


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43641] Update `ssl.PROTOCOL_TLSv1_2` docs since it is not the newest TLS version



Christian Heimes  added the comment:

Thanks!

--
nosy: +christian.heimes
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43284] sys.getwindowsversion().platform_version is incorrect



Shreyan Avigyan  added the comment:

@eryksun described "Apparently the developers do not want to guarantee that the 
version information on any particular system DLL can be used to get the system 
version. Apparently they also do not want to officially sanction using the 
"CurrentMajorVersionNumber", "CurrentMinorVersionNumber", and  
"CurrentBuildNumber" values in the registry."

If that's the case why not get the version from CurrentBuild registry key 
present in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion. 
This key at least goes back to Windows 7 and this is the key that's currently 
being used by winver command. And I don't think anyone at all uses windows 
version older than Windows 7. And chances are that the key maybe present even 
before Windows 7. At least the CurrentBuild is accurate.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43641] Update `ssl.PROTOCOL_TLSv1_2` docs since it is not the newest TLS version

2021-04-18 Thread miss-islington



miss-islington  added the comment:


New changeset 2798f247c0747d28cb857fa80803797b24696cb6 by Illia Volochii in 
branch 'master':
bpo-43641: Stop stating that TLS 1.2 is the most modern version in docs 
(GH-25041)
https://github.com/python/cpython/commit/2798f247c0747d28cb857fa80803797b24696cb6


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43362] Bad free in py_sha3_new_impl function



Christian Heimes  added the comment:

The issue affected Python 3.10a1 to latest alpha. 3.9 and earlier are not 
affected.

Thanks!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions:  -Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43362] Bad free in py_sha3_new_impl function



Christian Heimes  added the comment:


New changeset aa6da32edc3c6ddfda5e849561e20273b8d82771 by Christian Heimes in 
branch 'master':
bpo-43362: Fix invalid free and return check in _sha3 module (GH-25463)
https://github.com/python/cpython/commit/aa6da32edc3c6ddfda5e849561e20273b8d82771


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43877] Logging Cookbook ambiguity