[issue45382] platform() is not able to detect windows 11

2022-01-26 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 26.01.2022 01:29, Eryk Sun wrote:
> 
> Eryk Sun  added the comment:
> 
>> Bit wmic seems nice solution.
>> Is still working for windows lower than 11?
> 
> wmic.exe is still included in Windows 10 and 11, but it's officially 
> deprecated [1], which means it's no longer being actively developed, and it 
> might be removed in a future update. PowerShell is the preferred way to use 
> WMI.

All of these require shelling out to the OS, so why not stick to
`ver` as we've done in the past. This has existed for ages and
will most likely not disappear anytime soon.

Is there a good reason to prefer wmic or PowerShell (which are
less likely to be available or reliable) ?

> ---
> [1] 
> https://docs.microsoft.com/en-us/windows/deployment/planning/windows-10-deprecated-features
-- 
Marc-Andre Lemburg
eGenix.com

--

___
Python tracker 
<https://bugs.python.org/issue45382>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue46249] [sqlite3] move set lastrowid out of the query loop and enable it for executemany()

2022-01-12 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 11.01.2022 21:30, Erlend E. Aasland wrote:
> 
>> I'd suggest to deprecate the cursor.lastrowid attribute and
>> instead point people to the much more useful [...]
> 
> Yes, I think mentioning the RETURNING ROWID trick in the sqlite3 docs is a 
> very nice improvement. Mentioning the last_insert_rowid SQL function is 
> probably also worth consideration.
> 
> I'm reluctant to deprecate cursor.lastrowid, though. ATM, I'm leaning towards 
> just keeping the current behaviour.

Fair enough :-)

Perhaps just documenting that the value is not necessarily what
people may expect, when coming from other databases due to the
different semantics with SQLite, is enough.
 --
Marc-Andre Lemburg
eGenix.com

--

___
Python tracker 
<https://bugs.python.org/issue46249>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue46249] [sqlite3] move set lastrowid out of the query loop and enable it for executemany()

2022-01-11 Thread Marc-Andre Lemburg

Marc-Andre Lemburg  added the comment:

On 11.01.2022 20:46, Erlend E. Aasland wrote:
> 
> If we are to revert to this behaviour, we'll have to start examining the SQL 
> we are given (search for INSERT and REPLACE keywords, determine if they are 
> valid (i.e. not a comment, not part of a column or table name, etc.), which 
> will lead to a noticeable performance hit for every new statement (not for 
> statements reused via the LRU cache though). I'm not sure this is a good 
> idea. However I will give it a good thought.
>
> My first thought now, is that it would be better for the sqlite3 module to 
> align lastrowid with the behaviour of the C API sqlite3_last_insert_rowid() 
> (also available as an SQL function: last_insert_rowid). OTOH, the SQLite API 
> is tied to the _connection_ object, so it may not make sense to align it with 
> lastrowid which is a _cursor_ attribute.

I've had a look at the API description and find it less than useful,
to be honest:

https://sqlite.org/c3ref/last_insert_rowid.html

You don't know on which cursor the last row was inserted, it's
possible that this was or is done by a trigger and the last row
is not updated in case the INSERT does not succeed for some reason,
leaving it unchanged - without the user getting a notification of
this failure, since the .execute() call itself will succeed for
e.g. "INSERT INTO table SELECT ...;".

It also seems that the function really only works for INSERTs and
not for UPDATEs.

> Perhaps the Right Thing To Do™ is to be conservative and just leave it as it 
> is. I still want to apply the optimisation, though. It does not alter the 
> behaviour in any kind of way, and it speeds up executemany().

I'd suggest to deprecate the cursor.lastrowid attribute and
instead point people to the much more useful

"INSERT INTO t (name) VALUES ('two'), ('three') RETURNING ROWID;"

https://sqlite.org/lang_insert.html
https://sqlite.org/forum/forumpost/058ac49cc3

(good to know that SQLite has adopted this PostgreSQL variant as
well)

RETURNING is also available for UPDATES:

https://sqlite.org/lang_update.html

If people really want to use the sqlite3_last_insert_rowid()
functionality, they can use the SQL function of the same name:

https://www.sqlite.org/lang_corefunc.html#last_insert_rowid

which then has known semantics and doesn't conflict with the DB-API
specs.

But this is your call :-)

--

___
Python tracker 
<https://bugs.python.org/issue46249>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue46249] [sqlite3] move set lastrowid out of the query loop and enable it for executemany()

2022-01-11 Thread Marc-Andre Lemburg

Marc-Andre Lemburg  added the comment:

On 08.01.2022 21:56, Erlend E. Aasland wrote:
>  
> Marc-André: since Python 3.6, the sqlite3.Cursor.lastrowid attribute does no 
> longer comply with the recommendations of PEP 249:
> 
> Previously, lastrowid was set to None for operations other than INSERT or 
> REPLACE. This changed with ab994ed8b97e1b0dac151ec827c857f5e7277565 (in 
> Python 3.6), so that lastrowid is _unchanged_ for operations other than 
> INSERT or REPLACE, and it is set to 0 after the first valid SQL (that is not 
> INSERT/REPLACE) is executed on the cursor.
> 
> Now, PEP 249 only _recommends_ that lastrowid is set to None for operations 
> that do not modify a row, so it's probably not a big deal. No-one has ever 
> mentioned this change in behaviour; there have been no bug reports.
> 
> FTR, here is the relevant quote from PEP 249:
> 
> If the operation does not set a rowid or if the database does not support
> rowids, this attribute should be set to None.
> 
> (I interpret "should" as understood by RFC 2119.)

Well, it may be a little stronger than the SHOULD in the RFC, but then
again the whole DB-API is about conventions and if they don't make sense
for a database backend, it is possible to deviate from the spec, esp. for
optional extensions such as .lastrowid.

> So, my follow-up question becomes:
> I see no point in reverting to pre Python 3.6 behaviour. I would rather 
> change the default value to be 0 (to get rid of the dirty flag in GH-30380), 
> and to make the behaviour more consistent with how the actual SQLite API 
> behaves.
> 
> 
> Do you have an opinion about such a change (in behaviour)?

Is 0 a valid row ID in SQLite ? If not, then I guess this would be
an alternative to None as suggested by the DB-API.

If it is a valid row ID, I'd suggest to go back to resetting to None,
since otherwise code might get confused: if an UPDATE does not get
applied (e.g. a condition is false), code could then still take
.lastrowid as referring to the UPDATE and not a previous
operation, since code will now know whether the condition was met
or not.
 --
Marc-Andre Lemburg
eGenix.com

--
title: [sqlite3] lastrowid improvements -> [sqlite3] move set lastrowid out of 
the query loop and enable it for executemany()

___
Python tracker 
<https://bugs.python.org/issue46249>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue46338] libc_ver() runtime error when sys.executable is empty

2022-01-11 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 10.01.2022 23:01, Allie Hammond wrote:
> 
> libc_ver() in platform.py (called from platform()) causes a runtime error if 
> sys.executable returns null. In my case, FreeRADIUS offers a module 
> rlm_python3 which allows you to run python code from the C based FreeRADIUS 
> server - since this module doesn't use a python binary to execute 
> sys.executable returns null trigering this error.

Interesting. I guess rlm_python3 embeds Python. Is sys.executable an
empty string or None ?

--

___
Python tracker 
<https://bugs.python.org/issue46338>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue12756] datetime.datetime.utcnow should return a UTC timestamp

2022-01-10 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Hi Tony,

from practical experience, it is a whole lot better to not deal with
timezones in data processing code at all, but instead only use
naive UTC datetime values everywhere, expect when you have to
prepare reports or output which has a requirement to show datetime
value in local time or some specific timezone.

You convert all datetime values into UTC upon input, possibly
store the timezone somewhere, if that is relevant for later reporting,
and then forget about timezones.

Your code will run faster, become a lot easier to understand
and you avoid many pitfalls that TZs have, esp. when TZs are
silently dropped interfacing to e.g. numeric code, databases or
other external code.

There's a reason why cloud code (and a lot of other code, such
as data science code) has standardized on UTC :-)

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue12756>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue46249] [sqlite3] move set lastrowid out of the query loop

2022-01-04 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 04.01.2022 21:02, Erlend E. Aasland wrote:
> 
> Erlend E. Aasland  added the comment:
> 
>> If possible, it's usually better to have the .executemany() create a
>> cursor with an output array providing the row ids, e.g. using "INSERT ...
>> RETURNING ..." (PostgreSQL). That way you can access all row ids and
>> can also provide the needed detail in case the INSERTs happen out of
>> order to map them to the input data.
> 
> Hm, maybe so. But it will add a lot of overhead and complexity to 
> executemany(), and there haven't been requests for this feature for sqlite3. 
> AFAIK, there hasn't been request for lastrowid for executemany() at all. 
> OTOH, my proposal of modifying lastrowid to always show the rowid of the 
> actual last inserted row is a very cheap operation, _and_ it simplifies the 
> code (=> increased maintainability), so I think I'll go for that.

Sorry, I wasn't suggesting this for SQLite; it's just a better
and more flexible option than using cursor.lastrowid where
available. Sadly, the PG extension is not standards conform SQL.

>> For cases where you don't need sequence IDs, it's often better to
>> not rely on auto-increment columns for IDs, but instead use random
>> pre-generated IDs. Saves roundtrips to the database and works nicely
>> with cluster databases as well.
> 
> Yes, but in those cases you keep track of the row id yourself, so you 
> probably won't need lastrowid ;)

Indeed, and that's the point :-)

Those auto-increment ID fields are
not really such a good idea to begin with. Either you know that you
will need to manipulate the rows after inserting them (in which case
you can set an ID) or you don't care about the individual rows and
only want to aggregate or search in them based on other fields.

--

___
Python tracker 
<https://bugs.python.org/issue46249>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue46249] [sqlite3] move set lastrowid out of the query loop

2022-01-04 Thread Marc-Andre Lemburg

Marc-Andre Lemburg  added the comment:

On 04.01.2022 11:02, Erlend E. Aasland wrote:
> 
> Erlend E. Aasland  added the comment:
> 
> Thank you for your input Marc-André.
> 
> For SQLite, it's pretty simple: we use an API called 
> sqlite3_last_insert_rowid() which takes the database connection as it's 
> argument, not a statement pointer. This function returns "the rowid of the 
> most recent successful INSERT into a rowid table or virtual table on database 
> connection" (quote from SQLite docs). IMO, it would make sense to also use 
> this post executemany().

Sounds like a plan.

If possible, it's usually better to have the .executemany() create a
cursor with an output array providing the row ids, e.g. using "INSERT ...
RETURNING ..." (PostgreSQL). That way you can access all row ids and
can also provide the needed detail in case the INSERTs happen out of
order to map them to the input data.

For cases where you don't need sequence IDs, it's often better to
not rely on auto-increment columns for IDs, but instead use random
pre-generated IDs. Saves roundtrips to the database and works nicely
with cluster databases as well.

--

___
Python tracker 
<https://bugs.python.org/issue46249>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue46249] [sqlite3] move set lastrowid out of the query loop

2022-01-04 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 04.01.2022 00:49, Erlend E. Aasland wrote:
> 
> Erlend E. Aasland  added the comment:
> 
> I see that PEP 249 does not define the semantics of lastrowid for 
> executemany(). What's the precedence here, MAL? IMO, it would be nice to be 
> able to fetch the last row id after you've done a 1000 inserts using 
> executemany().

The DB-API deliberately leaves this undefined, since there are many ways you
could implement this, e.g.

- return the last row id for the last entry in the array passed to 
.executemany()
- return the last row id of the last actually modified/inserted row after
running .executemany()
- return an array of row ids, one for each row modified/inserted
- return a row id of one of the modified/inserted rows, without defining which
- always return None for .executemany()

Note that in some cases, the order of actions taken by the database is not
predefined (e.g. some databases run such inserts in chunks across
a cluster), so even the "last" semantics are not clear.

> So, another option would be to keep "set-lastrowid" in the query loop, and 
> just remove the condition; we set it every time (but of course only build a 
> PyObject of it when the loop is done).

Since the DB-API leaves this undefined, you are free to add your own
meaning, which makes most sense for SQLite, to always return None or
not implement it at all.

DB-API extensions such as Cursor.lastrowid are optional and don't need
to be implemented if they don't make sense for a particular use case:

https://www.python.org/dev/peps/pep-0249/#optional-db-api-extensions

--

___
Python tracker 
<https://bugs.python.org/issue46249>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41921] REDoS in parseentities

2021-12-06 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Interesting that the tool still exists. It uses mxTextTools, but in a 
non-packaged version, so it's been broken for two decades now :-)

I think it's safe to remove it from Tools\scripts.

--

___
Python tracker 
<https://bugs.python.org/issue41921>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45927] timeit accepts -c/--clock and -t/--time without any functionality

2021-11-29 Thread Marc-Andre Lemburg


New submission from Marc-Andre Lemburg :

>From the code:

opts, args = getopt.getopt(args, "n:u:s:r:tcpvh",
   ["number=", "setup=", "repeat=",
"time", "clock", "process",
"verbose", "unit=", "help"])

but the options -c and -t are not used in the subsequent code.

This can lead to situations where if you mistype e.g. -s as -c, you get 
completely wrong results (see https://bugs.python.org/issue45902 for an 
example).

--
components: Library (Lib)
messages: 407276
nosy: lemburg
priority: normal
severity: normal
status: open
title: timeit accepts -c/--clock and -t/--time without any functionality
versions: Python 3.10, Python 3.11

___
Python tracker 
<https://bugs.python.org/issue45927>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45902] Bytes and bytesarrays can be sorted with a much faster count sort.

2021-11-29 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 26.11.2021 10:56, Ruben Vorderman wrote:
> 
> $ python -m timeit -c "from bytes_sort import bytes_sort" "bytes_sort(b'')"
> 50 loops, best of 5: 495 nsec per loop

Shouldn't this read:

$ python -m timeit -s "from bytes_sort import bytes_sort" "bytes_sort(b'')"

(-s instead of -c) ?

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45902>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45921] codecs module doesn't support iso-8859-6-i, iso-8859-6-e, iso-8859-8-i or iso-8859-8-i

2021-11-29 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Even though these are IANA recognized encodings, we need to apply he same logic 
as we do for all new encodings, which essentially boils down to: Are these 
encoding in wider spread use today ?

Reading through the RFC 1556, it seems that the added -i or -e are just 
indications for applications on how to interpret BIDI information: either 
implicit by looking at the order of characters in the stream or explicit via 
control characters embedded in the stream. They are not new encodings, with new 
mappings.

If that's a correct interpretation, we could add those as aliases for the 
non-annotated encodings.

After more than 20 years with Unicode support in Python and the world moving 
towards UTF-8, I have become fairly reluctant towards adding more encoding 
support to Python.

If people are still using unsupported encodings, it's probably better to point 
them to other dedicated tools for converting text to UTF-8, e.g. iconv, than 
extending the pretty extensive support we already have in Python.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45921>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45476] [C API] Disallow using PyFloat_AS_DOUBLE() as l-value

2021-11-15 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 15.11.2021 10:54, STINNER Victor wrote:
> 
> I don't understand what you are trying to prove about compilers not inlining 
> when you explicitly ask them... not to inline.

I'm not trying to prove anything, Victor.

I'm only stating the fact that by switching from macros to inline
functions we are giving away control to the compilers and should not
be surprised that Python now suddenly runs a lot slower on systems
which either have inlining optimizations switched off or where the
compiler (wrongly) assumes that creating more assembler would result
in slower code.

I've heard all your arguments against macros, but don't believe the
blanket approach to convert to inline functions is warranted in all
cases, in particular not for code which is private to the interpreter
and where we know that we need the code inlined to not result in
unexpected performance regressions.

I also don't believe that we should assume that Python C extension
authors will unintentionally misuse Python API macros. If they do,
it's their business to sort out any issues, not ours. If we document
that macros may not be used as l-values, that's clear enough. We don't
need to use compiler restrictions to impose such limitations.

IMO, conversion to inline functions should only happen, when

a) the core language implementation has a direct benefit, and

b) it is very unlikely that compilers will not inline the code
   with -O2 settings, e.g. perhaps using a threshold of LOCs
   or by testing with the website Oleg mentioned.

Overall, I think the PEP 670 should get some more attentions from the
SC to have a guideline to use as basis for deciding whether or not
to use the static inline function approach. That way we could avoid
these discussions :-)

BTW: Thanks for the details about -O0 vs. -Og.

--

___
Python tracker 
<https://bugs.python.org/issue45476>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45476] [C API] Disallow using PyFloat_AS_DOUBLE() as l-value

2021-11-15 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 15.11.2021 08:54, Oleg Iarygin wrote:
> 
> Oleg Iarygin  added the comment:
> 
> Marc-Andre:
>> Inlining is something that is completely under the control of the
> used compilers. Compilers are free to not inline function marked for
> inlining [...]
> 
> I checked the following C snippet on gcc.godbolt.org using GCC 4.1.2 and 
> Clang 3.0.0 with /-O0/-O1/-Os, and both compilers inline a function 
> marked as static inline:
> 
> static inline int foo(int a)
> {
> return a * 2;
> }
> 
> int bar(int a)
> {
> return foo(a) < 0;
> }
> 
> So even with -O0, GCC from 2007 and Clang from 2011 perform inlining. Though, 
> old versions of CLang leave a dangling original copy of foo for some reason. 
> I hope a linker removes it later.

That's a great website :-) Thanks for sharing.

However, even with x86-64 gcc 11.2, I get assembler which does not inline
foo() without compiler options or with -O0: https://gcc.godbolt.org/z/oh6qnffh7

Only with -O1, the site reports inlining foo().

> As for other compilers, I believe that if somebody specifies -O0, that person 
> has a sound reason to do so (like per-line debugging, building precise flame 
> graphs, or other specific scenario where execution performance does not 
> matter), so inlining interferes here anyway.

Sure, but my point was a different one: even with higher optimization
levels, the compiler can decide whether or not to inline. We expect
the compiler to inline, but cannot be sure.

With macros the compiler has no choice and we are in control and even
when using -O0, you will still want e.g. Py_INCREF() and Py_DECREF()
inlined.

--

___
Python tracker 
<https://bugs.python.org/issue45476>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45698] Error on importing getopt

2021-11-03 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Could you please provide more details regarding the OS, whether you compiled
Python yourself and the env settings you used ?

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45698>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45653] Freeze the encodings module.

2021-10-30 Thread Marc-Andre Lemburg

Marc-Andre Lemburg  added the comment:

On 30.10.2021 17:54, Filipe Laíns wrote:
> 
> I just tested partially freezing the package, and it seems to working fine :)
FWIW: I think it's best not bother and simply freeze the whole thing.

It's mostly char mappings which compress well and there's a benefit
in sharing these using mmap (which the OS does for you with static
C data).

--

___
Python tracker 
<https://bugs.python.org/issue45653>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45548] Update Modules/Setup

2021-10-28 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 27.10.2021 22:58, Brett Cannon wrote:
> 
> Brett Cannon  added the comment:
> 
>> Could Brett or you please add those notes back ? There's no other place
> where such details are documented.
> 
> It really depends on what "details" you're referring to.

I had already listed some of those details.

> Most of what I removed were things like "Module by ", or saying 
> _json.c is for "json accelerator" which is obvious to me. Anything that 
> seemed pertinent to compilation I left in.
> 
> So if there's something you specifically want to add back in that you think 
> is important then please feel free to as I'm done editing the file for my 
> purposes at the moment.

Will do.

--

___
Python tracker 
<https://bugs.python.org/issue45548>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45653] Freeze the encodings module.

2021-10-28 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

encodings is a package. I think you first have to check whether mixing
frozen and non-frozen submodules are even supported. I've never tried
having only part of a package frozen.

Freezing the whole package certainly works.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45653>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45562] python -d creates lots of tokenizer messages

2021-10-28 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Thanks, Pablo :-)

--

___
Python tracker 
<https://bugs.python.org/issue45562>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45562] python -d creates lots of tokenizer messages

2021-10-28 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Hi Pablo,

I think you missed one instance:

   print_escape(stdout, tok->cur, tok->inp - tok->cur);

Cheers

--

___
Python tracker 
<https://bugs.python.org/issue45562>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42738] subprocess: don't close all file descriptors by default (close_fds=False)

2021-10-27 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Gregory P. Smith wrote:
> A higher level "best practices for launching child processes module" with 
> APIs reflecting explicit intents (performance vs security vs simplicity) 
> rather than requiring users to understand subprocess platform specific 
> details may be a good idea at this point (on PyPI I assume).

Interesting that you say that. subprocess was originally written with exactly 
this idea in mind - to create a module which deals with all the platform 
details, so that the user doesn't have to know about them: 
https://www.python.org/dev/peps/pep-0324/#motivation

On the topic itself: I believe Python should come with safe and usable 
defaults, but provide ways to enable alternative approaches which have more 
performance if the user so decides (by e.g. passing in a parameter). In such 
cases, the user then becomes responsible for understanding the implications.

Since there already is a parameter, expert users can already gain more 
performance by using it and then explicitly only closing FDs the users knows 
can/should be closed in the new process.

Perhaps there's a middle ground: have subprocess only loop over the first 64k 
FDs per default and not all possible ones.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue42738>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45548] Update Modules/Setup

2021-10-26 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 26.10.2021 20:47, Marc-Andre Lemburg wrote:
>> Brett removed a lot of stuff in 01cf4fb9c1aa567484c2ffb1b11f9b3fe9949b05 to 
>> make the file more readable. I removed unnecessary -D, -I, and -L to make 
>> the file even more readable. You can pass custom flags to ./configure.
> 
> Could Brett or you please add those notes back ? There's no other place
> where such details are documented. We've lost important information and
> I would like to get this back into Setup and ideally add more
> information to make it easier for users or admins to customize
> their build.

I could also edit the file and add those back, after you're done
with the refactoring.

--

___
Python tracker 
<https://bugs.python.org/issue45548>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45548] Update Modules/Setup

2021-10-26 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 26.10.2021 19:42, Christian Heimes wrote:
> 
> Brett removed a lot of stuff in 01cf4fb9c1aa567484c2ffb1b11f9b3fe9949b05 to 
> make the file more readable. I removed unnecessary -D, -I, and -L to make the 
> file even more readable. You can pass custom flags to ./configure.

Could Brett or you please add those notes back ? There's no other place
where such details are documented. We've lost important information and
I would like to get this back into Setup and ideally add more
information to make it easier for users or admins to customize
their build.

Regarding the -I and -L flags, my question was whether these are
now added elsewhere, since the code would not compile without
e.g. -I$(srcdir)/Include/internal.

I've gone through the Makefile and found that these are already
added via PY_CFLAGS_NODIST, so they are indeed not needed in Setup.
That's great :-)

Please be careful when moving e.g. -I or -L options which point to
non-Python directories. If such options get moved into e.g.
PY_CFLAGS_NODIST, there's no way to override them via options
given in Setup.

Having the -D defined in the relevant code is a lot better. Thanks
for that :-)

> Setup should not be edited by hand. Customizations go to Setup.local.

Editing Setup.local instead of Setup is not documented anywhere ?!
Both files are read, so I guess it's not all that relevant which
of the two are edited.

--

___
Python tracker 
<https://bugs.python.org/issue45548>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45548] Update Modules/Setup

2021-10-26 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 26.10.2021 16:17, Christian Heimes wrote:
> 
> Christian Heimes  added the comment:
> 
> Brett, we can use AM_CONDITIONAL() to conditionally enable/disable a feature 
> and AC_CONFIG_FILES() to create a Modules/Setup from a template:
> 
> Example:
> 
> The conditional
> 
> AM_CONDITIONAL([HAVE_SCPROXY], [test "$ac_sys_system" = "Darwin"])
> 
> sets HAVE_SCPROXY_FALSE and HAVE_NIS_SCPROXY based on the check.

I think it would be more helpful (and generate a lot fewer such
macros), if we'd just test for platforms and then use those in the
Setup.in template.

Some other things:

I saw that you stripped off lots of -I and -L options from the Setup
lines. Are those now unconditionally taken from somewhere else,
without possibility to override them ?

You also removed the structure of the listings, which makes things
like dependencies between e.g. _multibytecodec and the CJK codecs
unclear, the notes about audio, the comments on building the _md5
and sha* modules, etc.

Since Setup is a (potentially) hand edited file, it's good to leave
as much useful information in that file as possible, to not
accidentally create setups which don't work and to have references
which may help in finding the right compile options.

--

___
Python tracker 
<https://bugs.python.org/issue45548>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45395] Frozen stdlib modules are discarded if custom frozen modules added.

2021-10-26 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 26.10.2021 16:02, Eric Snow wrote:
> 
> FYI, I figured out the problem on my end.  I wasn't using an installed 
> python.  Once I did it worked fine.

Oh, you mean you tried using it directly from the source tree ?
I don't think I ever tried that direct route.

When building PyRun, I first install to a temporary directory and
then use this to run the freeze.py tool, generate the frozen .c
files and run make to have the executable built.

I've pretty much finished the port to 3.10.

I'll try the main version in the next couple of days. There's currently
a lot of work going on for the makesetup / Setup files
(https://bugs.python.org/issue45548). I'm waiting for that to stabilize.

--

___
Python tracker 
<https://bugs.python.org/issue45395>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45613] [sqlite3] set threadsafety attribute based on default SQLite threaded mode

2021-10-26 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

+1 on setting the attributes dynamically. Other DB-API modules use a
similar approach.

--

___
Python tracker 
<https://bugs.python.org/issue45613>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45548] Update Modules/Setup

2021-10-24 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 23.10.2021 21:30, Christian Heimes wrote:
> 
> The trick would move the math function back into the core. Mark moved the 
> math functions out of the core on purpose, see bpo-7518.

I don't follow you. With the _math.o target in Makefile.pre.in,
_math.c was always compiled into the main Python interpreter,
even with math and cmath built as shared libs.

And yes, it does export a symbol, but _Py_log1p is not going to
conflict with anything else out there :-)

The trick is essentially not changing the 3.10 status quo. It
only makes sure that _math.o is compiled in the same way as all
other Setup managed modules and moves the target from Makefile.pre.in
to the makesetup section of the final Makefile.

And it removes the warning of having multiple _math.o targets
ending up in the Makefile, which isn't problematic, since make
will always use the last definition (from the makesetup section),
but doesn't look nice either.

OTOH, _math.h and .c are really small, so perhaps it's better
to merge both into a single _math.h file and include that directly
into the modules. I believe that's what Brett's patch does, right ?

Today, only the _Py_log1p() code is actually used and then only
to address a very special case in a platform independent way
(log1p(-0.0) == -0.0), so the whole code boils down to some 10
lines of C being incorporated.

--

___
Python tracker 
<https://bugs.python.org/issue45548>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45548] Update Modules/Setup

2021-10-23 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 23.10.2021 20:04, Christian Heimes wrote:
> 
> PR GH-29179 or GH-29181 address the issue with _math.o

I think those patches are both taking things a bit too far.

This is a build problem, not a code problem. It's perfectly
good style to link a single file to multiple other object files
instead of copying the code into those object files.

The catch is that the makesetup logic is not smart enough to only
include the necessary Makefile line once.

The entry in Makefile.pre.in should not be needed, since the logic
for building math and cmath modules already exists in setup.py.

BTW: There's a simple trick to avoid the makesetup issue: simply
add the _math.c entry to some other module which is always linked
statically, e.g. _stat, and remove it from both math and cmath
entries in Setup, as well as the Makefile.pre.in.

--

___
Python tracker 
<https://bugs.python.org/issue45548>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45548] Update Modules/Setup

2021-10-23 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

I'm using a very simple conditional logic in Setup, which is based
on sed, mostly to add platform specific variables:

In Setup:

# @if macosx: TIME_DEFS=
# @if not macosx: TIME_DEFS=-lrt
time -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal timemodule.c
$(TIME_DEFS) # -lm # time operations and variables

In Makefile:

# Install the custom Modules/Setup file
if test "$(MACOSX_PLATFORM)"; then \
sed -e 's/# @if macosx: *//' \
$(PYRUNDIR)/$(MODULESSETUP) > 
$(PYTHONDIR)/Modules/Setup; \
elif test "$(FREEBSD_PLATFORM)"; then \
sed -e 's/# @if freebsd: *//' \
$(PYRUNDIR)/$(MODULESSETUP) > 
$(PYTHONDIR)/Modules/Setup; \
else \
sed -e 's/# @if not macosx: *//' \
-e 's/# @if not freebsd: *//' \
$(PYRUNDIR)/$(MODULESSETUP) > 
$(PYTHONDIR)/Modules/Setup; \
fi;

Setup used to be templated as well in the past, but that logic was removed
at some point. Perhaps it's time to reintroduce it.

--

___
Python tracker 
<https://bugs.python.org/issue45548>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45548] Update Modules/Setup

2021-10-23 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

FYI: I've been working with a fixed Setup file in PyRun for a long while. There 
are indeed a number of modules missing from Setup, since the whole logic was 
left behind a bit after things moved to setup.py.

The issue with _math.o is actually in the main Makefile.pre.in. The version 
listed there does not match the Makefile lines added via Setup. For PyRun, I 
had to comment out the one in Makefile.pre.in and then only add one instance to 
the math module and not the cmath one. This avoids a (harmless) warning during 
the build.

I'm not sure what the _math.o entry exists in Makefile.pre.in. It's only needed 
by those two modules, AFAIK.

Here's the list of modules I had to add in the past (taken from the 3.10 port):

"""
### Built-in extensions for which there are no entries in Setup.dist/Setup:

# _decimal needs more complex setup, punting on this for now
#DECIMAL_DEFS=-DCONFIG_64=1 -DASM=1
#_decimal \
#   _decimal/_decimal.c \
#   _decimal/libmpdec/basearith.c \
#   _decimal/libmpdec/constants.c \
#   _decimal/libmpdec/context.c \
#   _decimal/libmpdec/convolute.c \
#   _decimal/libmpdec/crt.c \
#   _decimal/libmpdec/difradix2.c \
#   _decimal/libmpdec/fnt.c \
#   _decimal/libmpdec/fourstep.c \
#   _decimal/libmpdec/io.c \
#   _decimal/libmpdec/memory.c \
#   _decimal/libmpdec/mpdecimal.c \
#   _decimal/libmpdec/numbertheory.c \
#   _decimal/libmpdec/sixstep.c \
#   _decimal/libmpdec/transpose.c \
#   $(DECIMAL_DEFS) \
#   -I$(srcdir)/Modules/_decimal \
#   -I$(srcdir)/Modules/_decimal/libmpdec \
#   -I$(prefix)/include -L$(exec_prefix)/lib

# _opcode
_opcode _opcode.c

# _ctypes needs to build libffi first - punting on this

# _lsprof
_lsprof _lsprof.c rotatingtree.c

# _sqlite3
SQLITE_DEFS=-DMODULE_NAME='"sqlite3"' -DSQLITE_OMIT_LOAD_EXTENSION
# @if freebsd: SQLITE_LIBS=-I/usr/local/include -L/usr/local/lib
# @if not freebsd: SQLITE_LIBS=
_sqlite3 \
_sqlite/module.c \
_sqlite/cache.c \
_sqlite/connection.c \
_sqlite/cursor.c \
_sqlite/microprotocols.c \
_sqlite/prepare_protocol.c \
_sqlite/row.c \
_sqlite/statement.c \
_sqlite/util.c \
$(SQLITE_DEFS) -I$(srcdir)/Modules/_sqlite \
$(SQLITE_LIBS) \
-I$(prefix)/include -L$(exec_prefix)/lib \
-lsqlite3

# bz2
_bz2 _bz2module.c -lbz2

# lzma
#
# Note: Adding this can cause serious issues, since the needed lib isn't
# universally installed everywhere. See #1793 and #1794.
#
#_lzma _lzmamodule.c -llzma

# multiprocessing
_multiprocessing \
_multiprocessing/semaphore.c \
_multiprocessing/multiprocessing.c \
-I$(srcdir)/Modules/_multiprocessing

# Optional add-on for multiprocessing to use shared memory
#POSIXSHMEM_LIBS=rt
POSIXSHMEM_LIBS=
_posixshmem \
_multiprocessing/posixshmem.c \
-I$(srcdir)/Modules/_multiprocessing \
$(POSIXSHMEM_LIBS)

# queue
_queue _queuemodule.c
"""

Not all modules are included, since I did not need all missing ones.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45548>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45563] inspect.getframeinfo() doesn't handle frames without lineno

2021-10-23 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Hmm, perhaps I should reopen the ticket, even though I now found the cause.

After all, it is possible that lineno is None and inspect.getframeinfo() cannot 
handle it :-)

And it may be worthwhile investigating why recreation of a code object using:

return types.CodeType(co.co_argcount,
  co.co_posonlyargcount,
  co.co_kwonlyargcount,
  co.co_nlocals, co.co_stacksize,
  co.co_flags, co.co_code, co.co_consts,
  co.co_names, co.co_varnames,
  co.co_filename, co.co_name,
  co.co_firstlineno, co.co_lnotab,
  co.co_freevars, co.co_cellvars)

does not necessarily create a valid copy of a code object co.

--
resolution: not a bug -> 
status: closed -> open

___
Python tracker 
<https://bugs.python.org/issue45563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45563] inspect.getframeinfo() doesn't handle frames without lineno

2021-10-22 Thread Marc-Andre Lemburg


Change by Marc-Andre Lemburg :


--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue45563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45563] inspect.getframeinfo() doesn't handle frames without lineno

2021-10-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Turns out this was a bug in the freeze.py script I was using. I had added a bug 
work-around for the modulefinder module and even though it should work as 
advertised, it seems to be missing some code object attributes when recreating 
the objects which fixed file paths.

The stdlib version uses the .replace() method which was added in Python 3.8 and 
that appears to work better.

Is it possible that code objects now have some extra attributes in 3.10 which 
aren't exposed ? E.g. things placed into ce_extras ?

--

___
Python tracker 
<https://bugs.python.org/issue45563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45563] inspect.getframeinfo() doesn't handle frames without lineno

2021-10-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

I've looked at how the importlib freeze logic works, compared to 
Tools/freeze/freeze.py.

The only difference I can spot is that importlib uses C to build the C array 
and does a compile followed by a marshal.dumps, whereas freeze.py loads all 
modules into memory and then runs marshal on module.__code__.

Could this cause issues with the line number calculations in 3.10 ?

--

___
Python tracker 
<https://bugs.python.org/issue45563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45563] inspect.getframeinfo() doesn't handle frames without lineno

2021-10-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

I see this in modules frozen by the Tools/freeze/ tool.

The line numbers shown for such frozen modules in the frameinfo stack are a bit 
off as well, compared normal Python. Could this be an indication that there's 
something not working quite right, which then leads to 
_PyCode_CheckLineNumber() returning -1 ?

The Tools/freeze/ doesn't do anything special, BTW. All it does is load the 
module and then store the marshal'ed code objects in C arrays. The information 
read from those C arrays should be the same as what Python reads from PYC files.

--

___
Python tracker 
<https://bugs.python.org/issue45563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45563] inspect.getframeinfo() doesn't handle frames without lineno

2021-10-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Update: I've been trying hard to find a short version which triggers the issue, 
but so far it seems to only trigger when using exec() from a frozen Python 
module.

There don't appear to be many ways frame->f_lineno can end up being -1 (which 
then gets translated into None in Python). _PyCode_CheckLineNumber() is one 
source I found.

Any hints where to look for the cause of this weird effect ?

--

___
Python tracker 
<https://bugs.python.org/issue45563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45563] inspect.getframeinfo() doesn't handle frames without lineno

2021-10-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

In the case of setuptools, this would be the file setup.py, but I think the 
specific file is not relevant. I can try to come up with a shorter example.

--

___
Python tracker 
<https://bugs.python.org/issue45563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45562] python -d creates lots of tokenizer messages

2021-10-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

I had left a comment on Github about using stderr instead of stdout, to make 
the output more consistent (other parser error messages go to stderr).

Note sure whether that's something you still want to change before closing the 
issue.

--
stage: patch review -> 

___
Python tracker 
<https://bugs.python.org/issue45562>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45563] inspect.getframeinfo() doesn't handle frames without lineno

2021-10-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

To add some more context:

This came up while porting eGenix PyRun to Python 3.10. While installing 
setuptools 58.2.0 via "pyrun setup.py install", an exception was raised in 
getframeinfo().

PyRun uses exec() to run Python code:

def pyrun_exec_code_file(filename, globals_dict, locals_dict=None):
with open(filename, 'r', encoding='utf-8') as file:
source = file.read()
code = compile(source, filename, 'exec', optimize=pyrun_optimized)
exec(code, globals_dict, locals_dict)

Using pdb, I then found that the top frame does not have f_lineno set in Python 
3.10.

--

___
Python tracker 
<https://bugs.python.org/issue45563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45562] python -d creates lots of tokenizer messages

2021-10-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Yes, I know that (at the moment) it's only documented to work in the parser, 
but since Py_DebugFlag is a general purpose flag, this use could easily be 
extended to other parts of the interpreter as well, e.g. for parsing the 
command line or instrumenting the interpreter to collect debug stats.

In any case, thanks for the quick fix, Pablo.

--

___
Python tracker 
<https://bugs.python.org/issue45562>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45395] Frozen stdlib modules are discarded if custom frozen modules added.

2021-10-21 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

I have an initial version of PyRun for Python 3.10 running as well.
This created a few more headaches in order to make it work with
setuptools and some glitches which appear to be bugs in 3.10
(https://bugs.python.org/issue45563 and https://bugs.python.org/issue45562).
Nothing major, though.

I'll have to check my version of the freeze tool against the one
in Python 3.9 and 3.10 to see whether there's anything in the
core versions which could cause the tool not to work.

BTW: (My) freeze.py uses this startup code as main():

int
main(int argc, char **argv)
{
extern int Py_FrozenMain(int, char **);

/* Disabled, since we want to default to non-optimized mode: */
/* Py_OptimizeFlag++; */
Py_NoSiteFlag++;/* Don't import site.py */

PyImport_FrozenModules = _PyImport_FrozenModules;
return Py_FrozenMain(argc, argv);
}

I still have to dig through the changes you have made, but this
suggests that it replaces PyImport_FrozenModules completely
with its own version, so the default freeze that you are
implementing gets overridden.

--

___
Python tracker 
<https://bugs.python.org/issue45395>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45563] inspect.getframeinfo() doesn't handle frames without lineno

2021-10-21 Thread Marc-Andre Lemburg


Change by Marc-Andre Lemburg :


--
components: +Interpreter Core, Library (Lib), Parser
nosy: +lys.nikolaou, pablogsal

___
Python tracker 
<https://bugs.python.org/issue45563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45562] python -d creates lots of tokenizer messages

2021-10-21 Thread Marc-Andre Lemburg


Change by Marc-Andre Lemburg :


--
components: +Parser
nosy: +lys.nikolaou, pablogsal

___
Python tracker 
<https://bugs.python.org/issue45562>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45562] python -d creates lots of tokenizer messages

2021-10-21 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

What's even worse is that those debug lines get written to stdout, not stderr.

--

___
Python tracker 
<https://bugs.python.org/issue45562>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45563] inspect.getframeinfo() doesn't handle frames without lineno

2021-10-21 Thread Marc-Andre Lemburg


New submission from Marc-Andre Lemburg :

In Python 3.10, it seems that top-level frames generated by running exec() have 
their f_lineno attribute set to None.

inspect.getframeinfo() tries to build context lines and fails on this line in 
such a case:

start = lineno - 1 - context//2

because lineno is None.

It's not clear whether this is a bug in inspect or the way such frames get 
their f_lineno attribute initialized.

The same code works just fine in Python 3.9.

--
messages: 404674
nosy: lemburg
priority: normal
severity: normal
status: open
title: inspect.getframeinfo() doesn't handle frames without lineno
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue45563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45562] python -d creates lots of tokenizer messages

2021-10-21 Thread Marc-Andre Lemburg


New submission from Marc-Andre Lemburg :

python3.9 -d:

Python 3.9.7 (default, Oct 21 2021, 20:51:19)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Loaded pyinteractive.py.

>>>

python3.10 -d:

Python 3.10.0 (default, Oct 21 2021, 23:13:32) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
line[1] = "\"\"\" pyinteractive.py\n"  tok->done = 10
line[2] = "\n"  tok->done = 10
line[3] = "This file is executed on interactive startup by Python\n"  
tok->done = 10
...

(not that in both cases a PYTHONINTERACTIVE script is loaded)

Is that intended ?

--
messages: 404673
nosy: lemburg
priority: normal
severity: normal
status: open
title: python -d creates lots of tokenizer messages
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue45562>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45395] Frozen stdlib modules are discarded if custom frozen modules added.

2021-10-20 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 20.10.2021 16:25, Eric Snow wrote:
> 
> Eric Snow  added the comment:
> 
> On Wed, Oct 20, 2021 at 6:01 AM Marc-Andre Lemburg
>  wrote:
>> I have PyRun mostly working with Python 3.9.
>> ...
>> No changes were necessary to Tools/freeze/.
> 
> Great!  Thanks for getting to that so quickly.  Are you going to take
> a look at 3.10 after you're happy with 3.9?

Yes, 3.10 is next, once I have 3.9 ironed out. And then I'll give
3.11 a try.

>> BTW: Why is test_embed even used for the PGO target ?
> 
> Perhaps I've missed something, but I'm not clear on why PGO would be a
> problem for test_embed.  Are you talking about a specific test in
> test_embed?

Sorry, I wasn't clear. PGO is not a problem for test_embed. I just
wonder why the test_embed tests are run for creating the PGO profile
files. test_embed is far from being a regular work load for Python
applications.

Well, I guess using the test suite for PGO is questionable anyway.
It's just that we don't have anything else handy to create those
profiles at Python build time :-)

--

___
Python tracker 
<https://bugs.python.org/issue45395>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45395] Frozen stdlib modules are discarded if custom frozen modules added.

2021-10-20 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 19.10.2021 18:47, Marc-Andre Lemburg wrote:
> 
>> On Sat, Oct 16, 2021 at 5:01 AM Marc-Andre Lemburg
>>  wrote:
>>> I can try to port PyRun to 3.9 and 3.10 to see whether I run into any 
>>> issues.
>>> Would that help ?
>>
>> Yeah, that would totally help.
> 
> Ok, I'll start looking into this and post updates here.

I have PyRun mostly working with Python 3.9. Still need to add a few
new C modules, but the basics work.

No changes were necessary to Tools/freeze/. The PGO build complains
about test_embed not working - no surprise there. I'll patch the suite
to ignore the test.

BTW: Why is test_embed even used for the PGO target ?

--

___
Python tracker 
<https://bugs.python.org/issue45395>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45395] Frozen stdlib modules are discarded if custom frozen modules added.

2021-10-19 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 16.10.2021 21:20, Eric Snow wrote:
> 
> On Sat, Oct 16, 2021 at 5:01 AM Marc-Andre Lemburg
>  wrote:
>> I can try to port PyRun to 3.9 and 3.10 to see whether I run into any issues.
>> Would that help ?
> 
> Yeah, that would totally help.

Ok, I'll start looking into this and post updates here.

--

___
Python tracker 
<https://bugs.python.org/issue45395>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue19459] Python does not support the GEORGIAN-PS charset

2021-10-19 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 19.10.2021 10:44, Serhiy Storchaka wrote:
> 
> Possible solutions (they can be combined):
> 
> 1. Add support for the GEORGIAN-PS charset and all other encodings used in 
> libc (issue22679). The problem is that it is difficult to get the official 
> information about these encodings.

As with all encodings we add: there has to be a real need to support
them natively in Python (as opposed to installing codecs via PyPI)
and we need a definite source for the encoding, e.g. a standards
document from an official body.

IMO, we should not really add more encodings to the stdlib, but instead
point people to e.g. the iconv package:

https://pypi.org/project/python-iconv/

Perhaps we ought to make it easier for such packages to provide
additional codecs even during the startup phase, e.g. via a special
env var which points Python to a list of codec packages to load
prior to initializing the I/O encoding... not sure whether this is
possible, though.

> 2. Falls back to utf-8 or ascii+surrogateescape in case of unsupported locale 
> encoding. But typos can slip unnoticed.

I think this would be a more general solution to such cases, provided
the startup logic issues a visible warning about the fallback.

--

___
Python tracker 
<https://bugs.python.org/issue19459>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45395] Frozen stdlib modules are discarded if custom frozen modules added.

2021-10-16 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 16.10.2021 01:31, Eric Snow wrote:
> 
> @MAL, who's maintaining Tools/freeze?  I'm not aware of who's using it (other 
> than you, of course).  It looks like PyRun isn't compatible with anything 
> newer than 3.5, so it seems like that isn't verifying that Tools/freeze still 
> works.  Neither does it have tests that run in the test suite (nor on 
> buildbots).
> 
> So could Tools/freeze have been broken for a while?  I ask because I haven't 
> been able to get it work work on the master branch (or on 3.10).

I don't know who maintains it, but it's been working fine up until Python 3.8,
which is the last version I ported PyRun to.

There have also been a couple of patches going into the freeze tool, so this is
still on the radar of at least some people other than me. It's also one of the
oldest tools we have in Python and dates back to the early days of Python. Guido
wrote the initial version.

I can try to port PyRun to 3.9 and 3.10 to see whether I run into any issues.
Would that help ?

--

___
Python tracker 
<https://bugs.python.org/issue45395>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45490] [meta][C API] Avoid C macro pitfalls and usage of static inline functions

2021-10-15 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Meta comment :-) ... wouldn't it be better to enable the Github wiki feature for
such collections ?

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45490>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45395] Frozen stdlib modules are discarded if custom frozen modules added.

2021-10-15 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 14.10.2021 22:56, Eric Snow wrote:
> 
> @MAL, what's the best way to make sure Tools/freeze is still working?  I 
> don't see any tests for it in the test suite.  I tried running the test in 
> Tools/freeze/test, but I can't get that to work on main (or on the 3.10 
> branch).

You'd have to create a frozen binary using the standard way freeze
works. I have never run those tests, so don't know whether they work,
but, of course, made sure that the freeze works as basis for PyRun
and patched it slightly to add features we needed.

One of these days, I need to refactor PyRun into a standalone project
and put it on Github (it's currently integrated into our internal
single repo setup). Then it'll be easier to see the changes I made.
For now, I can only reference the tar file:

https://www.egenix.com/products/python/PyRun/#Download
https://www.egenix.com/products/python/PyRun/#Installation

I can send you an updated version for Python 3.8, if there's
interest.

Essentially, you need to create a Python module which runs your
application, then point freeze.py to it and then compile the
generated .c files using the generated Makefile.

--

___
Python tracker 
<https://bugs.python.org/issue45395>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45476] [C API] Convert "AS" functions, like PyFloat_AS_DOUBLE(), to static inline functions

2021-10-15 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 15.10.2021 11:43, STINNER Victor wrote:
> Again, I'm not aware of any performance issue caused by short static inline 
> functions like Py_TYPE() or the proposed PyFloat_AS_DOUBLE(). If there is a 
> problem, it should be addressed, since Python uses more and more static 
> inline functions.
> 
> static inline functions is a common feature of C language. I'm not sure where 
> your doubts of bad performance come from.

Inlining is something that is completely under the control of the
used compilers. Compilers are free to not inline function marked for
inlining, which can result in significant slowdowns on platforms
which are e.g. restricted in RAM and thus emphasize on small code size,
or where the CPUs have small caches or not enough registers (think
micro-controllers).

The reason why we have those macros is because we want the developers to be
able to make a conscious decision "please inline this code unconditionally
and regardless of platform or compiler". The developer will know better
what to do than the compiler.

If the developer wants to pass control over to the compiler s/he can use
the corresponding C function, which is usually available (and then, in many
cases, also provides error handling).

> Using static inline functions has other advantages. It helps debugging and 
> profiling, since the function name can be retrieved by debuggers and 
> profilers when analysing the machine code. It also avoids macro pitfalls 
> (like abusing a macro to use it as an l-value ;-)).

Perhaps, but then I never had to profile macro use in the past. Instead,
what I typically found was that using macros results in faster code when
used in inner loops, so profiling usually guided me to use macros instead
of functions.

That said, the macros you have inlined so far were all really trivial,
so a compiler will most likely always inline them (the number of machine
code instructions for the call would be more than needed for
the actual operation).

Perhaps we ought to have a threshold for making such decisions, e.g.
number of machine code instructions generated for the macro or so, to
not get into discussions every time :-)

A blanket "static inline" is always better than a macro is not good
enough as an argument, though.

Esp. in PGO driven optimizations the compiler could opt for using
the function call rather than inlining if it finds that the code
in question is not used much and it needs to save space to have
loops fit into CPU caches.

--

___
Python tracker 
<https://bugs.python.org/issue45476>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45476] [C API] Convert "AS" functions, like PyFloat_AS_DOUBLE(), to static inline functions

2021-10-15 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

I am with Raymond on this one.

If "protecting against wrong use" is the only reason to go down the slippery 
path of starting to rely on compiler optimizations for performance critical 
operations, the argument is not good enough.

If people do use macros in l-value mode, it's their problem when their code 
breaks, not ours. Please don't forget that we are operating under the 
consenting adults principle: we expect users of the CPython API to use it as 
documented and expect them to take care of the fallout, if things break when 
they don't.

We don't need to police developers into doing so.

--

___
Python tracker 
<https://bugs.python.org/issue45476>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45382] platform() is not able to detect windows 11

2021-10-08 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 08.10.2021 02:15, Eryk Sun wrote:
> 
>> use the build number as reference instead of the major.minor
> 
> It could check the (major, minor, build) tuple, which allows reporting 10.1+ 
> as "post11" and minimizes hard coding of build numbers. For example, given 
> win32_ver() iterates by (major, minor, build) thresholds:

Great idea.

Could you prepare a PR for this ?

--

___
Python tracker 
<https://bugs.python.org/issue45382>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45395] Frozen stdlib modules are discarded if custom frozen modules added.

2021-10-07 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 07.10.2021 16:40, Eric Snow wrote:
> 
> On Thu, Oct 7, 2021 at 1:17 AM Marc-Andre Lemburg
>  wrote:
>> I'm not sure I follow, but in any case, please make sure that
>> the freeze tool in Tools/ continues to work with the new mechanism.
>>
>> The freeze tool would also need to know which modules are already
>> frozen via the new script, so that modules don't get included twice.
> 
> Will do.

Great, thanks, Eric.
 --
Marc-Andre Lemburg
eGenix.com

--

___
Python tracker 
<https://bugs.python.org/issue45395>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29410] Moving to SipHash-1-3

2021-10-07 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 07.10.2021 15:29, Christian Heimes wrote:
> 
> Christian Heimes  added the comment:
> 
> JP got back to me
> 
> On 07/10/2021 14.34, Jean-Philippe Aumasson wrote:
>> xxHash is much faster indeed, but collisions seem trivial to find, which 
>> might allow hash-flood DoS again (see for example 
>> https://github.com/Cyan4973/xxHash/issues/180 
>> <https://github.com/Cyan4973/xxHash/issues/180>). It's however unclear 
>> whether exploitable multicollisions can also be trivially found.
>>
>> If collisions don't matter and if the ~10x speed-up makes a difference, 
>> then probably a good option, but guess you'll need to keep SipHash (or 
>> some other safe hash) when DoS resistance is needed?
> 
> This information disqualifies xxHash for our use case.

The quoted issue was for an early version of XXH3.

Please see
https://github.com/Cyan4973/xxHash/wiki/Collision-ratio-comparison
as reference for collision analysis of the current xxHash versions.

The numbers are close to the expected case, meaning that
collisions are not more frequent than statistically to be
expected given the hash and the test sample size.

Looking at this older comparison, it would also make sense to
revisit the hybrid approach, e.g. use FNV for strings
up to 16 bytes, XXH128 for longer strings:

https://cglab.ca/~abeinges/blah/hash-rs/

Given that dictionaries often use relatively short string keys,
this should have a significant effect on applications where the
dictionaries don't just use interned strings.

It would also have an effect on Python startup time, since all
those interned strings need to have their hash calculated
during startup.

--

___
Python tracker 
<https://bugs.python.org/issue29410>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29410] Moving to SipHash-1-3

2021-10-07 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 07.10.2021 12:48, Christian Heimes wrote:
> 
>> I don't quite follow. Why is it fine that you discuss DoS, but it's not
> fine when others discuss DoS ?
> 
> But this BPO is not about discussing mitigations against DoS attacks in 
> general. It's about adding SipHash1-3- and following the example of Rust and 
> Ruby.
> 
> If you like to discuss DoS attacks on hashing of numeric types or other 
> mitigations, then please do this in a dedicated ticket. I like to keep this 
> BPO focused on a single topic.

The point that both Victor and I wanted to make is that we have
different views on the relevance of DoS attack mitigations
on selecting the default hash algorithm to use with Python strings
(and other objects which use pyhash.c).

The motivation for moving to siphash 1-3 is performance and we can
potentially get even better performance by looking at today's hash
algorithms and revisiting the decision to go with siphash.

This broadens the discussion, yes, but that can easily be addressed
by changing the title to e.g. "Revisiting the default hash algorithm
for strings".

Since siphash is a crypto hash function, whereas xxhash (and other
faster hash algorithms) are non-crypto hash functions, the topic of
hash collisions which can be used for DoS becomes relevant, so I
don't see why such discussions are off-topic.

With non-crypto hash algorithms available which exhibit good
collision stats and taking into account that DoS can be mitigated
using other ways (which is essential anyway, since Python doesn't
protect again hash based DoS in all cases), we get to a better Python.

More details on xxhash collision stats:
https://github.com/Cyan4973/xxHash/wiki/Collision-ratio-comparison#collision-study

--

___
Python tracker 
<https://bugs.python.org/issue29410>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29410] Moving to SipHash-1-3

2021-10-07 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

BTW: We already use (a slight variant of) xxHash for tuples: 
https://bugs.python.org/issue34751

The issues is an interesting read, in particular on how xxHash was eventually 
chosen, with a whole set of other hash algorithms in between.

--

___
Python tracker 
<https://bugs.python.org/issue29410>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29410] Moving to SipHash-1-3

2021-10-07 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 07.10.2021 12:16, Christian Heimes wrote:
> 
>> That's certainly true, but at the same time, just focusing on string
> hashes only doesn't really help either, e.g. it is very easy to
> create a DoS with numeric keys or other objects which use trivial
> hashing algorithms.
> 
> Marc-Andre, Victor, your postings are off-topic. Please move your discussion 
> to a new BPO.

I don't quite follow. Why is it fine that you discuss DoS, but it's not
fine when others discuss DoS ?

You're statement comes across as an attempt to shut down a possibly
fruitful discussion. Not sure if you intended it that way.

We could open a new issue to discuss faster alternatives to siphash
and then close this one, or simply rename this issue, if it bothers you
that we're not strictly focusing on siphash 1-3 :-)

--

___
Python tracker 
<https://bugs.python.org/issue29410>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29410] Moving to SipHash-1-3

2021-10-07 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 07.10.2021 11:49, Inada Naoki wrote:
> Hash DoS is not only for HTTP headers. Everywhere creating dict from 
> untrusted source can be attack vector.
> For example, many API servers receive JSON as HTTP request body. Limiting 
> HTTP header don't protect it.

That's certainly true, but at the same time, just focusing on string
hashes only doesn't really help either, e.g. it is very easy to
create a DoS with numeric keys or other objects which use trivial
hashing algorithms.

I wouldn't focus too much on this at the Python core level.
Server implementations have other ways to protect themselves against
DoS, e.g. by monitoring process memory, CPU load or runtime, applying
limits on incoming data.

IMO, it's much better to use application and use case specific methods
for this, than trying to fix basic data types in Python to address
the issue and making all Python application suffer as a result.

--

___
Python tracker 
<https://bugs.python.org/issue29410>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29410] Moving to SipHash-1-3

2021-10-07 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Since the days this was discussed, a lot of new and faster hash algorithms have 
been developed. It may be worthwhile looking at those instead.

E.g. xxHash is a lot more performant than siphash: 
https://github.com/Cyan4973/xxHash (the link also has a comparison of hash 
algorithms)

--

___
Python tracker 
<https://bugs.python.org/issue29410>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45382] platform() is not able to detect windows 11

2021-10-07 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

It's probably time to extend the marketing version detection mechanism to use
the build number as reference instead of the major.minor system version numbers.

Here's a good reference for this:

https://en.wikipedia.org/wiki/List_of_Microsoft_Windows_versions

MS resources:

https://docs.microsoft.com/en-us/windows/win32/sysinfo/operating-system-version
https://docs.microsoft.com/en-us/windows/release-health/release-information

--

___
Python tracker 
<https://bugs.python.org/issue45382>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26651] Deprecate register_adapter() and register_converter() in sqlite3

2021-10-07 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

FWIW: I'm -1 on removing the possibility to register conversion or adapter 
hooks in sqlite3. Such mechanisms have become a standard with Python database 
modules and are widely used to adapt them to applications or middleware using 
the modules.

The database module defaults don't always work well everywhere and there needs 
to be an efficient way to modify this behavior. Fixing all input to .execute() 
et al. and all output from .fetch*() is not efficient.

I'd suggest to close this as rejected. The deprecation won't do anyone good.

Related to the few such implementations in dbapi2.py of sqlite2, which I 
believe triggered this issue:

Those are not necessarily ideal, since they don't handle all possible cases.

The adapters (conversion from Python data type to SQLite data type) are always 
used, but the converters (conversion from SQLite data type to Python type) are 
not, since those rely on SQLite providing type information, which it only does 
if you pass in detect_types to the .connect() method. Many people don't notice 
the missing time offset support in the converters due to this (sqlite3 returns 
strings instead).

Modifying those preconfigured adapters / converters would need to be a separate 
issue, though. E.g. deprecating the date/timestamp converters would be a way 
forward, since applications will know better what to do in their particular use 
case. Such a deprecation would have to take a longer while, though, for the 
reasons stated by Raymond.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue26651>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45395] Frozen stdlib modules are discarded if custom frozen modules added.

2021-10-07 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

I'm not sure I follow, but in any case, please make sure that
the freeze tool in Tools/ continues to work with the new mechanism.

The freeze tool would also need to know which modules are already
frozen via the new script, so that modules don't get included twice.

--

___
Python tracker 
<https://bugs.python.org/issue45395>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45382] platform() is not able to detect windows 11

2021-10-05 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 05.10.2021 22:30, Steve Dower wrote:
> The version number for "Windows 11" still starts with 10.0. Just like how 
> Windows 5.x and 6.x were around for a very long time each ;)
> 
> There are tables in platform module that map the specific version to the 
> release name. These probably need to be updated to return "11" for versions 
> 10.0.22000 and greater.

Hmm, but the "ver" output seems to have more information than those
APIs.

Note: The tables for mapping to releases for Windows only take the
major.minor versions as key. Unfortunately, those did not change. It's
actually the build version which provides the indicator, it seems.

Any idea, whether a patch will fix this on Windows soonish ?

--

___
Python tracker 
<https://bugs.python.org/issue45382>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45382] platform() is not able to detect windows 11

2021-10-05 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

win32_ver() should be using the internal Windows APIs to figure out the 
version. I do wonder why those don't return the same version as the "ver" 
command line tool.

Adding our Windows experts to the noisy list.

--
nosy: +lemburg, paul.moore, steve.dower, tim.golden, zach.ware

___
Python tracker 
<https://bugs.python.org/issue45382>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45372] Unwarranted "certificate has expired" when urlopen-ing R3 sites

2021-10-05 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 05.10.2021 12:48, Aivar Annamaa wrote:
> 
> I can list the root certs with certmgr, but I'm not sure which piece to 
> investigate further. 

Check the certs in the LE chain as listed on the page you quoted
and compare them to the working installation.

> Even if there is problem with installed certs, it's interesting, why doesn't 
> it bother the browsers and requests? Maybe this is opportunity to make 
> something better in urllib?

Browsers and requests use their own list of trusted CAs.

--

___
Python tracker 
<https://bugs.python.org/issue45372>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45372] Unwarranted "certificate has expired" when urlopen-ing R3 sites

2021-10-05 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Are you sure that all updates on the failing machine have been correctly
installed ? It's possible that the list of CA root certs is not up to date
on the machine.

You can use certmgr.msc to check the list of installed CA root certs.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45372>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36819] Crash during encoding using UTF-16/32 and custom error handler

2021-09-29 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 29.09.2021 10:41, Serhiy Storchaka wrote:
> 
> Restricting the returned position to be strictly larger than start would 
> solve the problem with infinite loop and OOM. But this is a different issue.

Yes, this would make sense, since having the codec process
the same error location over and over again will not resolve the
error, so it's clearly a bug in the error handler.

--

___
Python tracker 
<https://bugs.python.org/issue36819>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36819] Crash during encoding using UTF-16/32 and custom error handler

2021-09-29 Thread Marc-Andre Lemburg


Change by Marc-Andre Lemburg :


--
nosy: +doerwalter

___
Python tracker 
<https://bugs.python.org/issue36819>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36819] Crash during encoding using UTF-16/32 and custom error handler

2021-09-29 Thread Marc-Andre Lemburg


Change by Marc-Andre Lemburg :


--
components: +Unicode
nosy: +ezio.melotti

___
Python tracker 
<https://bugs.python.org/issue36819>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36819] Crash during encoding using UTF-16/32 and custom error handler

2021-09-29 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Looking at the specs in PEP 293 (https://www.python.org/dev/peps/pep-0293/), it 
is certainly possible for the error handler to return a newpos outside the 
range start - end, meaning in most cases: a value >= end.

There's a good reason for this: the codec may not be able to correctly 
determine the end of the sequence and so the end value presented by the codec 
is not necessarily a valid start to continue encoding/decoding. The error 
handler can e.g. choose to skip more input characters by trying to find the 
next valid sequence.

In the example script, the handler returns start, so the value is within the 
range. A limit would not solve the problem.

It seems that the reallocation logic of the codecs is the main problem here.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue36819>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45020] Freeze all modules imported during startup.

2021-09-25 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 25.09.2021 18:20, STINNER Victor wrote:
> 
> STINNER Victor  added the comment:
> 
> Marc-Andre: I suppose that you're talking about LANDMARK in Modules/getpath.c 
> and PC/getpathp.c.

Now that you mention it: yes, that as well :-) But os.py is used in the
Python stdlib code as well, just search for "os.__file__" to see a few
such uses.

If you search for ".__file__" you'll find that there are quite a few
cases in the test suite expecting that attribute on other stdlib modules
as well. The attribute may officially be optional, but in reality a
lot of code expects to find it.

--

___
Python tracker 
<https://bugs.python.org/issue45020>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45020] Freeze all modules imported during startup.

2021-09-25 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Eric, I noticed that you are freezing os.py. Please be aware that
this module is often being used as indicator for where the stdlib
was installed (the stdlib itself does this in site.py to read the
LICENSE and the test suite also uses os.__file__ in a couple of
places).

It may be worth changing the stdlib to pick a different module
as landmark for this purpose.

Also: Unless you have added the .__file__ attribute to frozen
modules, much of this landmark checking code will fail... which is
the reason I added the attribute to frozen modules in PyRun.

--

___
Python tracker 
<https://bugs.python.org/issue45020>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

2021-09-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 22.09.2021 21:02, Raymond Hettinger wrote:
>> The language specification says that the dicts maintain insertion 
>> order, but the wording implies that this only to explicit 
>> dictionaries, not instance attribute or other namespace dicts.
> 
> That is a quite liberal reading of the spec.  I would object to making 
> instance and namespace dicts behave differently.  That would be a behavior 
> regression and we would forever have to wrestle with the difference.

I agree. Keeping the insertion order is essential for many common
use cases, including those where a class or instance dict is used,
e.g. namespaces used for data records, data caches, field
definitions in data records, etc. (and yes, those often can be
dynamically extended as well :-)).

I think for the case you mention, a documentation patch would be
better and more helpful for the programmers. Point them to slots
and the sharing problem should go away in most cases :-)

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue40116>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45020] Freeze all modules imported during startup.

2021-09-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 22.09.2021 20:47, Brett Cannon wrote:
> What about if there isn't a pre-computed location for __file__? I could 
> imagine a self-contained CPython build where there is no concept of a file 
> location on disk for anything using this.

This does work and is enough to make most code out there happy.

I use e.g. "/os.py" in PyRun. There is no os.py file to load,
but tracebacks and inspection tools work just fine with this.

--

___
Python tracker 
<https://bugs.python.org/issue45020>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45261] Unreliable (?) results from timeit (cache issue?)

2021-09-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On the topic average vs. minimum, it's interesting to see this pop up
every now and then. When I originally wrote pybench late in 1997, I used
average, since it gave good results on my PC at the time.

Later on, before pybench was added to Tools/ in Python 2.5 in 2006, people
started complaining about sometimes getting weird results (e.g. negative
times due to the calibration runs not being stable enough). A lot of noise
was made from the minimum fans, so I switched to minimum, which then made
the results more stable, but I left in the average figures as well.

Then some years later, people complained about pybench not being
good enough for comparing to e.g. PyPy, since those other implementations
optimized away some of the micro-benchmarks which were used in pybench.
It was then eventually removed, to not confuse people not willing
to try to understand what the benchmark suite was meant for, nor
understand the issues around running such benchmarks on more modern
CPUs.

CPUs have advanced a lot since the days pybench was written and so
reliable timings are not easy to get unless you invest in dedicated
hardware, custom OS and CPU settings and lots of time to calibrate
everything. See Victor's research for more details.

What we have here is essentially the same issue. timeit() is mostly
being used for micro-benchmarks, but those need to be run in dedicated
environments. timeit() is good for quick checks, but not really up
to the task of providing reliable timing results.

One of these days, we should ask the PSF or one of its sponsors to
provide funding and devtime to set up such a reliable testing
environment. One which runs not only high end machines, but also
average and lower end machines, and using different OSes,
so that we can detect performance regressions early and easily
on different platforms.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45261>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45213] Frozen modules are looked up using a linear search.

2021-09-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Perhaps a frozen dict could be used instead of the linear search.

This could then also be made available as sys.frozen_modules for inspection by 
applications and tools such as debuggers or introspection tools trying to find 
source code (and potentially failing at this).

Not urgent, though.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45213>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45255] sqlite3.connect() should check if the sqlite file exists and throw a FileNotFoundError if it doesn't

2021-09-22 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Such a change would be backwards incompatible and no longer in line with PEP 
249.

I also don't understand what you regard as confusing about the message "unable 
to open database file". The message could be extended to include the path, but 
apart from that, it's as clear as it can get :-)

--
nosy: +lemburg
status: pending -> open

___
Python tracker 
<https://bugs.python.org/issue45255>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45116] Performance regression 3.10b1 and later on Windows: Py_DECREF() not inlined in PGO build

2021-09-17 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

FWIW: Back in the days of Python 1.5.2, the ceval loop was too big for CPU 
caches as well and one of the things I experimented with at the time was 
rearranging the opcodes based on how often they were used and splitting the 
whole switch statement we had back then in two parts. This results in a 10-20% 
speedup.

CPU caches have since gotten much larger, but the size of the loop still is 
something to keep in mind and optimize for, as more and more logic gets added 
to the inner loop of Python.

IMO, we should definitely keep forced inlines / macros where they are used 
inside hot loops, perhaps even in all of the CPython code, since the conversion 
to inline functions is mostly for hiding internals from extensions, not to hide 
them from CPython itself.

@neonene: Could you provide more details about the CPU you're using to run the 
tests ?

BTW: Perhaps the PSF could get a few sponsors to add more hosts to 
speed.python.org, to provide a better overview. It looks as if the system is 
only compiling on Ubuntu 14.04 and running on an 11 year old system 
(https://speed.python.org/about/). If that's the case, the system uses a server 
CPU with 12MB cache 
(https://www.intel.com/content/www/us/en/products/sku/47916/intel-xeon-processor-x5680-12m-cache-3-33-ghz-6-40-gts-intel-qpi/specifications.html).

--

___
Python tracker 
<https://bugs.python.org/issue45116>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45232] ascii codec is used by default when LANG is not set

2021-09-17 Thread Marc-Andre Lemburg


Change by Marc-Andre Lemburg :


--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue45232>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45232] ascii codec is used by default when LANG is not set

2021-09-17 Thread Marc-Andre Lemburg

Marc-Andre Lemburg  added the comment:

On 17.09.2021 15:45, Olivier Delhomme wrote:
> 
> Olivier Delhomme  added the comment:
> 
> Hi Marc-Andre,
> 
> Please note that setting PYTHONUTF8 with "export PYTHONUTF8=1":
> 
> * Is external to the program and user dependent
> * It does not seems to work on my use case:
> 
>   $ unset LANG
>   $ export PYTHONUTF8=1
>   $ python3 
>   Python 3.6.4 (default, Jan 11 2018, 16:45:55) 
>   [GCC 4.8.5] on linux
>   Type "help", "copyright", "credits" or "license" for more information.
>   >>> machaine='help me if you can'
>  File "", line 0
> 
>^
>SyntaxError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal 
> not in range(128)

UTF-8 mode is only supported in Python 3.7 and later:

   https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-pep540
-- 
Marc-Andre Lemburg
eGenix.com

--

___
Python tracker 
<https://bugs.python.org/issue45232>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45232] ascii codec is used by default when LANG is not set

2021-09-17 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Yes, this is intended. ASCII is used as fallback in case Python
cannot determine the I/O encoding to use during startup. This is
also the reason why later changes to the environment have no
affect on this - the determination of the encoding has already
been applied.

You can force UTF-8 by enabling the UTF-8 mode:

export PYTHONUTF8=1

This will then have Python use UTF-8 regardless of the LANG
env var setting.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45232>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45120] Windows cp encodings "UNDEFINED" entries update

2021-09-17 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Just to be clear: The Python code page encodings are (mostly) taken from the 
unicode.org set of mappings (ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/). 
This is our standards body for such mappings, where possible. In some cases, 
the Unicode consortium does not provide such mappings and we resort to other 
standards (ISO, commonly used mapping files in OSes, Wikipedia, etc).

Changes to the existing mapping codecs should only be done in case corrections 
are applied to the mappings under those names by the standard bodies.

If you want to add variants such as the best fit ones from MS, we'd have to add 
them under a different name, e.g. bestfit1252 (see 
ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/).

Otherwise, interop with other systems would no longer.

>From Eryk's description it sounds like we should always add 
>WC_NO_BEST_FIT_CHARS as an option to MultiByteToWideChar() in order to make 
>sure it doesn't use best fit variants unless explicitly requested.

--

___
Python tracker 
<https://bugs.python.org/issue45120>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45020] Freeze all modules imported during startup.

2021-09-04 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

FWIW, I've not found the importer for frozen modules to be lacking
features. When using frozen modules, you don't expect to see source
code, so the whole part about finding source code is not really
relevant for that use case.

The only lacking part I found regarding frozen modules is support
for these in pkgutil.py. But that's easy to add:

--- /home/lemburg/egenix/projects/PyRun/Python-3.8.0/Lib/pkgutil.py
2019-10-14 1>
+++ ./Lib/pkgutil.py2019-11-17 11:36:38.404752218 +0100
@@ -315,20 +315,27 @@
 return self.etc[2]==imp.PKG_DIRECTORY

 def get_code(self, fullname=None):
+# eGenix PyRun needs pkgutil to also work for frozen modules,
+# since pkgutil is used by the runpy module, which is needed
+# to implement the -m command line switch.
+if self.code is not None:
+return self.code
 fullname = self._fix_name(fullname)
-if self.code is None:
-mod_type = self.etc[2]
-if mod_type==imp.PY_SOURCE:
-source = self.get_source(fullname)
-self.code = compile(source, self.filename, 'exec')
-elif mod_type==imp.PY_COMPILED:
-self._reopen()
-try:
-self.code = read_code(self.file)
-finally:
-self.file.close()
-elif mod_type==imp.PKG_DIRECTORY:
-self.code = self._get_delegate().get_code()
+mod_type = self.etc[2]
+if mod_type == imp.PY_FROZEN:
+self.code = imp.get_frozen_object(fullname)
+return self.code
+elif mod_type==imp.PY_SOURCE:
+source = self.get_source(fullname)
+self.code = compile(source, self.filename, 'exec')
+elif mod_type==imp.PY_COMPILED:
+self._reopen()
+try:
+self.code = read_code(self.file)
+finally:
+self.file.close()
+elif mod_type==imp.PKG_DIRECTORY:
+self.code = self._get_delegate().get_code()
 return self.code

 def get_source(self, fullname=None):

--

___
Python tracker 
<https://bugs.python.org/issue45020>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45020] Freeze all modules imported during startup.

2021-08-31 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 31.08.2021 20:14, Brett Cannon wrote:
> 
> Brett Cannon  added the comment:
> 
>> set __file__ (and __path__) on frozen modules?
> 
> See https://bugs.python.org/issue21736

The patch on that ticket is straight from PyRun, where the
__file__ location is set in a way which signals that the file
does not exist, but instead is baked into the executable:

>>> import os
>>> os.__file__
'/os.py'

Not doing this breaks too many tests in the test suite for no
good reason, which is why I mentioned "practicality beats
purity" in the ticket.

--

___
Python tracker 
<https://bugs.python.org/issue45020>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45020] Freeze all modules imported during startup.

2021-08-28 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 28.08.2021 06:06, Guido van Rossum wrote:
> 
>> With that in place, it'd be great to pre-cache all the .py files 
>> automatically read in at startup.
> 
> *All* the .py files? I think the binary bloat cause by deep-freezing the 
> entire stdlib would be excessive. In fact, Eric's approach freezes everything 
> in the encodings package, which turns out to be a lot of files and a lot of 
> code (lots of simple data tables expressed in code), and I found that for 
> basic startup time, it's best not to deep-freeze the encodings module except 
> for __init__.py, aliases.py and utf_8.py.

Eric's approach, as I understand it, is pretty much what PyRun does.
It freezes almost the entire stdlib. The main aim was to save space
and create a Python runtime with very few files for easy installation and
shipment of products written in Python.

For Python 3.8 (I haven't ported it to more recent Python versions yet),
the uncompressed stripped binary is 15MB. UPX compressed, it's only 5MB:

-rwxr-xr-x 1 lemburg lemburg  15M May 19 15:26 pyrun3.8
-rwxr-xr-x 1 lemburg lemburg  32M Aug 26  2020 pyrun3.8-debug
-rwxr-xr-x 1 lemburg lemburg 5.0M May 19 15:26 pyrun3.8-upx

There's no bloat, since you don't need the .py/.pyc files for the stdlib
anymore. In fact, you save quite a bit of disk space compared to a
full Python installation and additionally benefit from the memory
mapping the OS does for sharing access to the marshal'ed byte code
between processes.

That said, some things don't work with such an approach, e.g.
a few packages include additional data files which they expect to
find on disk. Since those are not available anymore, they fail.

For PyRun I have patched some of those packages to include the
data in form of Python modules instead, so that it gets frozen
as well, e.g. the Python grammar files.

Whether this is a good approach for Python in general is a different
question, though. PyRun is created on top of the existing released
Python distribution, so it doesn't optimize for being able to
work with the frozen code. In fact, early versions did not
even have a REPL, since the main point was to run a
single released app.

--

___
Python tracker 
<https://bugs.python.org/issue45020>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue45020] Freeze all modules imported during startup.

2021-08-26 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Not sure whether you are aware, but the PyRun project I'm maintaining
does that and goes beyond this by freezing almost the complete stdlib
and statically linking most C extensions into a single binary:

https://www.egenix.com/products/python/PyRun/

Startup is indeed better, but not as much as you might think.
You do save stat calls and can share resources across processes.

The big time consumer is turning marshal'ed code objects back
into Python objects, though. If that could be made faster by
e.g. using a more efficient storage format such as one which is
platform dependent, it'd be a much bigger win than the freezing
approach.

--
nosy: +lemburg

___
Python tracker 
<https://bugs.python.org/issue45020>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24339] iso6937 encoding missing

2021-06-29 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Right, the charmap codec was built with the Unicode Consortium mappings in mind.

However, you may have some luck decoding the two byte chars in ISO 6937 using 
combining code points in Unicode. With some extra post processing you could 
also normalize the output into single code points.

If I find time, I may have a look at gencodec.py again and update it to more 
modern interfaces. I've long given up maintenance of Unicode in Python and only 
try to help by giving some guidance based on the original implementation design.

--

___
Python tracker 
<https://bugs.python.org/issue24339>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24339] iso6937 encoding missing

2021-06-23 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Maarten, the code posted on bugs is copyrighted by the person who wrote it. We 
can only accept it for inclusion in Python after the CLA has been signed, since 
then we are allowed to relicense it.

As a result you can only take John's code and post it elsewhere, if John 
permits you to do so, since the files don't include a license.

Note: Creating a character map based codec is not hard using gencodec.py from 
the Tools/unicode/ dir and perhaps some added extra logic.

--

___
Python tracker 
<https://bugs.python.org/issue24339>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue9102] pybench: Cannot compare 2.x and 3.x benchmarks

2021-06-17 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Yes, closing.

--
stage: needs patch -> resolved
status: pending -> closed

___
Python tracker 
<https://bugs.python.org/issue9102>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43976] Allow Python distributors to add custom site install schemes

2021-05-05 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

Steve: I think the point of discussing whether "pip install" can
be used to manage system wide packages is moot. It's been like that
for ages, not only for pip, but also for the distutils setup.py install
process and the old Makefile.pre.in approach before that. People
have their reasons, it's what you'd expect to work as a Unix sysadmin
and won't go away anytime soon :-)

So back to the original point...

Filipe: Could you please explain why patching sysconfig.py is not a
long term solution ?

This doesn't involve any changes on the CPython side, is as flexible
as you can get (you can also patch functions defined in sysconfig.py
to do the necessary magic, not only provide a static dict),
doesn't create overhead for Python's startup, works with all the
different command line options for limiting sys.path additions and
avoids security issues with the Python import logic.

It's already clear that sysconfig.py will be the new golden source
for installation related APIs and schemes (perhaps this could be
made even clearer in the docs), so 3rd party packages will adapt
to this once 3.10 is out.

--

___
Python tracker 
<https://bugs.python.org/issue43976>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43976] Allow Python distributors to add custom site install schemes

2021-05-05 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 05.05.2021 10:29, Christian Heimes wrote:
> 
> I mean that Steve and you are talking about different things.

Could be. I was addressing the point Steve made about not allowing
or making it hard to run "pip install" as root user.

> Neither Steve nor you or I are are Linux distro packaging experts. I suggest 
> that we listen to the expertise of downstream packagers like Filipe or Miro. 
> They deal with packaging on a daily basis.

Agreed.

> By the way you are assuming that all container solutions work like Docker and 
> that all Docker and non-Docker based container solutions allow you to run 
> code as unrestricted, unconfined root. That's a) a incorrect, and b) offtopic 
> for this ticket.

I gave the Docker example as proof that running "pip install" as
root is a rather common scenario and needs to be supported.

Linux distros have been supporting this for many years and just
because distutils is deprecated should not mean that we no longer
provide ways to support this kind of setup.

BTW: I'm aware that other container solutions work in different ways,
e.g. Podman, LXC, etc. but I have yet to find a solution that doesn't
offer root permissions inside the containers (I'm not talking
about how the container is run in the host system).

--

___
Python tracker 
<https://bugs.python.org/issue43976>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43976] Allow Python distributors to add custom site install schemes

2021-05-05 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 05.05.2021 10:01, Christian Heimes wrote:
> 
>> "as root" imply that there's no user site-packages directory at all
>  ^  
> 
> Steve is talking about user site-packages, not global site-packages directory.

You mean "pip install --user" as root ? That's not how you typically
install Python packages as root in a Dockerfile, no, but, of course,
even as root, there is the possibility to install into /root/.local/.

The typical Unix way of installing non-system packages is either
into /usr/local, /opt/local or similar variants, not into /usr.
Python itself also defaults to /usr/local when running
"make install". System provided packages normally live
under /usr (or even directly under / for low level tools).

As a root user, I'd assume that "pip install" also installs into
a /usr/local based site-packages dir -- and that's what happens
at least on Debian based OSes. But it can only happen because
the distros patch the install scheme, since this would normally
install into the /usr based site-packages dir for a python binary
living in /usr/bin.

--

___
Python tracker 
<https://bugs.python.org/issue43976>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43976] Allow Python distributors to add custom site install schemes

2021-05-05 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 04.05.2021 22:58, Steve Dower wrote:
>> "pip install as root" will need to continue to work and thus distros
>> need to get a way to make sure that it doesn't corrupt the system
>> installed packages
> 
> Excuse my ignorance, but does "as root" imply that there's no user 
> site-packages directory at all?

Why should there be no site-packages dir ? All non-core packages get
installed into site-packages (or a similar dir which holds such packages)
by distutils / setuptools.

However, distros usually split this up further into packages which are
managed by the distro packager and ones which are managed by distutils /
setuptools and this is why the install schemes need to be patched.

> I'm not imagining a solution that doesn't require *users* to change their 
> commands, so if they're currently running "sudo pip install" because they 
> need to, but we change it so they shouldn't, then I'm okay with them having 
> to remove the "sudo". (At least for this discussion - we can evaluate 
> transition plans separately.)

I'm not sure I understand what you're suggesting.

For Docker, the instructions from the Dockerfile are run as root, so
there is no sudo involved. Whether you use sudo or not or how pip is
invoked is really not relevant for the discussion. The main point is
that the target of the installation is the system installation, not
a local user installation or a venv. That installation layout is what
sysconfig.py defines in the install schemes.

> And yeah, patching sysconfig.py seems easier. But then, adding a file to the 
> distro is even easier, and if it's easiest for Linux distros to do that via 
> configure than to add a copy step into their build (which is how I'll do it 
> for Windows distros that need it), then I'll leave that to others to 
> decide/implement.

You mean: put something like...

from _sysconfig_site import *
install_sysconfig_site()

at the end of sysconfig.py and then have distros add a
_sysconfig_site module ?

That would work as well, but details will have to be hashed out, since
this can be abused to hijack the system installation of Python (python
-S would have no effect on this). Patching sysconfig.py is definitely
safer.

--

___
Python tracker 
<https://bugs.python.org/issue43976>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue43976] Allow Python distributors to add custom site install schemes

2021-05-04 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 04.05.2021 22:29, Steve Dower wrote:
> 
> Would "pip install --user ..." in a Docker container also work, though? 
> Presumably all the filesystem paths are being redirected anyway, so is there 
> a difference?
> 
> (My assumption is that "--user" would essentially become the default if 
> you're using the OS provided pip/Python. If you do your own build/install of 
> it then you obviously get "default" behaviour, for better or worse.)

More modern Docker setups run the application itself under a non-root
user, but still install the packages and other dependencies as root.

See eg. Zammad's Dockerfile:
https://github.com/zammad/zammad-docker/blob/master/Dockerfile

Not sure whether that answers your question, though.

It's rather uncommon to install venvs inside Docker containers: one of the
main reasons for using containers is the added isolation, but it doesn't
make a lot of sense to add another layer of isolation inside the container.

"pip install as root" will need to continue to work and thus distros
need to get a way to make sure that it doesn't corrupt the system
installed packages. And perhaps distros can also patch pip to not
output those silly warnings anymore when using the system pip package :-)

Regarding the proposed solution: I'm not sure whether a new configure
option is the right way to go about this. Distros could simply patch
sysconfig.py, since that's the golden source of this information from
Python 3.10 onward.

setuptools' distutils version (and other packages which ship distutils)
will have to use this information instead of the copy which is/was
backed into distutils/sysconfig.py on Python 3.10+

--

___
Python tracker 
<https://bugs.python.org/issue43976>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-23 Thread Marc-Andre Lemburg


Marc-Andre Lemburg  added the comment:

On 23.04.2021 07:47, Inada Naoki wrote:
> 
> Inada Naoki  added the comment:
> 
> I think it is too late. Python 3.9 has been released already.  Reverting the 
> change is also breaking change.
> 
> PEP 100 says:
> "Search functions are expected to take one argument, the encoding name in all 
> lower case letters and with hyphens and spaces converted to underscores"
> https://www.python.org/dev/peps/pep-0100/#codecs-coder-decoders-lookup
> 
> But codecs.register() says:
> "Search functions are expected to take one argument, being the encoding name 
> in all lower case letters".
> 
> I don't know historical reason why two document are inconsistent.
> https://docs.python.org/3/library/codecs.html#codecs.register

I guess just an oversight on my part.

PEP 100 is certainly what I meant and implemented. I should have also
made it clear in PEP 100 that I meant lower case ASCII letters.

--

___
Python tracker 
<https://bugs.python.org/issue37751>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



  1   2   3   4   5   6   7   8   9   10   >