Re: how to discover what values produced an exception?

[Dieter Maurer via Python-list]

Johanne Fairchild wrote at 2024-5-3 10:56 -0300:
>How to discover what values produced an exception?  Or perhaps---why
>doesn't the Python traceback show the values involved in the TypeError?
>For instance:
>
>--8<>8---
 (0,0) < 4
>Traceback (most recent call last):
>  File "", line 1, in 
>TypeError: '<' not supported between instances of 'tuple' and 'int'
>--8<>8---
>
>It could have said something like:
>
>--8<>8---
>TypeError: '<' not supported between instances of 'tuple' and 'int'
>  in (0,0) < 4.
>--8<>8---
>
>We would know which were the values that caused the problem, which would
>be very helpful.

Typically, the traceback informs you about the source code line
where the exception has been raised.
When the source line contains literals, you see the values.
If not, then only in special (speak: rather simple) cases the
knowledge of the values will help much.
Usually, a more thorough analysis is required to find out the bug
location and how to fix the bug.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using a background thread with asyncio/futures with flask

[Dieter Maurer via Python-list]

dieter.mau...@online.de wrote at 2024-3-22 18:28 +0100:
>Thomas Nyberg wrote at 2024-3-22 11:08 +0100:
>> ... `future` use across thread boundaries ...
>> Here's an example using just the standard library that
>> exhibits the same issue:
> ...
>For use across thread boundaries, you likely will use
>`concurrent.Future` (not `asyncio.Future`).
>You can use `asyncio.futures._chain_futures` to associate
>an `asyncio.Future` with a `concurrent.Future`.
>Then the fate (result or exception set) of one will be reflected in the other.

You must not set the result/exception for a future in a "foreign" thread
("foreign" here means one not associated with the future's loop).
An aternative to the solution sketched above is to set the result
indirectly via `loop.call_soon_threadsafe`. This way, the
result is set in the futures "native" thread.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using a background thread with asyncio/futures with flask

[Dieter Maurer via Python-list]

Thomas Nyberg wrote at 2024-3-22 11:08 +0100:
> ... `future` use across thread boundaries ...
> Here's an example using just the standard library that
> exhibits the same issue:

I think all `asyncio` objects (futures, tasks, ...)
are meant to be used in a single thread.
If you use them across different threads, you must do special things.

Note that an `await(future)` registers a callback at *future*.
When the future gets its result or exception, the registered callback calls
are scheduled via `self._loop.call_soon`.
`call_soon` must be called from the `asyncio` thread (to which `self._loop`
belongs). A different thread may schedule activities for a loop
but it must use `call_soon_threadsafe` (not `call_soon`).

I would expect that the forbidden `call_soon` call raises an exception
which for reasons I do not know appears to be hidden.


For use across thread boundaries, you likely will use
`concurrent.Future` (not `asyncio.Future`).
You can use `asyncio.futures._chain_futures` to associate
an `asyncio.Future` with a `concurrent.Future`.
Then the fate (result or exception set) of one will be reflected in the other.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Popping key causes dict derived from object to revert to object

[Dieter Maurer via Python-list]

Loris Bennett wrote at 2024-3-21 10:56 +0100:
> ...
>So as I understand it, I need to convert the InstanceState-objects to,
>say, dicts, in order to print them.  However I also want to remove one
>of the keys from the output and assumed I could just pop it off each
>event dict, thus:
>
>event_dicts = [vars(e) for e in events]
>print(type(event_dicts[0]))
>event_dicts = [e.pop('_sa_instance_state', None) for e in event_dicts]
>print(type(event_dicts[0]))
>
>However, this prints
>
>  

`vars` typically returns a `dict`.

>  
This is what you have popped.
>
>If I comment out the third line, which pops the unwanted key, I get
Then you do not change `event_dicts`.

You problem likely is:
`pop` does not return the `dict` after the removal of a key
but the removed value.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Error in Module

[Dieter Maurer via Python-list]

Sanskar Mukeshbhai Joshi wrote at 2024-3-10 18:08 +:
>I had made my project in BCA in Python. When I had complete my project and run 
>the program, at that time I got the error in runnig my project. The error was 
>ModuleNotFoundError: No module named 'flask'.

`flask` is not part of the Python library; it has to be installed separately.

Apparently, you project is a `flask` project (a Web application platform).
In this case, you must run your program in a special `flask`
environment.

I suggest you contact your project mentor to get help.
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Extract lines from file, add to new files

[Dieter Maurer via Python-list]

Rich Shepard wrote at 2024-1-29 08:15 -0800:
> ...
>If this explanation is not sufficiently clear I'll re-write it. :-)

Have you read "https://docs.python.org/3/library/io.html#module-io;?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Await expressions (Posting On Python-List Prohibited)

[Dieter Maurer via Python-list]

>On 27/01/24 10:46 am, Stefan Ram wrote:
>>But your explanation seems to have no mention of the "something" /
>>"the awaitable object" part following the preposition "on". Shouldn't
>>this awaitable object play a rôle in the explanation of what happens?

You can explain a function call without saying much about the called function.
Similarly, you can explain "await " without saying much about
"".

Important is only: "" evaluates to an "awaitable".
An "awaitable" (usually an `asyncio.Future`, `asyncio.Task` or
call of an `async` function) eventuelly produces
a value and `await ` waits until
this happens and then returns this value.

Not everything which eventually returns a value is an
"awaitable" -- e.g. a call of `time.sleep` is not an "awaitable".
Special provisions are necessary to be able to wait for a value
(and meanwhile do other things).
`asyncio.sleep` has e.g. this provisions and a call of it is an "awaitable".

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Dieter Maurer via Python-list]

Frank Millman wrote at 2024-1-15 15:51 +0200:
>I have read that one should not have to worry about garbage collection 
>in modern versions of Python - it 'just works'.

There are still some isolated cases when not all objects
in an unreachable cycle are destroyed
(see e.g. step 2 of
"https://devguide.python.org/internals/garbage-collector/index.html#destroying-unreachable-objects;).
But Python's own objects (e.g. traceback cycles)
or instances of classes implemented in Python
should no longer be affected.

Thus, unless you use extensions implemented in C (with "legacy finalizer"s),
garbage collection should not make problems.


On the other hand, your application, too, must avoid memory leaks.
Caches of various forms (with data for several sessions) might introduce them.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: extend behaviour of assignment operator

[Dieter Maurer via Python-list]

Guenther Sohler wrote at 2024-1-9 08:14 +0100:
>when i run this code
>
>a = cube([10,1,1])
>b = a
>
>i'd like to extend the behaviour  of the assignment operator
>a shall not only contain the cube, but  the cube shall also know which
>variable name it
>was assigned to, lately. I'd like to use that for improved user interaction.

`Acquisition` (--> `PyPI`) implements something similar.

It does not work for variables -- but for attribute access.
Look at the following code:

```
from Acquisition import Implicit

class AccessAwareContainer(Implicit):
  ...

class AccessAwareContent(Implicit):
 ...

container = AccessAwareContainer()
container.content = AccessAwareContent()
```

When you now assign `content = container.content`, then
`content` knows that it has been accessed via `container`.


If fact `content` is not a true `AccessAwareContent` instance
but a wrapper proxy for it. It mostly behaves like an
`AccessAwareContent` object but has additional information
(e.g. it knows the access parent).


It works via a special `__getattribute__` method, essentially
implemented by:

```
   def __getattribute__(self, k):
 v = super().__getattribute__(k)
 return v.__of__(self) if hasattr(v, "__of__") else v
```

Your use case could be implemented similarly (again not for variables
and all objects, but for special classes (and maybe special objects)).

Your `__getattribute__` could look like:
```
   def __getattribute__(self, k):
 v = super().__getattribute__(k)
 try:
   v.name = k
 except TypeError: pass
 return v
```
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Context without manager

[Dieter Maurer via Python-list]

Piergiorgio Sartor wrote at 2023-11-25 22:15 +0100:
> ...
>Apparently, the "with" context manager is not usable
>in classes, at least not with __init__() & co.

You can use `with` in classes -- with any context manager.
However, you would usually not use `with` with a file you have opened
in `__init__`.

If a class defines `__enter__` and `__exit__` (i.e.
the "cntext manager protocol"), then its instances
can be used with the `with` statement.

The important use case for a context manager is the
situation:
set up a context (--> method `__enter__`)
perform some operations in this context (--> body of `with` statement)
tear down the context (--> method `__exit__`).
If you do not have this case (e.g. usually if you open the file
in a class's `__init__`), you do not use a context manager.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: __set_name__ equivalent for instance

[Dieter Maurer via Python-list]

Dom Grigonis wrote at 2023-11-16 21:11 +0200:
> ...
>> On 16 Nov 2023, at 21:00, Dieter Maurer  wrote:
>> ...
>> Methods are not bound during instance creation, they are bound during
>> access.
>
>Good to know. What is the criteria for binding then? Does it check if its type 
>is `vanilla` function? If yes, is there any way to simulate it for arbitrary 
>object?

Attribute access (including method binding) is documented in the
language reference.

Functions and descriptors accessed via an instance but found in a class (i.e.
not directly in the instance) are handled specially;
functions are bound. Other objects are returned as is.

`__getattribute__` can be used to take over control over the attribute
access.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: __set_name__ equivalent for instance

[Dieter Maurer via Python-list]

Dom Grigonis wrote at 2023-11-16 20:12 +0200:
>What I am interested in is a callback.
>Preferably just after methods get bound. So in `object.__new__`.

>I have done it via metaclass, but it is not ideal as there would be too much 
>overhead.
>
>I think what I am looking for is custom method binding.

Methods are not bound during instance creation, they are bound during
access.

You can use descriptors to implement "custom method binding".

Descriptors are defined on the class (and do not behave as
descriptors when defined on instances).
Thus, you would need a metaclass or `__inist_subclass__` is you
want your "custom method binding" globally.


For many methods (but usually not the `__...__` methods),
you can take over the binding in `__new__` or `__init__`
and populate the instance with prebound methods.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: __set_name__ equivalent for instance

[Dieter Maurer via Python-list]

Dom Grigonis wrote at 2023-11-15 18:44 +0200:
>So there is a method __set_name__ which is called on class creation.
>
>The functionality that I am interested in is not retrieving name, but the fact 
>that it also receives `owner` argument.
>
>Thus, allowing simulation of bound class method.
>
>I was wandering if there is an equivalent functionality of attribute to 
>receive `instance` argument on instance creation.

As PEP 487 describes, `__set_name__` essentially targets descriptors.
It is there to inform a descriptor about the name it is used for
in a class (and the class itself). There is no other (easy) way to allow
a descriptor to learn about this name.

If a descriptor is accessed via an instance, the descriptor (protocol)
methods get the instance as parameter.
Note that descriptors are stored in the class: they must not store
instance specific information in their attributes.
Therefore, a method informing an descriptor about instance creation
would not help: it cannot do anything with it.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Detect naming typos (AttributeError) in function names

[Dieter Maurer via Python-list]

c.bu...@posteo.jp wrote at 2023-11-6 12:47 +:
>I would like to know how to detect (e.g. via a linter) typos in function
>names imported from another module.

One option is a test suite (--> Python's "unittest" package)
with a sufficiently high coverage (near 100 %).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pip/pip3 confusion and keeping up to date

[Dieter Maurer via Python-list]

Karsten Hilbert wrote at 2023-11-5 23:19 +0100:
> ...
>do you happen to know where to read up on how to fit a pip
>constraint file into a Debian package creation workflow ?

I have only rudimentary `apt` knowledge.

I know it is quite flexible, e.g. it used to handle `flash`
in a special way. I expect it flexible enough to load from `PyPI`.

`apt` is essentially controlled by a set of `sources`.
Each "source" describes a list of package distributions.
A description contains metadata about the distribution
including name, version, description, dependencies, incompatibilities, etc.

Any integration would require you to define one or more
"source"s, listing the distributions which should come from `PyPI`.
This would allow you to specify potential upgrades at a central
place, the "source"s, you control.
It would not solve the more fundamental problem: to determine
which versions are compatible with the universe of (all the
"sources" described) package versions.


I know that debian packagers create debian packages
from Python distributions not using the approach sketched above
and likely they have their reasons.

You might want to discuss this on an `apt` related mailing list.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pip/pip3 confusion and keeping up to date

[Dieter Maurer via Python-list]

Karsten Hilbert wrote at 2023-11-3 14:47 +0100:
> ...
>> Are they not available in your system's package manager?
>
>... this clearly often answers to "no" for applications of
>any complexity.
>
>Is there a suggested proper path to deal with that (Debian is
>of interest to me here) ?

Complex applications may maintain a set of "known workable versions"
associated with the application's releases.
They may describe those "known workable versions" in a `pip` constraint file.
In this case, you can upgrade to a new application release
by using this constraint file.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pip/pip3 confusion and keeping up to date

[Dieter Maurer via Python-list]

Chris Green wrote at 2023-11-2 10:58 +:
> ...
>So, going on from this, how do I do the equivalent of "apt update; apt
>upgrade" for my globally installed pip packages?

`pip list -o` will tell you for which packages there are upgrades
available.
`pip install -U ...` will upgrade packages.

Be careful, though.
With `apt`, you usually have (`apt`) sources representing a consistent
package universe. Someone tests that package upgrades in this
universe do not break other packages (in this universe).
Because of this, upgrading poses low risk.

`PyPI` does not guarantes consistency. A new package version
may be incompatible to a previous one -- and with other
package you have installed.

I do not think that you would want to auto-upgrade all installed
packages.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to find any documentation for smbus?

[Dieter Maurer via Python-list]

Chris Green wrote at 2023-10-28 17:08 +0100:
>I am using the python3 smbus module, but it's hard work because of the
>lack of documentation.  Web searches confirm that the documentation is
>somewhat thin!
>
>If you do the obvious this is what you get:-
>
>>>> import smbus
>>>> dir (smbus)
>['SMBus', '__doc__', '__file__', '__loader__', '__name__', '__package__', 
> '__spec__']
>>>> help(smbus)
> ...

What does `help(smbus.SMBus`) tell you?

Almost surely, `SMBus` is a class providing the main access methods.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: NameError: name '__version__' is not defined

[Dieter Maurer via Python-list]

Loris Bennett wrote at 2023-10-27 09:29 +0200:
> ...
>For the application with the system Python this mechanism works, but for
>the non-system Python I get the error:
>
>  NameError: name '__version__' is not defined

If you get exceptions (they usually end in `Error` (such as `NameError`)),
look at the traceback. The traceback informs you about
the calling context that led to the exception, espeically where
in the code the exception occurred.

If you cannot resolve the problem at your own,
include the traceback in your call for assistence.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question(s)

[Dieter Maurer via Python-list]

o1bigtenor wrote at 2023-10-25 08:29 -0500:
> ...
>It would appear that something has changed.
>
>Went to the Eclipse download page, downloaded and verified (using sha-512).
>Expanded software to # opt .
>There is absolutely NO mention of anything python - - - java, c and
>its permutations,
>'scientific computing', and some others but nothing python.
>
>I may be missing something due to an extreme lack of knowledge.
>
>Please advise as to where I might find the 'python' environment in eclipse.

I entered "eclipse python download" in my favorite search engine
(=> "ecosia.org") and the second hit gave:
"https://marketplace.eclipse.org/content/pydev-python-ide-eclipse;.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question(s)

[Dieter Maurer via Python-list]

o1bigtenor wrote at 2023-10-25 07:50 -0500:
>> There are several others,
>> e.g. "ECLIPSE" can be used for Python development.
>
>Is 'Eclipse' a Windows oriented IDE?

No.
==> "https://en.wikipedia.org/wiki/Eclipse_(software)"
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question(s)

[Dieter Maurer via Python-list]

o1bigtenor wrote at 2023-10-25 06:44 -0500:
>On Wed, Oct 25, 2023 at 6:24?AM Dieter Maurer  wrote:
> ...
>> There are different kinds of errors.
>>
>> Some can be avoided by using an integrated development environment
>> (e.g. misspellings, type mismatches, ...).
>
>Haven't heard of a python IDE - - - doesn't mean that there isn't such - -
>just that I haven't heard of such. Is there a python IDE?

There are several.

Python comes with "IDLE".

There are several others,
e.g. "ECLIPSE" can be used for Python development.
Search for other alternatices.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Simple webserver

[Dieter Maurer via Python-list]

Frank Millman wrote at 2023-10-25 09:57 +0200:
> ...
>Based on this, I am considering the following -
>
>1. Replace my HTTP handler with Uvicorn. Functionality should be the
>same, but performance should be improved.
>
>2. Instead of running as a stand-alone server, run my app as a
>reverse-proxy using Nginx. I tested this a few years ago using Apache,
>and it 'just worked', so I am fairly sure that it will work with Nginx
>as well. Nginx can then provide the additional functionality that Dieter
>has mentioned.

Good ideas.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question(s)

[Dieter Maurer via Python-list]

o1bigtenor wrote at 2023-10-24 07:22 -0500:
> ...
>Is there a way to verify that a program is going to do what it is
>supposed to do even
>before all the hardware has been assembled and installed and tested?

Others have already noted that "verify" is a very strong aim.

There are different kinds of errors.

Some can be avoided by using an integrated development environment
(e.g. misspellings, type mismatches, ...).

Some can be found via tests.
Look at Python's "unittest" package for learn how to write tests.
"unittest.mock" can help you to mockup hardware you do not yet have.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Simple webserver

[Dieter Maurer via Python-list]

Janis Papanagnou wrote at 2023-10-21 04:03 +0200:
> ...
>I'd like to ask; where do you see the specific risks with Python
>(as language per se) and it's (web-socket-)libraries here?

The web server in Python's runtime library is fairly simple,
focusing only on the HTTP requirements.

You might want additional things for an HTTP server
exposed on the internet which should potentially handle high trafic:
e.g.

 * detection of and (partial) protection against denial of service attacks,
 * load balancing,
 * virtual hosting
 * proxing
 * URL rewriting
 * high throughput, low latency

Depending on your requirements, other web servers might be preferable.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to write list of integers to file with struct.pack_into?

[Dieter Maurer via Python-list]

Jen Kris wrote at 2023-10-2 00:04 +0200:
>Iwant to write a list of 64-bit integers to a binary file.  Everyexample I 
>have seen in my research convertsit to .txt, but I want it in binary.  I wrote 
>this code,based on some earlier work I have done:
>
>buf= bytes((len(qs_array)) * 8)
>
>for offset in range(len(qs_array)):
>  item_to_write= bytes(qs_array[offset])
>  struct.pack_into(buf,"
>But I get the error "struct.error: embedded null character."

You made a lot of errors:

 * the signature of `struct.pack_into` is
   `(format, buffer, offset, v1, v2, ...)`.
   Especially: `format` is the first, `buffer` the second argument

 * In your code, `offset` is `0`, `1`, `2`, ...
   but it should be `0 *8`, `1 * 8`, `2 * 8`, ...

 * The `vi` should be something which fits with the format:
   integers in your case. But you pass bytes.

Try `struct.pack_into("https://mail.python.org/mailman/listinfo/python-list

Re: Passing info to function used in re.sub

[Dieter Maurer via Python-list]

Jan Erik Moström wrote at 2023-9-3 18:10 +0200:
>I'm looking for some advice for how to write this in a clean way
> ...
>The "problem" is that I've currently written some code that works but it uses 
>global variables ... and I don't like global variables. I assume there is a 
>better way to write this, but how?

You could define a class with a `__call__` method
and use an instance of the class as replacement.
The class and/or instance can provide all relevant information via
attributes.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: GNU gettext: Print string translated and untranslated at the same time

[Dieter Maurer via Python-list]

c.bu...@posteo.jp wrote at 2023-8-17 07:10 +:
>I want to display one string in its original source (untranslated)
>version and in its translated version site by site without duplicating
>the string in the python source code?

You could try to translate into an unknown language: this
should give you the default translation.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: GNU gettext: Print string translated and untranslated at the same time

[Dieter Maurer via Python-list]

c.bu...@posteo.jp wrote at 2023-8-17 07:10 +:
>I want to display one string in its original source (untranslated)
>version and in its translated version site by site without duplicating
>the string in the python source code?

Is it an option for you to replace the `gettext` binding
by `zope.i18nmessageid`? Its `Message` instances provide
access to all interesting attributes (id, default, mapping, domain).
Together with other packages (--> `zope.i18n` and `i18ndude`)
it is compatible with `GNU gettext` translation catalogs.

If this is not an option, use Python's inspection functionality
to learn which attributes are made available by the binding's
message class.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to find the full class name for a frame

[Dieter Maurer via Python-list]

Jason Friedman wrote at 2023-8-3 21:34 -0600:
> ...
>my_frame = inspect.currentframe()
> ...
>My question is: let's say I wanted to add a type hint for my_frame.

`my_frame` will be an instance of `Types.FrameType`.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fallback for operator and other dunder methods

[Dieter Maurer via Python-list]

Dom Grigonis wrote at 2023-7-26 05:22 +0300:
> ...
>Is there a way to achieve it without actually implementing operators?
>I have looked at Proxy objects, but they do not seem suited to achieve this.

Proxying is a good approach:
you might have a look at `dm.reuse.proxy.OverridingProxy` (--> `dm.reuse`
on PyPI).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Meta Class Maybe?

[Dieter Maurer via Python-list]

Chris Nyland wrote at 2023-7-22 19:12 -0400:
>So I am stuck on a problem. I have a class which I want to use to create
>another class without having to go through the boiler plate of subclassing.

Do you know about `__init_subclass__`?
It is called whenever a class is subclassed and can be used to
check/update the newly created class.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Canonical list of Python security vulnerabilities

[Dieter Maurer via Python-list]

Bob Kline wrote at 2023-7-14 13:35 -0400:
>Can someone point me to the official catalog of security vulnerabilities in
>Python (by which I mean cpython and the standard libraries)? I found
>https://www.cvedetails.com/vulnerability-list/vendor_id-10210/product_id-18230/Python-Python.html
>but that isn't maintained by python.org.

I am active in the `Zope` community (a web application server
based on Python). This community has a security mailing list
for security related reports
and issues public CVE (= "Commun Vulnerabilities and Exposures") reports
(via a "GitHUB" service) as soon as a security risk has been resolved.

I expect that security risks for Python itself are handled in
a similar way (as, Python too, maintains its code on "GitHUB").
This means that the CVE dictionary should contain **ALL**
publicly announced security risk reports whether found by
the Pyhton community or packagers.

For details about CVE, read
"https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures;.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "Invalid literal for int() with base 10": is it really a literal?

[Dieter Maurer]

Chris Angelico wrote at 2023-5-26 18:29 +1000:
> ...
>However, if you want to change the wording, I'd be more inclined to
>synchronize it with float():
>
 float("a")
>Traceback (most recent call last):
>  File "", line 1, in 
>ValueError: could not convert string to float: 'a'

+1
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do subprocess.PIPE and subprocess.STDOUT sametime

[Dieter Maurer]

Horst Koiner wrote at 2023-5-9 11:13 -0700:
> ...
>For production i run the program with stdout=subprocess.PIPE and i can fetch 
>than the output later. For just testing if the program works, i run with 
>stdout=subprocess.STDOUT and I see all program output on the console, but my 
>program afterwards crashes since there is nothing captured in the python 
>variable. So I think I need to have the functionality of subprocess.PIPE and 
>subprcess.STDOUT sametime.

You might want to implement the functionality of the *nix programm
`tee` in Python.
`tee` reads from one file and writes the data to several files,
i.e. it multiplexes one input file to several output files.

Pyhton's `tee` would likely be implemented by a separate thread.

For your case, the input file could be the subprocess's pipe
and the output files `sys.stdout` and a pipe created by your own
used by your application in place of the subprocess's pipe.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What do these '=?utf-8?' sequences mean in python?

[Dieter Maurer]

Chris Green wrote at 2023-5-6 15:58 +0100:
>Chris Green  wrote:
>> I'm having a real hard time trying to do anything to a string (?)
>> returned by mailbox.MaildirMessage.get().
>>
>What a twit I am :-)
>
>Strings are immutable, I have to do:-
>
>newstring = oldstring.replace("_", " ")

The solution based on `email.Header` proposed by `jak` is better.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using loguru in a library

[Dieter Maurer]

Roy Hann wrote at 2023-4-30 15:40 -:
>Is there anyone using loguru (loguru 0.5.3 in my case) successfully in a
>library?
> ...
>  import mylib
>  logger.enable('mylib')
>
>expecting that it would report any log messages above level DEBUG, just
>as it does when I don't disable logging.

Have you configured the logging system?

Note that `logging.config.fileConfig` may do strange things
regarding disabling (due to its default parameter
`disable_existing_loggers=True`).
I had several cases of missing log entries because `fileConfig`
had disabled already existing loggers.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Embedded python is not 100% stable

[Dieter Maurer]

Guenther Sohler wrote at 2023-4-13 09:40 +0200:
> ...
>I have been working on adding embedded python into OpenSCAD (
>www.openscad.org)
>for some time already. For that i coded/added an additional Python Type
>Object
>which means to hold openscad geometric data.
>
>It works quite well but unfortunately its not 100% stable and i have been
>heavily checking
>all the functions which are referenced in the PyType Object and tried to
>match
>them with the documentation which i found in the python web site

The Python C/C++ interface is complex: it is easy to make
mistakes which may lead to crashes.

Often, `cython` (--> PyPI) can help you to define extension types
in a much safer way. Maybe, you check its documentation?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Problem in using libraries

[Dieter Maurer]

Pranav Bhardwaj wrote at 2023-4-3 22:13 +0530:
>Why can't I able to use python libraries such as numpy, nudenet, playsound,
>pandas, etc in my python 3.11.2. It always through the error "import
>'numpy' or any other libraries could not be resolved".

The "libraries" you speak of are extensions (i.e. not part of
the Python download).

Extensions are Python minor version specific.
You must install them for each Python minor version.
E.g. you can use an extension installation for Python 3.10 for
any Python 3.10.x,
but you must install it again for Python 3.11.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bug 3.11.x behavioral, open file buffers not flushed til file closed.

[Dieter Maurer]

aapost wrote at 2023-3-5 09:35 -0500:
> ...
>If a file is still open, even if all the operations on the file have
>ceased for a time, the tail of the written operation data does not get
>flushed to the file until close is issued and the file closes cleanly.

This is normal: the buffer is flushed if one of the following conditions
are met:
1. you call `flush`
2. the buffer overflows
3. the file is closed.
-- 
https://mail.python.org/mailman/listinfo/python-list

Look free ID genertion (was: Is there a more efficient threading lock?)

[Dieter Maurer]

Chris Angelico wrote at 2023-3-1 12:58 +1100:
> ...
> The
>atomicity would be more useful in that context as it would give
>lock-free ID generation, which doesn't work in Python.

I have seen `itertools.count` for that.
This works because its `__next__` is implemented in "C" and
therefore will not be interrupted by a thread switch.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: semi colonic

[Dieter Maurer]

Thomas Passin wrote at 2023-2-22 21:04 -0500:
>On 2/22/2023 7:58 PM, avi.e.gr...@gmail.com wrote:
> ...
>> So can anyone point to places in Python where a semicolon is part of a best
>> or even good way to do anything?
>
>Mostly I use it to run small commands on the command line with python
>-c.  e.g.
>
>python -c "import sys;print('\n'.join(sys.path))"
>
>This is handy enough that I wouldn't like to do without.
>
>Another place I use the semicolon (once in a while) is for quick
>debugging.  I might add as line like, perhaps,
>
>import os; print(os.path.exists(filename))

I also see, `;` occasionally in `*.pth` files.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: File write, weird behaviour

[Dieter Maurer]

Azizbek Khamdamov wrote at 2023-2-19 19:03 +0500:
> ...
>Example 2 (weird behaviour)
>
>file = open("D:\Programming\Python\working_with_files\cities.txt",
>'r+') ## contains list cities
># the following code DOES NOT add new record TO THE BEGINNING of the
>file IF FOLLOWED BY readline() and readlines()# Expected behaviour:
>new content should be added to the beginning of the file (as in
>Example 1)
>file.write("new city\n")
>
>file.readlines()
>file.close()
>
>I could not find anything in documentation to explain this strange
>behaviour. Why is this happening?

The effect of "r+" (and friends) is specified by the C standard.

The Linux doc (of `fopen`) tells us that ANSI C requires that
a file positioning command (e.g. `seek`) must intervene
between input and output operations. Your example above violates
this condition. Therefore, weird behavior is to be expected.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: asyncio questions

[Dieter Maurer]

Frank Millman wrote at 2023-1-26 12:12 +0200:
>I have written a simple HTTP server using asyncio. It works, but I don't
>always understand how it works, so I was pleased that Python 3.11
>introduced some new high-level concepts that hide the gory details. I
>want to refactor my code to use these concepts, but I am not finding it
>easy.
>
>In simple terms my main loop looked like this -
>
>     loop = asyncio.get_event_loop()
>     server = loop.run_until_complete(
>     asyncio.start_server(handle_client, host, port))
>     loop.run_until_complete(setup_companies())
>     session_check = asyncio.ensure_future(
>     check_sessions())  # start background task
>     print('Press Ctrl+C to stop')
>     try:
>     loop.run_forever()
>     except KeyboardInterrupt:
>     print()
>     finally:
>     session_check.cancel()  # tell session_check to stop running
>     loop.run_until_complete(asyncio.wait([session_check]))
>     server.close()
>     loop.stop()

Why does your code uses several `loop.run*` calls?

In fact, I would define a single coroutine and run that
with `asyncio.run`.
This way, the coroutine can use all `asyncio` features,
including `loop.create_task`.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Help Request] Embedding Python in a CPP Application Responsibly & Functionally

[Dieter Maurer]

John McCardle wrote at 2023-1-25 22:31 -0500:
> ...
>1) To get the compiled Python to run independently, I have to hack
>LD_LIBRARY_PATH to get it to execute. `LD_LIBRARY_PATH=./Python-3.11.1
>./Python-3.11.1/python` .

The need to set `LD_LIBRARY_PATH` usually can be avoided via
a link time option: it tells the linker to add library path
information into the created shared object.

Read the docs to find out which option this is (I think it
was `-r` but I am not sure).

>Even when trying to execute from the same
>directory as the binary & executable, I get an error, `/python: error
>while loading shared libraries: libpython3.11.so.1.0: cannot open shared
>object file: No such file or directory`.

It might be necessary, to provide the option mentioned above
for all shared libraries involved in your final application.

Alternatively, you could try to put the shared objects
into a stadard place (searched by default).

>2) When running the C++ program that embeds Python, I see these messages
>after initializing:
>`Could not find platform independent libraries 
>Could not find platform dependent libraries `

Again: either put your installation in a standard place
or tell the Python generation process about your non-standard place.


>This is seemingly connected to some issues regarding libraries: When I
>run the Python interpreter directly, I can get some of the way through
>the process of creating a virtual environment, but it doesn't seem to
>leave me with a working pip:
>
>`$ LD_LIBRARY_PATH=./Python-3.11.1 ./Python-3.11.1/python
> >>> import venv
> >>> venv.create("./venv", with_pip=True)
>subprocess.CalledProcessError: Command
>'['/home/john/Development/7DRL/cpp_embedded_python/venv/bin/python',
>'-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit
>status 127.`

Run the command manually and see what errors this gives.

> ...

>3) I'm not sure I even need to be statically linking the interpreter.

There should be no need (if all you want in the embedding).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments

[Dieter Maurer]

aapost wrote at 2023-1-10 22:15 -0500:
>On 1/4/23 12:13, aapost wrote:
>> On 1/4/23 09:42, Dieter Maurer wrote:
>> ...
>>> You might have a look at `PyXB`, too.
>>> It tries hard to enforce schema restrictions in Python code.
>> ...
>Unfortunately picking it apart for a while and diving deeper in to a
>rabbit hole, PyXB looks to be a no-go.
>
>PyXB while interesting, and I respect it's complexity and depth, is
>lacking in design consistency in how it operates if you are trying to
>modify and work with the resulting structure intuitively.
> ... problem with simple types ...

I use `PyXB` in `dm.saml2` and `dm.zope.saml2`, i.e. with
the SAML2 schema definitions (which include those
of XML signature and XML encryption).
I had no problems with simple types. I just assign them to attributes
of the Python objects representing the XML elements.
`PyXB` does the right thing when it serializes those objects into XML.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Mailing-Lists (pointer)

[Dieter Maurer]

Cameron Simpson wrote at 2023-1-11 08:37 +1100:
> ...
>There's a Discourse forum over at discuss.python.org. I use it in
>"mailing list mode" and do almost all my interactions via email, exactly
>as I do for python-list. Posts come to me and land in the same local
>mail folder I use for python-list. My replies land on the forum as
>expected (and of course also go by email to those members who have
>turned that mode on).

I am also using the Plone `Discourse` forum in "mailing list mode".
It now works quite well but it took some years before reaching this state.

For a very long time, my mail replies did not reach the forum reliably.
My latest complaint (more than half a year ago): when I had visited
the forum via `http` (I did this occasionally to verify
my reply has reached the forum), it sometimes thought, I had
seen a new message and did not inform me about it via mail.
Meanwhile, all replies seem to arrive reliably and I no longer
use `http` for access. Therefore, I do not know whether
the behavior described above still persists.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Mailing-Lists (pointer)

[Dieter Maurer]

Chris Green wrote at 2023-1-10 08:45 +:
> ...
>Yes, this is important I think.  Plus, if possible, if it's decided to
>move to a forum format make that accessible by E-Mail.

I much prefer a mailing list over an http based service.
With mailing lists, all interesting messages arrive in my email
reader, i.e. at a central place; with http based services,
I have to visit the various sites to learn whether there is
relevant new information.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: hello can I be in your group?

[Dieter Maurer]

Keith Thompson wrote at 2023-1-6 17:02 -0800:
>September Skeen  writes:
>> I was wondering if I could be in your group
>
>This is an unmoderated Usenet newsgroup.

In fact, there are several access channels, Usenet newsgroup is
one of them.

Another channel is the python-list mailing list.
You can subscribe to it on "python.org"
-- look for community/support --> mailing lists.
The maling list tends to have fewer spam then the newsgroup.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments

[Dieter Maurer]

aapost wrote at 2023-1-3 22:57 -0500:
> ...
>Consider the following:
>
>from lxml import objectify, etree
>schema = etree.XMLSchema(file="path_to_my_xsd_schema_file")
>parser = objectify.makeparser(schema=schema, encoding="UTF-8")
>xml_obj = objectify.parse("path_to_my_xml_file", parser=parser)
>xml_root = xml_obj.getroot()
>
>let's say I have a Version element, that is defined simply as a string
>in a 3rd party provided xsd schema
>
>

Does your schema include the third party schema?

You might have a look at `PyXB`, too.
It tries hard to enforce schema restrictions in Python code.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How make your module substitute a python stdlib module.

[Dieter Maurer]

Antoon Pardon wrote at 2022-12-27 14:25 +0100:
> ...
>> But a simple "sys.modules['threading'] = QYZlib.threaders" will work.
>> Of course, how *well* this works depends also on how well that module
>> manages to masquerade as the threading module, but I'm sure you've
>> figured that part out :)
>
>Well it is what will work for the moment. Thanks for the confirmation
>this will indeed work.

If you need to change a module in minor ways (e.g. only
provide a custom `thread_ident` function), you can use
a technique called "monkey patching" (which is patching at runtime).

You can usually assign new values to module variables.
Thus, you yould try `threading.thread_ident = `.

This would affect most uses of the function -- which may not be
a good idea.
Alternatively, you could monkey patch the `logging` module.
Look at its code and find out whether it accesses the function
directly or indirectly via `threading`. In the first case,
you would monkey patch the function, in the second the `threading`
variable.

You could also use `dm.reuse` (a package maintained on PyPI)
to override the method using the function. This way, your
change would be even more localized.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ContextVars in async context

[Dieter Maurer]

Marce Coll wrote at 2022-12-20 22:09 +0100:
>Hi python people, hope this is the correct place to ask this!
>
>For a transactional async decorator I'm building I am using contextvars in 
>order to know when a transaction is open in my current context.
>
>My understanding is that if given the following call stack
>
>A
>|- B
>|  |- C
>|- D
>   |- E
>
>If you set a context var at A with value 1, and then override it at B with 
>value 2, then A, D and E will see value 1 and B and C will se value 2. Very 
>similar (although a bit more manual) than dynamic scopes in common lisp.

This is not the way I understand context variables.
In my view (--> PEP 0567), the context is the coroutine not the call stack.
This means: all calls in the same coroutine share the same context
variables. In your example, if `B` overrides the context variable, then
all later calls in this coroutine will see the overridden value.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pip issue

[Dieter Maurer]

Gisle Vanem wrote at 2022-11-30 10:51 +0100:
>I have an issue with 'pip v. 22.3.1'. On any
>'pip install' command I get warning like this:
>   c:\> pip3 install asciinema
>   WARNING: Ignoring invalid distribution -arkupsafe 
> (f:\gv\python310\lib\site-packages)
>   WARNING: Ignoring invalid distribution -arkupsafe 
> (f:\gv\python310\lib\site-packages)
>   Collecting asciinema
> Downloading asciinema-2.2.0-py3-none-any.whl (92 kB)
>   ...
>
>Otherwise no issues. But where is this text "-arkupsafe" stored
>and how to get rid it it? I've searched through all of my .pth
>files and found no such string.

Have you looked at the content of the folder mentioned
in the warnings (e.g. `...\site-packages`).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Importlib behaves differently when importing pyd file

[Dieter Maurer]

Jach Feng wrote at 2022-11-15 22:52 -0800:
>My working directory d:\Works\Python\ has a package 'fitz' looks like this:
>
>fitz\
>__init__.py
>fitz.py
>utils.py
>_fitz.pyd
>
>There is a statement in fitz.py:
>return importlib.import_module('fitz._fitz')
>
>It works fine under Python 3.4 interpreter:
 import fitz

>
>But under Python 3.8 I get an exception:
 import fitz
>Traceback(...
>...
>...
>ImportError: DLL load failed while importing _fitz


The Python C-API is Python version dependent.
Your `_fitz.pyd` may need to be recreated for Python 3.8.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dealing with non-callable classmethod objects

[Dieter Maurer]

Ian Pilcher wrote at 2022-11-11 15:29 -0600:
> ...
>In searching, I've found a few articles that discuss the fact that
>classmethod objects aren't callable, but the situation actually seems to
>be more complicated.
>
> >>> type(DuidLLT._parse_l2addr)
>
> >>> callable(DuidLLT._parse_l2addr)
>True
>
>The method itself is callable, which makes sense.  The factory function
>doesn't access it directly, however, it gets it out of the _attrs
>dictionary.
>
> >>> type(DuidLLT._attrs['layer2_addr'])
>
> >>> callable(DuidLLT._attrs['layer2_addr'])
>False

Accessing an object via a `dict` does not change its type,
nor does putting it into a `dict`.
Thus, you did not put `DuidLLT._parse_l2addr` (of type `method`)
into your `_attrs` `dict` but something else (of type `classmethod`).

This narrows down the space for your investigation: why was
the object you have put into `_attr` was not what you have expected.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Superclass static method name from subclass

[Dieter Maurer]

Ian Pilcher wrote at 2022-11-11 10:21 -0600:
>Is it possible to access the name of a superclass static method, when
>defining a subclass attribute, without specifically naming the super-
>class?
>
>Contrived example:
>
>   class SuperClass(object):
>   @staticmethod
>   def foo():
>   pass
>
>   class SubClass(SuperClass):
>   bar = SuperClass.foo
> ^^
>
>Is there a way to do this without specifically naming 'SuperClass'?

Unless you overrode it, you can use `self.foo` or `SubClass.foo`;
if you overrode it (and you are using either Python 3 or
Python 2 and a so called "new style class"), you can use `super`.
When you use `super` outside a method definition, you must
call it with parameters.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: setup.py + cython == chicken and the egg problem

[Dieter Maurer]

Dan Stromberg wrote at 2022-8-16 14:03 -0700:
> ...
>I'm attempting to package up a python package that uses Cython.
> ...
>  Installing build dependencies ... error
>  error: subprocess-exited-with-error
>
>  ×? pip subprocess to install build dependencies did not run successfully.
>  ?? exit code: 1
>  > [3 lines of output]
>  Looking in indexes: https://test.pypi.org/simple/
>  ERROR: Could not find a version that satisfies the requirement
>setuptools (from versions: none)
>  ERROR: No matching distribution found for setuptools

The message tells you that there is a `setuptools` problem.
I would start to locate all `setuptools` requirement locations.


I am using `cython` for the package `dm.xmlsec.binding`.
I have not observed nor heard of a problem similar to yours
(but I have never tried `test.pypi.org`).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel(?) programming with python

[Dieter Maurer]

Dennis Lee Bieber wrote at 2022-8-10 14:19 -0400:
>On Wed, 10 Aug 2022 19:33:04 +0200, "Dieter Maurer" 
> ...
>>You could also use the `sched` module from Python's library.
>
>Time to really read the library reference manual again...
>
>   Though if I read this correctly, a long running action /will/ delay
>others -- which could mean the (FFT) process could block collecting new
>1-second readings while it is active. It also is "one-shot" on the
>scheduled actions, meaning those actions still have to reschedule
>themselves for the next time period.

Both true.

With `multiprocessing`, you can delegate long running activity
to a separate process.
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Parallel(?) programming with python

[Dieter Maurer]

Schachner, Joseph (US) wrote at 2022-8-9 17:04 +:
>Why would this application *require* parallel programming?   This could be 
>done in one, single thread program.   Call time to get time and save it as 
>start_time.   Keep a count of the number of 6 hour intervals, initialize it to 
>0.

You could also use the `sched` module from Python's library.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Trying to understand nested loops

[Dieter Maurer]

ojomooluwatolami...@gmail.com wrote at 2022-8-5 08:34 +0100:
>Hello, I’m new to learning python and I stumbled upon a question nested loops.

For future, more complex, questions of this kind,
you might have a look at the module `pdb` in Python's runtime library.
It implements a debugger which allows you (among other features)
to interactively run a program line by line
and explore the state of all involved variables.
There are also IDEs (= Integrated Development Environments) which
support this.

Whenever a program does things you do not understand,
debugging usually helps to bring light into the scene.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Register multiple excepthooks?

[Dieter Maurer]

Albert-Jan Roskam wrote at 2022-7-31 11:39 +0200:
>   I have a function init_logging.log_uncaught_errors() that I use for
>   sys.excepthook. Now I also want to call another function (ffi.dlclose())
>   upon abnormal termination. Is it possible to register multiple
>   excepthooks, like with atexit.register? Or should I rename/redefine
>   log_uncaught_errors() so it does both things?

`sys.excepthook` is a single function (not a list of them).
This means: at any moment a single `excepthook` is effective.

If you need a modular design, use a dispatcher function
as your `excepthook` associated with a registry (e.g. a `list`).
The dispatcher can then call all registered function.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fwd: timedelta object recursion bug

[Dieter Maurer]

Please stay on the list (such that others can help, too)

Ben Hirsig wrote at 2022-7-29 06:53 +1000:
>Thanks for the replies, I'm just trying to understand why this would be
>useful?
>
>E.g. why does max need a min/max/resolution, and why would these attributes
>themselves need a min/max/resolution, etc, etc?

`max` is a `timedelta` and as such inherits (e.g.) `resolution`
from the class (as any other `timedelta` instance).

Note that `timedelta` instances do not have a `max` (`min|resolution`)
slot. When `max` is looked up, it is first searched in the instance
(and not found), then in the class where it is found:
all `max` accesses result in the same object.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fwd: timedelta object recursion bug

[Dieter Maurer]

Ben Hirsig wrote at 2022-7-28 19:54 +1000:
>Hi, I noticed this when using the requests library in the response.elapsed
>object (type timedelta). Tested using the standard datetime library alone
>with the example displayed on
>https://docs.python.org/3/library/datetime.html#examples-of-usage-timedelta
>
>
>
>It appears as though the timedelta object recursively adds its own
>attributes (min, max, resolution) as further timedelta objects. I’m not
>sure how deep they go, but presumably hitting the recursion limit.

If you look at the source, you will see that `min`, `max`, `resolution`
are class level attributes. Their values are `timedelta` instances.
Therefore, you can access e.g. `timedelta(days=365).min.max.resolution`.
But this is nothing to worry about.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: argparse modify

[Dieter Maurer]

נתי שטרן wrote at 2022-6-24 08:28 +0300:
>I copied code from argparse library and modified it
>
>בתאריך יום חמישי, 23 ביוני 2022, מאת Dieter Maurer :
>
>> נתי שטרן wrote at 2022-6-23 15:31 +0300:
>> >how to solve this (argparse)
>> >
>> >
>> >traceback:
>> >Traceback (most recent call last):
>> >  File "u:\oracle\RTR.py", line 10, in 
>> >class sre_constants():
>> >  File "u:\oracle\RTR.py", line 77, in sre_constants
>> >MAXREPEAT = _NamedIntConstant(32,name=str(32))
>> >TypeError: 'name' is an invalid keyword argument for int()

The exception information tells you:
` _NamedIntConstant(32,name=str(32))` raises a `TypeError`:
`_NamedIntConstant` does not know the keyword parameter `name`.
Thus, something is wrong with the `_NamedIntConstant` definition.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: argparse modify

[Dieter Maurer]

נתי שטרן wrote at 2022-6-23 15:31 +0300:
>how to solve this (argparse)
>
>
>traceback:
>Traceback (most recent call last):
>  File "u:\oracle\RTR.py", line 10, in 
>class sre_constants():
>  File "u:\oracle\RTR.py", line 77, in sre_constants
>MAXREPEAT = _NamedIntConstant(32,name=str(32))
>TypeError: 'name' is an invalid keyword argument for int()

This does not look like an `argparse` problem:
the traceback comes from `oracle/RTR.py`.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Logging into single file from multiple modules in python when TimedRotatingFileHandler is used

[Dieter Maurer]

Chethan Kumar S wrote at 2022-6-21 02:04 -0700:
> ...
>I have a main process which makes use of different other modules. And these 
>modules also use other modules. I need to log all the logs into single log 
>file. Due to use of TimedRotatingFileHandler, my log behaves differently after 
>midnight. I got to know why it is so but couldn't get how I can solve it.
>Issue was because of serialization in logging when multiple processes are 
>involved.
>
>Below is log_config.py which is used by all other modules to get the logger 
>and log.
>import logging
>import sys
>from logging.handlers import TimedRotatingFileHandler
>
>FORMATTER = logging.Formatter("%(asctime)s — %(name)s — %(message)s")

The usual logging usage pattern is:
the individual components decide what to log
but how the logging happens it decided centrally - common
for all components.
This implies that usually the individual components do not handle
handlers or formatters but use the configuration set up centrally.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: traceback Shows path to my python libraries

[Dieter Maurer]

jsch...@sbcglobal.net wrote at 2022-6-20 13:49 -0500:
>I coded an application with a 64-bit executable using cython with the embed
>option and gcc and I received a traceback showing the path to my python
>installation.  Is that normal or does that mean the application is going
>outside of my executable to my python libraries?  I want it portable so it
>if is, then it's not portable.

The tracebacks are primarily for the developers.
Therefore, they identify source locations.
When you use `cython`, the compilation to "C" gets references
to the `cython` source (because those references are meaningful
for the developers). When a traceback is generated, the
`cython` source need not be available (you will then not
see the source line, only the line number and file information).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Issue sending data from C++ to Python

[Dieter Maurer]

Pablo Martinez Ulloa wrote at 2022-5-18 15:08 +0100:
>I have been using your C++ Python API, in order to establish a bridge from
>C++ to Python.

Do you know `cython`?
It can help very much in the implementation of bridges between
Python and C/C++.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Functionality like local static in C

[Dieter Maurer]

Cecil Westerhof wrote at 2022-4-14 17:02 +0200:
>In C when you declare a variable static in a function, the variable
>retains its value between function calls.
>The first time the function is called it has the default value (0 for
>an int).
>But when the function changes the value in a call (for example to 43),
>the next time the function is called the variable does not have the
>default value, but the value it had when the function returned.
>Does python has something like that?

In "C" a variable designates a storage location; assignment to
the variable changes the stored value.

In "Python" a variable designates an object.
Assignments to the variable do not change the object but
the association variable-object.

The im|mutability of the object determines whether the object
can or cannot have different values.
Mutable objects can behave similar to storage locations,
e.g.

class StaticVariable:
  def __init__(self, v): self.v = v
  def set(self, v): self.v = v
  def get(self): return self.v

static_emul = StaticVariable(...)

def f(...):
  ...
  static_emul.set(...)
  ...
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Comparing sequences with range objects

[Dieter Maurer]

Antoon Pardon wrote at 2022-4-7 17:16 +0200:
> ...
>Sorry I wasn't clear. The data contains information about persons. But not
>all records need to be complete. So a person can occur multiple times in
>the list, while the records are all different because they are missing
>different bits.
>
>So all records with the same firstname can be duplicates. But if I have
>a record in which the firstname is missing, it can at that point be
>a duplicate of all other records.

The description is still not clear enough. Especially, it does
not show where `range` objects come into play.


Answering on a more abstract level:
Apparently, you want to sort a list the elements of which can
be other lists or range objects.

List objects are ordered lexicographically, i.e. for lists l1, l2:
l1 <= l2 iff not l1 or (l2 and l1[0] <= l2[0] and l1[1:] <= l2[1:]).
If you want to sort a list containing list elements are compared using
this order.

For your case, you would need to use a `key` parameter for `sort` that
implements this order for `range` objects, too.
(Note that Python provides a function which transforms an order
definition into an appropriate `key` function).
A corresponding `sort` call may expand your range objects completely.

An alternative might be to not expand `range` objects but to put them
all at the start or end of the sorted list.
Of course, this would imply that their expansion does not influence
their order in the list -- which may or may not be acceptable
(depending on your use case).
If it is acceptable, it is likely possible to not put range objects
into the list to be sorted in the first place.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue47246] Race condition in `threadig.Thread._wait_for_tstate_lock`

[Dieter Maurer]

Dieter Maurer  added the comment:

The observation was caused by a bug which has been fixed in newer Python 
versions (3.9+ if I remember correctly). `isAlive` was called on a 
`_DummyThread` (while `_DummyThread` overides `is_alive` it had forgotten to 
override `isAlive` as well).

--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue47246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47246] Race condition in `threadig.Thread._wait_for_tstate_lock`

[Dieter Maurer]

Dieter Maurer  added the comment:

Apparently, the explanation is not that easy: `_stop` first sets `_is_stopped` 
to `True` and only then `_tstate_lock` to `None`. Therefore, the race should 
not cause the `AssertionError`.

I observed the `AssertionError` in Python 3.6. The related `threading` code is 
however almost identical to that in Python 3.11.

--

___
Python tracker 
<https://bugs.python.org/issue47246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47246] Race condition in `threadig.Thread._wait_for_tstate_lock`

[Dieter Maurer]

New submission from Dieter Maurer :

I have observed an `AssertionError (assert self._is_stopped)` in 
`threading.Thread._wait_for_tstate_lock`. This indicates that Python's internal 
state has been corrupted.

The analysis revealed the following race condition:
`_wait_for_tstate:lock` contains the code:
```
lock.release()
self._stop()
```
The `lock.release()` allows a conflicting call to execute. If this happens 
before `self._stop()` has executed `self._is_stopped = True`, the 
`AssertionError` is raised.

I suggest to give `_stop` an additional parameter `locked` with default 
`False`. In indicates whether the caller holds the `tstate_lock`. If this is 
the case, `_stop` releases the lock after it has ensured a consistent state 
(esspecially set `_is_stopped` to `True`). With this modification to `_stop` 
the two lines above can be replaced by `self._stop(locked=True)`.

--
components: Library (Lib)
messages: 416919
nosy: dmaurer
priority: normal
severity: normal
status: open
title: Race condition in `threadig.Thread._wait_for_tstate_lock`
type: behavior
versions: Python 3.10, Python 3.11, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue47246>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: dict.get_deep()

[Dieter Maurer]

Marco Sulla wrote at 2022-4-3 21:17 +0200:
>On Sun, 3 Apr 2022 at 18:57, Dieter Maurer  wrote:
>> You know you can easily implement this yourself -- in your own
>> `dict` subclass.
>
>Well, of course, but the question is if such a method is worth to be
>builtin, in a world imbued with JSON. I suppose your answer is no.
```
def mget(m, *keys):
  """return m[k1][k2]...[kn] or `None`"""
  for k in keys():
if k not in m: return
m = m[k]
  return m
```
Thus, it is so simple to get what you want.
No need to make it builtin.

--
Dieter
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: dict.get_deep()

[Dieter Maurer]

Marco Sulla wrote at 2022-4-2 22:44 +0200:
>A proposal. Very often dict are used as a deeply nested carrier of
>data, usually decoded from JSON. Sometimes I needed to get some of
>this data, something like this:
>
>data["users"][0]["address"]["street"]
>
>What about something like this instead?
>
>data.get_deep("users", 0, "address", "street")
>
>and also, instead of this
>
>try:
>result = data["users"][0]["address"]["street"]
>except KeyError, IndexError:
>result = "second star"
>
>write this:
>
>data.get_deep("users", 0, "address", "street", default="second star")

You know you can easily implement this yourself -- in your own
`dict` subclass.

You can also customize the JSON decoding (--> `object_hook`)
to determine how a JSON object is mapped to a Python object;
it defaults to `dict` but you could use your own `dict` subclass.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Exchange OWA using Python?

[Dieter Maurer]

Grant Edwards wrote at 2022-3-31 07:41 -0700:
>Is anybody aware of any Python code for the Exchange OWA protocol/API?

According to "https://en.wikipedia.org/wiki/Outlook.com#Mail_client_access;
Outlook.com (the modern name for OWA) supports "pop3" and "imap",
both supported by Python library modules.

Read the "Outlook.com" documentation about how you can connect
via those protocols.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Typing on child class' methods of a Generic base class

[Dieter Maurer]

Nicolas Haller wrote at 2022-3-12 12:05 -0500:
>On 2022-03-10 12:31, Dieter Maurer wrote:
>> Nicolas Haller wrote at 2022-3-9 10:53 -0500:
>>> ...
>>> The documentation about "user-defined generic types"[1] says that I can
>>> fix some types on a child class (class MyDict(Mapping[str, T]):) but
>>> doesn't say much about the signature of the methods I need to
>>> implement/override on that child class.
>>
>> I have the fealing that this is a case of (generic type) "specialization".
>> In this setup, `Mapping` would be a generic type with two type
>> variables `K` (the key type) and `V` (the value type).
>> The signatures of its methods would use those type variables.
>> In your example, you specialize the key type to `str` (and
>> leave the value type generic). The signatures of the methods
>> would automatically follow this specialization -- without the need
>> to do anything.
>
>If I understand correctly, you're saying I should use the first
>alternative by keeping the signature as it is in the base class:
>---
>T = TypeVar("T")
>
>
>class Base(Generic[T], metaclass=ABCMeta):
> """A base class."""
>
> @abstractmethod
> def _private_method(self, an_arg: T) -> T:
> ...
>
> def public_method(self, an_arg: T) -> T:
> from_private_method = self._private_method(an_arg)
> return from_private_method
>
>class Alternative1(Base[int]):
> def _private_method(self, an_arg: T) -> T:  # I keep T
> return 42
>---
>
>The problem with it is that mypy doesn´t seem quite happy with it:
>./scratch.py:22: error: Incompatible return value type (got "int",
>expected "T")
>
>Do you think this is error is incorrect?

No, I do not think so.

The `Base[int]` means that the generic type variable `T` was
specialized to `int`. This means that in this context
`_private_method` returns `int` (not a generic `T`).

You either do not type annotate the overriding methods
or you use the specialization (if any).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to test input via subprocess.Popen with data from file

[Dieter Maurer]

Loris Bennett wrote at 2022-3-11 07:40 +0100:
> ... I want to test the parsing ...
>Sorry if I was unclear but my question is:
>
>Given that the return value from Popen is a Popen object and given that
>the return value from reading a file is a single string or maybe a list
>of strings, what should the common format for the argument which is
>passed to the actual parsing function be?

What methods (of its input argument) does the parsing use?
If it uses `Popen` methods, then you pass a `POpen` object;
if it uses only (typical) file methods, then you pass a file object;
if it assumes its input to be a (line) interator, you pass
a (line) iterator (such as a "file" object).

I would design the parsing that it makes as few assumptions
about its input as possible -- to ease testing
and increase the chance for reuse.

That said, I would not design it to work with `Popen` objects
but likely to have a line iterator as input.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Typing on child class' methods of a Generic base class

[Dieter Maurer]

Nicolas Haller wrote at 2022-3-9 10:53 -0500:
> ...
>The documentation about "user-defined generic types"[1] says that I can
>fix some types on a child class (class MyDict(Mapping[str, T]):) but
>doesn't say much about the signature of the methods I need to
>implement/override on that child class.

I have the fealing that this is a case of (generic type) "specialization".
In this setup, `Mapping` would be a generic type with two type
variables `K` (the key type) and `V` (the value type).
The signatures of its methods would use those type variables.
In your example, you specialize the key type to `str` (and
leave the value type generic). The signatures of the methods
would automatically follow this specialization -- without the need
to do anything.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to test input via subprocess.Popen with data from file

[Dieter Maurer]

Loris Bennett wrote at 2022-3-10 13:16 +0100:
>I have a command which produces output like the
>following:
>
>  Job ID: 9431211
>  Cluster: curta
>  User/Group: build/staff
>  State: COMPLETED (exit code 0)
>  Nodes: 1
>  Cores per node: 8
>  CPU Utilized: 01:30:53
>  CPU Efficiency: 83.63% of 01:48:40 core-walltime
>  Job Wall-clock time: 00:13:35
>  Memory Utilized: 6.45 GB
>  Memory Efficiency: 80.68% of 8.00 GB
>
>I want to parse this and am using subprocess.Popen and accessing the
>contents via Popen.stdout.  However, for testing purposes I want to save
>various possible outputs of the command as text files and use those as
>inputs.

What do you want to test? the parsing? the "popen" interaction?
You can separately test both tasks (I, at your place, would do this).

For the parsing test, it is not relevant that the actual text
comes from an external process. You can directly read it from a file
or have it in your text.

In my view, you do not need a test for the `Popen` interaction:
if it works once, it will work always.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

[Dieter Maurer]

Martin Di Paola wrote at 2022-3-6 20:42 +:
>>Try to use `fork` as "start method" (instead of "spawn").
>
>Yes but no. Indeed with `fork` there is no need to pickle anything. In
>particular the child process will be a copy of the parent so it will
>have all the modules loaded, including the dynamic ones. Perfect.
>
>The problem is that `fork` is the default only in Linux. It works in
>MacOS but it may lead to crashes if the parent process is multithreaded
>(and the my is!) and `fork` does not work in Windows.

Then, you must put the initialization (dynamically loading the modules)
into the function executed in the foreign process.

You could wrap the payload function into a class instances to achieve this.
In the foreign process, you call the instance which first performs
the initialization and then executes the payload.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

[Dieter Maurer]

Martin Di Paola wrote at 2022-3-6 12:42 +:
>Hi everyone. I implemented time ago a small plugin engine to load code
>dynamically.
>
>So far it worked well but a few days ago an user told me that he wasn't
>able to run in parallel a piece of code in MacOS.
>
>He was using multiprocessing.Process to run the code and in MacOS, the
>default start method for such process is using "spawn". My understanding
>is that Python spawns an independent Python server (the child) which
>receives what to execute (the target function) from the parent process.
>
>In pseudo code this would be like:
>
>modules = loader() # load the plugins (Python modules at the end)
>objs = init(modules) # initialize the plugins
>
># One of the plugins wants to execute part of its code in parallel
># In MacOS this fails
>ch = multiprocessing.Process(target=objs[0].sayhi)
>ch.start()
>
>The code fails with "ModuleNotFoundError: No module named 'foo'" (where
>'foo' is the name of the loaded plugin).
>
>This is because the parent program sends to the serve (the child) what
>needs to execute (objs[0].sayhi) using pickle as the serialization
>mechanism.
>
>Because Python does not really serialize code but only enough
>information to reload it, the serialization of "objs[0].sayhi" just
>points to its module, "foo".
>
>Module which it cannot be imported by the child process.
>
>So the question is, what would be the alternatives and workarounds?

Try to use `fork` as "start method" (instead of "spawn").
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Dieter Maurer]

Avi Gross wrote at 2022-3-4 16:43 +:
>Your use is creative albeit it is not "needed" since all it does is make sure 
>your variable is initialized to something, specifically None.
>
>So would this not do the same thing?
>
>  eye = None
>
>  for eye in range(0):
>  print(eye)
>
>  eye

It would.

But, the `for` construct's main purpose is often to determine
a value for the loop variable and in this case, I like
that it itself can ensure that it gets a value.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Dieter Maurer]

Rob Cliffe wrote at 2022-3-4 00:13 +:
>I find it so hard to remember what `for ... else` means that on the very
>few occasions I have used it, I ALWAYS put a comment alongside/below the
>`else` to remind myself (and anyone else unfortunate enough to read my
>code) what triggers it, e.g.
>
>     for item in search_list:
>         ...
>         ... break
>     else: # if no item in search_list matched the criteria
>
>You get the idea.
>If I really want to remember what this construct means, I remind myself
>that `else` here really means `no break`.  Would have been better if it
>had been spelt `nobreak` or similar in the first place.

One of my use cases for `for - else` does not involve a `break`:
the initialization of the loop variable when the sequence is empty.
It is demonstrated by the following transscript:

```pycon
>>> for i in range(0):
...   pass
...
>>> i
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'i' is not defined
>>> for i in range(0):
...   pass
... else: i = None
...
>>> i
>>>
```

For this use case, `else` is perfectly named.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behavior of the for-else construct

[Dieter Maurer]

computermaster360 wrote at 2022-3-3 14:24 +0100:
>Do you find the for-else construct useful?

Yes.

>Have you used it in practice?

Yes
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: lxml empty versus self closed tag

[Dieter Maurer]

Robin Becker wrote at 2022-3-3 09:21 +:
>On 02/03/2022 18:39, Dieter Maurer wrote:
>> Robin Becker wrote at 2022-3-2 15:32 +:
>>> I'm using lxml.etree.XMLParser and would like to distinguish
>>>
>>> 
>>>
>>> from
>>>
>>> 
>>>
>>> I seem to have e.getchildren()==[] and e.text==None for both cases. Is 
>>> there a way to get the first to have e.text==''
>>
>> I do not think so (at least not without a DTD):
>
>I have a DTD which has
>
>
>
>so I guess the empty case is allowed as well as the self closed.

Potentially, something changes when `content` contains `PCDATA` (as
one possibility) (but I doubt it).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: lxml empty versus self closed tag

[Dieter Maurer]

Robin Becker wrote at 2022-3-2 15:32 +:
>I'm using lxml.etree.XMLParser and would like to distinguish
>
>
>
>from
>
>
>
>I seem to have e.getchildren()==[] and e.text==None for both cases. Is there a 
>way to get the first to have e.text==''

I do not think so (at least not without a DTD):
`' is just a shorthand notation for '' and
the difference has no influence on the DOM.

Note that `lxml` is just a Python binding for `libxml2`.
All the parsing is done by this library.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why does not Python accept functions with no names?

[Dieter Maurer]

Abdur-Rahmaan Janhangeer wrote at 2022-2-20 19:32 +0400:
>Out of curiosity, why doesn't Python accept
>def ():
>return '---'
>
>()
>
>Where the function name is ''?

Python knows about (somewhat restricted) anonymous functions:
it calls them `lambda` expressions (the body of those functions
can only be an expression).

Your example above can be expressed as
`(lambda: '---')()`.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Correct way to setup a package with both compiled C code and Python code?

[Dieter Maurer]

Christian Gollwitzer wrote at 2022-2-8 22:43 +0100:
>Am 08.02.22 um 18:57 schrieb Dieter Maurer:
>> Christian Gollwitzer wrote at 2022-2-7 20:33 +0100:
>>> we've developed a Python pacakge which consists of both a compiled
>>> extension module and some helper functions in Python. Is there a
>>> tutorial on how to package such an extension?
>>
>> Look at "https://package.python.org;,
>> especially 
>> "https://packaging.python.org/en/latest/guides/packaging-binary-extensions/;.
>
>Thank you, but that page is more like a high-level description, it talks
>a lot about the differences between C code and Python and how it can be
>combined using SWIG etc, but I already have a working extension.

Packaging and distribution was originally handled by `distutils`;
other tools, e.g. `setuptools`, extended `distutils` and have
gained wide spread acceptance. As a consequence, `distutils`
has been deprecated in favor of those other tools.
I assume the use of `setuptools` below.

Extensions are described via the `ext_modules` parameter of
the `setup` function. Its value is a sequence of
`setuptools.Extension` instances.

As of Python 3.11, `setuptools.Extension` is still a minor enhancement
of `distutils.extension.Extension`. This is documented via
docstrings in its source.
Read this documentation and come back if questions remain.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Correct way to setup a package with both compiled C code and Python code?

[Dieter Maurer]

Christian Gollwitzer wrote at 2022-2-7 20:33 +0100:
>we've developed a Python pacakge which consists of both a compiled
>extension module and some helper functions in Python. Is there a
>tutorial on how to package such an extension?

Look at "https://package.python.org;,
especially 
"https://packaging.python.org/en/latest/guides/packaging-binary-extensions/;.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ssl server: how to disable client cert verfication?

[Dieter Maurer]

Grant Edwards wrote at 2022-2-3 14:36 -0800:
>On 2022-02-03, Barry  wrote:
> ...
>I've looked through the ssl.Context documentation multiple times, and
>haven't been able to spot any option or flag that disables client
>certificate validation or allows the user to override the actual
>client certificate validation process.

Note that Python does not do the certificate validation itself
but delegates this to the underlying SSL library.
Thus, this library would need to support your use case.
It may not as your scenario is quite special.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PYTHONPATH vs Python Virtual Environment

[Dieter Maurer]

Sina Mobasheri wrote at 2022-2-4 15:55 +:
>it's not good title defiantly and I don't mean to compare apples and oranges
>
>when I start using python virtual environment it was because isolation 
>proposes and everyone say about its benefits in isolation and working with 
>different versions of the same package in different projects
>
>but recently I start using pip install --target  for 
>zipapp
> things, and then I use this pip's option (--target) and add its target folder 
>to PYTHONPATH and target folder's bin directory to PATH, so it's like virtual 
>environment to me
>
>I'm curious what is a problem with this approach (edges), what are other 
>benefits of the virtual environment that this way can't offer? most tutorials 
>talk about isolation and managing of packages versions but I think maybe it's 
>more than that maybe?

Usually, PYTHONPATH is global.
This can make a problem when you have different applications
which require different package versions.

Of course, you can use a wrapper for your application which locally
sets PYTHONPATH appropriately.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: http.client and dns lookups

[Dieter Maurer]

Michael Welle wrote at 2022-2-1 19:28 +0100:
> ...
>That doesn't happen when the 'real' issue occurs. Attaching strace to
>the Python process I can see that resolv.conf is stat'ed and open'ed. I
>guess now I'm more confused than before ;). There must be an additional
>condition that I'm missing.

The DNS service routinely uses caches. Outdated cache values
can cause (apparently non deterministic) failures.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: http.client and dns lookups

[Dieter Maurer]

Michael Welle wrote at 2022-1-30 09:18 +0100:
> ...
 The machine this is running on regularly switches
>its network configuration without restarting the Python application. Now
>it turns out that the application is still using an old, outdated dns
>server after such a network configuration switch.

It is unlikely that Python performs the host -> IP address translation
itself. Almost surely, the translation is delegated to functions
of the underlying C runtime library, e.g. "gethostbyname".
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "undefined symbol" in C extension module

[Dieter Maurer]

Robert Latest wrote at 2022-1-22 11:29 GMT:
>I've written some CPython extension modules in the past without problems. Now
>after moving to a new Archlinux box with Python3.10 installed, I can't build
>them any more. Or rather, I can build them but not use them due to "undefined
>symbols" during linking. Here's ldd's output when used on the "spam" example
>library from the docs:
>
>   linux-vdso.so.1 (0x7ffe2454a000)
>   libc.so.6 => /usr/lib/libc.so.6 (0x7fb6b6eb9000)
>   /usr/lib64/ld-linux-x86-64.so.2 (0x7fb6b70a4000)
>undefined symbol: PyObject_Init(./build/lib.linux-x86_64-3.10/
>spam.cpython-310-x86_64-linux-gnu.so)

There are important C-API changes for Python 3.10 (and above).
You may need to modify your extensions such that they work with
recent Python versions.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: preserving entities with lxml

[Dieter Maurer]

Robin Becker wrote at 2022-1-13 09:13 +:
>On 12/01/2022 20:49, Dieter Maurer wrote:
> ...
>> Apparently, the `resolve_entities=False` was not effective: otherwise,
>> your tree content should have more structure (especially some
>> entity reference children).
>>
>except that the tree knows not to expand the entities using ET.tostring so in 
>some circumstances resolve_entities=False
>does work.

I think this is a misunderstanding: `tostring` will represent the text 
character `&` as ``.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: preserving entities with lxml

[Dieter Maurer]

Robin Becker wrote at 2022-1-12 10:22 +:
>I have a puzzle over how lxml & entities should be 'preserved' code below 
>illustrates. To preserve I change & --> 
>in the source and add resolve_entities=False to the parser definition. The 
>escaping means we only have one kind of
>entity  which means lxml will preserve it. For whatever reason lxml won't 
>preserve character entities eg .
>
>The simple parse from string and conversion tostring shows that the parsing at 
>least took notice of it.
>
>However, I want to create a tuple tree so have to use tree.text, 
>tree.getchildren() and tree.tail for access.
>
>When I use those I expected to have to undo the escaping to get back the 
>original entities, but it seems they are
>already done.
>
>Good for me, but if the tree knows how it was created (tostring shows that) 
>why is it ignored with attribute access?
>
>if __name__=='__main__':
> from lxml import etree as ET
> #initial xml
> xml = b'a
>   A'
> #escaped xml
> xxml = xml.replace(b'&',b'')
>
> myparser = ET.XMLParser(resolve_entities=False)
> tree = ET.fromstring(xxml,parser=myparser)
>
> #use tostring
> print(f'using tostring\n{xxml=!r}\n{ET.tostring(tree)=!r}\n')
>
> #now access the items using text & children & text
> print(f'using 
> attributes\n{tree.text=!r}\n{tree.getchildren()=!r}\n{tree.tail=!r}')
>
>when run I see this
>
>$ python tmp/tlp.py
>using tostring
>xxml=b'a 
>mysym; lt; amp; gt;
>#33; A'
>ET.tostring(tree)=b'a mysym; lt; amp;
>gt; #33; A'
>
>using attributes
>tree.text='a  A'
>tree.getchildren()=[]
>tree.tail=None

Apparently, the `resolve_entities=False` was not effective: otherwise,
your tree content should have more structure (especially some
entity reference children).

Re: Script profiling details

[Dieter Maurer]

Joseph L. Casale wrote at 2022-1-10 18:43 +:
> ...
>I expected this given the implementation, but I was hoping to get some
>finer details so I can track down the specific module or at least the specific
>file so I have a place to start reviewing code for optimizations.
>
>Is there something I can use to analyze the existing profile output or to 
>generate
>it with more feedback?

You might try `py-spy`.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?

[Dieter Maurer]

Marco Sulla wrote at 2021-12-29 11:59 +0100:
>On Wed, 29 Dec 2021 at 09:12, Dieter Maurer  wrote:
>> `MutableMapping` is a so called abstract base class (--> `abc`).
>>
>> It uses the `__subclass_check__` (and `__instance_check__`) of
>> `abc.ABCMeta` to ensure `issubclass(dict, MutableMapping)`.
>> Those can be customized by overriding `MutableMapping.__subclasshook__`
>> to ensure that your `frozendict` class (and their subclasses)
>> are not considered subclasses of `MutableMapping`.
>
>It does not work:
> ...
>>>> issubclass(fd, Mm)
>True

There is a cache involved. The `issubclass` above,
brings your `fd` in the `Mn`'s subclass cache.

> ...
>>>> Mm.__subclasshook__ = _my_subclasshook
>>>> issubclass(fd, Mm)
>True

See above.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's the public API alternative to _PyObject_GC_IS_TRACKED()?

[Dieter Maurer]

Marco Sulla wrote at 2021-12-29 09:29 +0100:
>On second thought, I think I'll do this for the pure py version. But I
>will definitely not do this for the C extension

Are you sure you need to implement your type in C at all?

I made a small `timeit` test:
```
>>> class cd(dict): pass
...
>>> timeit("d[1]", "d={1:1}", globals=globals())
0.0247416019765
>>> timeit("d[1]", "d=cd({1:1})", globals=globals())
0.08281239100051607
```
This means that for the above trivial case, access is 3.5 times slower
(the difference is smaller for more complex cases when hashing
becomes more expensive) but it is still only 83 ns per access.
Thus, if your application is not highly dominated by dict accesses,
the overall difference will not be large.
-- 
https://mail.python.org/mailman/listinfo/python-list