Re: 'dn' saying "Welcome (to the list)"

[Chris Angelico]

On Sun, 26 Feb 2023 at 16:23, Hen Hanna  wrote:
>
>
> On Thursday, March 4, 2021  dn wrote:
> > Hi, and welcome to the list.
>
>
> note that this is the very same rude guy ('dn')   who is apparently the 
> rudest of them all.
>
> note that this is the very same rude guy ('dn')   who wants to shunt me away 
> to the   TUTOR   list.
>
>   ---  i guess i'd consider being TUTOR'ed by 'dn' (or 
> someone like it)
>if   i  can  be  paid$1000 / hour  for  it.
>
>
>
> Have others noticed this too?  (it's often the guy who says "Welcome" 
> that)
>
>
> Whenever someone  says  "Welcome"  to me (in this way),  my alarm goes off.
>
>
> - a good way to explain it (the Phenomenon)  is...   it's an 
> indication that he's itching to treat me as a Newbie...  that he 's  
> salivating because he senses  an opportunity to   MANsplain   and   
> WHITEsplain   things to me.
>

Fine, I won't say "welcome to the list" to anyone in the future.
Instead, I'll say "why don't you go and solve your own problems by
searching the web".

That's WAY more friendly, right? We should all start doing that.

It's true what they say - no good deed goes unpunished.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Chris Angelico]

On Sun, 26 Feb 2023 at 16:27, Dennis Lee Bieber  wrote:
>
> On Sat, 25 Feb 2023 15:41:52 -0600, Skip Montanaro
>  declaimed the following:
>
>
> >concurrent.futures.ThreadPoolExecutor() with the default number of workers (
> >os.cpu_count() * 1.5, or 12 threads on my system) to process each month, so
> >12 active threads at a time. Given that the process is pretty much CPU
> >bound, maybe reducing the number of workers to the CPU count would make
>
> Unless things have improved a lot over the years, the GIL still limits
> active threads to the equivalent of a single CPU. The OS may swap among
> which CPU as it schedules system processes, but only one thread will be
> running at any moment regardless of CPU count.

Specifically, a single CPU core *executing Python bytecode*. There are
quite a few libraries that release the GIL during computation. Here's
a script that's quite capable of saturating my CPU entirely - in fact,
typing this email is glitchy due to lack of resources:

import threading
import bcrypt
results = [0, 0]
def thrd():
for _ in range(10):
ok = bcrypt.checkpw(b"password",
b'$2b$15$DGDXMb2zvPotw1rHFouzyOVzSopiLIUSedO5DVGQ1GblAd6L6I8/6')
results[ok] += 1

threads = [threading.Thread(target=thrd) for _ in range(100)]
for t in threads: t.start()
for t in threads: t.join()
print(results)

I have four cores eight threads, and yeah, my CPU's not exactly the
latest and greatest (i7 6700k - it was quite good some years ago, but
outstripped now), but feel free to crank the numbers if you want to.

I'm pretty sure bcrypt won't use more than one CPU core for a single
hashpw/checkpw call, but by releasing the GIL during the hard number
crunching, it allows easy parallelization. Same goes for numpy work,
or anything else that can be treated as a separate operation.

So it's more accurate to say that only one CPU core can be
*manipulating Python objects* at a time, although it's hard to pin
down exactly what that means, making it easier to say that there can
only be one executing Python bytecode; it should be possible for any
function call into a C library to be a point where other threads can
take over (most notably, any sort of I/O, but also these kinds of
things).

As mentioned, GIL-removal has been under discussion at times, most
recently (and currently) with PEP 703
https://peps.python.org/pep-0703/ - and the benefits in multithreaded
applications always have to be factored against quite significant
performance penalties. It's looking like PEP 703's proposal has the
least-bad performance measurements of any GILectomy I've seen so far,
showing 10% worse performance on average (possibly able to be reduced
to 5%). As it happens, a GIL just makes sense when you want pure, raw
performance, and it's only certain workloads that suffer under it.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Chris Angelico]

On Sun, 26 Feb 2023 at 16:16, Jon Ribbens via Python-list
 wrote:
>
> On 2023-02-25, Paul Rubin  wrote:
> > The GIL is an evil thing, but it has been around for so long that most
> > of us have gotten used to it, and some user code actually relies on it.
> > For example, with the GIL in place, a statement like "x += 1" is always
> > atomic, I believe.  But, I think it is better to not have any shared
> > mutables regardless.
>
> I think it is the case that x += 1 is atomic but foo.x += 1 is not.
> Any replacement for the GIL would have to keep the former at least,
> plus the fact that you can do hundreds of things like list.append(foo)
> which are all effectively atomic.

The GIL is most assuredly *not* an evil thing. If you think it's so
evil, go ahead and remove it, because we'll clearly be better off
without it, right?

As it turns out, most GIL-removal attempts have had a fairly nasty
negative effect on performance. The GIL is a huge performance boost.

As to what is atomic and what is not... it's complicated, as always.
Suppose that x (or foo.x) is a custom type:

class Thing:
def __iadd__(self, other):
print("Hi, I'm being added onto!")
self.increment_by(other)
return self

Then no, neither of these is atomic, although if the increment itself
is, it probably won't matter. As far as I know, the only way that it
would be at all different for x+=1 and foo.x+=1 would be if the
__iadd__ method both mutates and returns something other than self,
which is quite unusual. (Most incrementing is done by either
constructing a new object to return, or mutating the existing one, but
not a hybrid.)

Consider this:

import threading
d = {0:0, 1:0, 2:0, 3:0}
def thrd():
for _ in range(1):
d[0] += 1
d[1] += 1
d[2] += 1
d[3] += 1

threads = [threading.Thread(target=thrd) for _ in range(50)]
for t in threads: t.start()
for t in threads: t.join()
print(d)

Is this code guaranteed to result in 50 in every slot in the
dictionary? What if you replace the dictionary with a four-element
list? Do you need a GIL for this, or some other sort of lock? What
exactly is it that is needed? To answer that question, let's look at
exactly what happens in the disassembly:

>>> def thrd():
... d[0] += 1
... d[1] += 1
...
>>> import dis
>>> dis.dis(thrd)
  1   0 RESUME   0

  2   2 LOAD_GLOBAL  0 (d)
 14 LOAD_CONST   1 (0)
 16 COPY 2
 18 COPY 2
 20 BINARY_SUBSCR
 30 LOAD_CONST   2 (1)
 32 BINARY_OP   13 (+=)
 36 SWAP 3
 38 SWAP 2
 40 STORE_SUBSCR

  3  44 LOAD_GLOBAL  0 (d)
 56 LOAD_CONST   2 (1)
 58 COPY 2
 60 COPY 2
 62 BINARY_SUBSCR
 72 LOAD_CONST   2 (1)
 74 BINARY_OP   13 (+=)
 78 SWAP 3
 80 SWAP 2
 82 STORE_SUBSCR
 86 LOAD_CONST   0 (None)
 88 RETURN_VALUE
>>>

(Your exact disassembly may differ, this was on CPython 3.12.)
Crucially, note these three instructions that occur in each block:
BINARY_SUBSCR, BINARY_OP, and STORE_SUBSCR. Those are a lookup
(retrieving the value of d[0]), the actual addition (adding one to the
value), and a store (putting the result back into d[0]). So it's
actually not guaranteed to be atomic; it would be perfectly reasonable
to interrupt that sequence and have something else do another
subscript.

Here's the equivalent with just incrementing a global:

>>> def thrd():
... x += 1
...
>>> dis.dis(thrd)
  1   0 RESUME   0

  2   2 LOAD_FAST_CHECK  0 (x)
  4 LOAD_CONST   1 (1)
  6 BINARY_OP   13 (+=)
 10 STORE_FAST   0 (x)
 12 LOAD_CONST   0 (None)
 14 RETURN_VALUE
>>>

The exact same sequence: load, add, store. Still not atomic.

General takeaway: The GIL is a performance feature, not a magic
solution, and certainly not an evil beast that must be slain at any
cost. Attempts to remove it always have to provide equivalent
protection in some other way. But the protection you think you have
might not be what you actually have.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

creepy email... Subj: Just to you.

[Hen Hanna]

   i received thiscreepy email...  Subj: Just to you.



>>>Just to you as I see little further reason to help you ...

>>>You have not taken hints that bashing the language as not doing what you 
>>> expect is not the way to get help so I leave you to others.


i'll check (!)  if he actually said anything helpful or informative !
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Dennis Lee Bieber]

On Sat, 25 Feb 2023 15:41:52 -0600, Skip Montanaro
 declaimed the following:


>concurrent.futures.ThreadPoolExecutor() with the default number of workers (
>os.cpu_count() * 1.5, or 12 threads on my system) to process each month, so
>12 active threads at a time. Given that the process is pretty much CPU
>bound, maybe reducing the number of workers to the CPU count would make

Unless things have improved a lot over the years, the GIL still limits
active threads to the equivalent of a single CPU. The OS may swap among
which CPU as it schedules system processes, but only one thread will be
running at any moment regardless of CPU count.

Common wisdom is that Python threading works well for I/O bound
systems, where each thread spends most of its time waiting for some I/O
operation to complete -- thereby allowing the OS to schedule other threads.

For CPU bound, use of the multiprocessing package may be more suited --
though you'll have to device a working IPC system transfer data to/from the
separate processes (no shared objects as possible with threads).


-- 
Wulfraed Dennis Lee Bieber AF6VN
wlfr...@ix.netcom.comhttp://wlfraed.microdiversity.freeddns.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

'dn' saying "Welcome (to the list)"

[Hen Hanna]

On Thursday, March 4, 2021  dn wrote:
> Hi, and welcome to the list.


note that this is the very same rude guy ('dn')   who is apparently the rudest 
of them all.

note that this is the very same rude guy ('dn')   who wants to shunt me away to 
the   TUTOR   list.

  ---  i guess i'd consider being TUTOR'ed by 'dn' (or someone 
like it)
   if   i  can  be  paid$1000 / hour  for  it.



Have others noticed this too?  (it's often the guy who says "Welcome" that)


Whenever someone  says  "Welcome"  to me (in this way),  my alarm goes off.


- a good way to explain it (the Phenomenon)  is...   it's an indication 
that he's itching to treat me as a Newbie...  that he 's  salivating because he 
senses  an opportunity to   MANsplain   and   WHITEsplain   things to me.
-- 
https://mail.python.org/mailman/listinfo/python-list

回复: Question about logging.config.dictConfig

[outlook_f8d50c5b9e059...@outlook.com]

    

    

   从 Windows 版[1]邮件发送

    

   发件人: [2]Ivan "Rambius" Ivanov
   发送时间: 2023年2月8日 23:29
   收件人: [3]python-list@python.org
   主题: Re: Question about logging.config.dictConfig

    

   On Tue, Feb 7, 2023 at 7:35 PM Peter J. Holzer  wrote:

   > 

   > On 2023-02-07 17:58:26 -0500, Ivan "Rambius" Ivanov wrote:

   > > I am trying to configure my loggers using dictConfig, but they do not

   > > print anything. Here are more details.

   > [...]

   > > from myloggingconf import configure_logging

   > >

   > > logger = logging.getLogger(os.path.basename(__file__))

   > [...]

   > >

   > > def main():

   > > configure_logging()

   > > dosmth()

   > [...]

   > > def configure_logging():

   > > config = {

   > > 'version': 1,

   > > 'disable_existing_loggers': True,

   > 

   >   'disable_existing_loggers': False,

    

   Thank you! That helped a lot!

    

   Regards

   rambius

    

   > 

   > I think I just found out why I've been cargo-culting this line since my

   > early Django days ;-)-

   > 

   > If you call getLogger very early (as you do), you don't want it disabled

   > by a later call to dictConfig().

   > 

   > 

   > > 'formatters': {

   > > 'default': {

   >   >   'fmt':   '%(asctime)s   %(levelname)-7s   %(name)s
   %(funcName)s %(message)s'

   >   'format'

   > > }

   > > },

   > [...]

   > > }

   > > logging.config.dictConfig(config)

   > >

   > > When I run uselog.py it prints nothing. I am wondering what is wrong

   > > with the second configuration.

   > 

   > See above.

   > 

   > hp

   > 

   > --

   >    _  | Peter J. Holzer    | Story must make more sense than reality.

   > |_|_) |    |

   > | |   | h...@hjp.at |    -- Charles Stross, "Creative writing

   > __/   | http://www.hjp.at/ |   challenge!"

   > --

   > https://mail.python.org/mailman/listinfo/python-list

    

    

    

   --

   Tangra Mega Rock: http://www.radiotangra.com

   --

   https://mail.python.org/mailman/listinfo/python-list

    

References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986
   2. mailto:rambiusparkisan...@gmail.com
   3. mailto:python-list@python.org
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Jon Ribbens via Python-list]

On 2023-02-25, Paul Rubin  wrote:
> Jon Ribbens  writes:
>>> 1) you generally want to use RLock rather than Lock
>> Why?
>
> So that a thread that tries to acquire it twice doesn't block itself,
> etc.  Look at the threading lib docs for more info.

Yes, I know what the docs say, I was asking why you were making the
statement above. I haven't used Lock very often, but I've literally
never once in 25 years needed to use RLock. As you say, it's best
to keep the lock-protected code brief, so it's usually pretty
obvious that the code can't be re-entered.

>> What does this mean? Are you saying the GIL has been removed?
>
> Last I heard there was an experimental version of CPython with the GIL
> removed.  It is supposed to take less of a performance hit due to
> INCREF/DECREF than an earlier attempt some years back.  I don't know its
> current status.
>
> The GIL is an evil thing, but it has been around for so long that most
> of us have gotten used to it, and some user code actually relies on it.
> For example, with the GIL in place, a statement like "x += 1" is always
> atomic, I believe.  But, I think it is better to not have any shared
> mutables regardless.

I think it is the case that x += 1 is atomic but foo.x += 1 is not.
Any replacement for the GIL would have to keep the former at least,
plus the fact that you can do hundreds of things like list.append(foo)
which are all effectively atomic.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: TypeError: can only concatenate str (not "int") to str

[Hen Hanna]

On Saturday, February 25, 2023 at 1:54:14 PM UTC-8, Paul Rubinwrote:
> Hen Hanna  writes: 
> > is this guy (dn) always this rude??? is he even more rude on this 
> > Python-Tutor list ?

> I'm not on either list but the purpose of the tutor list is to shunt 
> beginner questions away from the main list. Yes your questions do seem 
> excessively frequent even here on Usenet. I have mostly been ignoring 
> them after seeing the first few.


not sure...  what you mean by "either" list

this rude guy ('dn'  who writes poorly) is someone i want to see posting less 
(and fewer),

and (from what i've seen) ,   Paul Rubin  is someone i wish to see posting more.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: No module named _socket, on windows

[ofek shulberg]

On Monday, January 4, 2010 at 9:19:21 PM UTC+2, Gabriel Genellina wrote:
> En Mon, 04 Jan 2010 14:24:22 -0300, louisJ  escribi�:
> > I installed python 2.6 (from python.org) for windows XP, and then
> > Pylab.
> > When I type "import pylab" in a python shell it shows the error:
> >
> > ImportError: No module named _socket
> Open the Python command line, type the following lines, and tell us what 
> happens:
> Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit 
> (Intel)] on
> win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import socket
> >>> socket._socket
> 
> 
> -- 
> Gabriel Genellina


hi Gabriel,
i have the same problem and also pip is not installed for some reason, what can 
i do?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: TypeError: can only concatenate str (not "int") to str

[Hen Hanna]

On Saturday, February 25, 2023 at 4:54:36 PM UTC-8, Greg Ewing wrote:
> On 26/02/23 10:53 am, Paul Rubin wrote: 
> > I'm not on either list but the purpose of the tutor list is to shunt 
> >  beginner questions away from the main list.


> There's a fundamental problem with tutor lists. They rely on 
> experienced people, the ones capable of answering the questions, 
> to go out of their way to read the tutor list -- something that 
> is not of personal benefit to them. 
> 
> Also, pointing people towards tutor lists, if not done carefully, 
> can give the impression of saying "newcomers are not welcome here". 
> That's not a message we want to send to Python newcomers at all. 
> 
> -- 
> Greg



1.   i still think that  my original question  about   (not "int")   is  a good 
one.
  it may  seem like a  stupid  [beginner question] , 
but it really isn't.


2. the rude guy ('dn') hasn't  offered  a single word of comment   that's 
directly relevant to it.
     but he did  offer related stuff  which he thinks i 
should be  [grateful] for


3.   it's pretty clear  that  the rude guy ('dn')  intended-intends  to insult 
me as a Newbie... that much is certain.



___
 It'd be easy for the Error-Msg to include   the   VALUE   of the int.

A   LISP   programmer knows the value of everything,
  but the cost of nothing.



>>>   TypeError: can only join an iterable

  Here she's not even telling me the TYPE of the offending 
object.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Jon Ribbens via Python-list]

On 2023-02-25, Paul Rubin  wrote:
> Skip Montanaro  writes:
>> from threading import Lock
>
> 1) you generally want to use RLock rather than Lock

Why?

> 2) I have generally felt that using locks at the app level at all is an
> antipattern.  The main way I've stayed sane in multi-threaded Python
> code is to have every mutable strictly owned by exactly one thread, pass
> values around using Queues, and have an event loop in each thread taking
> requests from Queues.
>
> 3) I didn't know that no-gil was a now thing and I'm used to having the
> GIL.  So I would have considered the multiprocessing module rather than
> threading, for something like this.

What does this mean? Are you saying the GIL has been removed?
-- 
https://mail.python.org/mailman/listinfo/python-list

unpack, pack

[Hen Hanna]

when i unpack  the string (or Seed)USENEARSEYNE   , 
 i getwords like   Sense,  Sears, , Snare, 
Unary, 
 (sarsen, seesee, senary, serene, unease, uneasy, usneae, usneas)
 (sensyne)



How would i approach  the PACKING  function ?

e.g. ...   i want to find the string  X  that packs
Apple,  Pie,  Apricot,  Banana,  Candy

  (packing doesn't require an English dictionary)

and i want X to be short.  Thanks!

  (is this in Vol.3 of Knuth?)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: TypeError: can only concatenate str (not "int") to str

[Hen Hanna]

> PS are you aware that there is a Python-Tutor list for the use of people 
> learning Python?


is this guy (dn)  always this rude???is he even more rude  on this 
Python-Tutor list ?

he must have a reputation (for being rude)...  


On Friday, February 24, 2023 at 7:41:44 PM UTC-8, dn wrote:
> On 25/02/2023 09.36, Hen Hanna wrote: 
> > TypeError: can only concatenate str (not "int") to str 
> > 
> > thanks for the comments, --- esp. 2 or 3 (?) ppl who directly addressed it 
> > or commented on it.
> If you haven't already, please review the Python Software Foundation's 
> Code of Conduct https://www.python.org/psf/conduct/
> > py and pypy don't tell me what the (offending) int is 
> > 
> > (even tho' it'd be easy to show me the int, or number) 
> > 
> > because a Tuple, List... may produce a long output and they don't want to 
> > deal with ways of abbreviating it (with ... ) 
> > 
> > and because User-defined Class object may cause another error while 
> > printing it. 
> > 
> > 
> > 
> > _ 
> > TypeError: can only join an iterable 
> > 
> > Here too it'd help me if pypy (Jeannie) could tell me what the offending 
> > data object is.
> Observation: 
> - python doesn't tell you what you want 
> - correspondents are not telling you what you want and/or how you want 
> There's a common theme developing... 
> 
> Suggestion: if you want help, and at the frequency with which you've 
> been posting recently, would it be better not to sound like a 'troll'? 
> 
> 
> Please understand everyone here is volunteering time, in a bid to 
> provide you (and each-other) with benefit. 
> 
> Does the FACT that you have failed to answer several questions from 
> colleagues really entitle you to demand others only directly-address you 
> (and only your questions)? 
> 
> -- 
> Regards, 
> =dn
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Programming by contract.

[Cameron Simpson]

On 26Feb2023 02:44, Weatherby,Gerard  wrote:

The discussion of asserts got me thinking about Programming by Contract. Back 
in the 90s, I had occasion to learn Eiffel programming language. ( 
https://en.wikipedia.org/wiki/Eiffel_(programming_language) The concepts are 
intriguing, although Eiffel itself had barriers to widespread adoption.

Reviewing the topic, I found Wikipedia mentioned some Python implementations. I 
kicked the tires of “deal” and found it straightforward. I’m wondering if 
anyone on the list has experience using any of the Programming by Contract 
modules?


I've been using the icontract package happily.

from icontract import require, ensure

@typechecked
@require(
lambda name: is_valid_basename(name),  # pylint: 
disable=unnecessary-lambda
"name should be a clean file basename"
)
def TagSetClass(self, name: str) -> TaggedPath:
  ''' factory to create a `TaggedPath` from a `name`.
  '''
  fspath = joinpath(dirname(self.fspath), name)
  return TaggedPath(fspath, fstags=self.fstags)

@ensure(lambda result: result == normpath(result))
def keypath(self, fspath):
  ''' Compute the absolute path used to index a `TaggedPath` instance.

  This returns `realpath(fspath)` if `self.config.physical`,
  otherwise `abspath(fspath)`.
  '''
  return realpath(fspath) if self.config.physical else abspath(fspath)

You can stack the decorators just like deal.

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

RE: TypeError: can only concatenate str (not "int") to str

[avi.e.gross]

Greg,

Yes, the forum should be open. The first requests from the person were
replied to politely.

At some point a pattern was emerging of lots of fairly irreverent posts by
someone who is having trouble shifting programming paradigms. The suggestion
was then made as a SUGGESTION by several people that "some" of their
questions might be better asked on the tutor list where others new to python
may have similar questions and can learn.

This forum has all kinds of people and of course many topics are of more
interest to some that others. Programming styles differ too and I note some
here reacted to a suggestion that maybe constants could be more efficiently
be initiated in ways that use less resources. Some insisted it makes more
sense to be able to type what you want more compactly. Yes, of course,
multiple ways are equally valid especially as now, efficiency is not seen as
a major goal.

The reality is that several mailing lists are intended to be used for
occasional questions and people who have more serious needs should be using
local resources or taking courses and reading books as their main learning
method. An occasional question is welcomed. A barrage is an position and a
barrage where most of the answers are ignored or claimed to be wrong, can
generate an "attitude" some of us find less than appealing.

I continue to believe that a programmers job is to learn how to use a
language well, or switch languages, and not to keep moaning why it does not
do what you want or expect. Many answers have suggested how the OP can solve
some issues and apparently that is not of interest to them and they just
keep complaining.

I speak for nobody except myself. As I have said, I have chosen to not
respond and become frustrated.


-Original Message-
From: Python-list  On
Behalf Of Greg Ewing via Python-list
Sent: Saturday, February 25, 2023 7:54 PM
To: python-list@python.org
Subject: Re: TypeError: can only concatenate str (not "int") to str

On 26/02/23 10:53 am, Paul Rubin wrote:
> I'm not on either list but the purpose of the tutor list is to shunt 
> beginner questions away from the main list.

There's a fundamental problem with tutor lists. They rely on experienced
people, the ones capable of answering the questions, to go out of their way
to read the tutor list -- something that is not of personal benefit to them.

Also, pointing people towards tutor lists, if not done carefully, can give
the impression of saying "newcomers are not welcome here".
That's not a message we want to send to Python newcomers at all.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Programming by contract.

[Weatherby,Gerard]

The discussion of asserts got me thinking about Programming by Contract. Back 
in the 90s, I had occasion to learn Eiffel programming language. ( 
https://en.wikipedia.org/wiki/Eiffel_(programming_language) The concepts are 
intriguing, although Eiffel itself had barriers to widespread adoption.

Reviewing the topic, I found Wikipedia mentioned some Python implementations. I 
kicked the tires of “deal” and found it straightforward. I’m wondering if 
anyone on the list has experience using any of the Programming by Contract 
modules?

#!/usr/bin/env python3
import argparse
import deal

@deal.pre(lambda x: x > 7)
@deal.pre(lambda x: isinstance(x, int))
def func(x):
print(x)


parser = 
argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--nodeal', action='store_true', help="Disable deal")
parser.add_argument('bad_choice', choices=('type', 'value'))
args = parser.parse_args()
if args.nodeal:
deal.disable()
func(8)
if args.bad_choice == 'type':
func("8")
if args.bad_choice == 'value':
func(1)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why doesn't Python (error msg) tell me WHAT the actual (arg) values are ?

[Weatherby,Gerard]

“So the case where the assumption fails may not be easily
reproducable and the more information you can get post-mortem the
better”

That’s true for rare corner cases or esoteric race conditions. Usually, when I 
see asserts it's just because I was just plain stupid.

From: Python-list  on 
behalf of Peter J. Holzer 
Date: Saturday, February 25, 2023 at 5:21 PM
To: python-list@python.org 
Subject: Re: Why doesn't Python (error msg) tell me WHAT the actual (arg) 
values are ?
On 2023-02-25 21:58:18 +, Weatherby,Gerard wrote:
> I only use asserts for things I know to be true.

Yeah, that's what assers are for. Or rather for things that you *think*
are true.

> In other words, a failing assert means I have a hole in my program
> logic.

Yes, if you include your assumptions in your definition of "logic".


> For that use, the default behavior –telling me which line the assert
> is on, is more than sufficient. Depending on the circumstance, I’ll
> re-run the code with a breakpoint or replace the assert with an
> informative f-string Exception.

That may not always be practical. Things that we know (or think) are
true often have *are* true in most cases (otherwise we wouldn't think
so). So the case where the assumption fails may not be easily
reproducable and the more information you can get post-mortem the
better. For example, in C on Linux a failed assertion causes a core
dump. So you can inspect the complete state of the program.

hp

--
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: TypeError: can only concatenate str (not "int") to str

[Greg Ewing via Python-list]

On 26/02/23 10:53 am, Paul Rubin wrote:

I'm not on either list but the purpose of the tutor list is to shunt
beginner questions away from the main list.


There's a fundamental problem with tutor lists. They rely on
experienced people, the ones capable of answering the questions,
to go out of their way to read the tutor list -- something that
is not of personal benefit to them.

Also, pointing people towards tutor lists, if not done carefully,
can give the impression of saying "newcomers are not welcome here".
That's not a message we want to send to Python newcomers at all.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Barry]

Re sqlite and threads. The C API can be compiled to be thread safe from my
Reading if the sqlite docs. What I have not checked is how python’s bundled 
sqlite
is compiled. There are claims python’s sqlite is not thread safe.

Barry


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Thomas Passin]

On 2/25/2023 4:41 PM, Skip Montanaro wrote:

Thanks for the responses.

Peter wrote:


Which OS is this?


MacOS Ventura 13.1, M1 MacBook Pro (eight cores).

Thomas wrote:

 > I'm no expert on locks, but you don't usually want to keep a lock while
 > some long-running computation goes on.  You want the computation to be
 > done by a separate thread, put its results somewhere, and then notify
 > the choreographing thread that the result is ready.

In this case I'm extracting the noun phrases from the body of an email 
message(returned as a list). I have a collection of email messages 
organized by month(typically 1000 to 3000 messages per month). I'm using 
concurrent.futures.ThreadPoolExecutor() with the default number of 
workers (os.cpu_count() * 1.5, or 12 threads on my system)to process 
each month, so 12 active threads at a time. Given that the process is 
pretty much CPU bound, maybe reducing the number of workers to the CPU 
count would make sense. Processing of each email message enters that 
with block once.That's about as minimal as I can make it. I thought for 
a bit about pushing the textblob stuff into a separate worker thread, 
but it wasn't obvious how to set up queues to handle the communication 
between the threads created by ThreadPoolExecutor()and the worker 
thread. Maybe I'll think about it harder. (I have a related problem with 
SQLite, since an open database can't be manipulated from multiple 
threads. That makes much of the program's end-of-run processing 
single-threaded.)


If the noun extractor is single-threaded (which I think you mentioned), 
no amount of parallel access is going to help.  The best you can do is 
to queue up requests so that as soon as the noun extractor returns from 
one call, it gets handed another blob.  The CPU will be busy all the 
time running the noun-extraction code.


If that's the case, you might just as well eliminate all the threads and 
just do it sequentially in the most obvious and simple manner.


It would possibly be worth while to try this approach out and see what 
happens to the CPU usage and overall computation time.



 > This link may be helpful -
 >
 > https://anandology.com/blog/using-iterators-and-generators/ 



I don't think that's where my problem is. The lock protects the 
generation of the noun phrases. My loop which does the yielding operates 
outside of that lock's control. The version of the code is my latest, in 
which I tossed out a bunch of phrase-processing code (effectively dead 
end ideas for processing the phrases). Replacing the for loop with a 
simple return seems not to have any effect. In any case, the caller 
which uses the phrases does a fair amount of extra work with the 
phrases, populating a SQLite database, so I don't think the amount of 
time it takes to process a single email message is dominated by the 
phrase generation.


Here's timeitoutput for the noun_phrases code:

% python -m timeit -s 'text = """`python -m timeit --help`""" ; from 
textblob import TextBlob ; from textblob.np_extractors import 
ConllExtractor ; ext = ConllExtractor() ; phrases = TextBlob(text, 
np_extractor=ext).noun_phrases' 'phrases = TextBlob(text, 
np_extractor=ext).noun_phrases'

5000 loops, best of 5: 98.7 usec per loop

I process the output of timeit's help message which looks to be about 
the same length as a typical email message, certainly the same order of 
magnitude. Also, note that I call it once in the setup to eliminate the 
initial training of the ConllExtractor instance. I don't know if ~100us 
qualifies as long running or not.


I'll keep messing with it.

Skip


--
https://mail.python.org/mailman/listinfo/python-list

Re: Why doesn't Python (error msg) tell me WHAT the actual (arg) values are ?

[Peter J. Holzer]

On 2023-02-25 21:58:18 +, Weatherby,Gerard wrote:
> I only use asserts for things I know to be true.

Yeah, that's what assers are for. Or rather for things that you *think*
are true.

> In other words, a failing assert means I have a hole in my program
> logic.

Yes, if you include your assumptions in your definition of "logic".


> For that use, the default behavior –telling me which line the assert
> is on, is more than sufficient. Depending on the circumstance, I’ll
> re-run the code with a breakpoint or replace the assert with an
> informative f-string Exception.

That may not always be practical. Things that we know (or think) are
true often have *are* true in most cases (otherwise we wouldn't think
so). So the case where the assumption fails may not be easily
reproducable and the more information you can get post-mortem the
better. For example, in C on Linux a failed assertion causes a core
dump. So you can inspect the complete state of the program.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Weatherby,Gerard]

“I'm no expert on locks, but you don't usually want to keep a lock while
some long-running computation goes on.  You want the computation to be
done by a separate thread, put its results somewhere, and then notify
the choreographing thread that the result is ready.”

Maybe. There are so many possible threaded application designs I’d hesitate to 
make a general statement.

The threading.Lock.acquire method has flags for both a non-blocking attempt and 
a timeout, so a valid design could include a long-running computation with a 
main thread or event loop polling the thread. Or the thread could signal a main 
loop some other way.

I’ve written some code that coordinated threads by having a process talk to 
itself using a socket.socketpair. The advantage is that you can bundle multiple 
items (sockets, file handles, a polling timeout) into a select.select call 
which waits without consuming resources (at least on Linux) until
something interesting happens.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why doesn't Python (error msg) tell me WHAT the actual (arg) values are ?

[Weatherby,Gerard]

I only use asserts for things I know to be true. Nothing is harder to debug 
than when something you know to be true turns out to be… not True. Because I’ll 
check everything else instead of the cause of the bug.

In other words, a failing assert means I have a hole in my program logic.

For that use, the default behavior –telling me which line the assert is on, is 
more than sufficient. Depending on the circumstance, I’ll re-run the code with 
a breakpoint or replace the assert with an informative f-string Exception.



From: Python-list  on 
behalf of Peter J. Holzer 
Date: Saturday, February 25, 2023 at 9:22 AM
To: python-list@python.org 
Subject: Re: Why doesn't Python (error msg) tell me WHAT the actual (arg) 
values are ?
On 2023-02-25 09:10:06 -0500, Thomas Passin wrote:
> On 2/25/2023 1:13 AM, Peter J. Holzer wrote:
> > On 2023-02-24 18:19:52 -0500, Thomas Passin wrote:
> > > Sometimes you can use a second parameter to assert if you know what kind 
> > > of
> > > error to expect:
[...]
> > > With type errors, assert may actually give you the information needed:
> > >
> > > > > > c = {"a": a, "b": 2}
> > > > > > assert a > c
> > > Traceback (most recent call last):
> > >File "", line 1, in 
> > > TypeError: '>' not supported between instances of 'list' and 'dict'
> >
> > Actually in this case it isn't assert which gives you the information,
> > it's evaluating the expression itself. You get the same error with just
> >  a > c
> > on a line by its own.
>
> In some cases.  For my example with an explanatory string, you wouldn't want
> to write code like that after an ordinary line of code, at least not very
> often.  The assert statement allows it syntactically.

Yes, but if an error in the expression triggers an exception (as in this
case) the explanatory string will never be displayed.

hp

--
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Skip Montanaro]

Thanks for the responses.

Peter wrote:

> Which OS is this?

MacOS Ventura 13.1, M1 MacBook Pro (eight cores).

Thomas wrote:

> I'm no expert on locks, but you don't usually want to keep a lock while
> some long-running computation goes on.  You want the computation to be
> done by a separate thread, put its results somewhere, and then notify
> the choreographing thread that the result is ready.

In this case I'm extracting the noun phrases from the body of an email
message (returned as a list). I have a collection of email messages
organized by month (typically 1000 to 3000 messages per month). I'm using
concurrent.futures.ThreadPoolExecutor() with the default number of workers (
os.cpu_count() * 1.5, or 12 threads on my system) to process each month, so
12 active threads at a time. Given that the process is pretty much CPU
bound, maybe reducing the number of workers to the CPU count would make
sense. Processing of each email message enters that with block once. That's
about as minimal as I can make it. I thought for a bit about pushing the
textblob stuff into a separate worker thread, but it wasn't obvious how to
set up queues to handle the communication between the threads created by
ThreadPoolExecutor() and the worker thread. Maybe I'll think about it
harder. (I have a related problem with SQLite, since an open database can't
be manipulated from multiple threads. That makes much of the program's
end-of-run processing single-threaded.)

> This link may be helpful -
>
> https://anandology.com/blog/using-iterators-and-generators/

I don't think that's where my problem is. The lock protects the generation
of the noun phrases. My loop which does the yielding operates outside of
that lock's control. The version of the code is my latest, in which I
tossed out a bunch of phrase-processing code (effectively dead end ideas
for processing the phrases). Replacing the for loop with a simple return
seems not to have any effect. In any case, the caller which uses the
phrases does a fair amount of extra work with the phrases, populating a
SQLite database, so I don't think the amount of time it takes to process a
single email message is dominated by the phrase generation.

Here's timeit output for the noun_phrases code:

% python -m timeit -s 'text = """`python -m timeit --help`""" ; from
textblob import TextBlob ; from textblob.np_extractors import
ConllExtractor ; ext = ConllExtractor() ; phrases = TextBlob(text,
np_extractor=ext).noun_phrases' 'phrases = TextBlob(text,
np_extractor=ext).noun_phrases'
5000 loops, best of 5: 98.7 usec per loop

I process the output of timeit's help message which looks to be about the
same length as a typical email message, certainly the same order of
magnitude. Also, note that I call it once in the setup to eliminate the
initial training of the ConllExtractor instance. I don't know if ~100us
qualifies as long running or not.

I'll keep messing with it.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Peter J. Holzer]

On 2023-02-25 09:52:15 -0600, Skip Montanaro wrote:
> BLOB_LOCK = Lock()
> 
> def get_terms(text):
> with BLOB_LOCK:
> phrases = TextBlob(text, np_extractor=EXTRACTOR).noun_phrases
> for phrase in phrases:
> yield phrase
> 
> When I monitor the application using py-spy, that with statement is
> consuming huge amounts of CPU.

Another thought:

How accurate is py-spy? Is it possible that it assigns time actually
spent in 
phrases = TextBlob(text, np_extractor=EXTRACTOR).noun_phrases
to
with BLOB_LOCK:
?

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Thomas Passin]

On 2/25/2023 10:52 AM, Skip Montanaro wrote:

I have a multi-threaded program which calls out to a non-thread-safe
library (not mine) in a couple places. I guard against multiple
threads executing code there using threading.Lock. The code is
straightforward:

from threading import Lock

# Something in textblob and/or nltk doesn't play nice with no-gil, so just
# serialize all blobby accesses.
BLOB_LOCK = Lock()

def get_terms(text):
 with BLOB_LOCK:
 phrases = TextBlob(text, np_extractor=EXTRACTOR).noun_phrases
 for phrase in phrases:
 yield phrase

When I monitor the application using py-spy, that with statement is
consuming huge amounts of CPU. Does threading.Lock.acquire() sleep
anywhere? I didn't see anything obvious poking around in the C code
which implements this stuff. I'm no expert though, so could easily
have missed something.


I'm no expert on locks, but you don't usually want to keep a lock while 
some long-running computation goes on.  You want the computation to be 
done by a separate thread, put its results somewhere, and then notify 
the choreographing thread that the result is ready.


This link may be helpful -

https://anandology.com/blog/using-iterators-and-generators/

--
https://mail.python.org/mailman/listinfo/python-list

Re: Is there a more efficient threading lock?

[Peter J. Holzer]

On 2023-02-25 09:52:15 -0600, Skip Montanaro wrote:
> I have a multi-threaded program which calls out to a non-thread-safe
> library (not mine) in a couple places. I guard against multiple
> threads executing code there using threading.Lock. The code is
> straightforward:
> 
> from threading import Lock
> 
> # Something in textblob and/or nltk doesn't play nice with no-gil, so just
> # serialize all blobby accesses.
> BLOB_LOCK = Lock()
> 
> def get_terms(text):
> with BLOB_LOCK:
> phrases = TextBlob(text, np_extractor=EXTRACTOR).noun_phrases
> for phrase in phrases:
> yield phrase
> 
> When I monitor the application using py-spy, that with statement is
> consuming huge amounts of CPU.

Which OS is this?

> Does threading.Lock.acquire() sleep anywhere?

On Linux it calls futex(2), which does sleep if it can't get the lock
right away. (Of course if it does get the lock, it will return
immediately which may use a lot of CPU if you are calling it a lot.)

hp


-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Is there a more efficient threading lock?

[Skip Montanaro]

I have a multi-threaded program which calls out to a non-thread-safe
library (not mine) in a couple places. I guard against multiple
threads executing code there using threading.Lock. The code is
straightforward:

from threading import Lock

# Something in textblob and/or nltk doesn't play nice with no-gil, so just
# serialize all blobby accesses.
BLOB_LOCK = Lock()

def get_terms(text):
with BLOB_LOCK:
phrases = TextBlob(text, np_extractor=EXTRACTOR).noun_phrases
for phrase in phrases:
yield phrase

When I monitor the application using py-spy, that with statement is
consuming huge amounts of CPU. Does threading.Lock.acquire() sleep
anywhere? I didn't see anything obvious poking around in the C code
which implements this stuff. I'm no expert though, so could easily
have missed something.

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why doesn't Python (error msg) tell me WHAT the actual (arg) values are ?

[Peter J. Holzer]

On 2023-02-25 09:10:06 -0500, Thomas Passin wrote:
> On 2/25/2023 1:13 AM, Peter J. Holzer wrote:
> > On 2023-02-24 18:19:52 -0500, Thomas Passin wrote:
> > > Sometimes you can use a second parameter to assert if you know what kind 
> > > of
> > > error to expect:
[...]
> > > With type errors, assert may actually give you the information needed:
> > > 
> > > > > > c = {"a": a, "b": 2}
> > > > > > assert a > c
> > > Traceback (most recent call last):
> > >File "", line 1, in 
> > > TypeError: '>' not supported between instances of 'list' and 'dict'
> > 
> > Actually in this case it isn't assert which gives you the information,
> > it's evaluating the expression itself. You get the same error with just
> >  a > c
> > on a line by its own.
> 
> In some cases.  For my example with an explanatory string, you wouldn't want
> to write code like that after an ordinary line of code, at least not very
> often.  The assert statement allows it syntactically.

Yes, but if an error in the expression triggers an exception (as in this
case) the explanatory string will never be displayed.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why doesn't Python (error msg) tell me WHAT the actual (arg) values are ?

[Thomas Passin]

On 2/25/2023 1:13 AM, Peter J. Holzer wrote:

On 2023-02-24 18:19:52 -0500, Thomas Passin wrote:

On 2/24/2023 2:47 PM, dn via Python-list wrote:

On 25/02/2023 08.12, Peter J. Holzer wrote:

On 2023-02-24 16:12:10 +1300, dn via Python-list wrote:

In some ways, providing this information seems appropriate.
Curiously, this does not even occur during an assert exception -
despite the value/relationship being the whole point of using
the command!

  x = 1
  assert x == 2

AssertionError (and that's it)


Sometimes you can use a second parameter to assert if you know what kind of
error to expect:


a = [1,2,3]
b = [4,5]
assert len(a) == len(b), f'len(a): {len(a)} != len(b): {len(b)}'

Traceback (most recent call last):
   File "", line 1, in 
AssertionError: len(a): 3 != len(b): 2


Yup. That's very useful (but I tend to forget that).



With type errors, assert may actually give you the information needed:


c = {"a": a, "b": 2}
assert a > c

Traceback (most recent call last):
   File "", line 1, in 
TypeError: '>' not supported between instances of 'list' and 'dict'


Actually in this case it isn't assert which gives you the information,
it's evaluating the expression itself. You get the same error with just
 a > c
on a line by its own.


In some cases.  For my example with an explanatory string, you wouldn't 
want to write code like that after an ordinary line of code, at least 
not very often.  The assert statement allows it syntactically.


--
https://mail.python.org/mailman/listinfo/python-list