Re: UTF_16 question

[jak via Python-list]

Richard Damon ha scritto:

On Apr 29, 2024, at 12:23 PM, jak via Python-list  
wrote:

Hi everyone,
one thing that I do not understand is happening to me: I have some text
files with different characteristics, among these there are that they
have an UTF_32_le coding, utf_32be, utf_16_le, utf_16_be all of them
without BOM. With those utf_32_xx I have no problem but with the
UTF_16_xx I have. If I have an utf_16_le coded file and I read it with
encoding='utf_16_le' I have no problem I read it, with
encoding='utf_16_be' I can read it without any error even if the data I
receive have the inverted bytes. The same thing happens with the
utf_16_be codified file, I read it, both with encoding='utf_16_be' and
with 'utf_16_le' without errors but in the last case the bytes are
inverted. What did I not understand? What am I doing wrong?

thanks in advance

--
https://mail.python.org/mailman/listinfo/python-list


That is why the BOM was created. A lot of files can be “correctly” read as 
either UTF-16-LE or UTF-1-BE encoded, as most of the 16 bit codes are valid, so 
unless the wrong encoding happens to hit something that is invalid (most likely 
something looking like a Surrogage Pair without a match), there isn’t an error 
in reading the file. The BOM character was specifically designed to be an 
invalid code if read by the wrong encoding (if you ignore the possibility of 
the file having a NUL right after the BOM)

If you know the files likely contains a lot of “ASCII” characters, then you 
might be able to detect that you got it wrong, due to seeing a lot of 0xXX00 
characters and few 0x00XX characters, but that doesn’t create an “error” 
normally.



Thanks to you too for the reply. I was actually looking for a way to
distinguish "utf16le" texts from "utf16be" ones. Unfortunately, whoever
created this log file archive thought that the BOM was not important and
so omitted it. Now they want to switch to "utf8 " and also save the
previous. Fortunately I can be sure that the text of the log files
is in some European language, so after converting the file to "utf8" I
make sure that most of the bytes are less than the value 0x7F and if not
I reconvert them by replacing "utf16 " "le" with "be" or vice versa. The
strategy seems to be working. In the future, by writing files in "utf8"
they will no longer have problems like this.

--
https://mail.python.org/mailman/listinfo/python-list

Re: UTF_16 question

[Richard Damon via Python-list]

> On Apr 29, 2024, at 12:23 PM, jak via Python-list  
> wrote:
> 
> Hi everyone,
> one thing that I do not understand is happening to me: I have some text
> files with different characteristics, among these there are that they
> have an UTF_32_le coding, utf_32be, utf_16_le, utf_16_be all of them
> without BOM. With those utf_32_xx I have no problem but with the
> UTF_16_xx I have. If I have an utf_16_le coded file and I read it with
> encoding='utf_16_le' I have no problem I read it, with
> encoding='utf_16_be' I can read it without any error even if the data I
> receive have the inverted bytes. The same thing happens with the
> utf_16_be codified file, I read it, both with encoding='utf_16_be' and
> with 'utf_16_le' without errors but in the last case the bytes are
> inverted. What did I not understand? What am I doing wrong?
> 
> thanks in advance
> 
> --
> https://mail.python.org/mailman/listinfo/python-list

That is why the BOM was created. A lot of files can be “correctly” read as 
either UTF-16-LE or UTF-1-BE encoded, as most of the 16 bit codes are valid, so 
unless the wrong encoding happens to hit something that is invalid (most likely 
something looking like a Surrogage Pair without a match), there isn’t an error 
in reading the file. The BOM character was specifically designed to be an 
invalid code if read by the wrong encoding (if you ignore the possibility of 
the file having a NUL right after the BOM)

If you know the files likely contains a lot of “ASCII” characters, then you 
might be able to detect that you got it wrong, due to seeing a lot of 0xXX00 
characters and few 0x00XX characters, but that doesn’t create an “error” 
normally.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: UTF_16 question

[jak via Python-list]

Stefan Ram ha scritto:

jak  wrote or quoted:

 I read it, both with encoding='utf_16_be' and
with 'utf_16_le' without errors but in the last case the bytes are
inverted.


   I think the order of the octets (bytes) is exactly the difference
   between these two encodings, so your observation isn't really
   surprising. The computer can't report an error here since it
   can't infer the correct encoding from the file data. It's like
   that koan: "A bit has the value 1. What does that mean?".



Understood. They are just 2 bytes and there is no difference between
them.

Thank you.


--
https://mail.python.org/mailman/listinfo/python-list

UTF_16 question

[jak via Python-list]

Hi everyone,
one thing that I do not understand is happening to me: I have some text
files with different characteristics, among these there are that they
have an UTF_32_le coding, utf_32be, utf_16_le, utf_16_be all of them
without BOM. With those utf_32_xx I have no problem but with the
UTF_16_xx I have. If I have an utf_16_le coded file and I read it with
encoding='utf_16_le' I have no problem I read it, with
encoding='utf_16_be' I can read it without any error even if the data I
receive have the inverted bytes. The same thing happens with the
utf_16_be codified file, I read it, both with encoding='utf_16_be' and
with 'utf_16_le' without errors but in the last case the bytes are
inverted. What did I not understand? What am I doing wrong?

thanks in advance

--
https://mail.python.org/mailman/listinfo/python-list

Re: A question about import

[Cameron Simpson via Python-list]

On 16Feb2024 20:32, MRAB  wrote:

On 2024-02-16 20:07, Gabor Urban via Python-list wrote:

I need something about modules to be clarified.

Suppose I have written a module eg: ModuleA which imports an other
module, let us say the datetime.

If I import ModuleA in a script, will be datetime imported automatically?


Yes. When a module is imported it can import other modules.


But note that `datetime` does not magicly get put in the script's 
namespace.


Module A:

import datetime

Script:

import A

In the code in module A the name datetime is known and can be used.

In the code in the script the name A is known and can be used. Importing 
A does not magicly set the name datetime in the script's namespace - 
imagine the the pollution!


You _can_ access it as A.datetime because it is in the A module's 
namespace. But really if you just wanted datetime for direct use in the 
script you would import it there too:


import datetime
import A

Note that the datetime module is only actually loaded once. The import 
binds the name into your local namespace like any other variable.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: A question about import

[MRAB via Python-list]

On 2024-02-16 20:07, Gabor Urban via Python-list wrote:

Hi guys,

I need something about modules to be clarified.

Suppose I have written a module eg: ModuleA which imports an other
module, let us say the datetime.

If I import ModuleA in a script, will be datetime imported automatically?


Yes. When a module is imported it can import other modules.
--
https://mail.python.org/mailman/listinfo/python-list

A question about import

[Gabor Urban via Python-list]

Hi guys,

I need something about modules to be clarified.

Suppose I have written a module eg: ModuleA which imports an other
module, let us say the datetime.

If I import ModuleA in a script, will be datetime imported automatically?

Thanks in advance,

--
Urbán Gábor

Linux is like a wigwam: no Gates, no Windows and an Apache inside.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Left Right via Python-list]

So, here's some info about how to see what's going on with Python's
memory allocation: https://docs.python.org/3/library/tracemalloc.html
. I haven't looked into this in a long time, but it used to be the
case that you needed to compile native modules (and probably Python
itself?) so that instrumentation is possible (I think incref / decref
macros should give you a hint, because they would have to naturally
report some of that info).

Anyways.  The problem of tracing memory allocation / deallocation in
Python can be roughly split into these categories:

1. Memory legitimately claimed by objects created through Python
runtime, but not reclaimed due to programmer error. I.e. the
programmer wrote a program that keeps references to objects which it
will never use again.
2. Memory claimed through native objects obtained by means of
interacting with Python's allocator.  When working with Python C API
it's best to interface with Python allocator to deal with dynamic
memory allocation and release.  However, it's somewhat cumbersome, and
some module authors simply might not know about it, or wouldn't want
to use it because they prefer a different allocator.  Sometimes
library authors don't implement memory deallocation well.  Which
brings us to:
3. Memory claimed by any user-space code that is associated with the
Python process. This can be for example shared libraries loaded by
means of Python bindings, that is on top of the situation described
above.
4. System memory associated with the process.  Some system calls need
to allocate memory on the system side.  Typical examples are opening
files, creating sockets etc.  Typically, the system will limit the
number of such objects, and the user program will hit the numerical
limit before it hits the memory limit, but it can also happen that
this will manifest as a memory problem (one example I ran into was
trying to run conda-build and it would fail due to enormous amounts of
memory it requested, but the specifics of the failure were due to it
trying to create new sub-processes -- another system resource that
requires memory allocation).

There isn't a universal strategy to cover all these cases.  But, if
you have reasons to suspect (4), for example, you'd probably start by
using strace utility (on Linux) to see what system calls are executed.

For something like the (3), you could try to utilize Valgrind (but
it's a lot of work to set it up).  It's also possible to use jemalloc
to profile a program, but you would have to build Python with its
allocator modified to use jemalloc (I've seen an issue in the Python
bug tracker where someone wrote a script to do that, so it should be
possible).  Both of these are quite labor intensive and not trivial to
set up.

(2) could be often diagnosed with tracemalloc Python module and (1) is
something that can be helped with Python's gc module.

It's always better though to have an actual error and work from there.
Or, at least, have some monitoring data that suggests that your
application memory use increases over time.  Otherwise you could be
spending a lot of time chasing problems you don't have.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Frank Millman via Python-list]

On 2024-01-17 3:01 AM, Greg Ewing via Python-list wrote:

On 17/01/24 1:01 am, Frank Millman wrote:
I sometimes need to keep a reference from a transient object to a more 
permanent structure in my app. To save myself the extra step of 
removing all these references when the transient object is deleted, I 
make them weak references.


I don't see how weak references help here at all. If the transient
object goes away, all references from it to the permanent objects also
go away.

A weak reference would only be of use if the reference went the other
way, i.e. from the permanent object to the transient object.



You are right. I got my description above back-to-front. It is a pub/sub 
scenario. A transient object makes a request to the permanent object to 
be notified of any changes. The permanent object stores a reference to 
the transient object and executes a callback on each change. When the 
transient object goes away, the reference must be removed.


Frank

--
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Greg Ewing via Python-list]

On 17/01/24 1:01 am, Frank Millman wrote:
I sometimes need to keep a reference from a 
transient object to a more permanent structure in my app. To save myself 
the extra step of removing all these references when the transient 
object is deleted, I make them weak references.


I don't see how weak references help here at all. If the transient
object goes away, all references from it to the permanent objects also
go away.

A weak reference would only be of use if the reference went the other
way, i.e. from the permanent object to the transient object.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Greg Ewing via Python-list]

On 17/01/24 4:00 am, Chris Angelico wrote:

class Form:
 def __init__(self):
 self.elements = []

class Element:
 def __init__(self, form):
 self.form = form
 form.elements.append(self)


If you make the reference from Element to Form a weak reference,
it won't keep the Form alive after it's been closed.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Barry via Python-list]

> On 16 Jan 2024, at 12:10, Frank Millman via Python-list 
>  wrote:
> 
> My problem is that my app is quite complex, and it is easy to leave a 
> reference dangling somewhere which prevents an object from being gc'd.

What I do to track these problems down is use gc.get_objects() then summerize 
the number of each type. Part 2 is to print the delta after an interval of a 
2nd summary.
Leaks of objects show up as the count of a type increasing every time you 
sample.


Barry


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Barry via Python-list]

> On 16 Jan 2024, at 13:17, Thomas Passin via Python-list 
>  wrote:
> 
> The usual advice is to call deleteLater() on objects derived from PyQt 
> classes.  I don't know enough about PyQt to know if this takes care of all 
> dangling reference problems, though.

It works well and robustly.

Barry


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Chris Angelico via Python-list]

On Wed, 17 Jan 2024 at 01:45, Frank Millman via Python-list
 wrote:
>
> On 2024-01-16 2:15 PM, Chris Angelico via Python-list wrote:
> >
> > Where do you tend to "leave a reference dangling somewhere"? How is
> > this occurring? Is it a result of an incomplete transaction (like an
> > HTTP request that never finishes), or a regular part of the operation
> > of the server?
> >
>
> I have a class that represents a database table, and another class that
> represents a database column. There is a one-to-many relationship and
> they maintain references to each other.
>
> In another part of the app, there is a class that represents a form, and
> another class that represents the gui elements on the form. Again there
> is a one-to-many relationship.

I don't know when you'd be "done" with the table, so I won't try to
give an example, but I'll try this one and maybe it'll give some ideas
that could apply to both.

When you open the form, you initialize it, display it, etc, etc. This
presumably includes something broadly like this:

class Form:
def __init__(self):
self.elements = []

class Element:
def __init__(self, form):
self.form = form
form.elements.append(self)

frm = Form(...)
Element(frm, ...) # as many as needed
frm.show() # present it to the user

This is a pretty classic refloop. I don't know exactly what your setup
is, but most likely it's going to look something like this. Feel free
to correct me if it doesn't.

The solution here would be to trap the "form is no longer being
displayed" moment. That'll be some sort of GUI event like a "close" or
"delete" signal. When that comes through (and maybe after doing other
processing), you no longer need the form, and can dispose of it. The
simplest solution here is: Empty out frm.elements. That immediately
leaves the form itself as a leaf (no references to anything relevant),
and the elements still refer back to it, but once nothing ELSE refers
to the form, everything can be disposed of.

> A gui element that represents a piece of data has to maintain a link to
> its database column object. There can be a many-to-one relationship, as
> there could be more than one gui element referring to the same column.

Okay, so the Element also refers to the corresponding Column. If the
Form and Element aren't in a refloop, this shouldn't be a problem.
However, if this is the same Table and Column that you referred to
above, that might be the answer to my question. Are you "done" with
the Table at the same time that the form is no longer visible? If so,
you would probably have something similar where the Form refers to the
Table, and the Table and Columns refer to each other... so the same
solution hopefully should work: wipe out the Table's list of columns.

> There are added complications which I won't go into here. The bottom
> line is that on some occasions a form which has been closed does not get
> gc'd.
>
> I have been trying to reproduce the problem in my toy app, but I cannot
> get it to fail. There is a clue there! I think I have just
> over-complicated things.

Definitely possible.

> I will start with a fresh approach tomorrow. If you don't hear from me
> again, you will know that I have solved it!
>
> Thanks for the input, it definitely helped.

Cool cool, happy to help.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Frank Millman via Python-list]

On 2024-01-16 2:15 PM, Chris Angelico via Python-list wrote:


Where do you tend to "leave a reference dangling somewhere"? How is
this occurring? Is it a result of an incomplete transaction (like an
HTTP request that never finishes), or a regular part of the operation
of the server?



I have a class that represents a database table, and another class that 
represents a database column. There is a one-to-many relationship and 
they maintain references to each other.


In another part of the app, there is a class that represents a form, and 
another class that represents the gui elements on the form. Again there 
is a one-to-many relationship.


A gui element that represents a piece of data has to maintain a link to 
its database column object. There can be a many-to-one relationship, as 
there could be more than one gui element referring to the same column.


There are added complications which I won't go into here. The bottom 
line is that on some occasions a form which has been closed does not get 
gc'd.


I have been trying to reproduce the problem in my toy app, but I cannot 
get it to fail. There is a clue there! I think I have just 
over-complicated things.


I will start with a fresh approach tomorrow. If you don't hear from me 
again, you will know that I have solved it!


Thanks for the input, it definitely helped.

Frank


--
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Thomas Passin via Python-list]

On 1/16/2024 4:17 AM, Barry wrote:




On 16 Jan 2024, at 03:49, Thomas Passin via Python-list 
 wrote:

This kind of thing can happen with PyQt, also.  There are ways to minimize it 
but I don't know if you can ever be sure all Qt C++ objects will get deleted. 
It depends on the type of object and the circumstances.


When this has been seen in the past it has been promptly fixed by the 
maintainer.


The usual advice is to call deleteLater() on objects derived from PyQt 
classes.  I don't know enough about PyQt to know if this takes care of 
all dangling reference problems, though.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Chris Angelico via Python-list]

On Tue, 16 Jan 2024 at 23:08, Frank Millman via Python-list
 wrote:
>
> On 2024-01-15 3:51 PM, Frank Millman via Python-list wrote:
> > Hi all
> >
> > I have read that one should not have to worry about garbage collection
> > in modern versions of Python - it 'just works'.
> >
> > I don't want to rely on that. My app is a long-running server, with
> > multiple clients logging on, doing stuff, and logging off. They can
> > create many objects, some of them long-lasting. I want to be sure that
> > all objects created are gc'd when the session ends.
> >
>
> I did not explain myself very well. Sorry about that.
>
> My problem is that my app is quite complex, and it is easy to leave a
> reference dangling somewhere which prevents an object from being gc'd.
>
> This can create (at least) two problems. The obvious one is a memory
> leak. The second is that I sometimes need to keep a reference from a
> transient object to a more permanent structure in my app. To save myself
> the extra step of removing all these references when the transient
> object is deleted, I make them weak references. This works, unless the
> transient object is kept alive by mistake and the weak ref is never removed.
>
> I feel it is important to find these dangling references and fix them,
> rather than wait for problems to appear in production. The only method I
> can come up with is to use the 'delwatcher' class that I used in my toy
> program in my original post.
>
> I am surprised that this issue does not crop up more often. Does nobody
> else have these problems?
>

It really depends on how big those dangling objects are. My personal
habit is to not worry about a few loose objects, by virtue of ensuring
that everything either has its reference loops deliberately broken at
some point in time, or by keeping things small.

An example of deliberately breaking a refloop would be when I track
websockets. Usually I'll tag the socket object itself with some kind
of back-reference to my own state, but I also need to be able to
iterate over all of my own state objects (let's say they're
dictionaries for simplicity) and send a message to each socket. So
there'll be a reference loop between the socket and the state. But at
some point, I will be notified that the socket has been disconnected,
and that's when I go to its state object and wipe out its
back-reference. It can then be disposed of promptly, since there's no
loop.

It takes a bit of care, but in general, large state objects won't have
these kinds of loops, and dangling references haven't caused me any
sort of major issues in production.

Where do you tend to "leave a reference dangling somewhere"? How is
this occurring? Is it a result of an incomplete transaction (like an
HTTP request that never finishes), or a regular part of the operation
of the server?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Frank Millman via Python-list]

On 2024-01-15 3:51 PM, Frank Millman via Python-list wrote:

Hi all

I have read that one should not have to worry about garbage collection 
in modern versions of Python - it 'just works'.


I don't want to rely on that. My app is a long-running server, with 
multiple clients logging on, doing stuff, and logging off. They can 
create many objects, some of them long-lasting. I want to be sure that 
all objects created are gc'd when the session ends.




I did not explain myself very well. Sorry about that.

My problem is that my app is quite complex, and it is easy to leave a 
reference dangling somewhere which prevents an object from being gc'd.


This can create (at least) two problems. The obvious one is a memory 
leak. The second is that I sometimes need to keep a reference from a 
transient object to a more permanent structure in my app. To save myself 
the extra step of removing all these references when the transient 
object is deleted, I make them weak references. This works, unless the 
transient object is kept alive by mistake and the weak ref is never removed.


I feel it is important to find these dangling references and fix them, 
rather than wait for problems to appear in production. The only method I 
can come up with is to use the 'delwatcher' class that I used in my toy 
program in my original post.


I am surprised that this issue does not crop up more often. Does nobody 
else have these problems?


Frank



--
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Barry via Python-list]

> On 16 Jan 2024, at 03:49, Thomas Passin via Python-list 
>  wrote:
> 
> This kind of thing can happen with PyQt, also.  There are ways to minimize it 
> but I don't know if you can ever be sure all Qt C++ objects will get deleted. 
> It depends on the type of object and the circumstances.

When this has been seen in the past it has been promptly fixed by the 
maintainer.

Barry




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Thomas Passin via Python-list]

On 1/15/2024 9:47 PM, Akkana Peck via Python-list wrote:

I wrote:

Also be warned that some modules (particularly if they're based on libraries 
not written in Python) might not garbage collect, so you may need to use other 
methods of cleaning up after those objects.


Chris Angelico writes:

Got any examples of that?


The big one for me was gdk-pixbuf, part of GTK. When you do something like 
gtk.gdk.pixbuf_new_from_file(), there's a Python object that gets created, but 
there's also the underlying C code that allocates memory for the pixbuf. When 
the object went out of scope, the Python object was automatically garbage 
collected, but the pixbuf data leaked.


This kind of thing can happen with PyQt, also.  There are ways to 
minimize it but I don't know if you can ever be sure all Qt C++ objects 
will get deleted. It depends on the type of object and the circumstances.



Calling gc.collect() caused the pixbuf data to be garbage collected too.

There used to be a post explaining this on the pygtk mailing list: the link was
http://www.daa.com.au/pipermail/pygtk/2003-December/006499.html
but that page is gone now and I can't seem to find any other archives of that 
list (it's not on archive.org either). And this was from GTK2; I never checked 
whether the extra gc.collect() is still necessary in GTK3, but I figure leaving 
it in doesn't hurt anything. I use pixbufs in a tiled map application, so there 
are a lot of small pixbufs being repeatedly read and then deallocated.

 ...Akkana


--
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Chris Angelico via Python-list]

On Tue, 16 Jan 2024 at 13:49, Akkana Peck via Python-list
 wrote:
>
> I wrote:
> > > Also be warned that some modules (particularly if they're based on 
> > > libraries not written in Python) might not garbage collect, so you may 
> > > need to use other methods of cleaning up after those objects.
>
> Chris Angelico writes:
> > Got any examples of that?
>
> The big one for me was gdk-pixbuf, part of GTK. When you do something like 
> gtk.gdk.pixbuf_new_from_file(), there's a Python object that gets created, 
> but there's also the underlying C code that allocates memory for the pixbuf. 
> When the object went out of scope, the Python object was automatically 
> garbage collected, but the pixbuf data leaked. Calling gc.collect() caused 
> the pixbuf data to be garbage collected too.
>
> There used to be a post explaining this on the pygtk mailing list: the link 
> was
> http://www.daa.com.au/pipermail/pygtk/2003-December/006499.html
> but that page is gone now and I can't seem to find any other archives of that 
> list (it's not on archive.org either). And this was from GTK2; I never 
> checked whether the extra gc.collect() is still necessary in GTK3, but I 
> figure leaving it in doesn't hurt anything. I use pixbufs in a tiled map 
> application, so there are a lot of small pixbufs being repeatedly read and 
> then deallocated.
>

Okay, so to clarify: the Python object will always be garbage
collected correctly, but a buggy third-party module might have
*external* resources (in that case, the pixbuf) that aren't properly
released. Either that, or there is a reference loop, which doesn't
necessarily mean you NEED to call gc.collect(), but it can help if you
want to get rid of them more promptly. (Python will detect such loops
at some point, but not always immediately.) But these are bugs in the
module, particularly the first case, and should be considered as such.
2003 is fully two decades ago now, and I would not expect that a
serious bug like that has been copied into PyGObject (the newer way of
using GTK from Python).

So, Python's garbage collection CAN be assumed to "just work", unless
you find evidence to the contrary.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Akkana Peck via Python-list]

I wrote:
> > Also be warned that some modules (particularly if they're based on 
> > libraries not written in Python) might not garbage collect, so you may need 
> > to use other methods of cleaning up after those objects.

Chris Angelico writes:
> Got any examples of that?

The big one for me was gdk-pixbuf, part of GTK. When you do something like 
gtk.gdk.pixbuf_new_from_file(), there's a Python object that gets created, but 
there's also the underlying C code that allocates memory for the pixbuf. When 
the object went out of scope, the Python object was automatically garbage 
collected, but the pixbuf data leaked. Calling gc.collect() caused the pixbuf 
data to be garbage collected too.

There used to be a post explaining this on the pygtk mailing list: the link was
http://www.daa.com.au/pipermail/pygtk/2003-December/006499.html
but that page is gone now and I can't seem to find any other archives of that 
list (it's not on archive.org either). And this was from GTK2; I never checked 
whether the extra gc.collect() is still necessary in GTK3, but I figure leaving 
it in doesn't hurt anything. I use pixbufs in a tiled map application, so there 
are a lot of small pixbufs being repeatedly read and then deallocated.

...Akkana
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Chris Angelico via Python-list]

On Tue, 16 Jan 2024 at 06:32, Akkana Peck via Python-list
 wrote:
>
> > Frank Millman wrote at 2024-1-15 15:51 +0200:
> > >I have read that one should not have to worry about garbage collection
> > >in modern versions of Python - it 'just works'.
>
> Dieter Maurer via Python-list writes:
> > There are still some isolated cases when not all objects
> > in an unreachable cycle are destroyed
> > (see e.g. step 2 of
> > "https://devguide.python.org/internals/garbage-collector/index.html#destroying-unreachable-objects;).
>
> Also be warned that some modules (particularly if they're based on libraries 
> not written in Python) might not garbage collect, so you may need to use 
> other methods of cleaning up after those objects.
>

Got any examples of that?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Akkana Peck via Python-list]

> Frank Millman wrote at 2024-1-15 15:51 +0200:
> >I have read that one should not have to worry about garbage collection 
> >in modern versions of Python - it 'just works'.

Dieter Maurer via Python-list writes:
> There are still some isolated cases when not all objects
> in an unreachable cycle are destroyed
> (see e.g. step 2 of
> "https://devguide.python.org/internals/garbage-collector/index.html#destroying-unreachable-objects;).

Also be warned that some modules (particularly if they're based on libraries 
not written in Python) might not garbage collect, so you may need to use other 
methods of cleaning up after those objects.

...Akkana
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Dieter Maurer via Python-list]

Frank Millman wrote at 2024-1-15 15:51 +0200:
>I have read that one should not have to worry about garbage collection 
>in modern versions of Python - it 'just works'.

There are still some isolated cases when not all objects
in an unreachable cycle are destroyed
(see e.g. step 2 of
"https://devguide.python.org/internals/garbage-collector/index.html#destroying-unreachable-objects;).
But Python's own objects (e.g. traceback cycles)
or instances of classes implemented in Python
should no longer be affected.

Thus, unless you use extensions implemented in C (with "legacy finalizer"s),
garbage collection should not make problems.


On the other hand, your application, too, must avoid memory leaks.
Caches of various forms (with data for several sessions) might introduce them.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about garbage collection

[Skip Montanaro via Python-list]

> I do have several circular references. My experience is that if I do not
> take some action to break the references when closing the session, the
> objects remain alive. Below is a very simple program to illustrate this.
>
> Am I missing something? All comments appreciated.

Python has normal reference counting, but also has a cyclic garbage
collector. Here's plenty of detail about how it works:

https://devguide.python.org/internals/garbage-collector/index.html

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Question about garbage collection

[Frank Millman via Python-list]

Hi all

I have read that one should not have to worry about garbage collection 
in modern versions of Python - it 'just works'.


I don't want to rely on that. My app is a long-running server, with 
multiple clients logging on, doing stuff, and logging off. They can 
create many objects, some of them long-lasting. I want to be sure that 
all objects created are gc'd when the session ends.


I do have several circular references. My experience is that if I do not 
take some action to break the references when closing the session, the 
objects remain alive. Below is a very simple program to illustrate this.


Am I missing something? All comments appreciated.

Frank Millman

==

import gc

class delwatcher:
    # This stores enough information to identify the object being watched.
    # It does not store a reference to the object itself.
    def __init__(self, obj):
    self.id = (obj.type, obj.name, id(obj))
    print('***', *self.id, 'created ***')
    def __del__(self):
    print('***', *self.id, 'deleted ***')

class Parent:
    def __init__(self, name):
    self.type = 'parent'
    self.name = name
    self.children = []
    self._del = delwatcher(self)

class Child:
    def __init__(self, parent, name):
    self.type = 'child'
    self.parent = parent
    self.name = name
    parent.children.append(self)
    self._del = delwatcher(self)

p1 = Parent('P1')
p2 = Parent('P2')

c1_1 = Child(p1, 'C1_1')
c1_2 = Child(p1, 'C1_2')
c2_1 = Child(p2, 'C2_1')
c2_2 = Child(p2, 'C2_2')

input('waiting ...')

# if next 2 lines are included, parent and child can be gc'd
# for ch in p1.children:
# ch.parent = None

# if next line is included, child can be gc'd, but not parent
# p1.children = None

del c1_1
del p1
gc.collect()

input('wait some more ...')

--
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Karsten Hilbert via Python-list]

Am Sat, Jan 13, 2024 at 09:20:00PM +0100 schrieb Karsten Hilbert via 
Python-list:

> > I was wondering if
> > your type hint for queries shouldn't be the following.
> >
> > queries:list[dict[str,str]|dict[str,list]|dict[str,dict[str, dict[str, 
> > Ant]]]

Wait, not really. Let me give an example. Here's three times
the same query (as far as PostgreSQL is concerned, after
having been passed through psycopg2):

queries = [
{
'SQL': 'SELECT 1'
},
{
'SQL': 'SELECT %s',
'args': [1]
},
{
'SQL': 'SELECT %(value)s',
'args': {'value': 1}
}
]

The value for key "SQL" will always be str-like.

The value for "args" can be a list or a dict itself.

If "args" is a dict it will be of type [str, Any].

That's what I am trying to tell mypy.

Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Karsten Hilbert via Python-list]

Am Fri, Jan 12, 2024 at 02:23:43PM +0100 schrieb Antoon Pardon via Python-list:

> > queries:list[dict[str, str | list | dict[str, Any]]]=None,
> >
> >into
> >
> > "List[Dict[str, Union[str, List[Any], Dict[str, Any"
> >
> >seems accurate. I just don't understand why list[dict[str,
> >str]] should not pass that construct.
>
> Sorry for the late reaction

ne'er mind ya

> I was wondering if
> your type hint for queries shouldn't be the following.
>
> queries:list[dict[str,str]|dict[str,list]|dict[str,dict[str, dict[str, Ant]]]
>
> My impression at this moment is that you are write something like: dict[str, 
> str | int] as
> as shorthand for dict[str, str] | dict[str, int].

I do.

> But those two are different types.

A-ha ! In what way ?

Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Antoon Pardon via Python-list]

Op 29/12/2023 om 16:02 schreef Karsten Hilbert via Python-list:


Am Fri, Dec 29, 2023 at 07:49:17AM -0700 schrieb Mats Wichmann via Python-list:


I am not sure why mypy thinks this

gmPG2.py:554: error: Argument "queries" to "run_rw_queries" has incompatible type 
"List[Dict[str, str]]"; expected
"List[Dict[str, Union[str, List[Any], Dict[str, Any"  [arg-type]
 rows, idx = run_rw_queries(link_obj = conn, queries = 
queries, return_data = True)
   
^~~

should be flagged. The intent is for "queries" to be

a list
of dicts
with keys of str
and values of
str OR
list of anything OR
dict with
keys of str
and values of anything

I'd have thunk list[dict[str,str]] matches that ?

Dict[str, str] means the key type and value type should both be strings,

Indeed, I know that much, list[dict[str, str]] is what is getting
passed in in this particular invocation of run_rw_queries().

For what it's worth here's the signature of that function:

def run_rw_queries (
link_obj:_TLnkObj=None,
queries:list[dict[str, str | list | dict[str, Any]]]=None,
end_tx:bool=False,
return_data:bool=None,
get_col_idx:bool=False,
verbose:bool=False
) -> tuple[list[dbapi.extras.DictRow], dict[str, int] | None]:

Given that I would have thought that passing in
list[dict[str, str]] for "queries" ought to be type safe.
Mypy indicates otherwise which I am not grokking as to why.


but in your
retelling above you indicate lots of possible value types... actually the mypy 
guess
seems to be a pretty good recreation of your psuedo-code description.

I agree that mypy's grasp of my intent from

queries:list[dict[str, str | list | dict[str, Any]]]=None,

into

"List[Dict[str, Union[str, List[Any], Dict[str, Any"

seems accurate. I just don't understand why list[dict[str,
str]] should not pass that construct.


Sorry for the late reaction and may be I am missing something, but I was 
wondering if
your type hint for queries shouldn't be the following.

queries:list[dict[str,str]|dict[str,list]|dict[str,dict[str, dict[str, Ant]]]

My impression at this moment is that you are write something like: dict[str, 
str | int] as
as shorthand for dict[str, str] | dict[str, int]. But those two are different 
types.

--
Antoon Pardon.
--
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Karsten Hilbert via Python-list]

Thanks to all. I ended up using Sequence for the list part
and Mapping for the dict part, which does require "import
typing" which I would rather have avoided.

Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Greg Ewing via Python-list]

On 31/12/23 10:06 am, Thomas Passin wrote:
my suggestion above does 
work, *except* that you cannot mix-and-match different DictTypex types


Have you tried declaring the argument as a Mapping instead of a dict?
Seeing as Thomas Passin's Sequence experiment worked, it seems like this
should work too.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Aw: Re: mypy question

[Greg Ewing via Python-list]

On 31/12/23 8:05 am, Chris Angelico wrote:

Ah, I think you've hit on the problem there. Consider this:

def add_item(stuff: dict[str: str | int]):
 stuff["spam"] = "ham"
 stuff["vooom"] = 1_000_000


Yep, that's it exactly. It's not the union itself that's the problem,
but the fact that there's a *mutable container* containing that type.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Thomas Passin via Python-list]

On 12/30/2023 9:14 AM, Thomas Passin via Python-list wrote:

On 12/29/2023 10:02 AM, Karsten Hilbert via Python-list wrote:

I agree that mypy's grasp of my intent from

queries:list[dict[str, str | list | dict[str, Any]]]=None,

into

"List[Dict[str, Union[str, List[Any], Dict[str, Any"

seems accurate. I just don't understand why list[dict[str,
str]] should not pass that construct.


I made a tiny test program with your type signature, and got this error 
message from mypy:


c:\temp\python\typing_test.py:3: error: X | Y syntax for unions requires 
Python 3.10  [syntax]


Aside from that, this variation causes no mypy error (you must use 
Sequence instead of List), and is also more clear about what you are 
trying to get:


from typing import Union, Sequence, Dict

DictType1 = Dict[str, str]
DictType2 = Dict[str, Sequence]
DictType3 = Dict[str, Dict]
QueryType = Sequence[Union[DictType1, DictType2, DictType3]]

def test_typing(queries:QueryType=None):
     print(type(queries))

d1 = {'k1': 'v1', 'k2': 'v2'}
queries = [d1,]
test_typing(queries)

I'm not sure if this captures exactly what you want, but it avoids the 
problem where mypy does not regard str and Union[str, list] as 
equivalent types.  I tested this using Python 3.12.


In doing more testing, I have learned that my suggestion above does 
work, *except* that you cannot mix-and-match different DictTypex types 
within the same Sequence - meaning within the same query argument.  Any 
of the Union types is OK but they all have to be the same in any instance.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Aw: Re: mypy question

[Chris Angelico via Python-list]

On Sun, 31 Dec 2023 at 03:38, Thomas Passin via Python-list
 wrote:
> I am not very expert in Python type hints.  In working up the example
> program I just posted, I got an error message from mypy that remarked
> that "list" is invariant, and to try Sequence which is "covariant".  I
> don't know what that means (and I haven't looked into it yet), but when
> I changed from list to Sequence as suggested, mypy stopped complaining.
>

Ah, I think you've hit on the problem there. Consider this:

def add_item(stuff: dict[str: str | int]):
stuff["spam"] = "ham"
stuff["vooom"] = 1_000_000

Is it valid to pass this function a dict[str: str]? No, because it's
going to add an integer into it.

Hopefully that helps explain what's going on a bit.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Barry via Python-list]

> On 30 Dec 2023, at 15:11, Karsten Hilbert via Python-list 
>  wrote:
> 
> queries = [{'SQL': 'SELECT %(value)s', 'args': {'value': 1}}]
> 
> and
> 
> run_queries(conn, queries:list[str|dict[str, Any]]):

In cases like this I often use a wrapper class in place of a simple str.
If you have a class SqlString then your type becomes list[SqlString].

You may find that SqlString gains interesting methods over time.

 Barry
-- 
https://mail.python.org/mailman/listinfo/python-list

Aw: Re: Re: mypy question

[Karsten Hilbert via Python-list]

> I'm fairly sure your database queries don't actually give you strings or
> dicts, right?  You probably get lists (or iterators) of tuples and
> somewhere you convert them to the arguments you are feeding to
> run_queries().

Ah, no, those queries are enshrined within the middleware as Python strings
with placeholdders. When in need of being run they are assembled into
a list of dicts-enriched-with-arguments-per-query (and fed to run_queries()).

As to what queries *give* me: I have set up psycopg2 to, indeed, hand
me a list of dicts (DictRow instances, that is). Those are then used
in display rather than being fed to run_queries().

Karsten
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Thomas Passin via Python-list]

On 12/29/2023 10:02 AM, Karsten Hilbert via Python-list wrote:

Am Fri, Dec 29, 2023 at 07:49:17AM -0700 schrieb Mats Wichmann via Python-list:


I am not sure why mypy thinks this

gmPG2.py:554: error: Argument "queries" to "run_rw_queries" has incompatible type 
"List[Dict[str, str]]"; expected
"List[Dict[str, Union[str, List[Any], Dict[str, Any"  [arg-type]
 rows, idx = run_rw_queries(link_obj = conn, queries = 
queries, return_data = True)
   
^~~

should be flagged. The intent is for "queries" to be

a list
of dicts
with keys of str
and values of
str OR
list of anything OR
dict with
keys of str
and values of anything

I'd have thunk list[dict[str,str]] matches that ?


Dict[str, str] means the key type and value type should both be strings,


Indeed, I know that much, list[dict[str, str]] is what is getting
passed in in this particular invocation of run_rw_queries().

For what it's worth here's the signature of that function:

def run_rw_queries (
link_obj:_TLnkObj=None,
queries:list[dict[str, str | list | dict[str, Any]]]=None,
end_tx:bool=False,
return_data:bool=None,
get_col_idx:bool=False,
verbose:bool=False
) -> tuple[list[dbapi.extras.DictRow], dict[str, int] | None]:

Given that I would have thought that passing in
list[dict[str, str]] for "queries" ought to be type safe.
Mypy indicates otherwise which I am not grokking as to why.


but in your
retelling above you indicate lots of possible value types... actually the mypy 
guess
seems to be a pretty good recreation of your psuedo-code description.


I agree that mypy's grasp of my intent from

queries:list[dict[str, str | list | dict[str, Any]]]=None,

into

"List[Dict[str, Union[str, List[Any], Dict[str, Any"

seems accurate. I just don't understand why list[dict[str,
str]] should not pass that construct.


Maybe better to ask the mypy people directly.


Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B


--
https://mail.python.org/mailman/listinfo/python-list

Aw: Re: Re: mypy question

[Karsten Hilbert via Python-list]

> It occurs to me that you could simplify things if you converted those
> plain query strings to dicts:
>
> 'SELECT 1' --> {'SQL': 'SELECT 1'}

Ha, indeed. There's likely not that many "simple string SQL queries"
in that codebase so I shall take it as an opportunity to refactor them.

So, at least that much good has come from the mypy hint ;-)

Karsten
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Aw: Re: mypy question

[Thomas Passin via Python-list]

On 12/30/2023 10:08 AM, Karsten Hilbert via Python-list wrote:

Dear Thomas,

thanks for taking the time to look into my issue.

Maybe it helps if I explain what I want (sorry that my web mailer does not 
respect
indentation, I will insert dots).

I want a function to run SQL queries:

run_queries(conn, queries):
...for q in queries:
..conn.execute(q)

I now want to add type hints such that my large codebase can
be checked for silly doings. First, queries is to be a list
of the SQL to run:

run_queries(conn, queries:list):

Then, each list entry can be

...a string holding simple, complete SQL (say "SELECT 1")

run_queries(conn, queries:list[str]):


It occurs to me that you could simplify things if you converted those 
plain query strings to dicts:


'SELECT 1' --> {'SQL': 'SELECT 1'}

I'm fairly sure your database queries don't actually give you strings or 
dicts, right?  You probably get lists (or iterators) of tuples and 
somewhere you convert them to the arguments you are feeding to 
run_queries().  At least, that is how the standard Python db adapters 
work. If you change that conversion step, your arguments to 
run_queries() will all be lists of dicts, making your code simpler and 
reducing the complexity of the type hints.



or

...a dict holding the SQL and arguments for parameters

run_queries(conn, queries:list[dict]):

So, taken together:

run_queries(conn, queries:list[str|dict]):

(yes, this is in Python 3.11/3.12)

Now, when it is a list of dicts I want to further constrain the
dicts. Each is to contain the keys "SQL" and "args". So the keys
are of type str. The values for the keys will be of various types,
such that I chose Any as pseudo-type, so that each list entry that
is of type dict should be dict[str, Any], hence:

queries = [{'SQL': 'SELECT %(value)s', 'args': {'value': 1}}]

and

run_queries(conn, queries:list[str|dict[str, Any]]):

If I now call this function with a simple SQL query:

SQL_query = 'SELECT 1'  # should be type str ?
queries = [SQL_query]   # should be type list[str] ?
run_queries(conn, queries = queries)

and run mypy over that (at least inside my complex codebase) I will
get a type mismatch being hinted at.

So far I don't grasp at which point my reasoning above is faulty.

Karsten


--
https://mail.python.org/mailman/listinfo/python-list

Re: Aw: Re: mypy question

[Thomas Passin via Python-list]

On 12/30/2023 10:08 AM, Karsten Hilbert via Python-list wrote:

Dear Thomas,

thanks for taking the time to look into my issue.

Maybe it helps if I explain what I want (sorry that my web mailer does not 
respect
indentation, I will insert dots).

I want a function to run SQL queries:

run_queries(conn, queries):
...for q in queries:
..conn.execute(q)

I now want to add type hints such that my large codebase can
be checked for silly doings. First, queries is to be a list
of the SQL to run:

run_queries(conn, queries:list):

Then, each list entry can be

...a string holding simple, complete SQL (say "SELECT 1")

run_queries(conn, queries:list[str]):

or

...a dict holding the SQL and arguments for parameters

run_queries(conn, queries:list[dict]):

So, taken together:

run_queries(conn, queries:list[str|dict]):

(yes, this is in Python 3.11/3.12)

Now, when it is a list of dicts I want to further constrain the
dicts. Each is to contain the keys "SQL" and "args". So the keys
are of type str. The values for the keys will be of various types,
such that I chose Any as pseudo-type, so that each list entry that
is of type dict should be dict[str, Any], hence:

queries = [{'SQL': 'SELECT %(value)s', 'args': {'value': 1}}]

and

run_queries(conn, queries:list[str|dict[str, Any]]):

If I now call this function with a simple SQL query:

SQL_query = 'SELECT 1'  # should be type str ?
queries = [SQL_query]   # should be type list[str] ?
run_queries(conn, queries = queries)

and run mypy over that (at least inside my complex codebase) I will
get a type mismatch being hinted at.

So far I don't grasp at which point my reasoning above is faulty.

Karsten


I am not very expert in Python type hints.  In working up the example 
program I just posted, I got an error message from mypy that remarked 
that "list" is invariant, and to try Sequence which is "covariant".  I 
don't know what that means (and I haven't looked into it yet), but when 
I changed from list to Sequence as suggested, mypy stopped complaining.


Here is the exact error message, and it has a reference you might want 
to follow up with:


c:\temp\python\typing_test.py:16: note: "List" is invariant -- see 
https://mypy.readthedocs.io/en/stable/common_issues.html#variance
c:\temp\python\typing_test.py:16: note: Consider using "Sequence" 
instead, which is covariant


Before that, mypy insisted that str and Union[str, list] were 
incompatible argument types, which is something you are seeing, too.


I suggest that you build up your types as in my example, so that it's 
very clear what they are and very easy to change them, and use Sequence 
instead of List (or list).  See if that will do the job.


--
https://mail.python.org/mailman/listinfo/python-list

Aw: Re: mypy question

[Karsten Hilbert via Python-list]

Dear Thomas,

thanks for taking the time to look into my issue.

Maybe it helps if I explain what I want (sorry that my web mailer does not 
respect
indentation, I will insert dots).

I want a function to run SQL queries:

run_queries(conn, queries):
...for q in queries:
..conn.execute(q)

I now want to add type hints such that my large codebase can
be checked for silly doings. First, queries is to be a list
of the SQL to run:

run_queries(conn, queries:list):

Then, each list entry can be

...a string holding simple, complete SQL (say "SELECT 1")

run_queries(conn, queries:list[str]):

or

...a dict holding the SQL and arguments for parameters

run_queries(conn, queries:list[dict]):

So, taken together:

run_queries(conn, queries:list[str|dict]):

(yes, this is in Python 3.11/3.12)

Now, when it is a list of dicts I want to further constrain the
dicts. Each is to contain the keys "SQL" and "args". So the keys
are of type str. The values for the keys will be of various types,
such that I chose Any as pseudo-type, so that each list entry that
is of type dict should be dict[str, Any], hence:

queries = [{'SQL': 'SELECT %(value)s', 'args': {'value': 1}}]

and

run_queries(conn, queries:list[str|dict[str, Any]]):

If I now call this function with a simple SQL query:

SQL_query = 'SELECT 1'  # should be type str ?
queries = [SQL_query]   # should be type list[str] ?
run_queries(conn, queries = queries)

and run mypy over that (at least inside my complex codebase) I will
get a type mismatch being hinted at.

So far I don't grasp at which point my reasoning above is faulty.

Karsten
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Thomas Passin via Python-list]

On 12/29/2023 10:02 AM, Karsten Hilbert via Python-list wrote:

I agree that mypy's grasp of my intent from

queries:list[dict[str, str | list | dict[str, Any]]]=None,

into

"List[Dict[str, Union[str, List[Any], Dict[str, Any"

seems accurate. I just don't understand why list[dict[str,
str]] should not pass that construct.


I made a tiny test program with your type signature, and got this error 
message from mypy:


c:\temp\python\typing_test.py:3: error: X | Y syntax for unions requires 
Python 3.10  [syntax]


Aside from that, this variation causes no mypy error (you must use 
Sequence instead of List), and is also more clear about what you are 
trying to get:


from typing import Union, Sequence, Dict

DictType1 = Dict[str, str]
DictType2 = Dict[str, Sequence]
DictType3 = Dict[str, Dict]
QueryType = Sequence[Union[DictType1, DictType2, DictType3]]

def test_typing(queries:QueryType=None):
print(type(queries))

d1 = {'k1': 'v1', 'k2': 'v2'}
queries = [d1,]
test_typing(queries)

I'm not sure if this captures exactly what you want, but it avoids the 
problem where mypy does not regard str and Union[str, list] as 
equivalent types.  I tested this using Python 3.12.




--
https://mail.python.org/mailman/listinfo/python-list

Aw: Re: mypy question

[Karsten Hilbert via Python-list]

Hi Greg,

> dict[str, str] is not a subtype of dict[str, str | something_else]
> because you can assign a value of type something_else to the latter
> but not the former.

I understand what you are saying but I do not yet understand why this
applies to my situation.

I don't have Python at hand currently, so I'll write untested pseudocode:

def print_greeting(greeting:int|str):
   print(greeting)

print_greeting('hello')

The above snippet should be equivalent to my more complicated code over
which mypy complains to the equivalent of

   "input" is of type "str"
   but expected type "Union[str,int]

I do understand that "str" is formally more narrow than "Union [str,int]" and
the type system has every right to not consider them equivalent.

However, this seems like a very common use case: "allow passing in either str 
or int
and have type checking pass either input as valid" -- yet mypy doesn't seem
to share that idea.

Or else there's something I haven't wrapped my head around yet. But what ?

Karsten

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Greg Ewing via Python-list]

On 30/12/23 4:02 am, Karsten Hilbert wrote:


def run_rw_queries (
link_obj:_TLnkObj=None,
queries:list[dict[str, str | list | dict[str, Any]]]=None,



Given that I would have thought that passing in
list[dict[str, str]] for "queries" ought to be type safe.


dict[str, str] is not a subtype of dict[str, str | something_else]
because you can assign a value of type something_else to the latter
but not the former.

In this case it happens to be okay because the function is (presumably)
treating the dict passed in as immutable, but MyPy has no way to be sure
of that.

You could try declaring it as a collections.Mapping, which is immutable.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Karsten Hilbert via Python-list]

Am Fri, Dec 29, 2023 at 11:04:59AM -0700 schrieb Mats Wichmann via Python-list:

> >For what it's worth here's the signature of that function:
> >
> > def run_rw_queries (
> > link_obj:_TLnkObj=None,
> > queries:list[dict[str, str | list | dict[str, Any]]]=None,
> > end_tx:bool=False,
> > return_data:bool=None,
> > get_col_idx:bool=False,
> > verbose:bool=False
> > ) -> tuple[list[dbapi.extras.DictRow], dict[str, int] | None]:
> >
> >Given that I would have thought that passing in
> >list[dict[str, str]] for "queries" ought to be type safe.
> >Mypy indicates otherwise which I am not grokking as to why.
>
> ah... didn't grok what you were asking, sorry - ignore my attempt then.

Never mind, the attempt to help is appreciated.

> So you are passing something that has been typed more
> narrowly than the function parameter.

That would then sort of skirt on violation of the Liskov
principle, of which I learned while trying to research this
mypy behaviour.

However, I would not think the above to be a narrowing-down
as it just *selects* one of the explicitely "legal" options.

list[dict[str, str | list | dict[str, Any]]]

should AFAICT expand to:

list[dict[str, dict[str, Any]]]

OR

list[dict[str, list]]

OR

list[dict[str, str]]

the last of which should provide coverage of

[{'some key': 'some value'}]

> Can you use a TypeGuard here?

Not from what I understand about them...

Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Mats Wichmann via Python-list]

On 12/29/23 08:02, Karsten Hilbert via Python-list wrote:


Dict[str, str] means the key type and value type should both be strings,


Indeed, I know that much, list[dict[str, str]] is what is getting
passed in in this particular invocation of run_rw_queries().

For what it's worth here's the signature of that function:

def run_rw_queries (
link_obj:_TLnkObj=None,
queries:list[dict[str, str | list | dict[str, Any]]]=None,
end_tx:bool=False,
return_data:bool=None,
get_col_idx:bool=False,
verbose:bool=False
) -> tuple[list[dbapi.extras.DictRow], dict[str, int] | None]:

Given that I would have thought that passing in
list[dict[str, str]] for "queries" ought to be type safe.
Mypy indicates otherwise which I am not grokking as to why.


ah... didn't grok what you were asking, sorry - ignore my attempt then. 
So you are passing something that has been typed more narrowly than the 
function parameter. Can you use a TypeGuard here?

--
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Karsten Hilbert via Python-list]

Am Fri, Dec 29, 2023 at 07:49:17AM -0700 schrieb Mats Wichmann via Python-list:

> >I am not sure why mypy thinks this
> >
> >gmPG2.py:554: error: Argument "queries" to "run_rw_queries" has incompatible 
> >type "List[Dict[str, str]]"; expected
> >"List[Dict[str, Union[str, List[Any], Dict[str, Any"  [arg-type]
> > rows, idx = run_rw_queries(link_obj = conn, queries = 
> > queries, return_data = True)
> >   
> > ^~~
> >
> >should be flagged. The intent is for "queries" to be
> >
> >a list
> > of dicts
> > with keys of str
> > and values of
> > str OR
> > list of anything OR
> > dict with
> > keys of str
> > and values of anything
> >
> >I'd have thunk list[dict[str,str]] matches that ?
>
> Dict[str, str] means the key type and value type should both be strings,

Indeed, I know that much, list[dict[str, str]] is what is getting
passed in in this particular invocation of run_rw_queries().

For what it's worth here's the signature of that function:

def run_rw_queries (
link_obj:_TLnkObj=None,
queries:list[dict[str, str | list | dict[str, Any]]]=None,
end_tx:bool=False,
return_data:bool=None,
get_col_idx:bool=False,
verbose:bool=False
) -> tuple[list[dbapi.extras.DictRow], dict[str, int] | None]:

Given that I would have thought that passing in
list[dict[str, str]] for "queries" ought to be type safe.
Mypy indicates otherwise which I am not grokking as to why.

> but in your
> retelling above you indicate lots of possible value types... actually the 
> mypy guess
> seems to be a pretty good recreation of your psuedo-code description.

I agree that mypy's grasp of my intent from

queries:list[dict[str, str | list | dict[str, Any]]]=None,

into

"List[Dict[str, Union[str, List[Any], Dict[str, Any"

seems accurate. I just don't understand why list[dict[str,
str]] should not pass that construct.

Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Mats Wichmann via Python-list]

On 12/29/23 05:15, Karsten Hilbert via Python-list wrote:

Hi all,

I am not sure why mypy thinks this

gmPG2.py:554: error: Argument "queries" to "run_rw_queries" has incompatible type 
"List[Dict[str, str]]"; expected
"List[Dict[str, Union[str, List[Any], Dict[str, Any"  [arg-type]
 rows, idx = run_rw_queries(link_obj = conn, queries = 
queries, return_data = True)
   
^~~

should be flagged. The intent is for "queries" to be

a list
of dicts
with keys of str
and values of
str OR
list of anything OR
dict with
keys of str
and values of anything

I'd have thunk list[dict[str,str]] matches that ?


Dict[str, str] means the key type and value type should both be strings, 
but in your retelling above you indicate lots of possible value types... 
actually the mypy guess seems to be a pretty good recreation of your 
psuedo-code description.

--
https://mail.python.org/mailman/listinfo/python-list

Re: mypy question

[Karsten Hilbert via Python-list]

Am Fri, Dec 29, 2023 at 01:15:29PM +0100 schrieb Karsten Hilbert via 
Python-list:

> I am not sure why mypy thinks this
>
> gmPG2.py:554: error: Argument "queries" to "run_rw_queries" has incompatible 
> type "List[Dict[str, str]]"; expected
> "List[Dict[str, Union[str, List[Any], Dict[str, Any"  [arg-type]
> rows, idx = run_rw_queries(link_obj = conn, queries = 
> queries, return_data = True)
>   
> ^~~
>
> should be flagged. The intent is for "queries" to be
>
> a list
>   of dicts
>   with keys of str
>   and values of
>   str OR
>   list of anything OR
>   dict with
>   keys of str
>   and values of anything
>
> I'd have thunk list[dict[str,str]] matches that ?
>
> This is on Python 3.11.2 with mypy 1.0.1 on Debian.

For completeness, this was the mypy call signature:

mypy --pretty --allow-redefinition --no-strict-optional 
--ignore-missing-imports --follow-imports silent --show-error-codes 
--warn-unused-ignores gmPG2.py

Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

mypy question

[Karsten Hilbert via Python-list]

Hi all,

I am not sure why mypy thinks this

gmPG2.py:554: error: Argument "queries" to "run_rw_queries" has incompatible 
type "List[Dict[str, str]]"; expected
"List[Dict[str, Union[str, List[Any], Dict[str, Any"  [arg-type]
rows, idx = run_rw_queries(link_obj = conn, queries = 
queries, return_data = True)
  
^~~

should be flagged. The intent is for "queries" to be

a list
of dicts
with keys of str
and values of
str OR
list of anything OR
dict with
keys of str
and values of anything

I'd have thunk list[dict[str,str]] matches that ?

This is on Python 3.11.2 with mypy 1.0.1 on Debian.

Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Newline (NuBe Question)

[AVI GROSS via Python-list]

Dave, I gave an example, again, and make no deep claims so your comments may be 
valid, without any argument.

I mentioned CSV and a related family such as TSV as they were a common and 
simple data format that has long been used. There are oodles of others and yes, 
these days many people can read directly from formats like some from EXCEL. But 
for data that can be shared to almost anyone using anything, something like 
Comma Separated Values is often used.

And some programs that generate such data simply keep appending a line at a 
time to a file and do not have any header line. There are even some programs 
that may not tolerate a file with a header line, or comments or other optional 
things, and some where header lines you can create would cause problems such as 
using an extended character set or escaped characters.

I have worked with these files in many languages and environments and my 
thought process here focused on recent work in R, albeit much applies 
everywhere. My point was really not about CSV but the convenience and 
advantages of data structures you can access by name when you want and 
sometimes also by position when you want. Too many errors can happen when 
humans doing programming are not able to concentrate. It is similar to 
arguments about file names. In the old UNIX days, and the same for other 
systems like VMS, a filename tended to have a format where relatively few 
characters were allowed and it might have two parts with the latter being an 
extension of up to 3 characters, or whatever. So file names like A321G12.dat 
were common and also next to it similar unpronounceable other file names. It 
was easy to confuse them and even people who worked with them regularly would 
forget what it might mean or use the wrong one. 

Well, if I load in a CSV in a language like R and there is no header line, as 
with some other data structures, it may make up a placeholder set of names like 
V1, V2 and so on. Yes, there are ways to specify the names as they are read in 
or afterward and they can be changed. But I have seen lots of CSV files offered 
with way too many columns and no names as well as documentation suggesting what 
names can be added if you wish.

This may be a bit off topic, but I want to add a bit in this context about 
additional concepts regarding name. As mentioned, there is a whole set of 
add-ons people sometimes use and in R, I like the tidyverse family and it 
allows some fairly sophisticated things to be done using names. There are ways 
to specify you want a subset of a data.frame (sometimes a version called a 
tibble) and you can ask for say all columns starting with "xyz" or containing 
it or ending with it. That can be very helpful if say we wave columns 
containing the height and weight and other metrics of say people in three 
clinics and your column names embed the name of the clinic, or other such 
examples, and you want to select one grouping for processing. You cannot easily 
do that without external info is it is just positional. 

An extension of this is how compactly you can do fairly complex things such as 
asking to create lots of new columns using calculations. You can specify, as 
above, which sets of columns to do this too and that you want the results for 
each XYY in XYZ.mean and XYZ.std and so on. You can skip oodles of carefully 
crafted and nested loops because of the ability to manipulate using column 
names at a high and often abstract level. 

And, just FYI, many other structures such as lists in R also support names for 
components. It can be very useful. But the overall paradigm compared to Python 
has major differences and I see strengths and weaknesses and tradeoffs.

Your dictionary example is one of them as numpy/pandas often make good use of 
them as part of dealing with similar data.frame type structures that are often 
simpler or easier to code with.

There is lots of AI discussion these days and some of what you say is 
applicable in that additional info besides names might be useful in the storage 
format to make processing it more useful. That is available in formats related 
to XML where fairly arbitrary markup can be made available.

Have to head out as this is already long enough.



-Original Message-
From: 'DL Neil'  
Sent: Monday, November 27, 2023 2:49 AM
To: avi.e.gr...@gmail.com; python-list@python.org
Subject: Re: Newline (NuBe Question)

Avi,

On 11/27/2023 4:15 PM, avi.e.gr...@gmail.com wrote:
> Dave,
> 
> Back on a hopefully more serious note, I want to make a bit of an analogy
> with what happens when you save data in a format like a .CSV file.
> 
> Often you have a choice of including a header line giving names to the
> resulting columns, or not.
> 
> If you read in the data to some structure, often to some variation I would
> loosely call a data.frame or perhaps something like a matrix, then without
> headers you have to specify what you want positionally or

Re: Newline (NuBe Question)

['DL Neil' via Python-list]

Avi,

On 11/27/2023 4:15 PM, avi.e.gr...@gmail.com wrote:

Dave,

Back on a hopefully more serious note, I want to make a bit of an analogy
with what happens when you save data in a format like a .CSV file.

Often you have a choice of including a header line giving names to the
resulting columns, or not.

If you read in the data to some structure, often to some variation I would
loosely call a data.frame or perhaps something like a matrix, then without
headers you have to specify what you want positionally or create your own
names for columns to use. If names are already there, your program can
manipulate things by using the names and if they are well chosen, with no
studs among them, the resulting code can be quite readable. More
importantly, if the data being read changes and includes additional columns
or in a different order, your original program may run fine as long as the
names of the columns you care about remain the same.

Positional programs can be positioned to fail in quite subtle ways if the
positions no longer apply.


Must admit to avoiding .csv files, if possible, and working directly 
with the .xls? original (cf expecting the user to export the .csv - and 
NOT change the worksheet thereafter).


However, have recently been using the .csv format (as described) as a 
placeholder or introduction to formatting data for an RDBMS.


In a tabular structure, the expectation is that every field (column/row 
intersection) will contain a value. In the RDBMS-world, if the value is 
not-known then it will be recorded as NULL (equivalent of Python's None).


Accordingly, two points:
1 the special case of missing/unavailable data can be handled with ease,
2 most 'connector' interfaces will give the choice of retrieving data 
into a tuple or a dictionary (where the keys are the column-names). The 
latter easing data-identification issues (as described) both in terms of 
improving over relational-positioning and name-continuity (or column 
changes/expansions).



The point about data 'appearing' without headings should be considered 
carefully. The phrase "create your own names for columns" only vaguely 
accesses the problem. If someone else has created/provided the data, 
then we need to know the exact design (schema = rules). What is the 
characteristic of each component? Not only column-names, but also what 
is the metric (eg the infamous confusion between feet and meters)...




As I see it, many situations where some aspects are variable are not ideal
for naming. A dictionary is an example that is useful when you have no idea
how many items with unknown keys may be present. You can iterate over the
names that are there, or use techniques that detect and deal with keys from
your list that are not present. Not using names/keys here might involve a
longer list with lots of empty slots to designate missing items, This
clearly is not great when the data present is sparse or when the number of
items is not known in advance or cannot be maintained in the right order.


Agreed, and this is the draw-back incurred by folk who wish to take 
advantage of the schema-less (possibility) NoSQL DBs. The DB enjoys 
flexibility, but the downstream-coder has to contort and flex to cope.


In this case, JSON files are an easy place-holder/intro for NoSQL DBs - 
in fact, Python dicts and MongoDB go hand-in-glove.



The next issue raised is sparseness. In a table, the assumption is that 
all fields, or at least most of them, will be filled with values. 
However, a sparse matrix would make such very 'expensive' in terms of 
storage-space (efficacy).


Accordingly, there are other ways of doing things. All of these involve 
labeling each data-item (thus, the data expressed as a table needs to be 
at least 50% empty to justify the structural change).


In this case, one might consider a tree-type of structure - and if we 
have to continue the pattern, we might look at a Network Database 
methodology (as distinct from a DB on a network!)




There are many other situations with assorted tradeoffs and to insist on
using lists/tuples exclusively would be silly but at the same time, if you
are using a list to hold the real and imaginary parts of a complex number,
or the X/Y[/Z] coordinates of a point where the order is almost universally
accepted, then maybe it is not worth using a data structure more complex or
derived as the use may be obvious.


No argument (in case anyone thought I might...)

See @Peter's earlier advice.

Much of the consideration (apart from mutable/immutable) is likely to be 
ease of coding. Getting down 'into the weeds' is probably pointless 
unless questions are being asked about (execution-time) performance...



Isn't the word "obvious" where this discussion started? Whereas "studs" 
might be an "obvious" abbreviation for "students" to some, it is not to 
others (quite aside from the abbreviation being unnecessary in this 
day-and-age).


Curiously, whereas I DO happen to think a point as ( x, y, ) or 

RE: Newline (NuBe Question)

[AVI GROSS via Python-list]

Dave,

Back on a hopefully more serious note, I want to make a bit of an analogy
with what happens when you save data in a format like a .CSV file.

Often you have a choice of including a header line giving names to the
resulting columns, or not.

If you read in the data to some structure, often to some variation I would
loosely call a data.frame or perhaps something like a matrix, then without
headers you have to specify what you want positionally or create your own
names for columns to use. If names are already there, your program can
manipulate things by using the names and if they are well chosen, with no
studs among them, the resulting code can be quite readable. More
importantly, if the data being read changes and includes additional columns
or in a different order, your original program may run fine as long as the
names of the columns you care about remain the same. 

Positional programs can be positioned to fail in quite subtle ways if the
positions no longer apply.

As I see it, many situations where some aspects are variable are not ideal
for naming. A dictionary is an example that is useful when you have no idea
how many items with unknown keys may be present. You can iterate over the
names that are there, or use techniques that detect and deal with keys from
your list that are not present. Not using names/keys here might involve a
longer list with lots of empty slots to designate missing items, This
clearly is not great when the data present is sparse or when the number of
items is not known in advance or cannot be maintained in the right order. 

There are many other situations with assorted tradeoffs and to insist on
using lists/tuples exclusively would be silly but at the same time, if you
are using a list to hold the real and imaginary parts of a complex number,
or the X/Y[/Z] coordinates of a point where the order is almost universally
accepted, then maybe it is not worth using a data structure more complex or
derived as the use may be obvious.

I do recall odd methods sometimes used way back when I programmed in C/C++
or similar languages when some method was used to declare small constants
like:

#define FIRSTNAME 1
#define LASTNAME 2

Or concepts like "const GPA = 3"

And so on, so code asking for student_record[LASTNAME] would be a tad more
readable and if the order of entries somehow were different, just redefine
the constant.

In some sense, some of the data structures we are discussing, under the
hood, actually may do something very similar as they remap the name to a
small integer offset. Others may do much more or be slower but often add
value in other ways. A full-blown class may not just encapsulate the names
of components of an object but verify the validity of the contents or do
logging or any number of other things. Using a list or tuple does nothing
else.

So if you need nothing else, they are often suitable and sometimes even
preferable. 


-Original Message-
From: Python-list  On
Behalf Of DL Neil via Python-list
Sent: Sunday, November 26, 2023 5:19 PM
To: python-list@python.org
Subject: Re: Newline (NuBe Question)

On 11/27/2023 10:04 AM, Peter J. Holzer via Python-list wrote:
> On 2023-11-25 08:32:24 -0600, Michael F. Stemper via Python-list wrote:
>> On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote:
>>> Of course, for serious work, some might suggest avoiding constructs like
a
>>> list of lists and switch to using modules and data structures [...]
>>
>> Those who would recommend that approach do not appear to include Mr.
>> Rossum, who said:
>>Avoid overengineering data structures.
>^^^
> 
> The key point here is *over*engineering. Don't make things more
> complicated than they need to be. But also don't make them simpler than
> necessary.
> 
>>Tuples are better than objects (try namedtuple too though).
> 
> If Guido thought that tuples would always be better than objects, then
> Python wouldn't have objects. Why would he add such a complicated
> feature to the language if he thought it was useless?
> 
> The (unspoken?) context here is "if tuples are sufficient, then ..."


At recent PUG-meetings I've listened to a colleague asking questions and 
conducting research on Python data-structures*, eg lists-of-lists cf 
lists-of-tuples, etc, etc. The "etc, etc" goes on for some time! 
Respecting the effort, even as it becomes boringly-detailed, am 
encouraging him to publish his findings.

* sadly, he is resistant to OOP and included only a cursory look at 
custom-objects, and early in the process. His 'new thinking' has been to 
look at in-core databases and the speed-ups SQL (or other) might offer...

However, his motivation came from a particular application, and to 
create a naming-system so that he could distinguish a list-of-lists 
structure from some other tabular abstraction. The latter enables the 

Re: Newline (NuBe Question)

[Chris Angelico via Python-list]

On Mon, 27 Nov 2023 at 13:52, AVI GROSS via Python-list
 wrote:
>  Be that as it
> may, and I have no interest in this topic, in the future I may use the ever
> popular names of Primus, Secundus and Tertius and get blamed for using
> Latin.
>

Imperious Prima flashes forth her edict to "begin it". In gentler tone
Secunda hopes there will be nonsense in it. While Tertia interrupts
the tale not more than once a minute.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Newline (NuBe Question)

[AVI GROSS via Python-list]

Isn't it fascinating that a meaningless piece of code used to illustrate
something can be analyzed as if it was full of malicious content?

Yes, my choice of names was as expected. The numbers chosen had no special
meaning other than choosing one number in each of three equivalence classes.

But, if you want me to add subtle meaning for generations to examine as it
it were a literary work, I offer this:

Peter and Paul were studs who got Mary'd.

Can we now go back to our regularly scheduled talking about aspects of a
computer language?

P.S.
And just for history, Paul was really Noel Paul Stookey but Peter, Paul &
Mary sounded more like new testament characters and I think Noel signifies a
birth to Peter and Mary, sort of, which might have fit too unless it was a
computer program where a name with an umlaut was once not common. Another
interpretation is that Noel came from the Latin word for news. Be that as it
may, and I have no interest in this topic, in the future I may use the ever
popular names of Primus, Secundus and Tertius and get blamed for using
Latin.

-Original Message-
From: Python-list  On
Behalf Of DL Neil via Python-list
Sent: Sunday, November 26, 2023 4:58 PM
To: python-list@python.org
Subject: Re: Newline (NuBe Question)

On 11/27/2023 12:48 AM, Chris Angelico via Python-list wrote:
> On Sun, 26 Nov 2023 at 21:08, Michael F. Stemper via Python-list
>  wrote:
>>
>> On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote:
>>> Grizz[l]y,
>>>
>>> I think the point is not about a sorted list or sorting in general It is
>>> about reasons why maintaining a data structure such as a list in a
program
>>> can be useful beyond printing things once. There are many possible
examples
>>> such as having a list of lists containing a record where the third item
is a
>>> GPA for the student and writing a little list comprehension that selects
a
>>> smaller list containing only students who are Magna Cum Laude or Summa
Cum
>>> Laude.
>>>
>>> studs = [
>>> ["Peter", 82, 3.53],
>>> ["Paul", 77, 2.83],
>>> ["Mary", 103, 3.82]
>>> ]
>>
>> I've seen Mary, and she didn't look like a "stud" to me.
>>
> 
> That's what happens when you abbreviate "student" though :) Don't
> worry, there's far FAR worse around the place, and juvenile brains
> will always find things to snigger at, usually in mathematical
> libraries with "cumulative" functions.

The OP used an abbreviation: "studs". Why? Too lazy to type the full 
word? Abbreviation has full-meaning in the (narrow) domain? Was wanting 
something funny, or to snigger over?

Was the respondent sniggering? Perhaps he, like the OP, was also saving 
typing-time by making a joke, hoping that the OP would see the 
implicit-error in expecting others to understand that "studs" meant 
"students"?

Actually, Peter, Paul, and Mary were a band 
(https://www.peterpaulandmary.com/), so "studs" is even less expressive 
when the data also tells a story...

Working with "trainees", I avoid the word "student" even though some 
might see them as synonyms. In my mind, the abbreviation did not readily 
expand to the full word (mea culpa).

Accordingly, would not pass Code Review!
For the want of a few characters...
(https://en.wikipedia.org/wiki/For_Want_of_a_Nail)

--
Regards =dn
-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[DL Neil via Python-list]

On 11/27/2023 10:04 AM, Peter J. Holzer via Python-list wrote:

On 2023-11-25 08:32:24 -0600, Michael F. Stemper via Python-list wrote:

On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote:

Of course, for serious work, some might suggest avoiding constructs like a
list of lists and switch to using modules and data structures [...]


Those who would recommend that approach do not appear to include Mr.
Rossum, who said:
   Avoid overengineering data structures.

   ^^^

The key point here is *over*engineering. Don't make things more
complicated than they need to be. But also don't make them simpler than
necessary.


   Tuples are better than objects (try namedtuple too though).


If Guido thought that tuples would always be better than objects, then
Python wouldn't have objects. Why would he add such a complicated
feature to the language if he thought it was useless?

The (unspoken?) context here is "if tuples are sufficient, then ..."



At recent PUG-meetings I've listened to a colleague asking questions and 
conducting research on Python data-structures*, eg lists-of-lists cf 
lists-of-tuples, etc, etc. The "etc, etc" goes on for some time! 
Respecting the effort, even as it becomes boringly-detailed, am 
encouraging him to publish his findings.


* sadly, he is resistant to OOP and included only a cursory look at 
custom-objects, and early in the process. His 'new thinking' has been to 
look at in-core databases and the speed-ups SQL (or other) might offer...


However, his motivation came from a particular application, and to 
create a naming-system so that he could distinguish a list-of-lists 
structure from some other tabular abstraction. The latter enables the 
code to change data-format to speed the next process, without the coder 
losing-track of the data-type/format.


The trouble is, whereas the research reveals which is faster 
(in-isolation, and (only) on his 'platform'), my suspicion is that he 
loses all gains by reformatting the data between 'the most efficient' 
structure for each step. A problem of only looking at the 'micro', 
whilst ignoring wider/macro concerns.


Accordingly, as to the word "engineering" (above), a reminder that we 
work in two domains: code and data. The short 'toy examples' in training 
courses discourage us from a design-stage for the former - until we 
enter 'the real world' and meet a problem/solution too large to fit in a 
single human-brain. Sadly, too many of us are pre-disposed to be 
math/algorithmically-oriented, and thus data-design is rarely-considered 
(in the macro!). Yet, here we are...


--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[DL Neil via Python-list]

On 11/27/2023 1:08 AM, Roel Schroeven via Python-list wrote:
I prefer namedtuples or dataclasses over tuples. They allow you to refer 
to their fields by name instead of index: student.gpa is much clearer 
than student[2], and makes it less likely to accidentally refer to the 
wrong field.


+1
readability/comprehension!

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: RE: Newline (NuBe Question)

[Chris Angelico via Python-list]

On Mon, 27 Nov 2023 at 06:15,  wrote:
> But I learn from criticism. If I ever write a program like that and do not
> feel like typing, will this do?
>
> dents = [ ...]
>
> Or will that not include students who happen to be edentulous?
>

If they're learning to drive, this variable name would make complete sense.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[DL Neil via Python-list]

On 11/27/2023 12:48 AM, Chris Angelico via Python-list wrote:

On Sun, 26 Nov 2023 at 21:08, Michael F. Stemper via Python-list
 wrote:


On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote:

Grizz[l]y,

I think the point is not about a sorted list or sorting in general It is
about reasons why maintaining a data structure such as a list in a program
can be useful beyond printing things once. There are many possible examples
such as having a list of lists containing a record where the third item is a
GPA for the student and writing a little list comprehension that selects a
smaller list containing only students who are Magna Cum Laude or Summa Cum
Laude.

studs = [
["Peter", 82, 3.53],
["Paul", 77, 2.83],
["Mary", 103, 3.82]
]


I've seen Mary, and she didn't look like a "stud" to me.



That's what happens when you abbreviate "student" though :) Don't
worry, there's far FAR worse around the place, and juvenile brains
will always find things to snigger at, usually in mathematical
libraries with "cumulative" functions.


The OP used an abbreviation: "studs". Why? Too lazy to type the full 
word? Abbreviation has full-meaning in the (narrow) domain? Was wanting 
something funny, or to snigger over?


Was the respondent sniggering? Perhaps he, like the OP, was also saving 
typing-time by making a joke, hoping that the OP would see the 
implicit-error in expecting others to understand that "studs" meant 
"students"?


Actually, Peter, Paul, and Mary were a band 
(https://www.peterpaulandmary.com/), so "studs" is even less expressive 
when the data also tells a story...


Working with "trainees", I avoid the word "student" even though some 
might see them as synonyms. In my mind, the abbreviation did not readily 
expand to the full word (mea culpa).


Accordingly, would not pass Code Review!
For the want of a few characters...
(https://en.wikipedia.org/wiki/For_Want_of_a_Nail)

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Peter J. Holzer via Python-list]

On 2023-11-25 08:32:24 -0600, Michael F. Stemper via Python-list wrote:
> On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote:
> > Of course, for serious work, some might suggest avoiding constructs like a
> > list of lists and switch to using modules and data structures [...]
> 
> Those who would recommend that approach do not appear to include Mr.
> Rossum, who said:
>   Avoid overengineering data structures.
  ^^^

The key point here is *over*engineering. Don't make things more
complicated than they need to be. But also don't make them simpler than
necessary.

>   Tuples are better than objects (try namedtuple too though).

If Guido thought that tuples would always be better than objects, then
Python wouldn't have objects. Why would he add such a complicated
feature to the language if he thought it was useless?

The (unspoken?) context here is "if tuples are sufficient, then ..."

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: RE: Newline (NuBe Question)

[AVI GROSS via Python-list]

Just FYI, I deliberately chose that abbreviation for a sort of irony as for
some people college is about almost anything except learning and some people
think they are studs and just  party and ...

And I am very tired of gender discussions. Lots of words now include two or
even more genders. Women are often now "actors", not actresses. I see no
reason women cannot be studs!

But I learn from criticism. If I ever write a program like that and do not
feel like typing, will this do?

dents = [ ...]

Or will that not include students who happen to be edentulous?


-Original Message-
From: Python-list  On
Behalf Of Chris Angelico via Python-list
Sent: Sunday, November 26, 2023 6:49 AM
To: python-list@python.org
Subject: Re: RE: Newline (NuBe Question)

On Sun, 26 Nov 2023 at 21:08, Michael F. Stemper via Python-list
 wrote:
>
> On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote:
> > Grizz[l]y,
> >
> > I think the point is not about a sorted list or sorting in general It is
> > about reasons why maintaining a data structure such as a list in a
program
> > can be useful beyond printing things once. There are many possible
examples
> > such as having a list of lists containing a record where the third item
is a
> > GPA for the student and writing a little list comprehension that selects
a
> > smaller list containing only students who are Magna Cum Laude or Summa
Cum
> > Laude.
> >
> > studs = [
> >["Peter", 82, 3.53],
> >["Paul", 77, 2.83],
> >["Mary", 103, 3.82]
> > ]
>
> I've seen Mary, and she didn't look like a "stud" to me.
>

That's what happens when you abbreviate "student" though :) Don't
worry, there's far FAR worse around the place, and juvenile brains
will always find things to snigger at, usually in mathematical
libraries with "cumulative" functions.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

RE: RE: Newline (NuBe Question)

[AVI GROSS via Python-list]

via Python-list
Sent: Saturday, November 25, 2023 9:32 AM
To: python-list@python.org
Subject: Re: RE: Newline (NuBe Question)

On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote:
> Grizz[l]y,
> 
> I think the point is not about a sorted list or sorting in general It is
> about reasons why maintaining a data structure such as a list in a program
> can be useful beyond printing things once. There are many possible
examples
> such as having a list of lists containing a record where the third item is
a
> GPA for the student and writing a little list comprehension that selects a
> smaller list containing only students who are Magna Cum Laude or Summa Cum
> Laude.
> 
> studs = [
>["Peter", 82, 3.53],
>["Paul", 77, 2.83],
>["Mary", 103, 3.82]
> ]

I've seen Mary, and she didn't look like a "stud" to me.

> Of course, for serious work, some might suggest avoiding constructs like a
> list of lists and switch to using modules and data structures [...]

Those who would recommend that approach do not appear to include Mr.
Rossum, who said:
   
   Avoid overengineering data structures. Tuples are better than
   objects (try namedtuple too though). Prefer simple fields over
   getter/setter functions... Built-in datatypes are your friends.
   Use more numbers, strings, tuples, lists, sets, dicts. Also
   check out the collections library, eps. deque.[1]
   
I was nodding along with the people saying "list of lists" until I
reread this quote. A list of tuples seems most appropriate to me.

   
[1] <https://gist.github.com/hemanth/3715502>, as quoted by Bill
Lubanovic in _Introducing Python_

-- 
Michael F. Stemper
This sentence no verb.

-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Roel Schroeven via Python-list]

Michael F. Stemper via Python-list schreef op 25/11/2023 om 15:32:

On 24/11/2023 21.45,avi.e.gr...@gmail.com  wrote:
> Grizz[l]y,
> 
> I think the point is not about a sorted list or sorting in general It is

> about reasons why maintaining a data structure such as a list in a program
> can be useful beyond printing things once. There are many possible examples
> such as having a list of lists containing a record where the third item is a
> GPA for the student and writing a little list comprehension that selects a
> smaller list containing only students who are Magna Cum Laude or Summa Cum
> Laude.
> 
> studs = [

>["Peter", 82, 3.53],
>["Paul", 77, 2.83],
>["Mary", 103, 3.82]
> ]

> Of course, for serious work, some might suggest avoiding constructs like a
> list of lists and switch to using modules and data structures [...]

Those who would recommend that approach do not appear to include Mr.
Rossum, who said:

Avoid overengineering data structures. Tuples are better than

objects (try namedtuple too though). Prefer simple fields over
getter/setter functions... Built-in datatypes are your friends.
Use more numbers, strings, tuples, lists, sets, dicts. Also
check out the collections library, eps. deque.[1]

I was nodding along with the people saying "list of lists" until I

reread this quote. A list of tuples seems most appropriate to me.


I prefer namedtuples or dataclasses over tuples. They allow you to refer 
to their fields by name instead of index: student.gpa is much clearer 
than student[2], and makes it less likely to accidentally refer to the 
wrong field.


--
"Man had always assumed that he was more intelligent than dolphins because
he had achieved so much — the wheel, New York, wars and so on — whilst all
the dolphins had ever done was muck about in the water having a good time.
But conversely, the dolphins had always believed that they were far more
intelligent than man — for precisely the same reasons."
-- Douglas Adams
--
https://mail.python.org/mailman/listinfo/python-list

Re: RE: Newline (NuBe Question)

[Chris Angelico via Python-list]

On Sun, 26 Nov 2023 at 21:08, Michael F. Stemper via Python-list
 wrote:
>
> On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote:
> > Grizz[l]y,
> >
> > I think the point is not about a sorted list or sorting in general It is
> > about reasons why maintaining a data structure such as a list in a program
> > can be useful beyond printing things once. There are many possible examples
> > such as having a list of lists containing a record where the third item is a
> > GPA for the student and writing a little list comprehension that selects a
> > smaller list containing only students who are Magna Cum Laude or Summa Cum
> > Laude.
> >
> > studs = [
> >["Peter", 82, 3.53],
> >["Paul", 77, 2.83],
> >["Mary", 103, 3.82]
> > ]
>
> I've seen Mary, and she didn't look like a "stud" to me.
>

That's what happens when you abbreviate "student" though :) Don't
worry, there's far FAR worse around the place, and juvenile brains
will always find things to snigger at, usually in mathematical
libraries with "cumulative" functions.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: RE: Newline (NuBe Question)

[Michael F. Stemper via Python-list]

On 24/11/2023 21.45, avi.e.gr...@gmail.com wrote:

Grizz[l]y,

I think the point is not about a sorted list or sorting in general It is
about reasons why maintaining a data structure such as a list in a program
can be useful beyond printing things once. There are many possible examples
such as having a list of lists containing a record where the third item is a
GPA for the student and writing a little list comprehension that selects a
smaller list containing only students who are Magna Cum Laude or Summa Cum
Laude.

studs = [
   ["Peter", 82, 3.53],
   ["Paul", 77, 2.83],
   ["Mary", 103, 3.82]
]


I've seen Mary, and she didn't look like a "stud" to me.


Of course, for serious work, some might suggest avoiding constructs like a
list of lists and switch to using modules and data structures [...]


Those who would recommend that approach do not appear to include Mr.
Rossum, who said:
  
  Avoid overengineering data structures. Tuples are better than

  objects (try namedtuple too though). Prefer simple fields over
  getter/setter functions... Built-in datatypes are your friends.
  Use more numbers, strings, tuples, lists, sets, dicts. Also
  check out the collections library, eps. deque.[1]
  
I was nodding along with the people saying "list of lists" until I

reread this quote. A list of tuples seems most appropriate to me.

  
[1] , as quoted by Bill

Lubanovic in _Introducing Python_

--
Michael F. Stemper
This sentence no verb.

--
https://mail.python.org/mailman/listinfo/python-list

RE: Newline (NuBe Question)

[AVI GROSS via Python-list]

Grizz[l]y,

I think the point is not about a sorted list or sorting in general It is
about reasons why maintaining a data structure such as a list in a program
can be useful beyond printing things once. There are many possible examples
such as having a list of lists containing a record where the third item is a
GPA for the student and writing a little list comprehension that selects a
smaller list containing only students who are Magna Cum Laude or Summa Cum
Laude. 

studs = [
  ["Peter", 82, 3.53],
  ["Paul", 77, 2.83],
  ["Mary", 103, 3.82] 
]
  
magna = [stud for stud in studs if stud[2] >= 3.5 ]
summa = [stud for stud in studs if stud[2] >= 3.75 ]

print(studs, magna, summa, sep="\n")

OUTPUT:

>>> print(studs, magna, summa, sep="\n")
[['Peter', 82, 3.53], ['Paul', 77, 2.83], ['Mary', 103, 3.82]]
[['Peter', 82, 3.53], ['Mary', 103, 3.82]]
[['Mary', 103, 3.82]]

Of course, for serious work, some might suggest avoiding constructs like a
list of lists and switch to using modules and data structures that are often
more efficient to represent your data such as some form of matrix or
data.frame.

And, yes, you can sort something like the above by name or GPA or number of
credits taken but the point was responding to why bother making a list just
to print it. The answer is that many and even most programs do a bit more
than that and a good choice of data structure facilitates ...




-Original Message-
From: Python-list  On
Behalf Of Grizzy Adams via Python-list
Sent: Thursday, November 16, 2023 8:41 AM
To: python-list@python.org
Subject: Re: Newline (NuBe Question)

Thursday, November 16, 2023  at 7:47, Thomas Passin via Python-list wrote:
Re: Newline (NuBe Question) (at least in part)

>I wrote that you don't need the "students" list, which is correct.  But 
>there could be a use for a list.  It would let you change the order in 
>which students appear in the printed output.  Knowing how to do that is 
>a useful skill.  But that should be left for a later lesson, not mixed 
>in here.

I have a vague memory of seeing sorted list somewhere ;->)
-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Rimu Atkinson via Python-list]

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


\b - a word boundary.
[0-9]{2,7} - 2 to 7 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
\b - a word boundary.

Seems quite straightforward to me. I'll be impressed if you can write
that in Python in a way which is easier to read.



Now that I know what {} does, you're right, that IS straightforward! 
Maybe 2023 will be the year I finally get off my arse and learn regex.


Thanks :)

--
https://mail.python.org/mailman/listinfo/python-list

RE: Code improvement question

[AVI GROSS via Python-list]

Many features like regular expressions can be mini languages that are designed 
to be very powerful while also a tad cryptic to anyone not familiar.

But consider an alternative in some languages that may use some complex set of 
nested function calls that each have names like match_white_space(2, 5) and 
even if some are set up to be sort of readable, they can be a pain. Quite a few 
problems can be solved nicely with a single regular expression or several in a 
row with each one being fairly simple. Sometimes you can do parts using some of 
the usual text manipulation functions built-in or in a module for either speed 
or to simplify things so that the RE part is simpler and easier to follow.

And, as noted, Python allows ways to include comments in RE or ways to specify 
extensions such as PERL-style and so on. Adding enough comments above or within 
the code can help remind people or point to a reference and just explaining in 
English (or the language of your choice that hopefully others later can 
understand) can be helpful. You can spell out in whatever level of detail what 
you expect your data to look like and what you want to match or extract and 
then the RE may be easier to follow.

Of course the endless extensions added due to things like supporting UNICODE 
have made some RE much harder to create or understand and sometimes the result 
may not even be what you expected if something strange happens like the symbols 
①❹⓸ 

The above might match digits and maybe be interpreted at some point as 12 
dozen, which may even be appropriate but a bit of a surprise perhaps.

-Original Message-
From: Python-list  On 
Behalf Of Peter J. Holzer via Python-list
Sent: Friday, November 17, 2023 6:18 AM
To: python-list@python.org
Subject: Re: Code improvement question

On 2023-11-16 11:34:16 +1300, Rimu Atkinson via Python-list wrote:
> > > Why don't you use re.findall?
> > > 
> > > re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)
> > 
> > I think I can see what you did there but it won't make sense to me - or
> > whoever looks at the code - in future.
> > 
> > That answers your specific question. However, I am in awe of people who
> > can just "do" regular expressions and I thank you very much for what
> > would have been a monumental effort had I tried it.
> 
> I feel the same way about regex. If I can find a way to write something
> without regex I very much prefer to as regex usually adds complexity and
> hurts readability.

I find "straight" regexps very easy to write. There are only a handful
of constructs which are all very simple and you just string them
together. But then I've used regexps for 30+ years, so of course they
feel natural to me.

(Reading regexps may be a bit harder, exactly because they are to
simple: There is no abstraction, so a complicated pattern results in a
long regexp.)

There are some extensions to regexps which are conceptually harder, like
lookahead and lookbehind or nested contexts in Perl. I may need the
manual for those (especially because they are new(ish) and every
language uses a different syntax for them) or avoid them altogether.

Oh, and Python (just like Perl) allows you to embed whitespace and
comments into Regexps, which helps readability a lot if you have to
write long regexps.


> You might find https://regex101.com/ to be useful for testing your regex.
> You can enter in sample data and see if it matches.
> 
> If I understood what your regex was trying to do I might be able to suggest
> some python to do the same thing. Is it just removing numbers from text?

Not "removing" them (as I understood it), but extracting them (i.e. find
and collect them).

> > > re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)

\b - a word boundary.
[0-9]{2,7} - 2 to 7 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
\b - a word boundary.

Seems quite straightforward to me. I'll be impressed if you can write
that in Python in a way which is easier to read.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[jak via Python-list]

MRAB ha scritto:

Bare excepts are a very bad idea.


I know, you're right but to test the CAS numbers were inside a string
(txt) and instead of the 'open(file)' there was 'io.StingIO(txt)' so the
risk was almost null. When I copied it here I didn't think about it.
Sorry.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[MRAB via Python-list]

On 2023-11-17 09:38, jak via Python-list wrote:

Mike Dewhirst ha scritto:

On 15/11/2023 10:25 am, MRAB via Python-list wrote:

On 2023-11-14 23:14, Mike Dewhirst via Python-list wrote:

I'd like to improve the code below, which works. It feels clunky to me.

I need to clean up user-uploaded files the size of which I don't know in
advance.

After cleaning they might be as big as 1Mb but that would be super rare.
Perhaps only for testing.

I'm extracting CAS numbers and here is the pattern xx-xx-x up to
xxx-xx-x eg., 1012300-77-4

def remove_alpha(txt):

      """  r'[^0-9\- ]':

      [^...]: Match any character that is not in the specified set.

      0-9: Match any digit.

      \: Escape character.

      -: Match a hyphen.

      Space: Match a space.

      """

  cleaned_txt = re.sub(r'[^0-9\- ]', '', txt)

      bits = cleaned_txt.split()

      pieces = []

      for bit in bits:

      # minimum size of a CAS number is 7 so drop smaller clumps 
of digits


      pieces.append(bit if len(bit) > 6 else "")

      return " ".join(pieces)


Many thanks for any hints


Why don't you use re.findall?

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


I think I can see what you did there but it won't make sense to me - or 
whoever looks at the code - in future.


That answers your specific question. However, I am in awe of people who 
can just "do" regular expressions and I thank you very much for what 
would have been a monumental effort had I tried it.


That little re.sub() came from ChatGPT and I can understand it without 
too much effort because it came documented


I suppose ChatGPT is the answer to this thread. Or everything. Or will be.

Thanks

Mike


I respect your opinion but from the point of view of many usenet users
asking a question to chatgpt to solve your problem is truly an overkill.
The computer world overflows with people who know regex. If you had not
already had the answer with the use of 're' I would have sent you my
suggestion that as you can see it is practically identical. I am quite
sure that in this usenet the same solution came to the mind of many
people.

with open(file) as fp:
  try: ret = re.findall(r'\b\d{2,7}\-\d{2}\-\d{1}\b', fp.read())
  except: ret = []

The only difference is '\d' instead of '[0-9]' but they are equivalent.


Bare excepts are a very bad idea.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[jak via Python-list]

Mike Dewhirst ha scritto:

On 15/11/2023 10:25 am, MRAB via Python-list wrote:

On 2023-11-14 23:14, Mike Dewhirst via Python-list wrote:

I'd like to improve the code below, which works. It feels clunky to me.

I need to clean up user-uploaded files the size of which I don't know in
advance.

After cleaning they might be as big as 1Mb but that would be super rare.
Perhaps only for testing.

I'm extracting CAS numbers and here is the pattern xx-xx-x up to
xxx-xx-x eg., 1012300-77-4

def remove_alpha(txt):

      """  r'[^0-9\- ]':

      [^...]: Match any character that is not in the specified set.

      0-9: Match any digit.

      \: Escape character.

      -: Match a hyphen.

      Space: Match a space.

      """

  cleaned_txt = re.sub(r'[^0-9\- ]', '', txt)

      bits = cleaned_txt.split()

      pieces = []

      for bit in bits:

      # minimum size of a CAS number is 7 so drop smaller clumps 
of digits


      pieces.append(bit if len(bit) > 6 else "")

      return " ".join(pieces)


Many thanks for any hints


Why don't you use re.findall?

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


I think I can see what you did there but it won't make sense to me - or 
whoever looks at the code - in future.


That answers your specific question. However, I am in awe of people who 
can just "do" regular expressions and I thank you very much for what 
would have been a monumental effort had I tried it.


That little re.sub() came from ChatGPT and I can understand it without 
too much effort because it came documented


I suppose ChatGPT is the answer to this thread. Or everything. Or will be.

Thanks

Mike


I respect your opinion but from the point of view of many usenet users
asking a question to chatgpt to solve your problem is truly an overkill.
The computer world overflows with people who know regex. If you had not
already had the answer with the use of 're' I would have sent you my
suggestion that as you can see it is practically identical. I am quite
sure that in this usenet the same solution came to the mind of many
people.

with open(file) as fp:
try: ret = re.findall(r'\b\d{2,7}\-\d{2}\-\d{1}\b', fp.read())
except: ret = []

The only difference is '\d' instead of '[0-9]' but they are equivalent.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Thomas Passin via Python-list]

On 11/17/2023 9:46 AM, Peter J. Holzer via Python-list wrote:

On 2023-11-17 07:48:41 -0500, Thomas Passin via Python-list wrote:

On 11/17/2023 6:17 AM, Peter J. Holzer via Python-list wrote:

Oh, and Python (just like Perl) allows you to embed whitespace and
comments into Regexps, which helps readability a lot if you have to
write long regexps.


[...]

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


\b - a word boundary.
[0-9]{2,7} - 2 to 7 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
\b - a word boundary.

Seems quite straightforward to me. I'll be impressed if you can write
that in Python in a way which is easier to read.


And the re.VERBOSE (also re.X) flag can always be used so the entire
expression can be written line-by-line with comments nearly the same
as the example above


Yes. That's what I alluded to above.


I know, and I just wanted to make it explicit for people who didn't know 
much about Python regexes.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Peter J. Holzer via Python-list]

On 2023-11-17 07:48:41 -0500, Thomas Passin via Python-list wrote:
> On 11/17/2023 6:17 AM, Peter J. Holzer via Python-list wrote:
> > Oh, and Python (just like Perl) allows you to embed whitespace and
> > comments into Regexps, which helps readability a lot if you have to
> > write long regexps.
> > 
[...]
> > > > > re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)
> > 
> > \b - a word boundary.
> > [0-9]{2,7} - 2 to 7 digits
> > -  - a hyphen-minus
> > [0-9]{2}   - exactly 2 digits
> > -  - a hyphen-minus
> > [0-9]{2}   - exactly 2 digits
> > \b - a word boundary.
> > 
> > Seems quite straightforward to me. I'll be impressed if you can write
> > that in Python in a way which is easier to read.
> 
> And the re.VERBOSE (also re.X) flag can always be used so the entire
> expression can be written line-by-line with comments nearly the same
> as the example above

Yes. That's what I alluded to above.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Thomas Passin via Python-list]

On 11/17/2023 6:17 AM, Peter J. Holzer via Python-list wrote:

On 2023-11-16 11:34:16 +1300, Rimu Atkinson via Python-list wrote:

Why don't you use re.findall?

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


I think I can see what you did there but it won't make sense to me - or
whoever looks at the code - in future.

That answers your specific question. However, I am in awe of people who
can just "do" regular expressions and I thank you very much for what
would have been a monumental effort had I tried it.


I feel the same way about regex. If I can find a way to write something
without regex I very much prefer to as regex usually adds complexity and
hurts readability.


I find "straight" regexps very easy to write. There are only a handful
of constructs which are all very simple and you just string them
together. But then I've used regexps for 30+ years, so of course they
feel natural to me.

(Reading regexps may be a bit harder, exactly because they are to
simple: There is no abstraction, so a complicated pattern results in a
long regexp.)

There are some extensions to regexps which are conceptually harder, like
lookahead and lookbehind or nested contexts in Perl. I may need the
manual for those (especially because they are new(ish) and every
language uses a different syntax for them) or avoid them altogether.

Oh, and Python (just like Perl) allows you to embed whitespace and
comments into Regexps, which helps readability a lot if you have to
write long regexps.



You might find https://regex101.com/ to be useful for testing your regex.
You can enter in sample data and see if it matches.

If I understood what your regex was trying to do I might be able to suggest
some python to do the same thing. Is it just removing numbers from text?


Not "removing" them (as I understood it), but extracting them (i.e. find
and collect them).


re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


\b - a word boundary.
[0-9]{2,7} - 2 to 7 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
\b - a word boundary.

Seems quite straightforward to me. I'll be impressed if you can write
that in Python in a way which is easier to read.


And the re.VERBOSE (also re.X) flag can always be used so the entire 
expression can be written line-by-line with comments nearly the same as 
the example above


--
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Peter J. Holzer via Python-list]

On 2023-11-16 11:34:16 +1300, Rimu Atkinson via Python-list wrote:
> > > Why don't you use re.findall?
> > > 
> > > re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)
> > 
> > I think I can see what you did there but it won't make sense to me - or
> > whoever looks at the code - in future.
> > 
> > That answers your specific question. However, I am in awe of people who
> > can just "do" regular expressions and I thank you very much for what
> > would have been a monumental effort had I tried it.
> 
> I feel the same way about regex. If I can find a way to write something
> without regex I very much prefer to as regex usually adds complexity and
> hurts readability.

I find "straight" regexps very easy to write. There are only a handful
of constructs which are all very simple and you just string them
together. But then I've used regexps for 30+ years, so of course they
feel natural to me.

(Reading regexps may be a bit harder, exactly because they are to
simple: There is no abstraction, so a complicated pattern results in a
long regexp.)

There are some extensions to regexps which are conceptually harder, like
lookahead and lookbehind or nested contexts in Perl. I may need the
manual for those (especially because they are new(ish) and every
language uses a different syntax for them) or avoid them altogether.

Oh, and Python (just like Perl) allows you to embed whitespace and
comments into Regexps, which helps readability a lot if you have to
write long regexps.


> You might find https://regex101.com/ to be useful for testing your regex.
> You can enter in sample data and see if it matches.
> 
> If I understood what your regex was trying to do I might be able to suggest
> some python to do the same thing. Is it just removing numbers from text?

Not "removing" them (as I understood it), but extracting them (i.e. find
and collect them).

> > > re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)

\b - a word boundary.
[0-9]{2,7} - 2 to 7 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
-  - a hyphen-minus
[0-9]{2}   - exactly 2 digits
\b - a word boundary.

Seems quite straightforward to me. I'll be impressed if you can write
that in Python in a way which is easier to read.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Mike Dewhirst via Python-list]

On 16/11/2023 9:34 am, Rimu Atkinson via Python-list wrote:





Why don't you use re.findall?

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


I think I can see what you did there but it won't make sense to me - 
or whoever looks at the code - in future.


That answers your specific question. However, I am in awe of people 
who can just "do" regular expressions and I thank you very much for 
what would have been a monumental effort had I tried it.


I feel the same way about regex. If I can find a way to write 
something without regex I very much prefer to as regex usually adds 
complexity and hurts readability.


You might find https://regex101.com/ to be useful for testing your 
regex. You can enter in sample data and see if it matches.


If I understood what your regex was trying to do I might be able to 
suggest some python to do the same thing. Is it just removing numbers 
from text?


The for loop, "for bit in bits" etc, could be written as a list 
comprehension.


pieces = [bit if len(bit) > 6 else "" for bit in bits]

For devs familiar with other languages but new to Python this will 
look like gibberish so arguably the original for loop is clearer, 
depending on your team.


It's worth making the effort to get into list comprehensions though 
because they're awesome.


I agree qualitatively 100% but quantitively perhaps I agree 80% where 
readability is easy.


I think that's what you are saying anyway.








That little re.sub() came from ChatGPT and I can understand it 
without too much effort because it came documented


I suppose ChatGPT is the answer to this thread. Or everything. Or 
will be.


I am doubtful. We'll see!

R





--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Your
email software can handle signing.



OpenPGP_signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Rimu Atkinson via Python-list]

Why don't you use re.findall?

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


I think I can see what you did there but it won't make sense to me - or 
whoever looks at the code - in future.


That answers your specific question. However, I am in awe of people who 
can just "do" regular expressions and I thank you very much for what 
would have been a monumental effort had I tried it.


I feel the same way about regex. If I can find a way to write something 
without regex I very much prefer to as regex usually adds complexity and 
hurts readability.


You might find https://regex101.com/ to be useful for testing your 
regex. You can enter in sample data and see if it matches.


If I understood what your regex was trying to do I might be able to 
suggest some python to do the same thing. Is it just removing numbers 
from text?


The for loop, "for bit in bits" etc, could be written as a list 
comprehension.


pieces = [bit if len(bit) > 6 else "" for bit in bits]

For devs familiar with other languages but new to Python this will look 
like gibberish so arguably the original for loop is clearer, depending 
on your team.


It's worth making the effort to get into list comprehensions though 
because they're awesome.






That little re.sub() came from ChatGPT and I can understand it without 
too much effort because it came documented


I suppose ChatGPT is the answer to this thread. Or everything. Or will be.


I am doubtful. We'll see!

R


--
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[MRAB via Python-list]

On 2023-11-17 01:15, Mike Dewhirst via Python-list wrote:

On 15/11/2023 3:08 pm, MRAB via Python-list wrote:

On 2023-11-15 03:41, Mike Dewhirst via Python-list wrote:

On 15/11/2023 10:25 am, MRAB via Python-list wrote:

On 2023-11-14 23:14, Mike Dewhirst via Python-list wrote:
I'd like to improve the code below, which works. It feels clunky to 
me.


I need to clean up user-uploaded files the size of which I don't 
know in

advance.

After cleaning they might be as big as 1Mb but that would be super 
rare.

Perhaps only for testing.

I'm extracting CAS numbers and here is the pattern xx-xx-x up to
xxx-xx-x eg., 1012300-77-4

def remove_alpha(txt):

      """  r'[^0-9\- ]':

      [^...]: Match any character that is not in the specified set.

      0-9: Match any digit.

      \: Escape character.

      -: Match a hyphen.

      Space: Match a space.

      """

  cleaned_txt = re.sub(r'[^0-9\- ]', '', txt)

      bits = cleaned_txt.split()

      pieces = []

      for bit in bits:

      # minimum size of a CAS number is 7 so drop smaller 
clumps of digits


      pieces.append(bit if len(bit) > 6 else "")

      return " ".join(pieces)


Many thanks for any hints


Why don't you use re.findall?

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


I think I can see what you did there but it won't make sense to me - or
whoever looks at the code - in future.

That answers your specific question. However, I am in awe of people who
can just "do" regular expressions and I thank you very much for what
would have been a monumental effort had I tried it.

That little re.sub() came from ChatGPT and I can understand it without
too much effort because it came documented

I suppose ChatGPT is the answer to this thread. Or everything. Or 
will be.



\b  Word boundary
[0-9]{2,7}  2..7 digits
-   "-"
[0-9]{2}    2 digits
-   "-"
[0-9]{2}    2 digits
\b  Word boundary

The "word boundary" thing is to stop it matching where there are 
letters or digits right next to the digits.


For example, if the text contained, say, "123456789-12-1234", you 
wouldn't want it to match because there are more than 7 digits at the 
start and more than 2 digits at the end.



Thanks

I know I should invest some brainspace in re. Many years ago at a Perl
conferenceI did buy a coffee mug completely covered with a regex cheat
sheet. It currently holds pens and pencils on my desk. And spiders now I
look closely!

Then I took up Python and re is different.

Maybe I'll have another look ...

The patterns themselves aren't that different; Perl's just has more 
features than the re module's.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Mike Dewhirst via Python-list]

On 15/11/2023 3:08 pm, MRAB via Python-list wrote:

On 2023-11-15 03:41, Mike Dewhirst via Python-list wrote:

On 15/11/2023 10:25 am, MRAB via Python-list wrote:

On 2023-11-14 23:14, Mike Dewhirst via Python-list wrote:
I'd like to improve the code below, which works. It feels clunky to 
me.


I need to clean up user-uploaded files the size of which I don't 
know in

advance.

After cleaning they might be as big as 1Mb but that would be super 
rare.

Perhaps only for testing.

I'm extracting CAS numbers and here is the pattern xx-xx-x up to
xxx-xx-x eg., 1012300-77-4

def remove_alpha(txt):

      """  r'[^0-9\- ]':

      [^...]: Match any character that is not in the specified set.

      0-9: Match any digit.

      \: Escape character.

      -: Match a hyphen.

      Space: Match a space.

      """

  cleaned_txt = re.sub(r'[^0-9\- ]', '', txt)

      bits = cleaned_txt.split()

      pieces = []

      for bit in bits:

      # minimum size of a CAS number is 7 so drop smaller 
clumps of digits


      pieces.append(bit if len(bit) > 6 else "")

      return " ".join(pieces)


Many thanks for any hints


Why don't you use re.findall?

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


I think I can see what you did there but it won't make sense to me - or
whoever looks at the code - in future.

That answers your specific question. However, I am in awe of people who
can just "do" regular expressions and I thank you very much for what
would have been a monumental effort had I tried it.

That little re.sub() came from ChatGPT and I can understand it without
too much effort because it came documented

I suppose ChatGPT is the answer to this thread. Or everything. Or 
will be.



\b  Word boundary
[0-9]{2,7}  2..7 digits
-   "-"
[0-9]{2}    2 digits
-   "-"
[0-9]{2}    2 digits
\b  Word boundary

The "word boundary" thing is to stop it matching where there are 
letters or digits right next to the digits.


For example, if the text contained, say, "123456789-12-1234", you 
wouldn't want it to match because there are more than 7 digits at the 
start and more than 2 digits at the end.



Thanks

I know I should invest some brainspace in re. Many years ago at a Perl 
conferenceI did buy a coffee mug completely covered with a regex cheat 
sheet. It currently holds pens and pencils on my desk. And spiders now I 
look closely!


Then I took up Python and re is different.

Maybe I'll have another look ...

Cheers

Mike

--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Your
email software can handle signing.



OpenPGP_signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Grizzy Adams via Python-list]

Thursday, November 16, 2023  at 7:47, Thomas Passin via Python-list wrote:
Re: Newline (NuBe Question) (at least in part)

>I wrote that you don't need the "students" list, which is correct.  But 
>there could be a use for a list.  It would let you change the order in 
>which students appear in the printed output.  Knowing how to do that is 
>a useful skill.  But that should be left for a later lesson, not mixed 
>in here.

I have a vague memory of seeing sorted list somewhere ;->)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Thomas Passin via Python-list]

On 11/16/2023 1:19 AM, Grizzy Adams via Python-list wrote:

Wednesday, November 15, 2023  at 15:54, Thomas Passin via Python-list wrote:
Re: Newline (NuBe Question) (at least in part)


On 11/15/2023 2:04 PM, Grizzy Adams via Python-list wrote:

Wednesday, November 15, 2023  at 12:19, Pierre Fortin wrote:
Re: Newline (NuBe Question) (at least in part)
  

On Wed, 15 Nov 2023 16:51:09 - Grizzy Adams via Python-list wrote:



I don't give solutions; just a nudge...  you appear not to fully grok
"list"; your list is ONE list with no delineation between students. You
want a "list of lists"...
  

['Example High', 'Mary', 89.6, 'Pass', 'Example High', 'Matthew', 76.5, 'Fail', 
'Example High', 'Marie', 80.4, 'Fail', 'Example High', 'Manuel', 79.6, 'Fail', 
'Example High', 'Malala', 98.9, 'Pass']



Like this:
  

students = [
 ['Example High', 'Mary', 89.6, 'Pass'],
 ['Example High','Matthew', 76.5, 'Fail'],
 ['Example High', 'Marie', 80.4, 'Fail'],
 ['Example High', 'Manuel', 79.6, 'Fail'],
 ['Example High', 'Malala', 98.9, 'Pass']
]
  

for now I made a copt of code and altered to

students = []
grades = []



# In this design there is no point in the extra loop.
# also, the indentation is wrong.  Perhaps you inserted
# tabs?  Use only spaces in these posts.


I copy-pasted the code direct from IDLE, (to avoid any other typo's)


# Also you don't need the students list

for student in geographyClass:
# students.append(geographyStudent(s))

 s = geographyStudent(student)



  # for s in students:

   if s.finalGrade()>82: Result=("Pass")
   else: Result=("Fail")
   print(s.school, s.name, s.finalGrade(),Result)


I'll hive this a try (as a learning point)


I wrote that you don't need the "students" list, which is correct.  But 
there could be a use for a list.  It would let you change the order in 
which students appear in the printed output.  Knowing how to do that is 
a useful skill.  But that should be left for a later lesson, not mixed 
in here.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Grizzy Adams via Python-list]

Wednesday, November 15, 2023  at 15:54, Thomas Passin via Python-list wrote:
Re: Newline (NuBe Question) (at least in part)

>On 11/15/2023 2:04 PM, Grizzy Adams via Python-list wrote:
>> Wednesday, November 15, 2023  at 12:19, Pierre Fortin wrote:
>> Re: Newline (NuBe Question) (at least in part)
 
>>> On Wed, 15 Nov 2023 16:51:09 - Grizzy Adams via Python-list wrote:

>>> I don't give solutions; just a nudge...  you appear not to fully grok
>>> "list"; your list is ONE list with no delineation between students. You
>>> want a "list of lists"...
 
>>>> ['Example High', 'Mary', 89.6, 'Pass', 'Example High', 'Matthew', 76.5, 
>>>> 'Fail', 'Example High', 'Marie', 80.4, 'Fail', 'Example High', 'Manuel', 
>>>> 79.6, 'Fail', 'Example High', 'Malala', 98.9, 'Pass']

>>> Like this:
 
>>> students = [
>>> ['Example High', 'Mary', 89.6, 'Pass'],
>>> ['Example High','Matthew', 76.5, 'Fail'],
>>> ['Example High', 'Marie', 80.4, 'Fail'],
>>> ['Example High', 'Manuel', 79.6, 'Fail'],
>>> ['Example High', 'Malala', 98.9, 'Pass']
>>> ]
 
>> for now I made a copt of code and altered to
>> 
>> students = []
>> grades = []

># In this design there is no point in the extra loop.
># also, the indentation is wrong.  Perhaps you inserted
># tabs?  Use only spaces in these posts.

I copy-pasted the code direct from IDLE, (to avoid any other typo's) 

># Also you don't need the students list
>> for student in geographyClass:
>>  # students.append(geographyStudent(s))
> s = geographyStudent(student)
>> 
>  # for s in students:
>>   if s.finalGrade()>82: Result=("Pass")
>>   else: Result=("Fail")
>>   print(s.school, s.name, s.finalGrade(),Result)

I'll hive this a try (as a learning point)

Thanks
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Thomas Passin via Python-list]

On 11/15/2023 2:04 PM, Grizzy Adams via Python-list wrote:

Wednesday, November 15, 2023  at 12:19, Pierre Fortin wrote:
Re: Newline (NuBe Question) (at least in part)


On Wed, 15 Nov 2023 16:51:09 - Grizzy Adams via Python-list wrote:

I don't give solutions; just a nudge...  you appear not to fully grok
"list"; your list is ONE list with no delineation between students. You
want a "list of lists"...



['Example High', 'Mary', 89.6, 'Pass', 'Example High', 'Matthew', 76.5, 'Fail', 
'Example High', 'Marie', 80.4, 'Fail', 'Example High', 'Manuel', 79.6, 'Fail', 
'Example High', 'Malala', 98.9, 'Pass']



Like this:



students = [
['Example High', 'Mary', 89.6, 'Pass'],
['Example High','Matthew', 76.5, 'Fail'],
['Example High', 'Marie', 80.4, 'Fail'],
['Example High', 'Manuel', 79.6, 'Fail'],
['Example High', 'Malala', 98.9, 'Pass']
]


for now I made a copt of code and altered to

students = []
grades = []


# In this design there is no point in the extra loop.
# also, the indentation is wrong.  Perhaps you inserted
# tabs?  Use only spaces in these posts.
# Also you don't need the students list

for student in geographyClass:
# students.append(geographyStudent(s))

s = geographyStudent(student)



 # for s in students:

  if s.finalGrade()>82: Result=("Pass")
  else: Result=("Fail")
  print(s.school, s.name, s.finalGrade(),Result)


This may help get you headed in the right direction:



for s in students:
print( s )



Hint: look forward to learning about f-strings...


I will look forward to them, may even go search ahead,


--
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Grizzy Adams via Python-list]

Wednesday, November 15, 2023  at 12:19, Pierre Fortin wrote:
Re: Newline (NuBe Question) (at least in part)

>On Wed, 15 Nov 2023 16:51:09 - Grizzy Adams via Python-list wrote:
>
>I don't give solutions; just a nudge...  you appear not to fully grok
>"list"; your list is ONE list with no delineation between students. You
>want a "list of lists"...

>>['Example High', 'Mary', 89.6, 'Pass', 'Example High', 'Matthew', 76.5, 
>>'Fail', 'Example High', 'Marie', 80.4, 'Fail', 'Example High', 'Manuel', 
>>79.6, 'Fail', 'Example High', 'Malala', 98.9, 'Pass']

>Like this:

>students = [
>['Example High', 'Mary', 89.6, 'Pass'],
>['Example High','Matthew', 76.5, 'Fail'],
>['Example High', 'Marie', 80.4, 'Fail'],
>['Example High', 'Manuel', 79.6, 'Fail'],
>['Example High', 'Malala', 98.9, 'Pass']
>]

for now I made a copt of code and altered to 

students = []
grades = []
for s in geographyClass:
students.append(geographyStudent(s))

for s in students:
 if s.finalGrade()>82: Result=("Pass")
   else: Result=("Fail")
   print(s.school, s.name, s.finalGrade(),Result)

>This may help get you headed in the right direction:

>for s in students:
>print( s )

>Hint: look forward to learning about f-strings...

I will look forward to them, may even go search ahead, 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Pierre Fortin via Python-list]

On Wed, 15 Nov 2023 16:51:09 - Grizzy Adams via Python-list wrote:

I don't give solutions; just a nudge...  you appear not to fully grok
"list"; your list is ONE list with no delineation between students. You
want a "list of lists"...

>['Example High', 'Mary', 89.6, 'Pass', 'Example High', 'Matthew', 76.5, 
>'Fail', 'Example High', 'Marie', 80.4, 'Fail', 'Example High', 'Manuel', 79.6, 
>'Fail', 'Example High', 'Malala', 98.9, 'Pass']

Like this:

students = [
['Example High', 'Mary', 89.6, 'Pass'],
['Example High','Matthew', 76.5, 'Fail'],
['Example High', 'Marie', 80.4, 'Fail'],
['Example High', 'Manuel', 79.6, 'Fail'],
['Example High', 'Malala', 98.9, 'Pass']
]

This may help get you headed in the right direction:

for s in students:
print( s )

Hint: look forward to learning about f-strings...

HTH,
Pierre
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Grizzy Adams via Python-list]

Wednesday, November 15, 2023  at 9:45, Thomas Passin via Python-list wrote:
Re: Newline (NuBe Question) (at least in part)

>On 11/15/2023 2:25 AM, Grizzy Adams via Python-list wrote:
>> Hi & thanks for patience with what could be simple to you

>You may see responses that suggest various code alternatives.  But you 
>haven't shown us an example of what you want the output to look like,

Offered code got closer (sort of) my old code gave on long (only 5 records so 
far) list,

['Example High', 'Mary', 89.6, 'Pass', 'Example High', 'Matthew', 76.5, 'Fail', 
'Example High', 'Marie', 80.4, 'Fail', 'Example High', 'Manuel', 79.6, 'Fail', 
'Example High', 'Malala', 98.9, 'Pass']

offered code gave one tall column,

Example High
Mary
89.6
Pass
Example High
Matthew
76.5
Fail
Example High
Marie
80.4
Fail
Example High
Manuel
79.6
Fail
Example High
Malala
98.9
Pass

my ideal is one row for each student (I had edited manually to show this)

Example High, Mary, 89.6, Pass
Example High, Matthew, 76.5, Fail
Example High, Marie, 80.4, Fail
Example High, Manuel, 79.6, Fail
Example High, Malala, 98.9, Pass
 
>and you haven't said what else you plan to use the list for.  So anyone 
>who responds has to fly blind, without knowing key information.

I'll keep list for a while in case it gets used or reused later, for now it's 
just a test bed along with a few others, I can work thru as I learn 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Thomas Passin via Python-list]

On 11/15/2023 2:25 AM, Grizzy Adams via Python-list wrote:

Hi & thanks for patience with what could be simple to you

Have this (from an online "classes" tutorial)

--- Start Code Snippit  ---

students = []
grades = []
for s in geographyClass:
students.append(geographyStudent(s))
for s in students:
 grades.append(s.school)
 grades.append(s.name)
 grades.append(s.finalGrade())
 if s.finalGrade()>82:
 grades.append("Pass")
 else:
 grades.append("Fail")
print(grades)

--- End Code Snippit  ---

I have extended (from tutorial) it a bit, I would really like to have a newline

at end of each record, I have searched (and tested) but cant get "\n" to give a

newline, I get "Mydata\n"

Do I need to replace "append" with "print", or is there a way to get the
newline in as I append to list?


First of all, if this is an accurate representation of the course 
material, you need a better course.  There's no sense in appending all 
those values one after another in a single list since later it will be 
very inconvenient to detect the end of one student's info and the start 
of the next one's.  And if you don't need to know that, but just want to 
print out the data, you don't need to a list at all, just print it out 
in the loop.


A list that contains lists of each student's data, one per interior 
list, would make more sense.


Second, it is usual to append data to a list without print formatting, 
and then add your formatting when you go to print the list.  That way 
you can use the list for other things beyond just printing, and the code 
is clearer and simpler as well.


You may see responses that suggest various code alternatives.  But you 
haven't shown us an example of what you want the output to look like, 
and you haven't said what else you plan to use the list for.  So anyone 
who responds has to fly blind, without knowing key information.


Asking for help is like writing code, with an added social element.  You 
have to be clear about the requirements, inputs, and desired outputs, 
and you have to organize your request in a way that's easy for others to 
understand and be willing to help.  Your original post is partway there 
already.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Grizzy Adams via Python-list]

Wednesday, November 15, 2023  at 9:50, Alan Gauld via Python-list wrote:
Re: Newline (NuBe Question) (at least in part)

>On 15/11/2023 07:25, Grizzy Adams via Python-list wrote:

>> for s in students:
>> grades.append(s.school)
>> grades.append(s.name)
>> grades.append(s.finalGrade())
>> if s.finalGrade()>82:
>> grades.append("Pass")
>> else:
>> grades.append("Fail")
>> print(grades)
>> 
>> --- End Code Snippit  ---

>> Do I need to replace "append" with "print", or is there a way to get the 
>> newline in as I append to list?

>Firstly, it is usually a bad idea to mix formatting features(like
>newline) with the data. You will need to remove them again if you want
>to work with the data itself.

True, I onlt went that way when my (vain) attempts to add a newline any other 
way, my usual language is VB/VBA, Delphi or C++ at a push, so I'm really a nube 
here

>So, better to print the raw data and add the formatting during printing.

>There are a couple of options here (well more than a couple actually!)
>The simplest is to create another for loop and print each field with a
>newline automatically added by print()

>Another is to simply join everything in grades together separated by
>newlines. Python has a method to do that called join():

>print('\n'.join(grades))

>Unfortunately it seems your data has a mix of strings and numbers so
>that won't work without some tweaks:

>print('\n'.join(str(f) for f in grades))

that gets closer (sort of) my old code gave on long (only 5 records so far) 
list,

['Example High', 'Mary', 89.6, 'Pass', 'Example High', 'Matthew', 76.5, 'Fail', 
'Example High', 'Marie', 80.4, 'Fail', 'Example High', 'Manuel', 79.6, 'Fail', 
'Example High', 'Malala', 98.9, 'Pass']

your code gives one tall column, 

Example High
Mary
89.6
Pass
Example High
Matthew
76.5
Fail
Example High
Marie
80.4
Fail
Example High
Manuel
79.6
Fail
Example High
Malala
98.9
Pass

my ideal one row for each student (this I have edited manually)

Example High, Mary, 89.6, Pass
Example High, Matthew, 76.5, Fail
Example High, Marie, 80.4, Fail
Example High, Manuel, 79.6, Fail
Example High, Malala, 98.9, Pass

>However, I wonder if this really what you want? You have created grades as a
>long list containing all of the attributes of all of the students plus their
>Pass/Fail status. But you have no (easy)way to access the Pass/Fail value for
>each student. Do you really want to store the Pass/Fail in the student? And
>then print the students? Like so: 

I do want to keep the data in tact, incase it gets reused later in the 
tutorial(s)

>Just a thought...

>PS. There are neater ways to do this but you may not have covered
>those yet so I'll stick to basics.

I already jumped forward a bit (to python Classes)

Thanks
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Cameron Simpson via Python-list]

On 15Nov2023 07:25, Grizzy Adams  wrote:

Have this (from an online "classes" tutorial)


Response inline below.


students = []
grades = []
for s in geographyClass:
students.append(geographyStudent(s))
for s in students:
   grades.append(s.school)
   grades.append(s.name)
   grades.append(s.finalGrade())
   if s.finalGrade()>82:
   grades.append("Pass")
   else:
   grades.append("Fail")
print(grades)

--- End Code Snippit  ---

I have extended (from tutorial) it a bit, I would really like to have a newline
at end of each record, I have searched (and tested) but cant get "\n" 
to give a newline, I get "Mydata\n"


It would be useful to:
- see some of the things you've tried
- what their output actually was
- what output you actually want to achieve


Do I need to replace "append" with "print", or is there a way to get the
newline in as I append to list?


I think you're confusing output (print) with your data (the list of 
grades).


Is the code above genuinely what you're running?

I ask because it looks to me that you're:
- appending all the individual grade fields (school, name, ...) to one 
  long grades list containing all the data from all the students

- you're printing that single enormous list in one go at the end

What I'm imagine you want is one line of grade information per student.

Remember that indentation is important in Python. You're grades code 
looks like this:


grades = []
for s in students:
grades.append(s.school)
grades.append(s.name)
grades.append(s.finalGrade())
if s.finalGrade()>82:
grades.append("Pass")
else:
grades.append("Fail")
print(grades)

This:
- makes an empty list
- gathers up all of the student data
- prints the data in one go

I think you may want to do the first and last steps on a per student 
basis, not just once at the start and the end. So you might want to 
rearrange things:


for s in students:
grades = []
grades.append(s.school)
grades.append(s.name)
grades.append(s.finalGrade())
if s.finalGrade()>82:
grades.append("Pass")
else:
grades.append("Fail")
print(grades)

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[Alan Gauld via Python-list]

On 15/11/2023 07:25, Grizzy Adams via Python-list wrote:

> for s in students:
> grades.append(s.school)
> grades.append(s.name)
> grades.append(s.finalGrade())
> if s.finalGrade()>82:
> grades.append("Pass")
> else:
> grades.append("Fail")
> print(grades)
> 
> --- End Code Snippit  ---

> Do I need to replace "append" with "print", or is there a way to get the 
> newline in as I append to list?

Firstly, it is usually a bad idea to mix formatting features(like
newline) with the data. You will need to remove them again if you want
to work with the data itself.

So, better to print the raw data and add the formatting during printing.

There are a couple of options here (well more than a couple actually!)
The simplest is to create another for loop and print each field with a
newline automatically added by print()

Another is to simply join everything in grades together separated by
newlines. Python has a method to do that called join():

print('\n'.join(grades))

Unfortunately it seems your data has a mix of strings and numbers so
that won't work without some tweaks:

print('\n'.join(str(f) for f in grades))


However, I wonder if this really what you want? You have created grades
as a long list containing all of the attributes of all of the students
plus their Pass/Fail status. But you have no (easy)way to access the
Pass/Fail value for each student. Do you really want to store the
Pass/Fail in the student? And then print the students? Like so:

for s in students
if s.finalGrade() > 82:
   s.result = "Pass"
else:
   s.result = "Fail"
print(s.school)
print(s.name)
...
print(s.result)

Just a thought...

PS. There are neater ways to do this but you may not have covered
those yet so I'll stick to basics.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline (NuBe Question)

[dn via Python-list]

On 15/11/2023 20.25, Grizzy Adams via Python-list wrote:

Hi & thanks for patience with what could be simple to you

Have this (from an online "classes" tutorial)


There are lots of on-line classes!



--- Start Code Snippit  ---

students = []
grades = []
for s in geographyClass:
students.append(geographyStudent(s))
for s in students:
 grades.append(s.school)
 grades.append(s.name)
 grades.append(s.finalGrade())
 if s.finalGrade()>82:
 grades.append("Pass")
 else:
 grades.append("Fail")
print(grades)

--- End Code Snippit  ---

I have extended (from tutorial) it a bit, I would really like to have a newline

at end of each record, I have searched (and tested) but cant get "\n" to give a

newline, I get "Mydata\n"

Do I need to replace "append" with "print", or is there a way to get the
newline in as I append to list?


Don't know how "Mydata..." results - where is it in the code.

What do you see when grades is printed?
Do you really want data-values all mashed together?

Yes, what changed after removal of all the .append()-s, and instead, 
within the (second) for-loop print( school, name, ... ) was used?


Is it easier to go on from there?

--
Regards,
=dn
--
https://mail.python.org/mailman/listinfo/python-list

Newline (NuBe Question)

[Grizzy Adams via Python-list]

Hi & thanks for patience with what could be simple to you

Have this (from an online "classes" tutorial) 

--- Start Code Snippit  ---

students = []
grades = []
for s in geographyClass:
students.append(geographyStudent(s))
for s in students:
grades.append(s.school)
grades.append(s.name)
grades.append(s.finalGrade())
if s.finalGrade()>82:
grades.append("Pass")
else:
grades.append("Fail")
print(grades)

--- End Code Snippit  ---

I have extended (from tutorial) it a bit, I would really like to have a newline 

at end of each record, I have searched (and tested) but cant get "\n" to give a 

newline, I get "Mydata\n"

Do I need to replace "append" with "print", or is there a way to get the 
newline in as I append to list?

Thanks again
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[MRAB via Python-list]

On 2023-11-15 03:41, Mike Dewhirst via Python-list wrote:

On 15/11/2023 10:25 am, MRAB via Python-list wrote:

On 2023-11-14 23:14, Mike Dewhirst via Python-list wrote:

I'd like to improve the code below, which works. It feels clunky to me.

I need to clean up user-uploaded files the size of which I don't know in
advance.

After cleaning they might be as big as 1Mb but that would be super rare.
Perhaps only for testing.

I'm extracting CAS numbers and here is the pattern xx-xx-x up to
xxx-xx-x eg., 1012300-77-4

def remove_alpha(txt):

      """  r'[^0-9\- ]':

      [^...]: Match any character that is not in the specified set.

      0-9: Match any digit.

      \: Escape character.

      -: Match a hyphen.

      Space: Match a space.

      """

  cleaned_txt = re.sub(r'[^0-9\- ]', '', txt)

      bits = cleaned_txt.split()

      pieces = []

      for bit in bits:

      # minimum size of a CAS number is 7 so drop smaller clumps 
of digits


      pieces.append(bit if len(bit) > 6 else "")

      return " ".join(pieces)


Many thanks for any hints


Why don't you use re.findall?

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


I think I can see what you did there but it won't make sense to me - or
whoever looks at the code - in future.

That answers your specific question. However, I am in awe of people who
can just "do" regular expressions and I thank you very much for what
would have been a monumental effort had I tried it.

That little re.sub() came from ChatGPT and I can understand it without
too much effort because it came documented

I suppose ChatGPT is the answer to this thread. Or everything. Or will be.


\b  Word boundary
[0-9]{2,7}  2..7 digits
-   "-"
[0-9]{2}2 digits
-   "-"
[0-9]{2}2 digits
\b  Word boundary

The "word boundary" thing is to stop it matching where there are letters 
or digits right next to the digits.


For example, if the text contained, say, "123456789-12-1234", you 
wouldn't want it to match because there are more than 7 digits at the 
start and more than 2 digits at the end.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[Mike Dewhirst via Python-list]

On 15/11/2023 10:25 am, MRAB via Python-list wrote:

On 2023-11-14 23:14, Mike Dewhirst via Python-list wrote:

I'd like to improve the code below, which works. It feels clunky to me.

I need to clean up user-uploaded files the size of which I don't know in
advance.

After cleaning they might be as big as 1Mb but that would be super rare.
Perhaps only for testing.

I'm extracting CAS numbers and here is the pattern xx-xx-x up to
xxx-xx-x eg., 1012300-77-4

def remove_alpha(txt):

      """  r'[^0-9\- ]':

      [^...]: Match any character that is not in the specified set.

      0-9: Match any digit.

      \: Escape character.

      -: Match a hyphen.

      Space: Match a space.

      """

  cleaned_txt = re.sub(r'[^0-9\- ]', '', txt)

      bits = cleaned_txt.split()

      pieces = []

      for bit in bits:

      # minimum size of a CAS number is 7 so drop smaller clumps 
of digits


      pieces.append(bit if len(bit) > 6 else "")

      return " ".join(pieces)


Many thanks for any hints


Why don't you use re.findall?

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)


I think I can see what you did there but it won't make sense to me - or 
whoever looks at the code - in future.


That answers your specific question. However, I am in awe of people who 
can just "do" regular expressions and I thank you very much for what 
would have been a monumental effort had I tried it.


That little re.sub() came from ChatGPT and I can understand it without 
too much effort because it came documented


I suppose ChatGPT is the answer to this thread. Or everything. Or will be.

Thanks

Mike
--
https://mail.python.org/mailman/listinfo/python-list

Re: Code improvement question

[MRAB via Python-list]

On 2023-11-14 23:14, Mike Dewhirst via Python-list wrote:

I'd like to improve the code below, which works. It feels clunky to me.

I need to clean up user-uploaded files the size of which I don't know in
advance.

After cleaning they might be as big as 1Mb but that would be super rare.
Perhaps only for testing.

I'm extracting CAS numbers and here is the pattern xx-xx-x up to
xxx-xx-x eg., 1012300-77-4

def remove_alpha(txt):

      """  r'[^0-9\- ]':

      [^...]: Match any character that is not in the specified set.

      0-9: Match any digit.

      \: Escape character.

      -: Match a hyphen.

      Space: Match a space.

      """

  cleaned_txt = re.sub(r'[^0-9\- ]', '', txt)

      bits = cleaned_txt.split()

      pieces = []

      for bit in bits:

      # minimum size of a CAS number is 7 so drop smaller clumps of digits

      pieces.append(bit if len(bit) > 6 else "")

      return " ".join(pieces)


Many thanks for any hints


Why don't you use re.findall?

re.findall(r'\b[0-9]{2,7}-[0-9]{2}-[0-9]{2}\b', txt)

--
https://mail.python.org/mailman/listinfo/python-list

Code improvement question

[Mike Dewhirst via Python-list]

I'd like to improve the code below, which works. It feels clunky to me.

I need to clean up user-uploaded files the size of which I don't know in 
advance.


After cleaning they might be as big as 1Mb but that would be super rare. 
Perhaps only for testing.


I'm extracting CAS numbers and here is the pattern xx-xx-x up to 
xxx-xx-x eg., 1012300-77-4


def remove_alpha(txt):

    """  r'[^0-9\- ]':

    [^...]: Match any character that is not in the specified set.

    0-9: Match any digit.

    \: Escape character.

    -: Match a hyphen.

    Space: Match a space.

    """

cleaned_txt = re.sub(r'[^0-9\- ]', '', txt)

    bits = cleaned_txt.split()

    pieces = []

    for bit in bits:

    # minimum size of a CAS number is 7 so drop smaller clumps of digits

    pieces.append(bit if len(bit) > 6 else "")

    return " ".join(pieces)


Many thanks for any hints

Cheers

Mike
--
https://mail.python.org/mailman/listinfo/python-list

Re: Question(s)

[Greg Ewing via Python-list]

On 25/10/23 2:32 pm, Chris Angelico wrote:

Error correcting memory, redundant systems, and human
monitoring, plus the ability to rewrite the guidance software on the
fly if they needed to.


Although the latter couldn't actually be done with the AGC,
as the software was in ROM. They could poke values into RAM
to change its behaviour to some extent, and that got them out
of trouble a few times, but they couldn't patch the code.

It might have been possible with the Gemini computer, since
it loaded its code from tape. I don't know if it was ever
done, though.

--
Greg

--
https://mail.python.org/mailman/listinfo/python-list

Re: Question(s)

[Thomas Passin via Python-list]

On 10/26/2023 10:52 PM, avi.e.gr...@gmail.com wrote:

Thomas,

It looks like much of our discussion and attempts at help are not going to
be that helpful to Tenor as we may be way off bass about what he wants to do
and certainly RSTUDIO and quite a few other suggestions may not be available
in his microcontroller.

As I see it, some of his objective involves sampling a sensor in real time.
I have not heard what he wants to do with the data gathered and this may be
an example of where code needs to be running fast enough to keep up. Proving
the code will work, especially if you add logging or print statements or run
it in a monitored mode so you can follow what it is doing, presents special
challenges.

Now if he ever wants to read in a .CSV file and analyze the data and make
graphs and so on, I might chime in. For now, I am dropping out.



Avi


I'm there.  It's like someone is insisting on being instructed how to 
drive a race car in a race when he's only got a learner's permit.  You 
just have to go through the experience of actually driving on streets 
and in traffic first.  There is no substitute.


TomP


-Original Message-
From: Python-list  On
Behalf Of Thomas Passin via Python-list
Sent: Thursday, October 26, 2023 6:50 PM
To: python-list@python.org
Subject: Re: Question(s)

On 10/26/2023 6:36 PM, AVI GROSS via Python-list wrote:

I am not one for IDLE worship, Tenor. But if you have been getting a

message here, it is that there are an amazing number of programs that
support your use of python during the development phase and perhaps later. I
actually often use an environment called RSTUDIO (now part of a new name of
POSIT) because it has been expanded beyond supporting R and supports Python
and a growing number of other languages or combos that combine word
processing with inserts from multiple languages.

Excellent! I didn't know about this development.

[snip]




--
https://mail.python.org/mailman/listinfo/python-list

RE: Question(s)

[AVI GROSS via Python-list]

Thomas,

It looks like much of our discussion and attempts at help are not going to
be that helpful to Tenor as we may be way off bass about what he wants to do
and certainly RSTUDIO and quite a few other suggestions may not be available
in his microcontroller.

As I see it, some of his objective involves sampling a sensor in real time.
I have not heard what he wants to do with the data gathered and this may be
an example of where code needs to be running fast enough to keep up. Proving
the code will work, especially if you add logging or print statements or run
it in a monitored mode so you can follow what it is doing, presents special
challenges.

Now if he ever wants to read in a .CSV file and analyze the data and make
graphs and so on, I might chime in. For now, I am dropping out.

Avi

-Original Message-
From: Python-list  On
Behalf Of Thomas Passin via Python-list
Sent: Thursday, October 26, 2023 6:50 PM
To: python-list@python.org
Subject: Re: Question(s)

On 10/26/2023 6:36 PM, AVI GROSS via Python-list wrote:
> I am not one for IDLE worship, Tenor. But if you have been getting a
message here, it is that there are an amazing number of programs that
support your use of python during the development phase and perhaps later. I
actually often use an environment called RSTUDIO (now part of a new name of
POSIT) because it has been expanded beyond supporting R and supports Python
and a growing number of other languages or combos that combine word
processing with inserts from multiple languages.

Excellent! I didn't know about this development.

[snip]


-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list