Re: on the popularity of loops while and for
On Sun, 29 Aug 2021 at 00:04, Stefan Ram wrote:
> Stestagg writes:
> >If you're doing this analysis, I'd be pretty interested in how many of
> >those while loops where 'while True:'
>
> Here, about 40 %.
Thanks! That's an interesting stat. I might have to look more into how
while loops are actually used in other people's code.
>
> Below is a new version that is supposed to also count "if True:"
> and "if 1:" and also removes a bug with the exception handling
> (missing "import sys" and missing "UnicodeDecodeError"). I'm not
> experienced in using the AST module, so there might still be bugs
> in the program!
>
> import ast
> import pathlib
> import sys
> rootname=r'''PATH TO SOURCE DIR (WITHOUT A BACKQUOTE AT THE END)'''
> rootpath=pathlib.Path(rootname)
> rootiterable=rootpath.glob('**/*.py')
> WhileCount = 0
> TrueCount = 0
> ForCount = 0
> for filepath in rootiterable:
> try:
> with filepath.open(encoding="utf-8") as file:
> source=file.read()
> parse=ast.parse(source, filename=str(file))
> for entry in ast.walk(parse):
> if isinstance(entry, ast.While):
> WhileCount+=1
> if ( isinstance( entry.test, ast.Constant ) and
> hasattr( entry.test, "value") and
> ( isinstance( entry.test.value, bool )
> and entry.test.value == True or
> isinstance( entry.test.value, int )
> and entry.test.value != 0 )):
> TrueCount+=1
> print( f"{ForCount=}, {WhileCount=}, {TrueCount=}" )
> elif isinstance(entry, ast.For):
> ForCount+=1
> print( f"{ForCount=}, {WhileCount=}" )
> except ( SyntaxError, UnicodeDecodeError )as inst:
>print( f"skipping {filepath}." )
>print( f"{sys.exc_info()[ 0 ] =}" )
>print( f"{type( inst ) =}" )
>print( f"{inst.args =}" )
>print( f"{inst =}\n" )
> print( f"{ForCount=}, {WhileCount=}, {TrueCount=}" )
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for
If you're doing this analysis, I'd be pretty interested in how many of
those while loops where 'while True:'
I'd wager 75%. But may be completely off
Steve
On Sat, 28 Aug 2021 at 23:03, Hope Rouselle wrote:
> r...@zedat.fu-berlin.de (Stefan Ram) writes:
>
> > Hope Rouselle writes:
> >>Have you guys ever measured something like that in a casual or serious
> >
> > import ast
> > import pathlib
> > rootname=r''
> > rootpath=pathlib.Path(rootname)
> > rootiterable=rootpath.glob('**/*.py')
> > first = True
> > WhileCount = 0
> > ForCount = 0
> > for filepath in rootiterable:
> > try:
> > with filepath.open(encoding="utf-8") as file:
> > source=file.read()
> > parse=ast.parse(source, filename=str(file))
> > for entry in ast.walk(parse):
> > if isinstance(entry, ast.While):
> > WhileCount+=1
> > print( f"{ForCount=}, {WhileCount=}" )
> > elif isinstance(entry, ast.For):
> > ForCount+=1
> > print( f"{ForCount=}, {WhileCount=}" )
> > except SyntaxError as inst:
> >if first:
> >print( f"{sys.exc_info()[ 0 ] =}" )
> >print( f"{type( inst ) =}" )
> >print( f"{inst.args =}" )
> >print( f"{inst =}" )
> >print( f"skipping {filepath}." )
> > first=False
>
> You are so wonderful! Thanks quite a lot. Here's what I got:
>
> ForCount=18703, WhileCount=2505
>
> I have pretty much just the standard libraries and a couple more ---
> sympy and some xlsxwriter library.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: i just moved from bottleframework to flask. I changes what needed to be altered to convert the code and when i run it i just get "Internal server error" Running tail -f ../logs/error_log i get no
[my reply on the list this time :)] That's fine. Sorry, I should have provided more information. This type of error that you're gettting /can/ happen if you have the library correctly installed, but within your project, you have a different file of the same name as the library. Then when python tries to import 'flask', it actually imports your local file, NOT the installed flask. You see the error message is *not* saying that python can't find flask, but that it can't find a name `run` within the flask module: from flask import run, route, request, redirect ImportError: cannot import name 'run' Therefore python has successfully found a module named flask somewhere, just that it's probably not the module you were expecting it to find. This is almost always because of the file name issue I mentioned above, OR because the wrong version of flask has been installed, or something similar. One thing to try would be at the *very* top of counters.py, add the following lines: import flask print(flask) then run your command again, and see what the stdout output is. You should see something like: Does XX look like the right place for the flask installation? if not, then that's the problem. Steve On Thu, Jul 8, 2021 at 5:08 PM Νίκος Βέργος wrote: > No, the way i have installed flask was under root account with the command > pip3 install flask. > But even if i installed flask under my nikos user account its evene worse > as the file flask is not entered into PATH. > > So i guess as root i have to install flask, but then why cant it import > 'run" ? > > [root@superhost ~]# whereis flask > flask: /usr/local/bin/flask > > Στις Πέμ, 8 Ιουλ 2021 στις 7:02 μ.μ., ο/η Stestagg > έγραψε: > >> Hi >> >> Do you have a file or folder named 'flask' in the same directory as >> www.py by any chance? >> >> Steve >> >> On Thu, Jul 8, 2021 at 4:50 PM vergos@gmail.com < >> vergos.niko...@gmail.com> wrote: >> >>> i just moved from bottleframework to flask. I changes what needed to be >>> altered to convert the code and when i run it i just get "Internal server >>> error" >>> >>> Running tail -f ../logs/error_log i get no errors. >>> >>> How can i find out what is the culprit here? >>> >>> [nikos@superhost wsgi]$ export FLASK_APP=www.py >>> [nikos@superhost wsgi]$ export FLASK_ENV=development >>> [nikos@superhost wsgi]$ flask run >>> * Serving Flask app 'www.py' (lazy loading) >>> * Environment: development >>> * Debug mode: on >>> Usage: flask run [OPTIONS] >>> Try 'flask run --help' for help. >>> >>> Error: While importing 'www', an ImportError was raised: >>> >>> Traceback (most recent call last): >>> File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line >>> 256, in locate_app >>> __import__(module_name) >>> File "/home/nikos/wsgi/www.py", line 4, in >>> import counters >>> File "/home/nikos/wsgi/counters.py", line 6, in >>> from flask import run, route, request, redirect >>> ImportError: cannot import name 'run' >>> >>> and this is the error_log when tries to be displayed via browser: >>> >>> [Thu Jul 08 15:08:36.436195 2021] [wsgi:error] [pid 575443:tid >>> 139757752112896] [remote 89.210.199.119:6718] mod_wsgi (pid=575443): >>> Failed to exec Python script file '/home/nikos/wsgi/www.py'. >>> >>> [Thu Jul 08 15:08:36.436276 2021] [wsgi:error] [pid 575443:tid >>> 139757752112896] [remote 89.210.199.119:6718] mod_wsgi (pid=575443): >>> Exception occurred processing WSGI script '/home/nikos/wsgi/www.py'. >>> >>> [Thu Jul 08 15:08:36.436408 2021] [wsgi:error] [pid 575443:tid >>> 139757752112896] [remote 89.210.199.119:6718] Traceback (most recent >>> call last): >>> >>> [Thu Jul 08 15:08:36.436432 2021] [wsgi:error] [pid 575443:tid >>> 139757752112896] [remote 89.210.199.119:6718] File >>> "/home/nikos/wsgi/www.py", line 4, in >>> >>> [Thu Jul 08 15:08:36.436436 2021] [wsgi:error] [pid 575443:tid >>> 139757752112896] [remote 89.210.199.119:6718] import counters >>> >>> [Thu Jul 08 15:08:36.436441 2021] [wsgi:error] [pid 575443:tid >>> 139757752112896] [remote 89.210.199.119:6718] File >>> "/home/nikos/wsgi/counters.py", line 6, in >>> >>> [Thu Jul 08 15:08:36.436444 2021] [wsgi:error] [pid 575443:tid >>> 139757752
Re: i just moved from bottleframework to flask. I changes what needed to be altered to convert the code and when i run it i just get "Internal server error" Running tail -f ../logs/error_log i get no
Hi Do you have a file or folder named 'flask' in the same directory as www.py by any chance? Steve On Thu, Jul 8, 2021 at 4:50 PM vergos@gmail.com < vergos.niko...@gmail.com> wrote: > i just moved from bottleframework to flask. I changes what needed to be > altered to convert the code and when i run it i just get "Internal server > error" > > Running tail -f ../logs/error_log i get no errors. > > How can i find out what is the culprit here? > > [nikos@superhost wsgi]$ export FLASK_APP=www.py > [nikos@superhost wsgi]$ export FLASK_ENV=development > [nikos@superhost wsgi]$ flask run > * Serving Flask app 'www.py' (lazy loading) > * Environment: development > * Debug mode: on > Usage: flask run [OPTIONS] > Try 'flask run --help' for help. > > Error: While importing 'www', an ImportError was raised: > > Traceback (most recent call last): > File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line > 256, in locate_app > __import__(module_name) > File "/home/nikos/wsgi/www.py", line 4, in > import counters > File "/home/nikos/wsgi/counters.py", line 6, in > from flask import run, route, request, redirect > ImportError: cannot import name 'run' > > and this is the error_log when tries to be displayed via browser: > > [Thu Jul 08 15:08:36.436195 2021] [wsgi:error] [pid 575443:tid > 139757752112896] [remote 89.210.199.119:6718] mod_wsgi (pid=575443): > Failed to exec Python script file '/home/nikos/wsgi/www.py'. > > [Thu Jul 08 15:08:36.436276 2021] [wsgi:error] [pid 575443:tid > 139757752112896] [remote 89.210.199.119:6718] mod_wsgi (pid=575443): > Exception occurred processing WSGI script '/home/nikos/wsgi/www.py'. > > [Thu Jul 08 15:08:36.436408 2021] [wsgi:error] [pid 575443:tid > 139757752112896] [remote 89.210.199.119:6718] Traceback (most recent call > last): > > [Thu Jul 08 15:08:36.436432 2021] [wsgi:error] [pid 575443:tid > 139757752112896] [remote 89.210.199.119:6718] File > "/home/nikos/wsgi/www.py", line 4, in > > [Thu Jul 08 15:08:36.436436 2021] [wsgi:error] [pid 575443:tid > 139757752112896] [remote 89.210.199.119:6718] import counters > > [Thu Jul 08 15:08:36.436441 2021] [wsgi:error] [pid 575443:tid > 139757752112896] [remote 89.210.199.119:6718] File > "/home/nikos/wsgi/counters.py", line 6, in > > [Thu Jul 08 15:08:36.436444 2021] [wsgi:error] [pid 575443:tid > 139757752112896] [remote 89.210.199.119:6718] from flask import run, > route, request, redirect > > [Thu Jul 08 15:08:36.436458 2021] [wsgi:error] [pid 575443:tid > 139757752112896] [remote 89.210.199.119:6718] ImportError: cannot import > name 'run' > > [Thu Jul 08 15:08:40.075655 2021] [wsgi:error] [pid 575443:tid > 13975291008] [remote 89.210.199.119:13389] mod_wsgi (pid=575443): > Failed to exec Python script file '/home/nikos/wsgi/www.py'. > > [Thu Jul 08 15:08:40.075703 2021] [wsgi:error] [pid 575443:tid > 13975291008] [remote 89.210.199.119:13389] mod_wsgi (pid=575443): > Exception occurred processing WSGI script '/home/nikos/wsgi/www.py'. > > [Thu Jul 08 15:08:40.075785 2021] [wsgi:error] [pid 575443:tid > 13975291008] [remote 89.210.199.119:13389] Traceback (most recent > call last): > > [Thu Jul 08 15:08:40.075802 2021] [wsgi:error] [pid 575443:tid > 13975291008] [remote 89.210.199.119:13389] File > "/home/nikos/wsgi/www.py", line 4, in > > [Thu Jul 08 15:08:40.075805 2021] [wsgi:error] [pid 575443:tid > 13975291008] [remote 89.210.199.119:13389] import counters > > [Thu Jul 08 15:08:40.075810 2021] [wsgi:error] [pid 575443:tid > 13975291008] [remote 89.210.199.119:13389] File > "/home/nikos/wsgi/counters.py", line 6, in > > [Thu Jul 08 15:08:40.075813 2021] [wsgi:error] [pid 575443:tid > 13975291008] [remote 89.210.199.119:13389] from flask import run, > route, request, redirect > > [Thu Jul 08 15:08:40.075825 2021] [wsgi:error] [pid 575443:tid > 13975291008] [remote 89.210.199.119:13389] ImportError: cannot import > name 'run' > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Behaviour of pop() for dictionaries
You can do the following: _,v = d.popitem() Or: key, value = d.popitem() Steve On Mon, 14 Jun 2021 at 20:10, Greg Ewing wrote: > On 14/06/21 4:19 am, BlindAnagram wrote: > > Am I missing the obvious way to obtain the value (or the key) from a > > dictionary that is known to hold only one item? > > v = d.popitem()[1] > > > More importantly, is there a good reason why we don't have d.pop() for > > dictionaries? > > My guess is because it's not generally useful to get an arbitrary > value from a dict without its corresponding key. Hence the existence > of popitem(). > > -- > Greg > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: learning python ...
On Sun, 23 May 2021 at 20:37, hw wrote:
> On 5/23/21 7:28 PM, Peter Otten wrote:
> > On 23/05/2021 06:37, hw wrote:
> >>
> >> Hi,
> >>
> >> I'm starting to learn python and have made a little example program
> >> following a tutorial[1] I'm attaching.
> >>
> >> Running it, I'm getting:
> >>
> >>
> >> Traceback (most recent call last):
> >>File "[...]/hworld.py", line 18, in
> >> print(isinstance(int, float))
> >> TypeError: isinstance() arg 2 must be a type or tuple of types
> >>
> >>
> >> I would understand to get an error message in line 5 but not in 18.
> >> Is this a bug or a feature?
> >
> > It is a bug in your code (which you don't provide). Did you assign some
> > value to float, e. g.:
> >
> > >>> float = 42.0
> > >>> isinstance(int, float)
> > Traceback (most recent call last):
> >File "", line 1, in
> > isinstance(int, float)
> > TypeError: isinstance() arg 2 must be a type or tuple of types
> >
> > If you do not shadow the built-in you should get
> >
> > >>> isinstance(int, float)
> > False
> >
>
> Apparently the attachment was stripped from my message. I'll put a
> smaller version directly into this message instead of an attachment:
>
>
> #!/usr/bin/python
>
> print("world!")
>
> int = 17
> print("world", int)
>
> float = 6.670
> print("world", float)
>
> foo = 0
> print(type(int))
> print(type(float))
> print(type(foo))
>
> print(isinstance(foo, str))
> print(isinstance(int, float))
> print(isinstance(float, float))
>
>
> I don't know about shadowing.
Shadowing is effectively saying “within this bit of code, (scope) I’m going
to use an already-used name for my own value”
If I have defeated a whole variable type
> by naming a variable like a variable type, I would think it is a bad
> idea for python to allow this without warning.
There are some reasons why allowing this is quite nice. And there’s
actually a ton of corner cases to consider when thinking about changing the
rules
Interestingly python 3 made this a little bit better by stopping you from
rebinding (shadowing) a number of built ins, such as True and False.
In your case, I agree that it is super confusing. One thing to learn to
look out for is if you assign to a name, then use that name on a different
context, expecting it to be different, then that’s not likely to work as
you expect.
It seems like a recipie
> for creating chaos.
Luckily almost every python code checker and/or linter will highlight this
for you.
If you’re learning python, I’d highly recommend doing so in an ide or
editor that has a code checker running.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python script accessing own source code
On 2021-05-12 15:48, Michael F. Stemper wrote: > > On 12/05/2021 08.26, Dino wrote: > > > >> Hi, here's my (probably unusual) problem. Can a Python (3.7+) script > >> access its own source code? > > > > Here is a fairly simple python program that reads itself: > > > > > > #!/usr/bin/python > > > > import sys > > > > with open( sys.argv[0], "rt" ) as myself: > > for line in myself: > > junk = sys.stdout.write( "%s" % (line) ) > > > > sys.exit(0) > > > > > > It's not bullet-proof. If you put it in a directory in your $PATH and > > run it from somewhere else, it won't work. > > > > Here's a fairly simple option: == import inspect import sys print(inspect.getsource(sys.modules[__name__])) -- https://mail.python.org/mailman/listinfo/python-list
Re: Proposal: Disconnect comp.lang.python from python-list
Where's this discussion going? Let's not get too caught up on definitions or the sizes of everyone's respective .. newsgroups. Which of the practically possible options are best for this list <-> newsgroup setup? Thanks Steve On Thu, May 6, 2021 at 6:47 PM Jon Ribbens via Python-list < python-list@python.org> wrote: > On 2021-05-06, Richard Damon wrote: > > On 5/6/21 9:44 AM, Jon Ribbens via Python-list wrote: > >> Sounds like nearly all moderated lists/forums then. > > > > Then perhaps you have never been on a real Moderated mailing list or > > Forum. > > Ah, the "no true scotsforum" argument ;-) > > >>> While you could setup a robo-moderator to do a similar thing, Usenet > >>> posters will not have 'pre-subscribed' before posting, and the From > >>> address is no where near as relaible as invalid From addresses ARE > >>> allowed, and since the message comes via a NNTP injection source relay, > >>> non-verifiable. This make the job a LOT harder. > >> It makes essentially no difference at all. > > It sure does. Have you every actually TRIED to run a moderated Usenet > > group, or know anyone who has, especially a somewhat busy group? > > As I already mentioned, I am a moderator of a Usenet group. > > > I am presuming that the current gateway isn't bringing all the messages > > from Usenet into the mailing list. This is obvious as we don't see the > > noise here. The Cabal that runs the 'Big-8' doesn't really care what > > sort of filters are added at such a gateway. > > > > To setup a moderated group that defines similar filters in place for > > messages getting to Usenet, particularly for a group intended to replace > > a 'reasonably' working unmoderated group, is likely not going to be > > viewed well. > > Have you every actually TRIED to run a moderated Usenet group, or know > anyone who has, especially a somewhat busy group? Of *course* moderated > groups put filters on what they receive, what do you think group > moderation is *for* if not to block things? > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Not found in the documentation
On Tue, Apr 27, 2021 at 12:52 PM elas tica wrote: > > > However, in this case, the general information in the docs is > > absolutely sufficient, and the basic principle that the repr should > > (where possible) be a valid literal should explain what's needed. > > > This is a subjective statement. Recall: explicit is better implicit. Alas, > many parts of the Python docs are written in a conversational style, full > of implicit and fuzzy meanings. > The general rule of thumb that I use is: The *Library Reference* section of the Python docs contains info that cover 90% of use-cases and questions. (note built-in types are a section in the library reference, despite being built-in to the interpreter and not really part of the standard library, but this makes sense if you consider the library reference to be more 'friendly' documentation that tries to balance verbosity and covering every corner case explicitly against readability and approachability. This could be considered a 'conversational' style, but that style is generally considered a feature in the library reference section. The *Language Reference* is designed to be much more formally defined, and favors correctness and completeness over being easy to access by less technical readers. In this case, subjectively, I (and I think Chris agrees here) feel that python's behavior with calling str() is suitably non-surprising (especially when compared to almost every other modern language) that being described in the language reference seems entirely reasonable. There's an argument to be made that the description of the expected __str__() behavior of ints is not easy to find within section 3 of the language reference, and I imagine that a reasonable proposed change to this wording would be considered favourably. (python is accepting pull requests when they pass review :). Steve > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: ipython display figure
I'm guessing here a little bit, but it looks like the author expects you to run this code in a jupyter notebook, rather than in an ipython interactive console. The jupyter and ipython projects are related (the same?) which can cause some confusion, but if you run the code you shared in a jupyter lab environment, then the SVG should display. As an example, you should be able to view the expected SVG in the following google colab book: https://colab.research.google.com/drive/10KL9mV92L8yPe_5wL089C-m6i0N8ct7Q?usp=sharing Steve On Tue, Apr 13, 2021 at 3:12 PM Peng Yu wrote: > Hi, > > https://www.fuzzingbook.org/html/Grammars.html > > I am trying to follow an example on the above page. But it does not > show a figure. Could anybody let me know how to display the figure? > > $ ipython3 > Python 3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) > Type 'copyright', 'credits' or 'license' for more information > IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help. > > [ins] In [1]: from fuzzingbook.Grammars import * > > [ins] In [2]: from IPython.display import SVG, display > > [ins] In [3]: > SVG(show_diagram(syntax_diagram_expr(EXPR_GRAMMAR[''][0]))) > Out[3]: > > -- > Regards, > Peng > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Yield after the return in Python function.
On Wed, Apr 7, 2021 at 12:31 PM Chris Angelico wrote: > > I just realised that the whole eval/exec/namespace stuff is massive > overkill. All you need is an object that is inconsistent in its > boolification... > > Somewhat related: https://bugs.python.org/issue42899 Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
For completeness, from 3.5 onwards, you can also do the following:
[{'name': n, **d} for n, d in dod.items()]
On Tue, Mar 30, 2021 at 1:06 PM Chris Angelico wrote:
> On Tue, Mar 30, 2021 at 11:01 PM Jon Ribbens via Python-list
> wrote:
> >
> > On 2021-03-30, Chris Angelico wrote:
> > > I dunno about "canonical", but here's how I'd do it:
> > >
> > > lod = [info | {"name": name} for name, info in dod.items()]
> > >
> > > You could use {"name":name}|info instead if you prefer to have the
> > > name show up first in the dictionary.
> >
> > It's probably worth noting this method requires Python 3.9.
>
> True, and if you need 3.8 support, then the dict constructor with one
> kwarg is the way to do it. But this way has the flexibility that you
> can choose which way to resolve conflicts (if there's a name inside
> the info dict, should it override the key, or not?).
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Canonical conversion of dict of dicts to list of dicts
I'm not certain this is the clearest possible code pattern to use, but
depending on the structure of your larger code, it's possible to do this,
and the compactness may help with understandability (that's a judgement
call!):
[dict(d, name=n) for n, d in dod.items()]
On Tue, Mar 30, 2021 at 12:42 PM Loris Bennett
wrote:
> Hi,
>
> If I have dict of dicts, say
>
> dod = {
> "alice":
> {
> "lang": "python",
> "level": "expert"
> },
> "bob":
> {
> "lang": "perl",
> "level": "noob"
> }
> }
>
> is there a canonical, or more pythonic, way of converting the outer key
> to a value to get a list of dicts, e.g
>
> lod = [
> {
> "name": "alice",
> "lang": "python",
> "level": "expert"
> },
> {
> "name": "bob",
> "lang": "perl",
> "level": "noob"
> }
> ]
>
> than just
>
> lod = []
> for name in dod:
> d = dod[name]
> d["name"] = name
> lod.append(d)
>
> ?
>
> Cheers,
>
> Loris
>
> --
> This signature is currently under construction.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
> I'm sorry. I didn't understand your question right. If I have 4 workers, > they require 4Gb > in idle state and some extra memory when they execute other tasks. If I > increase workers > count up to 16, they`ll eat all the memory I have (16GB) on my machine and > will crash as soon > as system get swapped. > -- > It was my fault sorry, bad communication at my end. I was just trying to check to see that you had actually tested that running 16 does actually crash in your environment. Steve > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
> > 2. Can you try a test with 16 or 32 active workers (i.e. number of > > workers=2x available memory in GB), do they all still end up with 1gb > > usage? or do you get any other memory-related issues running this? > Yes. They will consume 1Gb each. It doesn't matter how many workers I > have, > they behave exactly the same. We can even forget about Flask and Celery. > If I run this code in Python console, behavior will remain the same. > > Woah, funky, so you got to a situation where your workers were allocating 2x more ram than your system had available? and they were still working? Were you hitting lots of swap? If no big swap load, then it looks like no problem, it's just that the metrics you're looking at aren't saying what they appear to be. > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: memory consumption
On Mon, Mar 29, 2021 at 2:32 PM Alexey wrote: > понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña: > > It looks like the problem is on celery. > > The mentioned issue is still open, so not sure if it was corrected. > > > > https://manhtai.github.io/posts/memory-leak-in-celery/ > > As I mentioned in my first message, I tried to run > this task(class) via Flask API calls, without Celery. > And results are the same. Flask worker receives the API call and > executes MyClass().run() inside of view. After a few calls > worker size increases to 1Gb of RAM. In production I have 8 workers, > so in idle they will hold 8Gb. > Memory statistics in modern OSs are incredibly complex to reason about. It's /possible/ that, while it certainly looks bad in the monitoring tools, there actually isn't a problem here. Some questions here to help understand more: 1. Do you have any actual problems caused by running 8 celery workers (beyond high memory reports)? What are they? 2. Can you try a test with 16 or 32 active workers (i.e. number of workers=2x available memory in GB), do they all still end up with 1gb usage? or do you get any other memory-related issues running this? Thanks Steve > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: port to PDOS (especially mainframe)
On Tue, Mar 23, 2021 at 3:20 PM Michael Torrie wrote: > On 3/23/21 5:19 AM, Paul Edwards wrote: > > Thanks for the tip. I don't actually need it to be > > light. I just need it to be C90-compliant. > > I guess the point with MicroPython is that since it can build on all > sorts of microcontrollers, a) it has a simpler build system and b) it is > definitely statically compiled. But I'm not sure whether it would be > applicable here. > > For example, micropython can be compiled with a standlone TCP/IP implementation: LWIP. lwip contains the relevant constants, along with the code needed to make them do stuff: https://github.com/lwip-tcpip/lwip/blob/0056522cc974d2be2005c324f37187b5b3695765/src/include/lwip/errno.h#L160 So if you're compiling for a non-posix compliant environment, this may be an easier route to go Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Why assert is not a function?
There is also the, I think seldom used, feature that calling python with '-O' removes all assert statements at parse/compile time (I believe for performance reasons) So for example: $ python -c 'assert False' Traceback (most recent call last): File "", line 1, in AssertionError But: $ python -O -c 'assert False' [no output] $ Ignoring the question about this feature being particularly useful, it would be much harder to implement if assert were a standard function (and re-bindable), as the 'arguments' in the assert construct are not even executed when '-O' is used (this is the performance gain): $ python -O -c 'assert DoesNotExist' Traceback (most recent call last): File "", line 1, in NameError: name 'DoesNotExist' is not defined vs. $ python -O -c 'assert DoesNotExist' [no output] $ so the compiler would have to do a lot of work to identify assert calls and remove them. Steve On Tue, Mar 2, 2021 at 9:04 PM Chris Angelico wrote: > On Wed, Mar 3, 2021 at 7:53 AM Marco Sulla > wrote: > > > > I have a curiosity. Python, as many languages, has assert as a > > keyword. Can't it be implemented as a function? Is there an advantage > > to have it as a keyword? > > It could, but it would need some magic. A lot of test frameworks have > their own set of assertions and you have to pick the one you want - > for instance: > > assertEqual(x, y) > > instead of > > assert x == y > > The trouble is that, if you write a function for this, it will just > get True or False, without any explanation of the cause - making it > somewhat unhelpful when something goes wrong. To compensate, function > equivalents inevitably have to have myriad types of assertions, where > the language just needs one. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there a way to subtract 3 from every digit of a number?
With numpy and pandas, it's almost always best to start off with the
simple, obvious solution.
In your case, I would recommend defining a function, and calling
`Series.map`, passing in that function.
Sometimes, however, with large datasets, it's possible to use some
pandas/numpy tricks to significantly speed up execution.
Here's an example that runs about 5x faster than the previous example on a
1-million item array. (non-scientific testing). The downside of this
approach is you have to do some mucking about with numpy's data types to do
it, which can end up with hard-to-read solutions:
def sub3(series):
chars = series.values.astype(str)
intvals = chars.view('int8')
subbed = np.mod(intvals - ord('0') - 3, 10) + ord('0')
subbed_chars = np.where(intvals, subbed, intvals).view(chars.dtype)
return pd.Series(subbed_chars)
For example, let's create a series with 1 million items:
num_lengths = np.power(10, np.random.randint(1, 7, size=1_000_000))
test_series = pd.Series(np.random.randint(0, num_lengths,
size=1_000_000)).astype(str)
calling: sub3(test_series)
takes ~ 600ms and returns the array.
Whereas the lambda technique takes 3s for me:
test_series.map(lambda a: int("".join(map(lambda x: str((int(x)-3)%10)
,list(str(a))
How the numpy approach works:
1. series.values returns a numpy array of the series data
2. .astype(str) - makes numpy turn the array into a native string array (as
opposed to an array of python string onbjects). Numpy arrays are
fixed-width (dynamically chosen width to hold the largest string in the
array), so each element is zero-byte padded to be the correct width. In
this case, this is fine, as we're dealing with numbers
3. chars.view('int8') - creates a zero-copy view of the array, with each
byte as its own element. I'm assuming you're using ascii numbers here, so
this is safe. For example the digit '1' will be represented as an array
element with value 49 (ascii '1').
4. To convert, say, '49' to '1', we can subtract the ascii value for '0'
(48) from each element. Most mathematical operations in numpy are
performed element-wise, so for example 'my_array - 3' subtracts 3 from each
item
5. The actual per-character maths is done using element-wise operations,
before we add the ord('0') back to each number to convert the decimal value
back to equivalent ascii character.
6. Now we have a large 1-d array of the correctly adjusted digits, but we
need to reconstruct the original joined-up numbers.
7. np.where(intvals, a, b) is a really nice numpy builtin:
https://numpy.org/doc/stable/reference/generated/numpy.where.html. It's
used here to put back the 'zero byte' padding values from the original
numpy array
8. .view(chars.dtype) creates a fixed-size string view, of the corect
dimensions based on the original chars array.
9. Finally convert the numpy array back to a pandas series object and
return.
On Sun, Feb 21, 2021 at 3:37 PM Avi Gross via Python-list <
python-list@python.org> wrote:
> Ah, that is an interesting, Mike, but not an informative answer. My
> question is where the specific problem came from. Yes, someone used to R
> and coming to Python might work at adjusting to what is different and how
> to
> get things done. I do that all the time as one hobby is learning lots of
> languages and assessing them against each other.
>
> So care to share your solution in R which I would assume you could do
> easily?
>
> My quick and dirty attempt, of no interest to the python community, using
> the R form of integer that has a maximum, and the pipe operator that will
> not be in standard R for another iteration, is this:
>
> ## R code using pipes to convert an integer
> ## to another integer by subtracting 3
> ## from each digit and wrapping around from
> ## 2 to 9 and so on, meaning modulo 10
>
> ## Load libraries to be used
> library(dplyr)
>
> ## define function to return subtraction by N mod 10
> rotdown <- function(dig, by=3) (dig -by) %% 10
>
> start <- 123456789L
>
> ## Using pipes that send output between operators
>
> start %>%
> as.character %>%
> strsplit(split="") %>%
> unlist %>%
> as.integer %>%
> rotdown %>%
> as.character %>%
> paste(collapse="") %>%
> as.integer
>
> When run:
>
> > start %>%
> + as.character %>%
> + strsplit(split="") %>%
> + unlist %>%
> + as.integer %>%
> + rotdown %>%
> + as.character %>%
> + paste(collapse="") %>%
> + as.integer
> [1] 890123456
>
> The above is not meant to be efficient and I could do better if I take more
> than a few minutes but is straightforward and uses the vectorized approach
> so no obvious loops are needed.
>
>
>
>
>
>
> -Original Message-
> From: Python-list On
> Behalf Of C W
> Sent: Sunday, February 21, 2021 9:48 AM
> To: Chris Angelico
> Cc: Python
> Subject: Re: Is there a way to subtract 3 from every digit of a number?
>
> Hey Avi,
>
> I am a long time R user now using Python. So, this is my attempt to master
> the language.
>
> The
Re: I need some help interpreting this error
Some sources, in case they help: Message.get() calls policy.header_fetch_parse ( https://github.com/python/cpython/blob/cd80f430daa7dfe7feeb431ed34f88db5f64aa30/Lib/email/message.py#L471 ) Compat32.header_fetch_parse calls self._sanitize_header ( https://github.com/python/cpython/blob/cd80f430daa7dfe7feeb431ed34f88db5f64aa30/Lib/email/_policybase.py#L311 ) _sanitize_header calls _has_surrogates ( https://github.com/python/cpython/blob/cd80f430daa7dfe7feeb431ed34f88db5f64aa30/Lib/email/_policybase.py#L287 ) _has_surrogates check: https://github.com/python/cpython/blob/cd80f430daa7dfe7feeb431ed34f88db5f64aa30/Lib/email/utils.py#L51 On Wed, Feb 17, 2021 at 5:42 PM Stestagg wrote: > I don't particularly like to encourage this shotgun help request because, > as previous commenter suggests, debugging this yourself is best. > > Sometimes debugging is super hard, and especially so when uncommon > situations occur, but it's always far easier to debug things when you have > visibility into the system under test. > > However, in this case, the email code is super complex, and this scenario > also looks very uncommon, but not unique: ( > https://github.com/Sydius/mbox-to-txt/issues/2), so you successfully > nerd-sniped me :). > > My *guess*, from reading the python standard library source code is that > you came across an email with some content in the subject line that is > considered a "surrogate", roughly, some badly encoded unicode or binary > data in it. > > When this happens, the code in some situations (depending on the > policy...) may return a header.Header() instance, rather than a > headerregistry.UniqueUnstructuredHeader (which would have had a > headerregistry.BaseHeader (mro: str) dynamically attached). > > header.Header() does not inherit from str, and thus would throw the > traceback you observed. > > Your suggestion of a try: catch: may make sense, alternately, you could > wrap the result in a call to str(): > > if sbstrip in str(msghdr["subject"]): > > which should attempt to encode the binary into some form of string object > for comparison (I haven't checked exactly what would happen, beyond: it > tries). > > It should be possible to create a test mbox with some funky bytes in the > subject, and try to reproduce it that way. > > Steve > > > On Wed, Feb 17, 2021 at 5:07 PM Chris Green wrote: > >> Stefan Ram wrote: >> > Chris Green writes: >> > >But msghdr["subject"] is surely just a string isn't it? Why is it >> > >complaining about something of type 'Header'? >> > >> > What would you do to debug-print the type of an object? >> > >> I don't know, what would I do? :-) >> >> Without knowing what provokes the problem I could be waiting for days >> or weeks even before I see the error again. As I'd need to print the >> type for every message I'd get some big logs or I'd need to add a try: >> except to trap the specific error. >> >> -- >> Chris Green >> · >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > -- https://mail.python.org/mailman/listinfo/python-list
Re: I need some help interpreting this error
I don't particularly like to encourage this shotgun help request because, as previous commenter suggests, debugging this yourself is best. Sometimes debugging is super hard, and especially so when uncommon situations occur, but it's always far easier to debug things when you have visibility into the system under test. However, in this case, the email code is super complex, and this scenario also looks very uncommon, but not unique: ( https://github.com/Sydius/mbox-to-txt/issues/2), so you successfully nerd-sniped me :). My *guess*, from reading the python standard library source code is that you came across an email with some content in the subject line that is considered a "surrogate", roughly, some badly encoded unicode or binary data in it. When this happens, the code in some situations (depending on the policy...) may return a header.Header() instance, rather than a headerregistry.UniqueUnstructuredHeader (which would have had a headerregistry.BaseHeader (mro: str) dynamically attached). header.Header() does not inherit from str, and thus would throw the traceback you observed. Your suggestion of a try: catch: may make sense, alternately, you could wrap the result in a call to str(): if sbstrip in str(msghdr["subject"]): which should attempt to encode the binary into some form of string object for comparison (I haven't checked exactly what would happen, beyond: it tries). It should be possible to create a test mbox with some funky bytes in the subject, and try to reproduce it that way. Steve On Wed, Feb 17, 2021 at 5:07 PM Chris Green wrote: > Stefan Ram wrote: > > Chris Green writes: > > >But msghdr["subject"] is surely just a string isn't it? Why is it > > >complaining about something of type 'Header'? > > > > What would you do to debug-print the type of an object? > > > I don't know, what would I do? :-) > > Without knowing what provokes the problem I could be waiting for days > or weeks even before I see the error again. As I'd need to print the > type for every message I'd get some big logs or I'd need to add a try: > except to trap the specific error. > > -- > Chris Green > · > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: UTF-16 or something else?
Try setting encoding to: "utf-8-sig". 'eb bb bf' is the byte order mark for UTF8 (most systems do not include this in UTF-8 encoded files) Python will correctly read UTF8 BOMs if you use the 'utf-8-sig' encoding when reading files Steve On Tue, Feb 9, 2021 at 2:56 PM Skip Montanaro wrote: > I downloaded US hospital ICU capacity data this morning from this page: > > > https://healthdata.gov/dataset/covid-19-reported-patient-impact-and-hospital-capacity-facility > > (The download link is about halfway down the page.) > > Trying to read it using my personal CSV tools without specifying an > encoding, it failed to understand the first column, hospital_pk. That is > apparently because the file isn't simply ASCII or UTF-8. There are a few > bytes ahead of the "h". However, if I open the file using "utf-16" as the > encoding, Python complains there is no BOM. od(1) suggests there is > *something* ahead of the first column name, but it's three bytes, not two: > > % od -A x -t x1z -v < > > reported_hospital_capacity_admissions_facility_level_weekly_average_timeseries_20210207.csv > | head > 00 *ef bb bf* 68 6f 73 70 69 74 61 6c 5f 70 6b 2c 63 > >...hospital_pk,c< > 10 6f 6c 6c 65 63 74 69 6f 6e 5f 77 65 65 6b 2c 73 >ollection_week,s< > 20 74 61 74 65 2c 63 63 6e 2c 68 6f 73 70 69 74 61 >tate,ccn,hospita< > ... > > I'm opening the file like so: > > inf = open(args[0], "r", encoding=encoding) > > where encoding is passed on the command line. I know I can simply edit out > those bytes and probably be good-to-go, but I'd prefer not to. What should > I be passing for the encoding? > > Skip, who thought everybody had effectively settled on utf-8 at this point, > but apparently not... > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: advice on debugging a segfault
I would normally agree, except... This is a refcount issue (I was able to reproduce the problem, gbd shows a free error ) And I wouldn't recommend DGBing a refcount issue as a beginner to debugging. The other mailing list identified a PIL bug that messes up the refcount for True, but this refcount issue is for some tuple object, so may possibly be a different problem. Steve On Sun, Jan 17, 2021 at 9:27 PM Barry wrote: > Run python under gdb and when the segv happens use > the gdb bt command to get a stack trace. > > Also if gdb says that it needs debug symbols install you will need to > do that. Otherwise the not will not contain symbols. > > Barry > > > On 17 Jan 2021, at 19:58, Robin Becker wrote: > > > > I have a segfault in the 3.10 alpha 4 when running the reportlab > document generation; all the other tests seem to have worked. I would like > to ask experts here how to approach getting the location of the problem. I > run recent archlinux. > > > > Below is the output of a test run with -Xdev -Xtracemalloc; the main > process is almost finished so the error appears to come from trying to free > resources > > > >> $ python -Xdev -Xtracemalloc genuserguide.py > /home/robin/devel/reportlab/.py310/lib/python3.10/distutils/__init__.py:1: > DeprecationWarning: the imp module is deprecated in favour of importlib; > see the module's documentation for alternative uses > >> import imp > >> Built story contains 1843 flowables... > >> Saved > "/home/robin/devel/reportlab/REPOS/reportlab/docs/reportlab-userguide.pdf" > >> [ Top 10 ] > >> :630: size=10.5 MiB, > count=105832, average=104 B > >> /home/robin/devel/reportlab/reportlab/lib/abag.py:19: size=7161 KiB, > count=27126, average=270 B > >> /home/robin/devel/reportlab/reportlab/platypus/paraparser.py:3141: > size=3364 KiB, count=5980, average=576 B > >> > /home/robin/devel/reportlab/.py310/lib/python3.10/site-packages/pyphen/__init__.py:160: > size=2905 KiB, count=41797, average=71 B > >> /home/robin/devel/reportlab/reportlab/platypus/paragraph.py:776: > size=1386 KiB, count=32208, average=44 B > >> /home/robin/devel/reportlab/reportlab/platypus/paragraph.py:92: > size=1384 KiB, count=26248, average=54 B > >> /home/robin/devel/reportlab/.py310/lib/python3.10/copy.py:280: > size=1232 KiB, count=8827, average=143 B > >> /home/robin/devel/reportlab/reportlab/platypus/frames.py:155: size=1101 > KiB, count=1173, average=962 B > >> :241: size=915 KiB, count=8146, > average=115 B > >> /home/robin/devel/reportlab/reportlab/platypus/paragraph.py:773: > size=881 KiB, count=16104, average=56 B > >> sys:1: ResourceWarning: unclosed file <_io.BufferedReader > name='../images/replogo.gif'> > >> sys:1: ResourceWarning: unclosed file <_io.BufferedReader > name='../images/testimg.gif'> > >> Debug memory block at address p=0x5617c33effe0: API '' > >>0 bytes originally requested > >>The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd): > >>at p-7: 0x00 *** OUCH > >>at p-6: 0x00 *** OUCH > >>at p-5: 0x00 *** OUCH > >>at p-4: 0x00 *** OUCH > >>at p-3: 0x00 *** OUCH > >>at p-2: 0x00 *** OUCH > >>at p-1: 0x00 *** OUCH > >>Because memory is corrupted at the start, the count of bytes > requested > >> may be bogus, and checking the trailing pad bytes may segfault. > >>The 8 pad bytes at tail=0x5617c33effe0 are not all FORBIDDENBYTE > (0xfd): > >>at tail+0: 0x00 *** OUCH > >>at tail+1: 0x00 *** OUCH > >>at tail+2: 0x00 *** OUCH > >>at tail+3: 0x00 *** OUCH > >>at tail+4: 0x00 *** OUCH > >>at tail+5: 0x00 *** OUCH > >>at tail+6: 0x00 *** OUCH > >>at tail+7: 0x00 *** OUCH > >> Enable tracemalloc to get the memory block allocation traceback > >> Fatal Python error: _PyMem_DebugRawFree: bad ID: Allocated using API > '', verified using API 'o' > >> Python runtime state: finalizing (tstate=0x5617c53c8b30) > >> Current thread 0x7fd5742b3740 (most recent call first): > >> > >> Aborted (core dumped) > > > > > > for comparison here is the 3.9.1 output> $ python -Xdev -Xtracemalloc > genuserguide.py > >> > /home/robin/devel/reportlab/.py39/lib/python3.9/distutils/__init__.py:1: > DeprecationWarning: the imp module is deprecated in favour of importlib; > see the module's documentation for alternative uses > >> import imp > >> > /home/robin/devel/reportlab/.py39/lib/python3.9/site-packages/fontTools/misc/py23.py:11: > DeprecationWarning: The py23 module has been deprecated and will be removed > in the next release. Please update your code. > >> warnings.warn( > >> Built story contains 1843 flowables... > >> Saved > "/home/robin/devel/reportlab/REPOS/reportlab/docs/reportlab-userguide.pdf" > >> [ Top 10 ] > >> :587: size=12.4 MiB, > count=128010, average=102 B > >> /home/robin/devel/reportlab/reportlab/lib/abag.py:19: size=7132 KiB, > count=26951, average=271 B > >> /home/robin/devel/reportlab/reportlab/platypus/paraparser.py:3141: >
constructor classmethods
Hi I was hoping to canvas opinion on using classmethods as constructors over __init__. We've got a colleague who is very keen that __init__ methods don't contain any logic/implementation at all, and if there is any, then it should be moved to a create() classmethod. As a concrete example, one change proposed was to go from this: ``` def __init__(self, ...): self.queue = Queue.Queue() to this: def __init__(self, queue): self.queue = queue @classmethod def create(cls, ...): return cls(Queue.Queue()) ``` The rationale is that it decouples the class from the queue implementation (no evidence or suggestion that we would actually ever want to change impl in this case), and makes testing easier. I get the underlying principal, and it's one that a strict OOp approach would suggest, but my gut feeling is that this is not a pythonic approach at all. What do people feel about this? Thanks Steve -- https://mail.python.org/mailman/listinfo/python-list
