Re: convert script awk in python
Christian Gollwitzer writes: > The closest equivalent I can come up with in Python is this: > > == > import sys > > s=0 > for line in sys.stdin: > try: > s += float(line.split()[1]) > except: > pass > print(s) > === > > > I don't want to cram this into a python -c " " line, if it even is > possible; how do you handle indentation levels and loops?? > I agree. Perhaps we need a ‘awk’ module/package. I see that there is one in PyPI but that was last updated in 2016. -- Regards, Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: How to implement logging for an imported module?
"Joseph L. Casale" writes: >> I couldn't find any information on how to implement logging in a >> library that doesn't know the name of the application that uses >> it. How is that done? > Create (get) a root logger (you don't have to use it) and set the > level, and attach a handler to it. Then get the logger your library > uses and set the level to what you want. So does that mean if we change the following code to get a logger without any name then it will work? Without any further change. >> logger = logging.getLogger('foo') >> logger.addHandler(ch) >> logger.setLevel(logging.DEBUG) -- Regards, Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: How to uninstall python 2 package specifically?
Chris Green writes: > I have two (or maybe even three) versions of Click installed:- > > /usr/local/lib/python2.7/dist-packages/click > /usr/local/lib/python3.7/dist-packages/click > /usr/lib/python3/dist-packages/click > > I run [x]ubuntu. > > How can I uninstall those extra versions of click (which have > presumably been installed by pip, how would I know?). For python2 package use pip, for python3 packages use pip3. -- https://mail.python.org/mailman/listinfo/python-list
[HN] How to Make Python Wait
https://news.ycombinator.com/item?id=21834408 -- https://mail.python.org/mailman/listinfo/python-list
Re: Developers are advised to purge these malicious packages
Christian Heimes writes: > On 04/12/2019 18.59, David Lowry-Duda wrote: >> I notice that "python3-dateutil" is in over 4000 github repositories >> [1]. That sounds like a disaster. >> >> [1]: https://github.com/search?q=python3-dateutil=Code > > At least the first pages are packaging files for Debian, Fedora, and > other Linux distributions. Downstream distributions provide a Python > package under multiple names. For example the Fedora's build spec [1] > creates python2-dateutil and python3-dateutil packages from the > python-dateutil upstream project. > > Attackers abuse the fact and try to typo-squat packages in hope that > somebody uses the Linux distribution package name "python3-dateutil" > instead of the upstream name "python-dateutil" in requirements.txt > Nice explanation. Thanks. -- https://mail.python.org/mailman/listinfo/python-list
Developers are advised to purge these malicious packages
``` The Python security team removed two trojanized Python libraries from PyPI (Python Package Index) that were caught stealing SSH and GPG keys from the projects of infected developers. The first is "python3-dateutil," which imitated the popular "dateutil" library. The second is "jeIlyfish" (the first L is an I), which mimicked the "jellyfish" library. ``` https://www.zdnet.com/article/two-malicious-python-libraries-removed-from-pypi/ Regards, -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: tab replace to space 4
황병희 writes: > usally i write python code in gnu emacs on ubuntu 18.04 sometimes i > re-edit the code vim in same machine so often when i do run the code in > shell like as ./test.py i meet consol error -- which line wrong! > > so i am considering how can i replace all tab to space 4 within python > code. if there is solution in google i am very sorry. In Emacs, use "M-x untabify". And "M-x tabify" if you want to do the reverse. Regards, -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to retrieve data from Juypter notebook
> Any help is appreciated.. Could you please elaborate a little bit? I didn't get from where you want to retrieve data? Is this somewhere hosted? Or you want to see raw files from your own Jupyter notebook running on your own machine. Regards, -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: Can we use Python for hacking?
>> Someone requested my answer to the question: "Can we use Python for >> hacking?" > Sigh. I suppose it's a lost battle to reclaim that word. So true. I still remember the ESR article that I used to share twenty years ago. -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: Using Makefiles in Python projects
Chris Angelico writes: > On Sun, Nov 10, 2019 at 2:10 AM Thomas Jollans wrote: >> On 07/11/2019 20:20, Vitaly Potyarkin wrote: >> > What do you think of using Makefiles for automating common chores in >> > Python projects? Like linting, type checking and testing? >> > >> > I've come up with a reusable Makefile for automating virtual environment >> > management in Python projects. I think it can be useful for simplifying >> > the onboarding of new developers (both new to project and new to Python) >> > and for documenting project's development practices. >> > >> > Here it is: >> > - Repo: https://github.com/sio/Makefile.venv >> > - Demo screencast: https://asciinema.org/a/279646 >> > >> > What do you think? Is this useful or I'm just unaware of some tool that >> > abstracts venv chores away better? >> > >> As others have said, make is a useful tool and many people use it for >> different purposes in their Python projects. Nothing wrong with that. >> >> HOWEVER, at risk of stating the obvious, using Makefiles written for/on >> *nix systems on Windows is a bit of a hassle. If, for some reason, your >> software is *nix-only anyway, that's fine. If not, using make means >> sacrificing some portability. >> >> If your software runs on Windows, of you think it might run on Windows >> in the future, maybe consider writing simple Python scripts for >> platform-independent tasks rather than makefiles and shell scripts. >> > > Are you assuming that every Windows system has Python, but that you > can't get make or bash? Because neither half of that is true. I've > happily used makefiles on Windows, and these days, bash is as easy to > get hold of as Python is. Actually, the proposal to use Makefile is great. I have always been thinking about it and using it in my own projects. At my previous workplace there was compulsion to use Windows, so I had to setup Cygwin. And I had same level of difficulty setting up Python on Windows. So the difficulty level is same for Python and Make setup. Another advantage of using Make is that it is easy to configure portable mixed native programs with Python. To work on modern projects similar to NumPy, TensorFlow, PyCharm where half of the code is native and half Python, it is much easier to manage native part with Make. Autoconf, automake etc. utilities tells you in advance what native libraries are missing from the system - Windows or Linux. We should not forget that Data Science, Machine Learning etc. fields have helped in proliferation of Python lately. And the above mentioned libraries have great role. We should make it even more easier to manage mixed native projects portably. GNU Make is the way to go. Thanks for raising this. Regards, -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: Pip not available
Ethan Woo writes: > I was using Python 3.7.4 for a project, and I needed to install the > *playsound* function. I looked through online and saw that I needed to use > pip. However, it didn't work. I looked online and saw that I needed to > install it through the application, however, now this problem pops up. > [image: image.png] > Can you help me with this? > python3 -m pip install -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: python -m pip install and pip install
Chris Angelico writes: > On Fri, Oct 11, 2019 at 3:37 PM אורי wrote: >> >> When you upgrade pip, you have to write: >> >> python -m pip install --upgrade pip >> >> When you install or upgrade anything else, you can write "pip install". >> >> You can't upgrade pip using "pip install --upgrade pip". >> > > Only a consideration on Windows. Other platforms are absolutely fine > upgrading pip either way. > Oh! I wasn't aware of this Windows thing. Thanks. -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: python -m pip install and pip install
Michael Torrie writes: > On 10/10/19 1:21 AM, Pankaj Jangid wrote: >> So the scripts will just work fine if you simply use ~import pip~ >> and work with it. >> >> Suppose you were writing bash scripts around python programs. Then >> what will be the behaviour of, >> >> pip2 install mod >> >> under a python3 environment. > > If Python 2 wasn't installed, pip2 wouldn't be found at all. If Python 2 > was available, it would happily install mod under the default/current > Python 2 environment. > I am talking about environments where both python2 and python3 installed. >> Similarly, there may be other commands specific to python2 and >> python3. All of them work flawlessly if you just replace python2 with >> python3. > > Not sure what you're getting at there. I am just trying to think about scenarios where one approach is better than other. Nothing more. -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: python -m pip install and pip install
Hongyi Zhao writes: > Hongyi Zhao 于2019年10月8日周二 下午4:53写道: >> >> Cameron Simpson 于2019年10月8日周二 下午12:25写道: >> > >> > On 08Oct2019 02:49, Hongyi Zhao wrote: >> > >On Tue, 08 Oct 2019 06:28:05 +0530, Pankaj Jangid wrote: >> > >> A very good use-case is when you have both, python2 and python3 >> > >> installed. >> > >> python2 -m pip install mod >> > >> python3 -m pip install mod >> > >> will install the package in the corresponding PYTHONPATH. >> > >> >> > > >> > >If so, why not just: >> > >pip2 install mod >> > >and using: >> > >pip3 install mod >> > >> > Because of the slight disconnect between "pip2" and "python2", etc. Do >> > you _know_ they both use the same Python install? With "pythonX -m pip" >> > you're using the same python executable which will be accessing what you >> > just installed. >> >> I use pyenv + pip, which will do the trick. > > And nowadays, the pyenv + vurtualenv + pip + pipenv is the suggested > env management method for python. > In this way, the pipenv will solve the version dependence of the > underlying tool chain and corresponding used packages/modules. > Exactly. So the scripts will just work fine if you simply use ~import pip~ and work with it. Suppose you were writing bash scripts around python programs. Then what will be the behaviour of, pip2 install mod under a python3 environment. Similarly, there may be other commands specific to python2 and python3. All of them work flawlessly if you just replace python2 with python3. -- Regards, Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: python -m pip install and pip install
Hongyi Zhao writes: > Hi, > > What's the diff: > > python -m pip install mod > and > pip install mod A very good use-case is when you have both, python2 and python3 installed. python2 -m pip install mod and python3 -m pip install mod will install the package in the corresponding PYTHONPATH. -- Regards Pankaj -- https://mail.python.org/mailman/listinfo/python-list
Re: Obtain the file's path.
Cameron Simpson writes: > On 17Sep2019 15:09, Hongyi Zhao wrote: >>See the following two methods for obtaining the file's path: >> >>os.path.realpath(file) >>or >>os.path.abspath(os.path.expanduser(file)) >> >>Which is more robust? > > My inclination is often to use abspath, because it may better express > the "intent" of the filename by not "undoing" the effects of symlinks. > > So abspath(expanduser("~/media/foo.mp4")) might expand to > "/home/cameron/media/foo.mp4". > > Conversely, realpath(expanduser("~/media/foo.mp4")) might expand to > "/raid_volume/cameron/media/foo.mp4". > > You might ask yourself, why do you need to know the absolute path at > all? A relative path is usually just fine; it isn't like it won't work > unless you use it from another directory. > Somewhat related to the OP's question. So what is a good strategy in an application? I am now inclined to use relative path and working directory both. And when there is change of runtime context just change the working directory and assemble the absolute path at runtime. And to save the working directory use "abspath" as suggested by Cameron. -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: kmeans clustering
Sharan Basappa writes: > Can someone please help me to clarify the different between fit and > predict functions of kmeans? > What is the significance of `predict' in K-means? It is an unsupervised clustering algorithm. My intuition says that the cluster composition itself might change if you add a new example and re-run K-means. On the other hand if you don't want to change the cluster compositions and just want to find a cluster for a new example then it is not K-means. In that case it is Knearest classifier. -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: phyton
inhahe writes: > On Tue, Sep 10, 2019 at 8:41 AM Pankaj Jangid > wrote: > >> You can do something like this ;-) >> >> >> import math >> >> def add_vat(a, b): >> return math.ceil(100*(a * 0.57 + sum([ord(c) for c in list(b)]) * >> 0.15538))/100 >> >> print(add_vat(101, 'high')) >> print(add_vat(101, 'low')) >> >> -- >> Pankaj Jangid >> -- >> https://mail.python.org/mailman/listinfo/python-list > > > BAD BAD BAD don't do Pankaj's suggestion, tim, it's DEVILSPEAK :-( what did I do? This is giving correct ans. hahahaha -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: phyton
tim.g...@quicknet.nl writes: > For school i need to write the right code to get the following outcome. > Can someone help me with this > I can't find a solution to link the word high to 1.21. > > 11 print(add_vat(101, 'high')) > 12 print(add_vat(101, 'low')) > > Outcome: > > 122.21 > 110.09 > You can do something like this ;-) import math def add_vat(a, b): return math.ceil(100*(a * 0.57 + sum([ord(c) for c in list(b)]) * 0.15538))/100 print(add_vat(101, 'high')) print(add_vat(101, 'low')) -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: local variable 'p' referenced before assignment
David writes: > https://docs.python.org/3/faq/programming.html#what-are-the-rules-for-local-and-global-variables-in-python > > " If a variable is assigned a value anywhere within the function’s body, > it’s assumed to be a local unless explicitly declared as global." > Coming with a baggage of other languages. :-) I should have searched it. Thanks a lot for sharing this. Regards. -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
local variable 'p' referenced before assignment
Why this code is giving local variable access error when I am accessing a global variable? p = 0 def visit(): m = 1 if m > p: p = m visit() print(p) If I change the variable assignment inside the function to q = m then it works fine. Like this p = 0 def visit(): m = 1 if m > p: q = m# Changed 'p' to 'q' visit() print(p) -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: How to only read words within brackets/ parentheses (in .txt file) using Python
A S writes: > I understand that reading lines in .txt files would look something like this > in Python: > > > with open('filename','r') as fd: >lines = fd.readlines() > > > However, how do I run my code to only read the words in my .txt files that > are within each balanced parenthesis? > > I am not sure how to go about it, let's say my .txt file contents lines like > this: > > k; > > select xx("xE'", PUT(xx..),"'") jdfjhf:jhfjj from _x_xx_L ; > quit; > > The main idea is to read only these portions of the .txt file (i.e. Those > within parentheses): > This should work for the outer parenthesis: import re p = re.compile(r"\((.+)\)", re.VERBOSE) with open('filename','r') as fd: lines = fd.readlines() for line in lines: m = p.findall(line) for s in m: print(s) -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: Hi how do I import files inside a txt file?
Spencer Du writes: > How do i import files inside a txt file if they exist in the current > directory? > > Here is the current code but I dont know how to do it correctly. > > import paho.mqtt.client as mqtt > from mqtt import * > import importlib > import os > import os.path > # from stateMachine import * > > with open("list_of_devices.txt", "r") as reader: > for item in reader: > try: > os.getcwd() > print("hi") > except: > print("error") > > This is "list_of_devices.txt": > test1,test2 > > Each name refers to a python file. > My interpretation is that you want to read a file (list_of_devices.txt) and this file contains names of other files and you want to read those files as well and do something with them (read or print or whatever). You can approach it like this: write a function to read a file and work on it. Like this, def fn(fname): with open(fname, "r") as f: try: # work with f except: print("error") Then use this function in your code that you have writen. Like this with open("list_of_devices.txt", "r") as reader: for item in reader: try: fn(item) except: print("error") In the example that you gave, you have written contents of "list_of_devices.txt" as test1,test2 Take care to read them as comma separated. Or if you have control then write them on separate lines. Regards. -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: Python unit test framework to fire 1000 unique concurrent requests
vishwatheworld...@gmail.com writes: > Looking for a unit test framework with below requirements: > 1. Send 1000 unique URL requests in parallel ( Not sequential and not > repetitive requests) > 2. Verify HTTP response status code ( eg: 200, 403, 404 etc) of each > requests in parallel > 3. Verify Response headers of each URL requests > 4. Login to remote machine ( to which requests is fired) and verify logs > 1st is a load testing. You can use something like Apache JMeter. For other advanced validations use Desktop version of vREST - https://desktop.vrest.io/. Regards. -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: Using the same data for both validation and prediction in Keras
Amirreza Heidari writes: > I was reading a tutorial for time series prediction by Neural > Networks. I found that this code have used the same test data in the > following code for validation, and later also for prediction. > > history = model.fit(train_X, train_y, epochs=50, batch_size=72, > validation_data=(test_X, test_y), verbose=2, shuffle=False) > > Does it mean that the validation and test data are the same, or there is a > default percentage to split the data into validation and prediction? As per Prof. Andrew Ng, training, cross-validation and testing should have three different data-sets. If you have a small example set (for example 10,000 or may be 50,000) then you can split the example set into 60:20:20 ratio for train:validation:testing. But if you have a very large data-set (1 million, 10 million) then consider using 1% or may be lesser for validation and testing. -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: python requests get from API and post to another API and remote u'
Noah writes: > I place a get request and the response from the API is "{'id': 32, > 'description': u'Firewall Outside', 'address': u'10.10.10.230/30'}" > > I then take that information and attempt post it to the other API. > The data is not accepted and the result is an HTTP 400 code. > > I tried posting with postman and still rejected. > > I can post to the second API if I change the data to look like this: > {"id": 32, "description": "Firewall Outside", "address": > "10.10.10.230/30"} This could be python version issue at your end. But I am thinking why this is getting rejected by Postman as well. Try with vREST. -- Pankaj Jangid -- https://mail.python.org/mailman/listinfo/python-list
Re: Meanwhile Norwegian trolls created ...
Abdur-Rahmaan Janhangeer writes: > Was browsing when i came across this hilarious piece of text: > > source: http://enki-editor.org/2014/08/23/Pyqt_mem_mgmt.html > I haven't searched for it though, but I guess if there is a Qt like framework in Rust then a Python wrapper around it would behave more meaningfully. -- Pankaj Planet Earth. -- https://mail.python.org/mailman/listinfo/python-list
Re: Your IDE's?
Ben Finney writes: > John Doe writes: > >> What is your favorite Python IDE? > > Maybe the Atom editor will get there some day, though for now I hear > many complaints that with many plug-ins active it's just too slow when > doing the kind of complex tasks we expect of a programmer's editor like > Vim or GNU Emacs. I also use Emacs for all my coding requirements. Atom will probably be abandonded now. Github is now acquired by Microsoft and hence their is no point for them to run two similar projects. The community might fork it though. -- Pankaj Planet Earth. -- https://mail.python.org/mailman/listinfo/python-list
[issue33587] inspect.getsource performs unnecessary filesystem stat call
Pankaj Pandey added the comment: I simply profiled the script with "python -m cProfile -s cumtime inspect_stack_perf.py" And yes, I should move the linecache check first, that would be a better idea. Though getmodule() is also fast enough and reordering it does not significantly change the performance (for the common case of no PEP302 modules). -- ___ Python tracker <https://bugs.python.org/issue33587> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33587] inspect.getsource performs unnecessary filesystem stat call which is ignored in some circumstances
Pankaj Pandey <panka...@gmail.com> added the comment: Here's the patch performance difference before and after the patch: Before: Sun May 20 21:42:32 2018prof1.stat 1188991 function calls (1188851 primitive calls) in 4.821 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 18/10.0060.0004.8244.824 {built-in method builtins.exec} 10.0000.0004.8244.824 inspect_stack_perf.py:1() 10.0010.0014.7904.790 inspect_stack_perf.py:22(load_nodes) 4210.0010.0004.7870.011 inspect_stack_perf.py:5(__init__) 4210.0010.0004.7860.011 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1492(stack) 4210.0110.0004.7850.011 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1464(getouterframes) 46300.0310.0004.7700.001 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1425(getframeinfo) 4200.0010.0004.7390.011 inspect_stack_perf.py:9(add_child) 139944.1590.0004.1590.000 {built-in method nt.stat} 46300.0420.0003.2230.001 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:760(findsource) 93220.0430.0002.9600.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:680(getsourcefile) 93220.0110.0002.8320.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\genericpath.py:16(exists) 46300.0160.0001.3390.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\linecache.py:53(checkcache) 46300.0970.0000.3640.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:714(getmodule) 46920.0080.0000.1350.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:702(getabsfile) 47540.0080.0000.0910.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\ntpath.py:538(abspath) 47540.0360.0000.0740.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\ntpath.py:471(normpath) 1626000.0710.0000.0710.000 {built-in method builtins.hasattr} After the patch: Sun May 20 21:39:44 2018prof2.stat 2639991 function calls (2639727 primitive calls) in 2.841 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 18/10.0050.0002.8442.844 {built-in method builtins.exec} 10.0000.0002.8442.844 inspect_stack_perf.py:1() 10.0010.0012.8142.814 inspect_stack_perf.py:22(load_nodes) 4210.0010.0002.8120.007 inspect_stack_perf.py:5(__init__) 4210.0010.0002.8110.007 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1492(stack) 4210.0100.0002.8100.007 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1464(getouterframes) 4200.0010.0002.8020.007 inspect_stack_perf.py:9(add_child) 46300.0290.0002.7950.001 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1425(getframeinfo) 46300.0400.0002.3800.001 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:760(findsource) 46741.6310.0001.6310.000 {built-in method nt.stat} 46300.0140.0001.6300.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\linecache.py:53(checkcache) 13952/138900.2810.0000.9070.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:714(getmodule) 9322/92600.0380.0000.7030.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:680(getsourcefile) 139520.0180.0000.2590.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:702(getabsfile) 140140.0170.0000.2150.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\ntpath.py:538(abspath) 4781660.1770.0000.1770.000 {built-in method builtins.hasattr} Total runtime reduced from 4.8 s to 2.8 s, and the major difference can be seen in the `nt.stat` call. This is on a machine with SSD. In workplace where python is installed in cluster on nfs, I'm sure the performance is worse. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33587> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33587] inspect.getsource performs unnecessary filesystem stat call which is ignored in some circumstances
New submission from Pankaj Pandey <panka...@gmail.com>: The specific os.path.exists() call is shown here: https://github.com/python/cpython/pull/6805 If the filename is already in linecache or the module has a PEP 302 loader, then the os.path.exists() call is ignored. Hence it is better to call it only if its value is needed. Attached a very simple case which shows the impact of the change: -- files: inspect_stack_perf.py messages: 317186 nosy: Pankaj Pandey priority: normal pull_requests: 6667 severity: normal status: open title: inspect.getsource performs unnecessary filesystem stat call which is ignored in some circumstances Added file: https://bugs.python.org/file47607/inspect_stack_perf.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33587> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: why does memory consumption keep growing?
On Thu, Oct 5, 2017 at 17:07 Fetchinson . via Python-list < python-list@python.org> wrote: > Hi folks, > > I have a rather simple program which cycles through a bunch of files, > does some operation on them, and then quits. There are 500 files > involved and each operation takes about 5-10 MB of memory. As you'll > see I tried to make every attempt at removing everything at the end of > each cycle so that memory consumption doesn't grow as the for loop > progresses, but it still does. > > import os > > for f in os.listdir( '.' ): > > x = [ ] > > for ( i, line ) in enumerate( open( f ) ): > > import mystuff > x.append( mystuff.expensive_stuff( line ) ) > del mystuff > > import mystuff > mystuff.some_more_expensive_stuff( x ) > del mystuff > del x > > > What can be the reason? I understand that mystuff might be leaky, but > if I delete it, doesn't that mean that whatever memory was allocated > is freed? Similary x is deleted so that can't possibly make the memory > consumption go up. > > Any hint would be much appreciated, > Daniel You are not closing f anywhere. Better to use a context manager, so it does the clean up on exit. for f in os.listdir( '.' ): with open(f) as fh: for (i, line) in enumerate(fh): #your code. #for loop done #context manager flushes, closes fh nicely. P > > > > -- > Psss, psss, put it down! - http://www.cafepress.com/putitdown > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Looking for Python Engineering Manager or Python experts for world's leading Artificial Intelligence Company in San Francisco,CA
Hi, I am looking to hire for an excellent opportunity for an Python Engineering Manager or Python experts for world's leading Arti-0878ficial Intelligence Company in San Francisco,CA. Please do refer or contact me at cell 510-396-0878. Both Full time Perm and Contract opportunities are open. Appreciate any help. Thanks so much, Pankaj Director Best Practices Openmind Technologies Inc Cell 510-396-0878. -- https://mail.python.org/mailman/listinfo/python-list
[issue25321] Null Dereference "subname" in zipimport.c
New submission from Pankaj Sharma: Hi , The issues reported as String NULL dereferecne at line Python-2.7.10/Modules/zipimport.c:266 for "subname" which assigned by get_subname(fullname) at line 264.so there should be check "subname" for NULL and return error,if it is NULL. the respective patch have been attached ,please review it. Thanks -- components: Library (Lib) files: Python-2.7.10-zipimport-01.patch keywords: patch messages: 252377 nosy: benjamin.peterson, pankaj.s01, twouters priority: normal severity: normal status: open title: Null Dereference "subname" in zipimport.c type: crash versions: Python 2.7 Added file: http://bugs.python.org/file40695/Python-2.7.10-zipimport-01.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24854] Null check handle return by new_string()
New submission from Pankaj Sharma: The issue reported in python-2.7.10/Parser/tokenizer.c:237 to handle NULL return by new_string() if PyMem_MALLOC() failed. So need to check for NULL and return to prevent from crash happened in get_normal_name().this issue related with issue18470 has been taken care by setting error code E_NOMEM in 3.4.X. i have attached patch, please review it. -- files: Python-2.7.10-tokenizer.patch keywords: patch messages: 248498 nosy: benjamin.peterson, pankaj.s01 priority: normal severity: normal status: open title: Null check handle return by new_string() type: crash versions: Python 2.7 Added file: http://bugs.python.org/file40172/Python-2.7.10-tokenizer.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24854 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24734] Dereferencing a null returning value
New submission from Pankaj Sharma: Hi, there is an issue reported for dereferencing a null return by function PyString_FromStringAndSize() file:Python-2.7.10/Python/compile.c .so there should be check for NULL before dereferencing in function compiler_nameop(). Please review attached patch. Thanks! -- components: Interpreter Core, Library (Lib) files: Python-2.7.10_compile.patch keywords: patch messages: 247459 nosy: benjamin.peterson, pankaj.s01 priority: normal severity: normal status: open title: Dereferencing a null returning value type: crash versions: Python 2.7 Added file: http://bugs.python.org/file40036/Python-2.7.10_compile.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24734 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24719] Resourc Leak in cPickle Module
New submission from Pankaj Sharma: Hi, There is resource leak in cPickle module in function Pickle_getvalue(). so need to free before return . I have attached patched ,please review it. Thnaks! -- components: Extension Modules, Library (Lib) files: Python-2.7.10-cPickle.patch keywords: patch messages: 247345 nosy: benjamin.peterson, pankaj.s01 priority: normal severity: normal status: open title: Resourc Leak in cPickle Module type: resource usage versions: Python 2.7 Added file: http://bugs.python.org/file40016/Python-2.7.10-cPickle.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24719 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24704] Dereferencing a Null Pointer
New submission from Pankaj Sharma: Hi, Reporting bugs for dereferencing a pointer m might be NULL. the respective patch have been attached ,please check and review it. thanks! -- components: Extension Modules, Library (Lib) files: Python-2.7.10-json.patch keywords: patch messages: 247262 nosy: benjamin.peterson, pankaj.s01 priority: normal severity: normal status: open title: Dereferencing a Null Pointer type: crash versions: Python 2.7 Added file: http://bugs.python.org/file40005/Python-2.7.10-json.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24704 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24302] Dead Code of Handler check in function faulthandler_fatal_error()
Changes by Pankaj Sharma pankaj@samsung.com: -- nosy: +benjamin.peterson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24302 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: OOP for System Administrator
Yes Michael, your guess is right. I once was a Java Programmer :) Thanks for the advice and link. Regards, -- https://mail.python.org/mailman/listinfo/python-list
OOP for System Administrator
Hi, I'm a Linux system administrator and my work requires me to write bash scripts (100-500 lines) for process monitoring, server health check and automate some manual processes. Now I've started to learn python as I want to write scripts in python rather than bash or any shell. My questions is Do I need to learn Object oriented Python in order to be good at python scripting. I know basics of OOP but personally didn't like it much. Thanks Pankaj -- https://mail.python.org/mailman/listinfo/python-list
[issue15108] ERROR: SystemError: ./../Objects/tupleobject.c:118: bad argument to internal function
Pankaj D p...@comcast.net added the comment: I believe I have found the root-cause for this issue. It is occurring due to the use of the garbage collector in another “memMonitor” thread (we run it periodically to get stats on objects, track mem leaks, etc). Since _pysqlite_fetch_one_row() releases the GIL before calling PyTuple_SetItem(), if the memMonitor is scheduled to run and, say, calls gc.get_objects(), it increments the refcount on all tracked objects (via append_objects()-PyList_Append()-app1()-PY_INCREF()).I have stack traces to confirm. This seems to rule out the use of gc methods (such as get_objects(), get_referrers/referents()) in multi-threaded programs or have them handle SystemError arising from such usage. Agree? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15108 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15108] ERROR: SystemError: ./../Objects/tupleobject.c:118: bad argument to internal function
Pankaj D p...@comcast.net added the comment: Wondering the same thing myself, and yes sqlite3_column_type() by itself doesn't seem expensive. I assumed in general it was to allow more responsiveness for apps with huge number of columns (i.e. large tuple size). But we have about 20-25 columns and so I was going to try removing it and seeing the results. In any case, it seems, fewer GIL acquire/releases will help with throughput. Are there any guidelines on when GIL should be released? Re PyTuple_SET_ITEM...yes that's also a possibility but it would then hide genuine bugs. -- versions: +Python 2.6 -Python 2.7, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15108 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15108] ERROR: SystemError: ./../Objects/tupleobject.c:118: bad argument to internal function
New submission from Pankaj D p...@comcast.net: Hi, Sporadically, while running an sqlite3 query, the above error is seen. In order to debug, I modified Objects/tupleobject.c, PyTuple_SetItem() as follows: if (!PyTuple_Check(op) || op-ob_refcnt != 1) { Py_XDECREF(newitem); /* * Temp: Bug XYZ Generate core so that we can debug * * PyErr_BadInternalCall(); * return -1; */ char errmsg[200]; sprintf(errmsg, Bug XYZ: PyTuple_Check(op) = %d op-ob_refcnt = %d; see core\n, PyTuple_Check(op), op-ob_refcnt); Py_FatalError(errmsg); } This generates a core with the following bt. Showing the top few frames only: (gdb) bt #0 0x000800acd3fc in thr_kill () at thr_kill.S:3 #1 0x000800b5e283 in abort () at /build/mnt/src/lib/libc/stdlib/abort.c:65 #2 0x00494acf in Py_FatalError (msg=Variable msg is not available. ) at ./../Python/pythonrun.c:1646 #3 0x0044e740 in PyTuple_SetItem (op=0x80c6e6308, i=16, newitem=0x80a80d780) at ./../Objects/tupleobject.c:128 #4 0x000807298866 in _pysqlite_fetch_one_row (self=0x80b846e48) at _sqlite/cursor.c:402 #5 0x000807298bf5 in pysqlite_cursor_iternext (self=0x80b846e48) at _sqlite/cursor.c:898 #6 0x00476943 in PyEval_EvalFrameEx (f=0x80a94d420, throwflag=Variable throwflag is not available. ) at ./../Python/ceval.c:2237 #7 0x0047acbf in PyEval_EvalFrameEx (f=0x80a94b820, throwflag=Variable throwflag is not available. ) at ./../Python/ceval.c:3765 #8 0x0047bf09 in PyEval_EvalCodeEx (co=0x808d2ac60, globals=Variable globals is not available. ) at ./../Python/ceval.c:2942 ... (gdb) fr 4 #4 0x000807298866 in _pysqlite_fetch_one_row (self=0x80b846e48) at _sqlite/cursor.c:402 402 PyTuple_SetItem(row, i, converted); Current language: auto; currently c (gdb) l 397 converted = buffer; 398 } (gdb) p *(PyTupleObject *)row $11 = {ob_refcnt = 2, ob_type = 0x60fee0, ob_size = 22, ob_item = {0x80a534030}} 'row' was allocated via PyTuple_New() but, somehow its refcount has become 2 while setting the 16th item!!! Is this a known issue? If not, what can I do to debug. Thanks, Pankaj -- messages: 163224 nosy: pxd priority: normal severity: normal status: open title: ERROR: SystemError: ./../Objects/tupleobject.c:118: bad argument to internal function type: behavior versions: Python 2.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15108 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15108] ERROR: SystemError: ./../Objects/tupleobject.c:118: bad argument to internal function
Pankaj D p...@comcast.net added the comment: sorry, 2.7, 3.2 is not an option currently but I am hoping someone can provide enough info to help probe this more efficiently. There seem to be references to this issue on the web but no root-cause. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15108 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
exp_continue
I am using pexpect module in python I would like to know how I can use exp_continue here. what is the alternative of exp_continue of perl/tcl in pexpect. -- http://mail.python.org/mailman/listinfo/python-list
exp_internal in pexpect
how to enable expect log in pexpect ? (similar as exp_internal in expect) -- http://mail.python.org/mailman/listinfo/python-list
timeout restart in pexpect
I am using pexpect to automate some procedures. My procedures are time cosuming, some messages coming out from that, in failure cases it will not output anything. So I would like to give small timeout value but when some message is received, it should restart the timeout, similar like restart_timeout_upon_receive in perl. Is it possible in Python using pexpect ? -- http://mail.python.org/mailman/listinfo/python-list
Re: symbolic links, aliases, cls clear
its true for the processes, gone to the system mode and from there the process is not considering the shell variable any more. In case of deamons also it is the same case. For Phython programs, I am not sure , may they are also nohops, so might be not accessing the shell variables. But i think you have defined the alias to some other place the error is reported by shell that cls not found. This could be investigated if you do some tests with the system. Like alias is shell script you can open it and read it , how it invokes the command for creating aliases. etc etc. Alias: When the shell is in picture it directly replaces the string cls to clear and execute the clear. Symbolic links: are reference to the inode of the actual files, so it should work in your case. just run #which clear (O/P , you will get the location of binary say /usr/bin/clear) Now you can create the symbolic link named as cls at location /usr/bin. this symbolic link should be a soft link to /usr/bin/clear. you can also put the symbolic links to any of the location that is displayed by $PATH variable. for any command you execute in shell, it functions in following order: (Not exactly the same, need to confirm, i might miss some of the steps) 1) check with the shell variable(whether any alias to this command exists or not!) 2) search it in the locations $PATH and execute the command is it is found. so on... -- http://mail.python.org/mailman/listinfo/python-list
Calling C++ function from python script
The module which i am creating is like Part A: 1. It does some processing by using python code. 2. The result of this python code execution is written to a text file. [This part is already compelete]] Part B: 1. I read a text file which is outputted by above python script in a C++ program 2. and again output of this c++ code is text file [This part is as well complete with graph data structure construction part involved, which i feel can be done in better way in C++ than in python] Now i want to integrate this flow. The communication between part A and part B is by call to a function present in C++ How should i do that? Kindly suggest some ways. Regards Pankaj -- http://mail.python.org/mailman/listinfo/python-list
Re: Calling C++ function from python script
In my case, i want to use python script as parent script and call C++ function from within the python script. If this is the case, then what am i doing? 1. extending 2. embedding. ? Ravi Teja wrote: Pankaj wrote: The module which i am creating is like Part A: 1. It does some processing by using python code. 2. The result of this python code execution is written to a text file. [This part is already compelete]] Part B: 1. I read a text file which is outputted by above python script in a C++ program 2. and again output of this c++ code is text file [This part is as well complete with graph data structure construction part involved, which i feel can be done in better way in C++ than in python] Now i want to integrate this flow. The communication between part A and part B is by call to a function present in C++ How should i do that? Kindly suggest some ways. Regards Pankaj If the bulk of your code is in Python, write an extension module in C++. You can use Swig or Boost here. If the bulk of your code is in C++, embed Python. The general rule is extend, not embed Python documentation http://docs.python.org/ext/ext.html Unless you know C/C++ well, these are not easy topics. Boost http://www.boost.org/libs/python/doc/tutorial/doc/html/index.html Swig, SIP are other good alternatives. All the above are relatively easy. If your C++ can be simplified to C, Pyrex is remarkably easy to write extensions in. In this case you will import/include a C header containing your function and write a Pyrex wrapper around it rather than directly using Pyrex as a language. If you are on MS Windows, you can always write a COM server in either language (COM in C++ is not easy unless if you are using something like C++ Builder). Or you can simply use popen from Python or use pipes or just read stdout from either side. As you can see, there are a number of options. -- http://mail.python.org/mailman/listinfo/python-list
Re: Calling C++ function from python script
The examples given are too complicated. So, if it can be explained using my sample example. Would be thankful for that. /* 1.c File **/ func( char a[10] ) { int i; for( i =0; i 10; i++) printf(\n array element is: %c, a[i]); } /* 1.py File **/ f = open( x.txt, r) while 1: l = f.readline( ) = Here i want to call func() as func( l) -- http://mail.python.org/mailman/listinfo/python-list
Re: Calling C++ function from python script
See, i tell u, nothing is difficult in this world. I achieved the thing which i was asking for. Thanks for the advice. I used this paper: http://www.swig.org/tutorial.html for creating python modules from C code. Which is what i needed. In order to interface both things, we need convert atleast one thing to other so i choosed to convert C code to python module. The steps which i followed were: These steps were to create a python module from C code. So a mandatory condition for .c file is : it should not have main function, and any variable in any function called from main function should be declared global. 1. Creating wrapper from .i : swig -python TestCase.i (where TestCase.i is interface file containing declarations of functions and variables) 2. Creating .o's : gcc -c TestCase.c TestCase_wrap.c (TestCase_wrap.c is file genereted by swig and is a wrapper for creating a python module) 3. Shared Library: ld -shared TestCase.o TestCase_wrap.o TestCase.so Module was inserted as: 1. export LD_LIBRARY_PATH=$PWD:$LD_LIBRARY_PATH 2. python Tc.py ( In this python file i did: import TestCase and then used it as TestCase.main_module() ** So, we don't need to write any code to do this. -- http://mail.python.org/mailman/listinfo/python-list
Re: Calling C++ function from python script
One small thing was incorrect there. gcc -c TestCase.c TestCase_wrap.c -I /usr/include/python2.2/ This include path is required for Python.h file -- http://mail.python.org/mailman/listinfo/python-list
Executing python script in non-interactive mode on cygwin (windows)
I am facing a very basic problem as any new bie would face. I know perl and now i want to use python In perl, it is very simple , just perl scriptname.pl will execute the script. But i think python is worse than perl It takes to interactive mode , which i don;t want to use. python scriptname.py does not execute python script It was, i don't know executing it or not. 1. I placed these contents in a file named 1.py a,b=0,1 n=5 while n=0: print b a=b b=a+b n=n-1 print n Then 2. on cygwin shell in windows i gave python 1.py hoping that it will exeucte and give me output, but nothing was printed Does python not support teh non-interactive mode of execution I have the manuals and they dont' specify this thing anywhere. they tell use python -c filename Please help. A newbie's request. Pankaj i gav -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script in non-interactive mode on cygwin (windows)
Thanks it was my mistake -- http://mail.python.org/mailman/listinfo/python-list
Backreferences in python ?
I have something like below in perl and i am searching for equivalent in python: ::: Perl ::: *** while( FILEHANDLE ) { line = $_; pattern = printf\( \$lineNo \ \),; line =~ s/for(.*)\((*.)\;(.*)/for$1\($pattern$2\;$3/g; } This is used to search for :for ( i = 0; i 10; i++) Replace with: for( printf( 10 ), i =0; i 10; i++) Where 10 is the line no. What i tried in python was:: f = open( ./1.c, r) fNew = open( ./1_new.c, w) for l in f: print l lineno = lineno + 1 strToFind = for\((.*)\;(.*) ## For Converting int to string, i.e. line no. to string lineNoClone = lineno pattern = printf(\ + str( lineNoClone) + \), print pattern strToReplace = for\( + pattern + \1\; fNew.write( l.replace( strToFind, strToReplace) ) print l fNew.close() -- http://mail.python.org/mailman/listinfo/python-list
Re: Backreferences in python ?
My tries have with re have not yielded results:: { strToFind = 'for*;*' ## Converting int to string, i.e. line no. to string lineNoClone = lineno pattern = printf(\ + str( lineNoClone) + \), regObj = re.compile( strToFind) m = regObj.search( l) if ( m != None ) : subStrPattern1_hasInitialization = \1 #m.group(1) subStrPattern2_hasRestTillEnd = \2 #m.group(2) strToReplace = for( + pattern + subStrPattern1_hasInitialization + ; + subStrPattern2_hasRestTillEnd fNew.write( regObj.sub( strToFind, strToReplace ) ) else: fNew.write( l) } Here problem is , i am not getting backreferences using \1 and \2 The string : for( i =0; i 10; i++) is getting replace by: for *;* { I don't believe this, they have given that \1 and \2 store backreferences, then where r they?? -- http://mail.python.org/mailman/listinfo/python-list
Re: Backreferences in python ?
I got my answer if ( m != None ) : subStrPattern1_hasInitialization = m.group(1) subStrPattern2_hasRestTillEnd = m.group(2) str = subStrPattern1_hasInitialization + subStrPattern2_hasRestTillEnd strToReplace = for( + pattern + str This gave me my solution But to tell u, i have not got that, why i should concatenate and then only place it . while i was trying the same thing by concatenation it straight to replace string, it was not working Any body has reasons ??? -- http://mail.python.org/mailman/listinfo/python-list
Accesing Global variables in python
Hi All, I have a service written in python (on Apache 2.0) which Initializes the global variables (or objects) when request comes for it. I want to use the same objects (or varaibles) in the subsequent requests. How could this one be done in Apache 2.0? can it be done by doing some configuration ? Sincere Thanks, Pankaj -- http://mail.python.org/mailman/listinfo/python-list
Accesing Global variables in multiple Requests (Apache 2.0)
Hi All, I have a service written in python (on Apache 2.0) which Initializes the global variables (or objects) when request comes for it. I want to use the same objects (or varaibles) in the subsequent requests. How could this one be done in Apache 2.0? can it be done by doing some configuration ? Sincere Thanks, Pankaj -- http://mail.python.org/mailman/listinfo/python-list
['ext.IsDOMString', 'ext.SplitQName']
Hi, I am new to Python-XML programming. I am using python 2.3.4, py2exe 0.5.3, pyXML 0.8.4. While making executable by py2exe I am getting folowing warning:- The following modules appear to be missing ['ext.IsDOMString', 'ext.SplitQName'] Even after trying the option:- setup.py py2exe --includes xml.sax.drivers2.drv_pyexpat I am getting the same Warning. Please, can anybody help me out.. I will be highly obliged. Thanks and regards Pankaj Jindal Aeroflex E-Newsletter Subscribe now to receive Aeroflex newsletters. As a subscriber, you will receive all the latest Aeroflex and Racal Instruments Wireless Solutions news including upcoming products, specials and other Aeroflex related information. Sign up now for our FREE monthly newsletter http://www.aeroflex.com/aboutus/enewsletter/signup-form.cfm This email and any attached files are confidential and intended solely for the use of the individual/s or entity to whom it is addressed. If you have received this email in error please notify:- [EMAIL PROTECTED] Racal Instruments Wireless Solutions Limited. An AEROFLEX Company. Visit our website at:- http://www.aeroflex.com/riws/ This footnote also confirms that this email message has been swept for the presence of computer viruses. -- http://mail.python.org/mailman/listinfo/python-list