Re: general module auditing
On 4-7-2014 1:09, Rita wrote:
here is what I am doing now,
egrep 'from|import' *.py | wc -l which is giving me that. But this does not
give me the number of times the particular module gets called. I was
thinking of adding a logging feature to all of my modules so every time
they get called it will be written to a log file with corresponding host
and username. Is there an easy way to do that?
Okay I've read up a bit on Python import hooks and came up with the following
code.
It hooks into Python's import mechanim by putting a custom loader into
sys.meta_path
that logs the time, the user, the machine and the name of the module being
imported.
Theoretically you could load this as a startup module (sitecustomize.py?) and
make it
log to a central location or something like that.
The code at the end of this message outputs the following on my machine:
$ python audit.py
Hello I'm about to import a module.
2014-07-04 19:01:12,321 [irmen@Neptune] importing /cgi
2014-07-04 19:01:12,323 [irmen@Neptune] importing /urlparse
2014-07-04 19:01:12,328 [irmen@Neptune] importing /mimetools
[... and some more modules...]
Bye.
Code follows:
import sys
import logging
import socket
import os
import getpass
def setup_import_logging():
class ContextFilter(logging.Filter):
def filter(self, record):
record.hostname = socket.gethostname()
if sys.version_info (3, 0):
record.username = getpass.getuser()
else:
record.username = os.getlogin()
return True
# configure the logger, adapt as desired:
logging.basicConfig(level=logging.DEBUG, format=%(asctime)s
[%(username)s@%(hostname)s] %(message)s)
class AuditingImporter(object):
log = logging.getLogger(auditingimporter)
log.setLevel(logging.DEBUG)
log.addFilter(ContextFilter())
def find_module(self, fullname, path):
return self.find_spec(fullname, path)
def find_spec(self, fullname, path, target=None):
self.log.debug(importing {path}/{fullname}.format(path=path or ,
fullname=fullname))
return None
sys.meta_path.insert(0, AuditingImporter())
setup_import_logging()
print(Hello I'm about to import a module.)
import cgi # will generate a load of logging entries
print(Bye.)
--
https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On 4-7-2014 19:05, Irmen de Jong wrote: The code at the end of this message outputs the following on my machine: [...] hmm the formatting got screwed up a bit it seems. Here's the same code: https://gist.github.com/irmen/c3d07118a8e1a00367f5 Irmen -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On 03/07/2014 02:17, Rita wrote: On Wed, Jul 2, 2014 at 2:46 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl wrote: On 2-7-2014 4:04, Rita wrote: yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. (Please don't top-post, it makes the discussion harder to follow.) On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list [...] Why would the fact that you upgrade or remove a package, affect another developer in your group? Are you all using the same machine to develop on, with one Python installation? I think you'll have to tell us some more details about the way you work together before we can give a meaningful answer to your question. Irmen -- https://mail.python.org/mailman/listinfo/python-list we have a shared mount point which has our python install. we have 3 servers on one part of the campus and 2 in another part. I want to find out what packages our user base is using thats the final goal. I can figure out who is using python by writing a wrapper but not what module. -- --- Get your facts first, then you can distort them as you please.-- You can check every users's program for import statements but do you really need to, why not check what's in the site-packages folder for your python install? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On Thu, Jul 3, 2014 at 2:54 AM, Mark Lawrence breamore...@yahoo.co.uk wrote: On 03/07/2014 02:17, Rita wrote: On Wed, Jul 2, 2014 at 2:46 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl wrote: On 2-7-2014 4:04, Rita wrote: yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. (Please don't top-post, it makes the discussion harder to follow.) On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list [...] Why would the fact that you upgrade or remove a package, affect another developer in your group? Are you all using the same machine to develop on, with one Python installation? I think you'll have to tell us some more details about the way you work together before we can give a meaningful answer to your question. Irmen -- https://mail.python.org/mailman/listinfo/python-list we have a shared mount point which has our python install. we have 3 servers on one part of the campus and 2 in another part. I want to find out what packages our user base is using thats the final goal. I can figure out who is using python by writing a wrapper but not what module. -- --- Get your facts first, then you can distort them as you please.-- You can check every users's program for import statements but do you really need to, why not check what's in the site-packages folder for your python install? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list how can i get frequency of the module usage? thats the end goal. -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On 03/07/2014 10:27, Rita wrote: On Thu, Jul 3, 2014 at 2:54 AM, Mark Lawrence breamore...@yahoo.co.uk mailto:breamore...@yahoo.co.uk wrote: On 03/07/2014 02:17, Rita wrote: On Wed, Jul 2, 2014 at 2:46 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl__ wrote: On 2-7-2014 4:04, Rita wrote: yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. (Please don't top-post, it makes the discussion harder to follow.) On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl__ wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list [...] Why would the fact that you upgrade or remove a package, affect another developer in your group? Are you all using the same machine to develop on, with one Python installation? I think you'll have to tell us some more details about the way you work together before we can give a meaningful answer to your question. Irmen -- https://mail.python.org/__mailman/listinfo/python-list https://mail.python.org/mailman/listinfo/python-list we have a shared mount point which has our python install. we have 3 servers on one part of the campus and 2 in another part. I want to find out what packages our user base is using thats the final goal. I can figure out who is using python by writing a wrapper but not what module. -- --- Get your facts first, then you can distort them as you please.-- You can check every users's program for import statements but do you really need to, why not check what's in the site-packages folder for your python install? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence how can i get frequency of the module usage? thats the end goal. -- --- Get your facts first, then you can distort them as you please.-- Count the number of imports or count the times a given program gets run for the number of imports depending on what you mean. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On Thu, Jul 3, 2014 at 8:36 AM, Mark Lawrence breamore...@yahoo.co.uk wrote: On 03/07/2014 10:27, Rita wrote: On Thu, Jul 3, 2014 at 2:54 AM, Mark Lawrence breamore...@yahoo.co.uk mailto:breamore...@yahoo.co.uk wrote: On 03/07/2014 02:17, Rita wrote: On Wed, Jul 2, 2014 at 2:46 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl__ wrote: On 2-7-2014 4:04, Rita wrote: yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. (Please don't top-post, it makes the discussion harder to follow.) On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl mailto:irmen.nos...@xs4all.nl __ wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list [...] Why would the fact that you upgrade or remove a package, affect another developer in your group? Are you all using the same machine to develop on, with one Python installation? I think you'll have to tell us some more details about the way you work together before we can give a meaningful answer to your question. Irmen -- https://mail.python.org/__mailman/listinfo/python-list https://mail.python.org/mailman/listinfo/python-list we have a shared mount point which has our python install. we have 3 servers on one part of the campus and 2 in another part. I want to find out what packages our user base is using thats the final goal. I can figure out who is using python by writing a wrapper but not what module. -- --- Get your facts first, then you can distort them as you please.-- You can check every users's program for import statements but do you really need to, why not check what's in the site-packages folder for your python install? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence how can i get frequency of the module usage? thats the end goal. -- --- Get your facts first, then you can distort them as you please.-- Count the number of imports or count the times a given program gets run for the number of imports depending on what you mean. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list here is what I am doing now, egrep 'from|import' *.py | wc -l which is giving me that. But this does not give me the number of times the particular module gets called. I was thinking of adding a logging feature to all of my modules so every time they get called it will be written to a log file with corresponding host and username. Is there an easy way to do that? -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On 04/07/2014 00:09, Rita wrote: here is what I am doing now, egrep 'from|import' *.py | wc -l which is giving me that. But this does not give me the number of times the particular module gets called. I was thinking of adding a logging feature to all of my modules so every time they get called it will be written to a log file with corresponding host and username. Is there an easy way to do that? Start here https://docs.python.org/3/library/logging.html#module-logging -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On 2-7-2014 4:04, Rita wrote: yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. (Please don't top-post, it makes the discussion harder to follow.) On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list [...] Why would the fact that you upgrade or remove a package, affect another developer in your group? Are you all using the same machine to develop on, with one Python installation? I think you'll have to tell us some more details about the way you work together before we can give a meaningful answer to your question. Irmen -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On Wed, Jul 2, 2014 at 2:46 PM, Irmen de Jong irmen.nos...@xs4all.nl wrote: On 2-7-2014 4:04, Rita wrote: yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. (Please don't top-post, it makes the discussion harder to follow.) On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list [...] Why would the fact that you upgrade or remove a package, affect another developer in your group? Are you all using the same machine to develop on, with one Python installation? I think you'll have to tell us some more details about the way you work together before we can give a meaningful answer to your question. Irmen -- https://mail.python.org/mailman/listinfo/python-list we have a shared mount point which has our python install. we have 3 servers on one part of the campus and 2 in another part. I want to find out what packages our user base is using thats the final goal. I can figure out who is using python by writing a wrapper but not what module. -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
general module auditing
i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list appdirs (1.3.0) colorama (0.3.1) cytoolz (0.6.1) docutils (0.11) Jinja2 (2.7.3) MarkupSafe (0.23) pip (1.5.6) py (1.4.20) pyflakes (0.8.1) Pygments (1.6) serpent (1.5, d:\projects\serpent) setuptools (4.0.1) smartypants (1.8.3) Sphinx (1.2.2) Sphinx-PyPI-upload (0.2.1) tox (1.7.1) virtualenv (1.11.6) wheel (0.23.0) You can even use pip freeze and use the resulting output as a requirements.txt file to easily install everything using pip install -r. Hope this is an idea to get you started? Irmen -- https://mail.python.org/mailman/listinfo/python-list
Re: general module auditing
yes, this helps. But I want to know who uses the module, serpent. So, when I upgrade it or remove it they won't be affected adversely. On Tue, Jul 1, 2014 at 2:16 PM, Irmen de Jong irmen.nos...@xs4all.nl wrote: On 1-7-2014 12:38, Rita wrote: i work in a group of developers (15 or so) who are located globally. I would like to know what modules everyone is uses if I ever have to upgrade my python. Is there mechanism which will let me see who is using what? ie, tom,matplotlib bob, pylab nancy, numpy nancy, matplotlib etc... Well, if your group is all using Pip (and perhaps even virtualenv), you could use pip list. In my case: $ pip list appdirs (1.3.0) colorama (0.3.1) cytoolz (0.6.1) docutils (0.11) Jinja2 (2.7.3) MarkupSafe (0.23) pip (1.5.6) py (1.4.20) pyflakes (0.8.1) Pygments (1.6) serpent (1.5, d:\projects\serpent) setuptools (4.0.1) smartypants (1.8.3) Sphinx (1.2.2) Sphinx-PyPI-upload (0.2.1) tox (1.7.1) virtualenv (1.11.6) wheel (0.23.0) You can even use pip freeze and use the resulting output as a requirements.txt file to easily install everything using pip install -r. Hope this is an idea to get you started? Irmen -- https://mail.python.org/mailman/listinfo/python-list -- --- Get your facts first, then you can distort them as you please.-- -- https://mail.python.org/mailman/listinfo/python-list
