Re: [Numpy-discussion] profiling line by line
On Sat, Sep 27, 2008 at 2:36 AM, Robert Kern [EMAIL PROTECTED] wrote: On Wed, Sep 17, 2008 at 07:12, Arnar Flatberg [EMAIL PROTECTED] wrote: On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern [EMAIL PROTECTED] wrote: There is now! Excellent work! I've had no problem with using the ipython-support (lprun), and it works for all my usecases, so far. Feel free to add it as highly recommended in the IPython installation instructions, though. :-) Most certainly will. It would be better to have a more stable release/download link than your hg repo. Do you have a page or a Pypi link for it yet? A little readme? I thought the hg-repos distribution was painless. The doc string says it all for usage, however a note on the ipy_user_conf setup should probably be documented some other place than just this thread. Thanks, Arnar ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Sat, Sep 27, 2008 at 00:50, Fernando Perez [EMAIL PROTECTED] wrote:
Hey,
On Fri, Sep 26, 2008 at 5:36 PM, Robert Kern [EMAIL PROTECTED] wrote:
There is now! Add
import line_profiler
ip.expose_magic('lprun', line_profiler.magic_lprun)
to your ipy_user_conf.py .
Mmh, should we ship this out of the box in ipython? The C dependency
is the only thing that gives me a bit of pause. There goes our pure
C, easy to build binaries for... On the other hand, me wants :)
What's your take on that?
I'd prefer not to. The exclusion of C modules is an eminently
reasonable one. You would inevitably make it optional, and my opinion
is that optional functionality is best handled by separate packages.
Feel free to add it as highly recommended in the IPython
installation instructions, though. :-)
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Wed, Sep 17, 2008 at 07:12, Arnar Flatberg [EMAIL PROTECTED] wrote:
On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern [EMAIL PROTECTED] wrote:
So, I could use some comments on the workflow. Does this look sensible
to everyone? How else would you like to use it?
Works for me. I would love to use it as a part of an ipython session.
Initially, I sprinkled som $profile in
a library of my own, then I thought I would use the -s (SETUP) option to
initialize some data to pass into the $profiled functions. However, I have
probably misunderstood the meaning of the -s option. Could you give an
example of its use? I ended in doing a %edit in ipython to write a small
test script with testdata and library imports, used kernprof.py and
view_line_prof.py from ipython and captured the output from view_line_prof
into ipython. Pheew now, there's perhaps an easier way?
There is now! Add
import line_profiler
ip.expose_magic('lprun', line_profiler.magic_lprun)
to your ipy_user_conf.py .
In [1]: %lprun?
Type: Magic function
Base Class: type 'instancemethod'
Namespace:IPython internal
File: /Users/rkern/hg/line_profiler/line_profiler.py
Definition: %lprun(self, parameter_s='')
Docstring:
Execute a statement under the line-by-line profiler from the
line_profiler module.
Usage:
%lprun -f func1 -f func2 statement
The given statement (which doesn't require quote marks) is run via the
LineProfiler. Profiling is enabled for the functions specified by the -f
options. The statistics will be shown side-by-side with the code through the
pager once the statement has completed.
Options:
-f function: LineProfiler only profiles functions and methods it is told
to profile. This option tells the profiler about these functions. Multiple
-f options may be used. The argument may be any expression that gives
a Python function or method object. However, one must be careful to avoid
spaces that may confuse the option parser. Additionally, functions defined
in the interpreter at the In[] prompt or via %run currently cannot be
displayed. Write these functions out to a separate file and import them.
One or more -f options are required to get any useful results.
-D filename: dump the raw statistics out to a marshal file on disk. The
usual extension for this is .lprof. These statistics may be viewed later
by running line_profiler.py as a script.
-T filename: dump the text-formatted statistics with the code side-by-side
out to a text file.
-r: return the LineProfiler object after it has completed profiling.
BTW, in the requirements you may want to mention argparse
(view_line_prof.py)
'tis gone now. The script and the requirement for argparse. Now you
just use line_profiler.py as a script.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
Hey,
On Fri, Sep 26, 2008 at 5:36 PM, Robert Kern [EMAIL PROTECTED] wrote:
There is now! Add
import line_profiler
ip.expose_magic('lprun', line_profiler.magic_lprun)
to your ipy_user_conf.py .
Mmh, should we ship this out of the box in ipython? The C dependency
is the only thing that gives me a bit of pause. There goes our pure
C, easy to build binaries for... On the other hand, me wants :)
What's your take on that?
f
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
Robert Kern wrote: On Fri, Sep 19, 2008 at 07:00, Robert Cimrman [EMAIL PROTECTED] wrote: Robert Kern wrote: Ah, found it. T_LONGLONG is a #define from structmember.h which is used to describe the types of attributes. Apparently, this was not added until Python 2.5. That particular member didn't actually need to be long long, so I've fixed that. Great, I will try it after it appears on the web page. Oops! It's now pushed. Thanks, it builds ok now! But then cProfile is also new in 2.5. - I should probably upgrade and stop bothering you :-) r. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Sun, Sep 21, 2008 at 02:09, Robert Cimrman [EMAIL PROTECTED] wrote: Robert Kern wrote: On Fri, Sep 19, 2008 at 07:00, Robert Cimrman [EMAIL PROTECTED] wrote: Robert Kern wrote: Ah, found it. T_LONGLONG is a #define from structmember.h which is used to describe the types of attributes. Apparently, this was not added until Python 2.5. That particular member didn't actually need to be long long, so I've fixed that. Great, I will try it after it appears on the web page. Oops! It's now pushed. Thanks, it builds ok now! But then cProfile is also new in 2.5. - I should probably upgrade and stop bothering you :-) Actually, if you pull again, I've just pushed my changes to remove the hard dependency on cProfile. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
Robert Kern wrote: On Sun, Sep 21, 2008 at 02:09, Robert Cimrman [EMAIL PROTECTED] wrote: Robert Kern wrote: On Fri, Sep 19, 2008 at 07:00, Robert Cimrman [EMAIL PROTECTED] wrote: Robert Kern wrote: Ah, found it. T_LONGLONG is a #define from structmember.h which is used to describe the types of attributes. Apparently, this was not added until Python 2.5. That particular member didn't actually need to be long long, so I've fixed that. Great, I will try it after it appears on the web page. Oops! It's now pushed. Thanks, it builds ok now! But then cProfile is also new in 2.5. - I should probably upgrade and stop bothering you :-) Actually, if you pull again, I've just pushed my changes to remove the hard dependency on cProfile. Cool, then I have another one: $ ./kernprof.py -l pystone.py Traceback (most recent call last): File ./kernprof.py, line 173, in ? sys.exit(main(sys.argv)) File ./kernprof.py, line 138, in main import line_profiler File /home/share/software/packages/line_profiler-2ea515d0cbc1/line_profiler.py, line 9, in ? from _line_profiler import LineProfiler as CLineProfiler File _line_profiler.pyx, line 3, in _line_profiler (_line_profiler.c:2246) from cProfile import label ImportError: No module named cProfile Or do I call the profiler in a wrong way? r. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Sun, Sep 21, 2008 at 02:26, Robert Cimrman [EMAIL PROTECTED] wrote: Cool, then I have another one: $ ./kernprof.py -l pystone.py Traceback (most recent call last): File ./kernprof.py, line 173, in ? sys.exit(main(sys.argv)) File ./kernprof.py, line 138, in main import line_profiler File /home/share/software/packages/line_profiler-2ea515d0cbc1/line_profiler.py, line 9, in ? from _line_profiler import LineProfiler as CLineProfiler File _line_profiler.pyx, line 3, in _line_profiler (_line_profiler.c:2246) from cProfile import label ImportError: No module named cProfile Or do I call the profiler in a wrong way? Nope, I just had a brainfart and put the implementation of label() in the wrong file. Pull again. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
Robert Kern wrote: On Sun, Sep 21, 2008 at 02:26, Robert Cimrman [EMAIL PROTECTED] wrote: Cool, then I have another one: $ ./kernprof.py -l pystone.py Traceback (most recent call last): File ./kernprof.py, line 173, in ? sys.exit(main(sys.argv)) File ./kernprof.py, line 138, in main import line_profiler File /home/share/software/packages/line_profiler-2ea515d0cbc1/line_profiler.py, line 9, in ? from _line_profiler import LineProfiler as CLineProfiler File _line_profiler.pyx, line 3, in _line_profiler (_line_profiler.c:2246) from cProfile import label ImportError: No module named cProfile Or do I call the profiler in a wrong way? Nope, I just had a brainfart and put the implementation of label() in the wrong file. Pull again. :-) The example works now in Python 2.4. r. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
Robert Kern wrote: On Thu, Sep 18, 2008 at 06:01, Robert Cimrman [EMAIL PROTECTED] wrote: Hi Robert, Robert Kern wrote: On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg [EMAIL PROTECTED] wrote: That would make me an extremely happy user, I've been looking for this for years! I can't imagine I'm the only one who profiles some hundred lines of code and ends up with 90% of total time in the dot-function For the time being, you can grab it here: http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ It requires Cython and a C compiler to build. I'm still debating myself about the desired workflow for using it, but for now, it only profiles functions which you have registered with it. I have made the profiler work as a decorator to make this easy. E.g., many thanks for this! I have wanted to try out the profiler but failed to build it (changeset 6 0de294aa75bf): $ python setup.py install --root=/home/share/software/ running install running build running build_py creating build creating build/lib.linux-i686-2.4 copying line_profiler.py - build/lib.linux-i686-2.4 running build_ext cythoning _line_profiler.pyx to _line_profiler.c building '_line_profiler' extension creating build/temp.linux-i686-2.4 i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c -o build/temp.linux-i686-2.4/_line_profiler.o _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a function) error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. It uses the #define'd macro PY_LONG_LONG. Go through your Python headers to see what this gets expanded to. I have Python 2.4.4 in pyconfig.h #define HAVE_LONG_LONG 1 in pyport.h: #ifdef HAVE_LONG_LONG #ifndef PY_LONG_LONG #define PY_LONG_LONG long long #endif #endif /* HAVE_LONG_LONG */ so it seems compatible with 'ctypedef long long PY_LONG_LONG' r. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
Ondrej Certik wrote: On Thu, Sep 18, 2008 at 4:12 PM, Ryan May [EMAIL PROTECTED] wrote: Ondrej Certik wrote: On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman [EMAIL PROTECTED] wrote: It requires Cython and a C compiler to build. I'm still debating myself about the desired workflow for using it, but for now, it only profiles functions which you have registered with it. I have made the profiler work as a decorator to make this easy. E.g., many thanks for this! I have wanted to try out the profiler but failed to build it (changeset 6 0de294aa75bf): $ python setup.py install --root=/home/share/software/ running install running build running build_py creating build creating build/lib.linux-i686-2.4 copying line_profiler.py - build/lib.linux-i686-2.4 running build_ext cythoning _line_profiler.pyx to _line_profiler.c building '_line_profiler' extension creating build/temp.linux-i686-2.4 i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c -o build/temp.linux-i686-2.4/_line_profiler.o _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a function) error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. I am telling you all the time Robert to use Debian that it just works and you say, no no, gentoo is the best. :) And what's wrong with that? :) Once you get over the learning curve, Gentoo works just fine. Must be Robert K.'s fault. :) Well, I think if Robert C. hasn't yet get over the learning curve after so many years of hard work, maybe the learning curve is too steep. :) This is most probably not related to Gentoo at all and certainly not related to me knowing Gentoo or not :) (and no, learning Gentoo is not that hard.) r. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Fri, Sep 19, 2008 at 03:33, Robert Cimrman [EMAIL PROTECTED] wrote: I have Python 2.4.4 in pyconfig.h #define HAVE_LONG_LONG 1 in pyport.h: #ifdef HAVE_LONG_LONG #ifndef PY_LONG_LONG #define PY_LONG_LONG long long #endif #endif /* HAVE_LONG_LONG */ so it seems compatible with 'ctypedef long long PY_LONG_LONG' Ah, found it. T_LONGLONG is a #define from structmember.h which is used to describe the types of attributes. Apparently, this was not added until Python 2.5. That particular member didn't actually need to be long long, so I've fixed that. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Fri, Sep 19, 2008 at 10:37 AM, Robert Cimrman [EMAIL PROTECTED] wrote: Ondrej Certik wrote: On Thu, Sep 18, 2008 at 4:12 PM, Ryan May [EMAIL PROTECTED] wrote: Ondrej Certik wrote: On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman [EMAIL PROTECTED] wrote: It requires Cython and a C compiler to build. I'm still debating myself about the desired workflow for using it, but for now, it only profiles functions which you have registered with it. I have made the profiler work as a decorator to make this easy. E.g., many thanks for this! I have wanted to try out the profiler but failed to build it (changeset 6 0de294aa75bf): $ python setup.py install --root=/home/share/software/ running install running build running build_py creating build creating build/lib.linux-i686-2.4 copying line_profiler.py - build/lib.linux-i686-2.4 running build_ext cythoning _line_profiler.pyx to _line_profiler.c building '_line_profiler' extension creating build/temp.linux-i686-2.4 i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c -o build/temp.linux-i686-2.4/_line_profiler.o _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a function) error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. I am telling you all the time Robert to use Debian that it just works and you say, no no, gentoo is the best. :) And what's wrong with that? :) Once you get over the learning curve, Gentoo works just fine. Must be Robert K.'s fault. :) Well, I think if Robert C. hasn't yet get over the learning curve after so many years of hard work, maybe the learning curve is too steep. :) This is most probably not related to Gentoo at all and certainly not related to me knowing Gentoo or not :) (and no, learning Gentoo is not that hard.) Let us know where the problem was. :) I am just using common sense, if something works on Debian and macosx and doesn't work on gentoo, I thought it was safe to say it was gentoo related, but I may well be wrong. :)) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Wed, Sep 17, 2008 at 18:29, Robert Kern [EMAIL PROTECTED] wrote: On Wed, Sep 17, 2008 at 18:09, Ondrej Certik [EMAIL PROTECTED] wrote: This is what I am getting: $ ./kernprof.py -l pystone.py Wrote profile results to pystone.py.lprof $ ./view_line_prof.py pystone.py.lprof Timer unit: 1e-06 s $ So I think you meant: $ ./kernprof.py -l mystone.py 20628 Wrote profile results to mystone.py.lprof $ ./view_line_prof.py mystone.py.lprof Timer unit: 1e-06 s File: pystone.py Function: Proc0 at line 79 Total time: 13.0803 s [...] Now it works. No, I meant pystone.py. My script-finding code may have (incorrectly) found a different, uninstrumented pystone.py file somewhere else, though. Try with ./pystone.py. There was a bug in how I was constructing the munged namespaces. Fixed now. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
Robert Kern wrote: Ah, found it. T_LONGLONG is a #define from structmember.h which is used to describe the types of attributes. Apparently, this was not added until Python 2.5. That particular member didn't actually need to be long long, so I've fixed that. Great, I will try it after it appears on the web page. Thank you, r. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Fri, Sep 19, 2008 at 07:00, Robert Cimrman [EMAIL PROTECTED] wrote: Robert Kern wrote: Ah, found it. T_LONGLONG is a #define from structmember.h which is used to describe the types of attributes. Apparently, this was not added until Python 2.5. That particular member didn't actually need to be long long, so I've fixed that. Great, I will try it after it appears on the web page. Oops! It's now pushed. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Thu, Sep 18, 2008 at 1:29 AM, Robert Kern [EMAIL PROTECTED] wrote:
On Wed, Sep 17, 2008 at 18:09, Ondrej Certik [EMAIL PROTECTED] wrote:
On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern [EMAIL PROTECTED] wrote:
On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg [EMAIL PROTECTED] wrote:
That would make me an extremely happy user, I've been looking for this for
years!
I can't imagine I'm the only one who profiles some hundred lines of code
and
ends up with 90% of total time in the dot-function
For the time being, you can grab it here:
http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
It requires Cython and a C compiler to build. I'm still debating
myself about the desired workflow for using it, but for now, it only
profiles functions which you have registered with it. I have made the
profiler work as a decorator to make this easy. E.g.,
from line_profiler import LineProfiler
profile = LineProfiler()
@profile
def my_slow_func():
...
profile.dump_stats('my_slow_func.lprof')
This is kind of inconvenient, so I have a little script that handles
everything except for the @profile itself. It started as a script to
run cProfile nicely, so it actually does that by default.
I took pystone.py from the Python source and added a couple of
@profile decorators to demonstrate:
[line_profiler]$ ./kernprof.py --help
Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg]
...
Options:
-h, --helpshow this help message and exit
-l, --line-by-lineUse the line-by-line profiler from the line_profiler
module instead of cProfile. Implies --builtin.
-b, --builtin Put 'profile' in the builtins. Use 'profile.enable()'
and 'profile.disable()' in your code to turn it on
and
off, or '@profile' to decorate a single function, or
'with profile:' to profile a single section of code.
-o OUTFILE, --outfile=OUTFILE
Save stats to outfile
-s SETUP, --setup=SETUP
Code to execute before the code to profile
[line_profiler]$ ./kernprof.py -l pystone.py
Pystone(1.1) time for 5 passes = 11.34
This machine benchmarks at 4409.17 pystones/second
Wrote profile results to pystone.py.lprof
[line_profiler]$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 8.46516 s
[...]
This is what I am getting:
$ ./kernprof.py -l pystone.py
Wrote profile results to pystone.py.lprof
$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
$
So I think you meant:
$ ./kernprof.py -l mystone.py
20628
Wrote profile results to mystone.py.lprof
$ ./view_line_prof.py mystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 13.0803 s
[...]
Now it works.
No, I meant pystone.py. My script-finding code may have (incorrectly)
found a different, uninstrumented pystone.py file somewhere else,
though. Try with ./pystone.py.
This is an excellent piece of software! Nice job. Just today I needed
such a thing!
How do you easily install it?
python setup.py install should have installed the module. I haven't
done anything with the scripts, yet.
I usually do python setup.py install
--home=~/lib and I have the PYTHONPATH + PATH setup in my .bashrc,
but I then need to manually remove the stuff from my ~/lib if I want
to uninstall, which sucks. So this time I just did python setup.py
build and moved the .so file manually to the current dir. But there
must be a better way. What is your workflow?
For things I am developing on, I build them in-place, using the
setuptools develop command to add the appropriate path to the
easy-install.pth file. To remove, I would just edit that file.
For thing I'm not developing on, I usually build and install an egg if
at all possible.
But then, I'm typically on a single-user box where I'm root, so I
sometimes do nasty and unsanitary things like chown -R rkern:rkern
/usr/local/.
Anyway, so I used it on my code and here is what I got:
File: hermes1d/mesh.py
Function: integrate_function at line 119
Total time: 0.647412 s
Line # Hits Time % Time Line Contents
=
119 @profile
120 def integrate_function(self, f):
121
122 Integrate the function
f on the element.
123
12496 1091 0.2 from numpy import array
12596 461070 71.2 from scipy.integrate
import quadrature
12696 496 0.1 a, b =
self.nodes[0].x, self.nodes[1].x
12796 418 0.1
Re: [Numpy-discussion] profiling line by line
On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman [EMAIL PROTECTED] wrote: Hi Robert, Robert Kern wrote: On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg [EMAIL PROTECTED] wrote: That would make me an extremely happy user, I've been looking for this for years! I can't imagine I'm the only one who profiles some hundred lines of code and ends up with 90% of total time in the dot-function For the time being, you can grab it here: http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ It requires Cython and a C compiler to build. I'm still debating myself about the desired workflow for using it, but for now, it only profiles functions which you have registered with it. I have made the profiler work as a decorator to make this easy. E.g., many thanks for this! I have wanted to try out the profiler but failed to build it (changeset 6 0de294aa75bf): $ python setup.py install --root=/home/share/software/ running install running build running build_py creating build creating build/lib.linux-i686-2.4 copying line_profiler.py - build/lib.linux-i686-2.4 running build_ext cythoning _line_profiler.pyx to _line_profiler.c building '_line_profiler' extension creating build/temp.linux-i686-2.4 i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c -o build/temp.linux-i686-2.4/_line_profiler.o _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a function) error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. I am telling you all the time Robert to use Debian that it just works and you say, no no, gentoo is the best. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
Ondrej Certik wrote: On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman [EMAIL PROTECTED] wrote: It requires Cython and a C compiler to build. I'm still debating myself about the desired workflow for using it, but for now, it only profiles functions which you have registered with it. I have made the profiler work as a decorator to make this easy. E.g., many thanks for this! I have wanted to try out the profiler but failed to build it (changeset 6 0de294aa75bf): $ python setup.py install --root=/home/share/software/ running install running build running build_py creating build creating build/lib.linux-i686-2.4 copying line_profiler.py - build/lib.linux-i686-2.4 running build_ext cythoning _line_profiler.pyx to _line_profiler.c building '_line_profiler' extension creating build/temp.linux-i686-2.4 i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c -o build/temp.linux-i686-2.4/_line_profiler.o _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a function) error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. I am telling you all the time Robert to use Debian that it just works and you say, no no, gentoo is the best. :) And what's wrong with that? :) Once you get over the learning curve, Gentoo works just fine. Must be Robert K.'s fault. :) Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
So the timing raises a lot. For obvious reasons, that's the overhead of the profiler. But the problem is that then the timings just don't fit, e.g. if I sum the total time spent in subfunctions, it doesn't account for all the time printed on the respective line in the parent function. I don't know if there is any way to fix it, or even worth fixing. So I guess one should just use the profiling info to determine which line to fix. Do you think it's worthy to implement line profiling for all lines and then make sure that all timings match? What is your experience. The reason I want this is so that I can determine just by looking at the timing how much time is spent at the line and how much time is spent at the functions that are being called at the line. I think it is doable, what do you think? One would trace how much time was wasted in python_trace_callback and then just substract this time from all parent function's lines timings. Btw Robin, how is Matlab doing it? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Wed, Sep 17, 2008 at 10:33 PM, Robert Kern [EMAIL PROTECTED] wrote: On Wed, Sep 17, 2008 at 07:12, Arnar Flatberg [EMAIL PROTECTED] wrote: On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern [EMAIL PROTECTED] wrote: It should be straightforward to make an ipython %magic, yes. There are some questions as to how that would work, though. Do you want the top-level code in the file to be profiled? I dont think that will be my primary use. Or do you just want functions defined inside to be profiled? That was my initial thought, yes Do you want to be able to specify functions defined outside of the code to be profiled? Sounds complicated, but that would be nice. I'll play around with ipythons %magic to see if I can get a workflow Im comfortable with. I was thinking something in the line of ipython's %prun for a quick looksy, and the possibility to easily write small scripts to test different inputs (small/large arrays etc.) to a specific function. Any recommendations are very welcome. I do not have much experience in profiling code. Arnar ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Thu, Sep 18, 2008 at 06:01, Robert Cimrman [EMAIL PROTECTED] wrote: Hi Robert, Robert Kern wrote: On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg [EMAIL PROTECTED] wrote: That would make me an extremely happy user, I've been looking for this for years! I can't imagine I'm the only one who profiles some hundred lines of code and ends up with 90% of total time in the dot-function For the time being, you can grab it here: http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ It requires Cython and a C compiler to build. I'm still debating myself about the desired workflow for using it, but for now, it only profiles functions which you have registered with it. I have made the profiler work as a decorator to make this easy. E.g., many thanks for this! I have wanted to try out the profiler but failed to build it (changeset 6 0de294aa75bf): $ python setup.py install --root=/home/share/software/ running install running build running build_py creating build creating build/lib.linux-i686-2.4 copying line_profiler.py - build/lib.linux-i686-2.4 running build_ext cythoning _line_profiler.pyx to _line_profiler.c building '_line_profiler' extension creating build/temp.linux-i686-2.4 i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c -o build/temp.linux-i686-2.4/_line_profiler.o _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a function) error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. It uses the #define'd macro PY_LONG_LONG. Go through your Python headers to see what this gets expanded to. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Thu, Sep 18, 2008 at 10:03, Ondrej Certik [EMAIL PROTECTED] wrote: Anyway, back to work: Robert K., I noticed that if I profile some function, I get results like this for example: 40 307246952 6.6 [x,w] = p_roots(n) 41 307224192 3.4 x = real(x) 42 307234470 4.8 ainf, binf = map(isinf,(a,b)) 43 3072 6565 0.9 if ainf or binf: 44 raise ValueError, Gaussian quadrature is only available for \ 45finite limits. 46 3072 5093 0.7 if not reference: 47 x = (b-a)*(x+1)/2.0 + a 48 3072 594190 83.5 return (b-a)/2.0*sum(w*func(x,*args),0) --- Then if I turn profiling of the func() method, I get this: 40 307246999 4.6 [x,w] = p_roots(n) 41 307224313 2.4 x = real(x) 42 307234327 3.4 ainf, binf = map(isinf,(a,b)) 43 3072 6190 0.6 if ainf or binf: 44 raise ValueError, Gaussian quadrature is only available for \ 45finite limits. 46 3072 4918 0.5 if not reference: 47 x = (b-a)*(x+1)/2.0 + a 48 3072 906876 88.6 return (b-a)/2.0*sum(w*func(x,*args),0) --- So the timing raises a lot. For obvious reasons, that's the overhead of the profiler. But the problem is that then the timings just don't fit, e.g. if I sum the total time spent in subfunctions, it doesn't account for all the time printed on the respective line in the parent function. I don't know if there is any way to fix it, or even worth fixing. So I guess one should just use the profiling info to determine which line to fix. So here's what going on: I'm being clever (and possibly too clever). When tracing is enabled, Python will call my tracing function just before each new line gets executed. If tracing isn't enabled for this function, I return. Otherwise, I grab the current time. Then, I look for the last line and time I recorded for this function. I look up the accumulator for the (code, old line) pair and record the time delta. Then I grab the current time *again*, and store the current line and this new time for the next go 'round. This way, I exclude most of the time spent inside the profiler itself and just record the time being spent in the code. The total time reported is just a sum of the recorded times, not the sum of wall-clock times spent in the function. Now, this does break down in your use case where you are profiling both the caller and callee and trying to determine how much of a line's time is being spent just by calling the function. I could record wall-clock times between the start and end of a function call, but I think that's fragile. For example, suppose you are profiling A() and B() but not C(), and both A() and C() call B(). Using the wall-clock time spent in B() will tell you that you spent more time in B() than the appropriate line (or lines!) that called it in A(). I think the most robust way to figure this out is to rewrite your code to pull out such calls onto their own lines. This is like breaking up your functions into tiny little one-liners in order to use cProfile only it doesn't suck *nearly* as much. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Thu, Sep 18, 2008 at 02:54:13PM -0500, Robert Kern wrote: So here's what going on: I'm being clever (and possibly too clever). Oh no. Robert K. is too clever. We knew that, right ;). Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern [EMAIL PROTECTED] wrote: So, I could use some comments on the workflow. Does this look sensible to everyone? How else would you like to use it? Works for me. I would love to use it as a part of an ipython session. Initially, I sprinkled som $profile in a library of my own, then I thought I would use the -s (SETUP) option to initialize some data to pass into the $profiled functions. However, I have probably misunderstood the meaning of the -s option. Could you give an example of its use? I ended in doing a %edit in ipython to write a small test script with testdata and library imports, used kernprof.py and view_line_prof.py from ipython and captured the output from view_line_prof into ipython. Pheew now, there's perhaps an easier way? BTW, in the requirements you may want to mention argparse (view_line_prof.py) Thanks for sharing ! Arnar ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Wed, Sep 17, 2008 at 07:12, Arnar Flatberg [EMAIL PROTECTED] wrote: On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern [EMAIL PROTECTED] wrote: So, I could use some comments on the workflow. Does this look sensible to everyone? How else would you like to use it? Works for me. I would love to use it as a part of an ipython session. Initially, I sprinkled som $profile in a library of my own, then I thought I would use the -s (SETUP) option to initialize some data to pass into the $profiled functions. However, I have probably misunderstood the meaning of the -s option. Could you give an example of its use? It's mostly for using cProfile on a whole script. Since line_profiler is already only active for certain functions, it's probably not all that useful. At the time I added the feature, I was trying to profile (part of) a GUI program using wx and VTK, both of which take a long time to load and taking up a good chunk of my profile time. If I had a script, load_wx_vtk.py: import wx from enthought.tvtk import tvtk Then I could just do: $ kernprof.py -s load_wx_vtk.py my_script.py The script at the end still needs to be the main thing that executes. I ended in doing a %edit in ipython to write a small test script with testdata and library imports, used kernprof.py and view_line_prof.py from ipython and captured the output from view_line_prof into ipython. Pheew now, there's perhaps an easier way? It should be straightforward to make an ipython %magic, yes. There are some questions as to how that would work, though. Do you want the top-level code in the file to be profiled? Or do you just want functions defined inside to be profiled? Do you want to be able to specify functions defined outside of the code to be profiled? BTW, in the requirements you may want to mention argparse (view_line_prof.py) Oops! Yes. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern [EMAIL PROTECTED] wrote:
On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg [EMAIL PROTECTED] wrote:
That would make me an extremely happy user, I've been looking for this for
years!
I can't imagine I'm the only one who profiles some hundred lines of code and
ends up with 90% of total time in the dot-function
For the time being, you can grab it here:
http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
It requires Cython and a C compiler to build. I'm still debating
myself about the desired workflow for using it, but for now, it only
profiles functions which you have registered with it. I have made the
profiler work as a decorator to make this easy. E.g.,
from line_profiler import LineProfiler
profile = LineProfiler()
@profile
def my_slow_func():
...
profile.dump_stats('my_slow_func.lprof')
This is kind of inconvenient, so I have a little script that handles
everything except for the @profile itself. It started as a script to
run cProfile nicely, so it actually does that by default.
I took pystone.py from the Python source and added a couple of
@profile decorators to demonstrate:
[line_profiler]$ ./kernprof.py --help
Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ...
Options:
-h, --helpshow this help message and exit
-l, --line-by-lineUse the line-by-line profiler from the line_profiler
module instead of cProfile. Implies --builtin.
-b, --builtin Put 'profile' in the builtins. Use 'profile.enable()'
and 'profile.disable()' in your code to turn it on and
off, or '@profile' to decorate a single function, or
'with profile:' to profile a single section of code.
-o OUTFILE, --outfile=OUTFILE
Save stats to outfile
-s SETUP, --setup=SETUP
Code to execute before the code to profile
[line_profiler]$ ./kernprof.py -l pystone.py
Pystone(1.1) time for 5 passes = 11.34
This machine benchmarks at 4409.17 pystones/second
Wrote profile results to pystone.py.lprof
[line_profiler]$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 8.46516 s
[...]
This is what I am getting:
$ ./kernprof.py -l pystone.py
Wrote profile results to pystone.py.lprof
$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
$
So I think you meant:
$ ./kernprof.py -l mystone.py
20628
Wrote profile results to mystone.py.lprof
$ ./view_line_prof.py mystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 13.0803 s
[...]
Now it works.
This is an excellent piece of software! Nice job. Just today I needed
such a thing!
How do you easily install it? I usually do python setup.py install
--home=~/lib and I have the PYTHONPATH + PATH setup in my .bashrc,
but I then need to manually remove the stuff from my ~/lib if I want
to uninstall, which sucks. So this time I just did python setup.py
build and moved the .so file manually to the current dir. But there
must be a better way. What is your workflow?
Anyway, so I used it on my code and here is what I got:
File: hermes1d/mesh.py
Function: integrate_function at line 119
Total time: 0.647412 s
Line # Hits Time % Time Line Contents
=
119 @profile
120 def integrate_function(self, f):
121
122 Integrate the function
f on the element.
123
12496 1091 0.2 from numpy import array
12596 461070 71.2 from scipy.integrate
import quadrature
12696 496 0.1 a, b =
self.nodes[0].x, self.nodes[1].x
12796 418 0.1 def func(x):
128 #print x
129 #print
array([f.f(y) for y in x])
130 return
array([f.f(y) for y in x])
13196 183903 28.4 val, err =
quadrature(func, a, b)
132 #print val, a, b
133 #stop
13496 434 0.1 return val
This is interesting, because 70% of time is spent on from
scipy.integrate import quadrature. So this is not good.
So I moved it out and voila! My code is 3.4x faster. I really didn't
know that importing (scipy.integrate) is *so* slow.
File: hermes1d/mesh.py
Function: integrate_function at line 121
Total time: 0.189429 s
Line # Hits Time % Time Line Contents
Re: [Numpy-discussion] profiling line by line
On Wed, Sep 17, 2008 at 18:09, Ondrej Certik [EMAIL PROTECTED] wrote:
On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern [EMAIL PROTECTED] wrote:
On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg [EMAIL PROTECTED] wrote:
That would make me an extremely happy user, I've been looking for this for
years!
I can't imagine I'm the only one who profiles some hundred lines of code and
ends up with 90% of total time in the dot-function
For the time being, you can grab it here:
http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
It requires Cython and a C compiler to build. I'm still debating
myself about the desired workflow for using it, but for now, it only
profiles functions which you have registered with it. I have made the
profiler work as a decorator to make this easy. E.g.,
from line_profiler import LineProfiler
profile = LineProfiler()
@profile
def my_slow_func():
...
profile.dump_stats('my_slow_func.lprof')
This is kind of inconvenient, so I have a little script that handles
everything except for the @profile itself. It started as a script to
run cProfile nicely, so it actually does that by default.
I took pystone.py from the Python source and added a couple of
@profile decorators to demonstrate:
[line_profiler]$ ./kernprof.py --help
Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg]
...
Options:
-h, --helpshow this help message and exit
-l, --line-by-lineUse the line-by-line profiler from the line_profiler
module instead of cProfile. Implies --builtin.
-b, --builtin Put 'profile' in the builtins. Use 'profile.enable()'
and 'profile.disable()' in your code to turn it on and
off, or '@profile' to decorate a single function, or
'with profile:' to profile a single section of code.
-o OUTFILE, --outfile=OUTFILE
Save stats to outfile
-s SETUP, --setup=SETUP
Code to execute before the code to profile
[line_profiler]$ ./kernprof.py -l pystone.py
Pystone(1.1) time for 5 passes = 11.34
This machine benchmarks at 4409.17 pystones/second
Wrote profile results to pystone.py.lprof
[line_profiler]$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 8.46516 s
[...]
This is what I am getting:
$ ./kernprof.py -l pystone.py
Wrote profile results to pystone.py.lprof
$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
$
So I think you meant:
$ ./kernprof.py -l mystone.py
20628
Wrote profile results to mystone.py.lprof
$ ./view_line_prof.py mystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 13.0803 s
[...]
Now it works.
No, I meant pystone.py. My script-finding code may have (incorrectly)
found a different, uninstrumented pystone.py file somewhere else,
though. Try with ./pystone.py.
This is an excellent piece of software! Nice job. Just today I needed
such a thing!
How do you easily install it?
python setup.py install should have installed the module. I haven't
done anything with the scripts, yet.
I usually do python setup.py install
--home=~/lib and I have the PYTHONPATH + PATH setup in my .bashrc,
but I then need to manually remove the stuff from my ~/lib if I want
to uninstall, which sucks. So this time I just did python setup.py
build and moved the .so file manually to the current dir. But there
must be a better way. What is your workflow?
For things I am developing on, I build them in-place, using the
setuptools develop command to add the appropriate path to the
easy-install.pth file. To remove, I would just edit that file.
For thing I'm not developing on, I usually build and install an egg if
at all possible.
But then, I'm typically on a single-user box where I'm root, so I
sometimes do nasty and unsanitary things like chown -R rkern:rkern
/usr/local/.
Anyway, so I used it on my code and here is what I got:
File: hermes1d/mesh.py
Function: integrate_function at line 119
Total time: 0.647412 s
Line # Hits Time % Time Line Contents
=
119 @profile
120 def integrate_function(self, f):
121
122 Integrate the function
f on the element.
123
12496 1091 0.2 from numpy import array
12596 461070 71.2 from scipy.integrate
import quadrature
12696 496 0.1 a, b =
self.nodes[0].x, self.nodes[1].x
12796 418 0.1 def func(x):
128 #print x
129
Re: [Numpy-discussion] profiling line by line
On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg [EMAIL PROTECTED] wrote:
That would make me an extremely happy user, I've been looking for this for
years!
I can't imagine I'm the only one who profiles some hundred lines of code and
ends up with 90% of total time in the dot-function
For the time being, you can grab it here:
http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
It requires Cython and a C compiler to build. I'm still debating
myself about the desired workflow for using it, but for now, it only
profiles functions which you have registered with it. I have made the
profiler work as a decorator to make this easy. E.g.,
from line_profiler import LineProfiler
profile = LineProfiler()
@profile
def my_slow_func():
...
profile.dump_stats('my_slow_func.lprof')
This is kind of inconvenient, so I have a little script that handles
everything except for the @profile itself. It started as a script to
run cProfile nicely, so it actually does that by default.
I took pystone.py from the Python source and added a couple of
@profile decorators to demonstrate:
[line_profiler]$ ./kernprof.py --help
Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ...
Options:
-h, --helpshow this help message and exit
-l, --line-by-lineUse the line-by-line profiler from the line_profiler
module instead of cProfile. Implies --builtin.
-b, --builtin Put 'profile' in the builtins. Use 'profile.enable()'
and 'profile.disable()' in your code to turn it on and
off, or '@profile' to decorate a single function, or
'with profile:' to profile a single section of code.
-o OUTFILE, --outfile=OUTFILE
Save stats to outfile
-s SETUP, --setup=SETUP
Code to execute before the code to profile
[line_profiler]$ ./kernprof.py -l pystone.py
Pystone(1.1) time for 5 passes = 11.34
This machine benchmarks at 4409.17 pystones/second
Wrote profile results to pystone.py.lprof
[line_profiler]$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 8.46516 s
Line # Hits Time % Time Line Contents
=
79 @profile
80 def Proc0(loops=LOOPS):
81 global IntGlob
82 global BoolGlob
83 global Char1Glob
84 global Char2Glob
85 global Array1Glob
86 global Array2Glob
87 global PtrGlb
88 global PtrGlbNext
89
90 1 11 0.0 starttime = clock()
91 50001 128471 1.5 for i in range(loops):
92 5 125580 1.5 pass
93 1 28 0.0 nulltime = clock() - starttime
94
95 1 21 0.0 PtrGlbNext = Record()
96 18 0.0 PtrGlb = Record()
97 12 0.0 PtrGlb.PtrComp = PtrGlbNext
98 12 0.0 PtrGlb.Discr = Ident1
99 12 0.0 PtrGlb.EnumComp = Ident3
100 13 0.0 PtrGlb.IntComp = 40
101 13 0.0 PtrGlb.StringComp =
DHRYSTONE PROGRAM, SOME STRING
102 12 0.0 String1Loc = DHRYSTONE
PROGRAM, 1'ST STRING
103 15 0.0 Array2Glob[8][7] = 10
104
105 16 0.0 starttime = clock()
106
107 50001 123728 1.5 for i in range(loops):
108 5 220345 2.6 Proc5()
109 5 238459 2.8 Proc4()
110 5 119789 1.4 IntLoc1 = 2
111 5 123639 1.5 IntLoc2 = 3
112 5 116081 1.4 String2Loc =
DHRYSTONE PROGRAM, 2'ND STRING
113 5 119487 1.4 EnumLoc = Ident2
114 5 653711 7.7 BoolGlob = not
Func2(String1Loc, String2Loc)
115 5 130518 1.5 while IntLoc1 IntLoc2:
116 5 132916 1.6 IntLoc3 = 5 *
IntLoc1 - IntLoc2
117 5 235608 2.8 IntLoc3 =
Proc7(IntLoc1, IntLoc2)
118 5 147952 1.7 IntLoc1 = IntLoc1 + 1
119 5 664949 7.9 Proc8(Array1Glob,
Array2Glob, IntLoc1, IntLoc3)
120 5 1710897 20.2 PtrGlb = Proc1(PtrGlb)
121
Re: [Numpy-discussion] profiling line by line
On Mon, Sep 15, 2008 at 05:27, Robin [EMAIL PROTECTED] wrote: Hi, I am using the prun feature of Ipython which is very helpful. I was wondering though if theres anything for Python that would allow line-by-line profiling (ie a time for each line of code) like the MATLAB profiler? I don't think there is anything out there at the moment. Python does have a hook for line-by-line tracing, and one can make a line-by-line profiler from it. To do it with the least overhead, you need to use the C API for it and a C callback. I do have some Cython code that does this. It needs a little bit more work, though. I'll try to push it out soonish. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Mon, Sep 15, 2008 at 5:18 PM, Robert Kern [EMAIL PROTECTED] wrote: I do have some Cython code that does this. It needs a little bit more work, though. I'll try to push it out soonish. That would make me an extremely happy user, I've been looking for this for years! I can't imagine I'm the only one who profiles some hundred lines of code and ends up with 90% of total time in the dot-function Arnar ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Mon, Sep 15, 2008 at 5:13 PM, Arnar Flatberg [EMAIL PROTECTED] wrote: On Mon, Sep 15, 2008 at 5:18 PM, Robert Kern [EMAIL PROTECTED] wrote: I do have some Cython code that does this. It needs a little bit more work, though. I'll try to push it out soonish. That would make me an extremely happy user, I've been looking for this for years! I can't imagine I'm the only one who profiles some hundred lines of code and ends up with 90% of total time in the dot-function Yes, that would be terrific - the profiler is about the only thing I miss from Matlab... Especially after an afternoon of refactoring everything into tiny functions to get anything useful out of the normal profiler and see where the bottleneck in my code was. Robin ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
The hotshot profiler used to do this, but I don't think it is really supported anymore... I have not used it in a while, but agree that a line-by-line profiler can be very nice. Michael. On Sep 15, 2008, at 6:27 AM, Robin wrote: Hi, I am using the prun feature of Ipython which is very helpful. I was wondering though if theres anything for Python that would allow line-by-line profiling (ie a time for each line of code) like the MATLAB profiler? Cheers Robin ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
Michael McNeil Forbes wrote: The hotshot profiler used to do this, but I don't think it is really supported anymore... I have not used it in a while, but agree that a line-by-line profiler can be very nice. You can do it with kcachegrind, which is arguably more powerful, but it is a bit a pain to set up. http://jcalderone.livejournal.com/21124.html cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Mon, Sep 15, 2008 at 21:43, David Cournapeau [EMAIL PROTECTED] wrote: Michael McNeil Forbes wrote: The hotshot profiler used to do this, but I don't think it is really supported anymore... I have not used it in a while, but agree that a line-by-line profiler can be very nice. You can do it with kcachegrind, which is arguably more powerful, but it is a bit a pain to set up. http://jcalderone.livejournal.com/21124.html No, that's with cProfile, which doesn't support line-by-line profiling. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
Robert Kern wrote: On Mon, Sep 15, 2008 at 21:43, David Cournapeau [EMAIL PROTECTED] wrote: You can do it with kcachegrind, which is arguably more powerful, but it is a bit a pain to set up. http://jcalderone.livejournal.com/21124.html No, that's with cProfile, which doesn't support line-by-line profiling. You're right, I did it with hotshot, I did not paste the right link, sorry. The right one should be: http://ddaa.net/blog/python/lsprof-calltree Both profiles (cProfile and hotshot) formats are usable with kcachegrind, but you have to convert first the profiles, so it is not as integrated as if everything was in python. It does give you information similar as matlab, which is one thing I missed from matlab too when I started using python. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] profiling line by line
-> [Numpy-discussion] profiling line by line numpy-discussion -- Thread -- -- Date -- <!-- google_ad_client = "pub-7266757337600734"; google_alternate_ad_url = "http://www.mail-archive.com/blank.png"; google_ad_width = 160; google_ad_height = 600; google_ad_format = "160x600_as"; google_ad_channel = "8427791634"; google_color_border = "FF"; google_color_bg = "FF"; google_color_link = "006792"; google_color_url = "006792"; google_color_text = "00"; //--> [Numpy-discussion] profiling line by line Robin Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line Arnar Flatberg Re: [Numpy-discussion] profiling line by line Robin Re: [Numpy-discussion] profiling line by line Michael McNeil Forbes Re: [Numpy-discussion] profiling line by line David Cournapeau Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line David Cournapeau Reply via email to <!-- google_ad_client = "pub-7266757337600734"; google_alternate_ad_url = "http://www.mail-archive.com/blank.png"; google_ad_width = 160; google_ad_height = 600; google_ad_format = "160x600_as"; google_ad_channel = "8427791634"; google_color_border = "FF"; google_color_bg = "FF"; google_color_link = "006792"; google_color_url = "006792"; google_color_text = "00"; //--> [Numpy-discussion] profiling line by line Robin Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line Arnar Flatberg Re: [Numpy-discussion] profiling line by line Robin Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line Michael McNeil Forbes Re: [Numpy-discussion] profiling line by line David Cournapeau Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line David Cournapeau Reply via email to <!-- google_ad_client = "pub-7266757337600734"; google_alternate_ad_url = "http://www.mail-archive.com/blank.png"; google_ad_width = 160; google_ad_height = 600; google_ad_format = "160x600_as"; google_ad_channel = "8427791634"; google_color_border = "FF"; google_color_bg = "FFFFFF"; google_color_link = "006792"; google_color_url = "006792"; google_color_text = "00"; //--> [Numpy-discussion] profiling line by line Robin Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line Arnar Flatberg Re: [Numpy-discussion] profiling line by line Robin Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line Arnar Flatberg Re: [Numpy-discussion] profiling line by line Michael McNeil Forbes Re: [Numpy-discussion] profiling line by line David Cournapeau Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line David Cournapeau Reply via email to <!-- google_ad_client = "pub-7266757337600734"; google_alternate_ad_url = "http://www.mail-archive.com/blank.png"; google_ad_width = 160; google_ad_height = 600; google_ad_format = "160x600_as"; google_ad_channel = "8427791634"; google_color_border = "FF"; google_color_bg = "FFFFFF"; google_color_link = "006792"; google_color_url = "006792"; google_color_text = "00"; //--> [Numpy-discussion] profiling line by line Robin Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line Arnar Flatberg Re: [Numpy-discussion] profiling line by line Robin Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line Arnar Flatberg Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line Michael McNeil Forbes Re: [Numpy-discussion] profiling line by line David Cournapeau Re: [Numpy-discussion] profiling line by line Robert Kern Re: [Numpy-discussion] profiling line by line David Cournapeau Reply via email to <!-- google_ad_client = "pub-7266757337600734"; google_alternate_ad_url = "http://www.mail-archive.com/blank.png"; google_ad_width = 160; google_ad_height = 600; googl
