Re: [C++-sig] profiling python extension
Thank you, Ralf. All that sounds pretty promising. I took a look on their web-site - indeed it does what I whant. However, I am still woundering if there any options to do this with bjam? Moreover I was successful to compile and run the extension with variant=profile option and I suspect that there should be some information generated, but I do not know how to see it. So if someone here knows how to do this - that should be very valuable knowledge. Anyways it looks that up to now the oprofile - is my only option. Best wishes Alexey 2010/2/27 Ralf W. Grosse-Kunstleve > I usually use oprofile. Typically, I write a small piece of python to call > the > extension multiple times so it runs the critical code for about a minute > (10 > seconds will probably do, too), then the extension code is the only thing > showing up near the top of the oprofile stats. > It is important that nothing else is running on the machine when you run > your extension for profiling, since oprofile is a kernel-level tool and > sees > the entire system. > Last time I looked at the docs (ca. 5 years ago...) you had to be root for > "opcontrol --start". I'm not sure if that's still a requirement. > The commands I use are: > > opcontrol --reset; opcontrol --start # root > # run as usual (not root) > opcontrol --shutdown # root > opreport -l > report > opannotate --source > annotated > grep '^[ 0-9][ 0-9][ 0-9][ 0-9][ 0-9][0-9] ' annotated | sort -n > > You may need "yum install oprofile" or similar. > > -- > *From:* Alexey Akimov > *To:* Development of Python/C++ integration > *Sent:* Sat, February 27, 2010 7:52:17 PM > *Subject:* [C++-sig] profiling python extension > > Dear all, > > Does anyone know how one may profile a python extention? There is a bunch > of literature on profiling of pure binaries (programs and even libraries), > pure python code, etc. However the profiling of a python module from python > script seems a tricky procedure. > What I tried is to pass variant=profile to bjam. Everything works fine and > I get .so library corresponding to my module (placed into /profile > sub-directory of this particular build). When I execute the python script > invoking my module's functions (performance of those functions is most > interesting for me) there is no any output generated (something similar to > gmon.out). So that is basically my question - how can I retrieve the > profiling information about my extension? Of course I can compile a normal > executable using the functions in the module and then use a standard > approach (and of course gprof), but I hope that there should be more > straightforward and easy way to do this (hopefully by passing some arguments > to bjam). > Thanks in advance. > > Alexey > > ___ > Cplusplus-sig mailing list > [email protected] > http://mail.python.org/mailman/listinfo/cplusplus-sig > ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] profiling python extension
> Date: Sat, 27 Feb 2010 21:52:17 -0600 > From: Alexey Akimov > To: "Development of Python/C++ integration" > Subject: [C++-sig] profiling python extension > Message-ID: > <[email protected]> > Content-Type: text/plain; charset="iso-8859-1" > > Dear all, > > Does anyone know how one may profile a python extention? There is a bunch of > literature on profiling of pure binaries (programs and even libraries), pure > python code, etc. However the profiling of a python module from python > script seems a tricky procedure. > What I tried is to pass variant=profile to bjam. Everything works fine and I > get .so library corresponding to my module (placed into /profile > sub-directory of this particular build). When I execute the python script > invoking my module's functions (performance of those functions is most > interesting for me) there is no any output generated (something similar to > gmon.out). So that is basically my question - how can I retrieve the > profiling information about my extension? Of course I can compile a normal > executable using the functions in the module and then use a standard > approach (and of course gprof), but I hope that there should be more > straightforward and easy way to do this (hopefully by passing some arguments > to bjam). > Thanks in advance. > > Alexey If you're using OSX, then the Shark tool seems to work really well, and is really easy to use. I does not require that you compile in -pg mode. http://developer.apple.com/tools/shark_optimize.html Amos. ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] profiling python extension
Thank you, Amos I forget to mention - I am working Linux, so the shark tool probably will not be suitable for me. But anyway thanks for you reply. Best wishes Alexey 2010/2/28 Amos Anderson > > Date: Sat, 27 Feb 2010 21:52:17 -0600 > > From: Alexey Akimov > > To: "Development of Python/C++ integration" > > Subject: [C++-sig] profiling python extension > > Message-ID: > ><[email protected]> > > Content-Type: text/plain; charset="iso-8859-1" > > > > Dear all, > > > > Does anyone know how one may profile a python extention? There is a bunch > of > > literature on profiling of pure binaries (programs and even libraries), > pure > > python code, etc. However the profiling of a python module from python > > script seems a tricky procedure. > > What I tried is to pass variant=profile to bjam. Everything works fine > and I > > get .so library corresponding to my module (placed into /profile > > sub-directory of this particular build). When I execute the python script > > invoking my module's functions (performance of those functions is most > > interesting for me) there is no any output generated (something similar > to > > gmon.out). So that is basically my question - how can I retrieve the > > profiling information about my extension? Of course I can compile a > normal > > executable using the functions in the module and then use a standard > > approach (and of course gprof), but I hope that there should be more > > straightforward and easy way to do this (hopefully by passing some > arguments > > to bjam). > > Thanks in advance. > > > > Alexey > > > If you're using OSX, then the Shark tool seems to work really well, > and is really easy to use. I does not require that you compile in -pg > mode. > > http://developer.apple.com/tools/shark_optimize.html > > Amos. > ___ > Cplusplus-sig mailing list > [email protected] > http://mail.python.org/mailman/listinfo/cplusplus-sig > ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] profiling python extension
Alexey Akimov wrote: Thank you, Amos I forget to mention - I am working Linux, so the shark tool probably will not be suitable for me. But anyway thanks for you reply. Or this: valgrind --tool=callgrind mypythonscript.py kcachegrind -t ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] profiling python extension
Thank you, Troy I'll try the valgrind tool as well (in addition to oprofile). Best wishes Alexey 2010/2/28 troy d. straszheim > Alexey Akimov wrote: > >> Thank you, Amos >> I forget to mention - I am working Linux, so the shark tool probably will >> not be suitable for me. But anyway thanks for you reply. >> >> > Or this: > > valgrind --tool=callgrind mypythonscript.py > kcachegrind > > -t > > ___ > Cplusplus-sig mailing list > [email protected] > http://mail.python.org/mailman/listinfo/cplusplus-sig > ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] profiling python extension
Alexey Akimov wrote: Does anyone know how one may profile a python extention? I've used the google profiler on Linux successfully in the past. It is quite straightforward to set up inside an extension. http://goog-perftools.sourceforge.net/doc/cpu_profiler.html John. ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] profiling python extension
Thank you John, I just installed the tool you suggested and tried to use it. However I do not see any output. Could you give some more detailes of how the profiling process with the google-profiler should look like? What I do is: 1) bjam variant=profile toolset=gcc cxxflags=-fno-for-scope linkflags=-lprofiler test_ext 2) env CPUPROFILE=/path/to/my/dir/out.prof python ./mypythonscript.py This runs my python script which invokes the extension. However it does not produce any output files (which i expect to be out.prof in my working directory). What should I do in order to see the profiling information? By the way, by this time I've tried other two tools suggested. Here are my feelings: 1) oprofile - looks and works good. it is fast enough and produces much useful information, there is a bunch of options to play with. The only inconvenience is that I need to start and finish it only being a root. This means I either have to switch back and forth oftenly or do all profiling (and development as a root). 2) valgrind - also is pretty convenient tool and produces a lot of information (the raw output loooks quite difficult to understand, but i guess there is a play around options). It gives you many options of profiling. The only bad thing - it is very slow. 3) google-profiling - is kind a promising for my purposes, but by this time i still have not managed how to see the output information. Best wishes Alexey 2010/2/28 John Reid > Alexey Akimov wrote: > >> Does anyone know how one may profile a python extention? >> > > I've used the google profiler on Linux successfully in the past. It is > quite straightforward to set up inside an extension. > > http://goog-perftools.sourceforge.net/doc/cpu_profiler.html > > John. > > > ___ > Cplusplus-sig mailing list > [email protected] > http://mail.python.org/mailman/listinfo/cplusplus-sig > ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] profiling python extension
Alexey Akimov wrote: 2) valgrind - also is pretty convenient tool and produces a lot of information (the raw output loooks quite difficult to understand, but i guess there is a play around options). It gives you many options of profiling. use kcachegrind to inspect the call graph. ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] profiling python extension
> The only inconvenience is that I need to start and finish it only being a
> root. This means
> I either have to switch back and forth oftenly or do all profiling (and
> development as a root).
I'm in an environment with relatively few users which I fully trust. To get
around the
inconvenience I run the script below which installs two set-user-id-root
commands
"opcontrol_start" and "opcontrol_shutdown".
Ralf
#! /bin/csh -f
set verbose
gcc -o setuid_linux setuid_linux.c
cp setuid_linux /usr/bin/opcontrol_start
mv setuid_linux /usr/bin/opcontrol_shutdown
cat << EOT > /usr/bin/opcontrol_start.script
#! /bin/bash -r
set -v
opcontrol --reset; opcontrol --start
EOT
cat << EOT > /usr/bin/opcontrol_shutdown.script
#! /bin/bash -r
set -v
opcontrol --shutdown
EOT
chmod 6711 /usr/bin/opcontrol_start /usr/bin/opcontrol_shutdown
chmod 755 /usr/bin/opcontrol_start.script /usr/bin/opcontrol_shutdown.script
opcontrol --no-vmlinux
% cat setuid_linux.c
#include
#include
#include
#include
#include
static char **cpargv(int argc, char *argv[], const char *cmd_name)
{
int i;
char **cargv;
cargv = malloc((argc + 1) * sizeof (*cargv));
if (cargv == NULL)
return NULL;
cargv[0] = (char *) cmd_name;
for (i = 1; i < argc; i++)
cargv[i] = argv[i];
cargv[i] = NULL;
return cargv;
}
static const char *StripFileName(const char *PathName, int Separator)
{
const char *FileName;
for (FileName = PathName; *PathName; PathName++)
if (*PathName == Separator)
FileName = PathName + 1;
return FileName;
}
static const char *TrustedCommands[] =
{
"opcontrol_start",
"opcontrol_shutdown",
NULL
};
int main(int argc, char *argv[])
{
const char *progn, **tc, *cmd_dir, *cmd_ext;
char*cmd_path, **run_cmd_argv;
int n;
progn = StripFileName(argv[0], '/');
for (tc = TrustedCommands; *tc; tc++)
if (strcmp(progn, *tc) == 0)
break;
if (*tc == NULL)
{
fprintf(stderr, "%s: Not a trusted command.\n", progn);
exit(1);
}
if (setuid(0) != 0) {
perror(progn);
exit(1);
}
if (setgid(0) != 0) {
perror(progn);
exit(1);
}
run_cmd_argv = cpargv(argc, argv, progn);
if (run_cmd_argv == NULL)
{
fprintf(stderr, "%s: Not enough core.\n", progn);
exit(1);
}
cmd_dir = "/usr/bin/";
cmd_ext = ".script";
n = strlen(cmd_dir) + strlen(progn) + strlen(cmd_ext) + 1;
cmd_path = malloc(n * sizeof (*cmd_path));
if (cmd_path == NULL)
{
fprintf(stderr, "%s: Not enough core.\n", progn);
exit(1);
}
strcpy(cmd_path, cmd_dir);
strcat(cmd_path, progn);
strcat(cmd_path, cmd_ext);
(void) execv(cmd_path, run_cmd_argv);
perror(progn);
exit (1);
return 0;
}
___
Cplusplus-sig mailing list
[email protected]
http://mail.python.org/mailman/listinfo/cplusplus-sig
