On Mon, 7 May 2001, Gilles Detillieux wrote:
> Date: Mon, 7 May 2001 15:07:32 -0500 (CDT)
> From: Gilles Detillieux <[EMAIL PROTECTED]>
> To: Joe R. Jah <[EMAIL PROTECTED]>
> Cc: Gilles Detillieux <[EMAIL PROTECTED]>,
[EMAIL PROTECTED]
> Subject: Re: [htdig-dev] Profiling htdig-3.2.0b3 on BSDi-4.2
>
> According to Joe R. Jah:
> > On Mon, 30 Apr 2001, Gilles Detillieux wrote:
> > > According to Alexander Cohen:
> > > > > With different versions of htdig:
> > > > >
> > > > > 3.1.5 11 Minutes
> > > > > 3.2.0b3 9 1/2 hours
> > > > > 3.2.0b4-031201 29 hours and 20 minutes
> > > > > 3.2.0b4-040801 > 12 days
> > > >
> > > > Is it just me or there a pattern developing here? ;]
> > >
> > > There sure seems to be, but we're nowhere closer to knowing why this is
> > > happening. It doesn't seem to be an issue with any systems other than
> > > BSDI. Some profiling, to know where htdig is spending its time, could be
> > > quite helpful. Joe, is there any possibility you could figure out how to
> > > compile the code on your system with profiling enabled, to get some stats
> > > on this?
> >
> > OK, I compiled htdig-3.2.0b3 with profiling enabled and randig. Here are
> > the stats:
> > ___________________________________________________
> > htdig: Start Digging: Mon May 7 08:56:50 PDT 2001
> > htpurge: Start Purging: Mon May 7 09:57:16 PDT 2001
> > htmerge: Start Merging: Mon May 7 09:58:09 PDT 2001
> > rundig: End time: Mon May 7 09:58:10 PDT 2001
>
> So now you're back down to just over an hour of indexing time, which doesn't
> seem that outlandish. It is still 4 times slower than 3.1.5, though, so it
> would be helpful to know where it's spending most of its time.
Oh no;) I wish that was the case. I only ran it on a folder with 269
Documents total.
> > And attached is the gziped profile created by gprof.
>
> You gave us the gmon.out file, which is pretty much useless on any
> system other than yours. It needs to be run through gprof, along with
> your htdig binary, to get a listing of where htdig is spending its time.
Whoops;( I sent you the wrong file. Attached is the correct one.
> Make sure your gmon.out is from htdig, though, and not from htmerge, as
> it's htdig that is taking a really long time. I think if you simply run
> "rundig", then when htmerge starts after htdig finishes, htmerge will
> overwrite htdig's gmon.out with its own.
I will run htdig without rundig and send you the profile later, but here
is the one from rundig, just in case it was not overwritten.
Regards,
Joe
--
_/ _/_/_/ _/ ____________ __o
_/ _/ _/ _/ ______________ _-\<,_
_/ _/ _/_/_/ _/ _/ ......(_)/ (_)
_/_/ oe _/ _/. _/_/ ah [EMAIL PROTECTED]
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls Ts/call Ts/call name
50.00 0.01 0.01 __swsetup
50.00 0.02 0.01 ng_minimize
0.00 0.02 0.00 4260 0.00 0.00 Retriever::got_word(char const
*, int, int)
0.00 0.02 0.00 2886 0.00 0.00 Retriever::GetLocalUser(String
const &, StringList *)
0.00 0.02 0.00 209 0.00 0.00 Retriever::got_href(URL &, char
const *, int)
0.00 0.02 0.00 193 0.00 0.00 __svfscanf
0.00 0.02 0.00 63 0.00 0.00 B_CreateAlgorithmObject
0.00 0.02 0.00 63 0.00 0.00 B_DestroyAlgorithmObject
0.00 0.02 0.00 62 0.00 0.00 Retriever::GetLocal(String const
&)
0.00 0.02 0.00 62 0.00 0.00 String virtual table
0.00 0.02 0.00 45 0.00 0.00 lbnDiv_16
0.00 0.02 0.00 26 0.00 0.00 lbnMontReduce_16
0.00 0.02 0.00 21 0.00 0.00 lbnExpMod_16
0.00 0.02 0.00 18 0.00 0.00 makeservent
0.00 0.02 0.00 16 0.00 0.00 Retriever::IsValidURL(String
const &)
0.00 0.02 0.00 14 0.00 0.00 parse_hes_list
0.00 0.02 0.00 10 0.00 0.00 Square
0.00 0.02 0.00 10 0.00 0.00 strtoll
0.00 0.02 0.00 7 0.00 0.00 __tcf_2
0.00 0.02 0.00 6 0.00 0.00 pw_res_get
0.00 0.02 0.00 4 0.00 0.00 __sfseek
0.00 0.02 0.00 4 0.00 0.00 lbnDoubleExpMod_16
0.00 0.02 0.00 4 0.00 0.00 tzparse
0.00 0.02 0.00 3 0.00 0.00 Retriever::got_redirect(char
const *, DocumentRef *)
0.00 0.02 0.00 3 0.00 0.00 strtod
0.00 0.02 0.00 2 0.00 0.00 b2d
0.00 0.02 0.00 2 0.00 0.00 ulp
0.00 0.02 0.00 1 0.00 0.00 nw_res_get
0.00 0.02 0.00 1 0.00 0.00 shaTransform
% the percentage of the total running time of the
time program used by this function.
cumulative a running sum of the number of seconds accounted
seconds for by this function and those listed above it.
self the number of seconds accounted for by this
seconds function alone. This is the major sort for this
listing.
calls the number of times this function was invoked, if
this function is profiled, else blank.
self the average number of milliseconds spent in this
ms/call function per call, if this function is profiled,
else blank.
total the average number of milliseconds spent in this
ms/call function and its descendents per call, if this
function is profiled, else blank.
name the name of the function. This is the minor sort
for this listing. The index shows the location of
the function in the gprof listing. If the index is
in parenthesis it shows where it would appear in
the gprof listing if it were to be printed.
Call graph (explanation follows)
granularity: each sample hit covers 4 byte(s) for 50.00% of 0.02 seconds
index % time self children called name
<spontaneous>
[1] 50.0 0.01 0.00 ng_minimize [1]
-----------------------------------------------
<spontaneous>
[2] 50.0 0.01 0.00 __swsetup [2]
-----------------------------------------------
[3] 0.0 0.00 0.00 330+7161 <cycle 1 as a whole> [3]
0.00 0.00 4260+1 Retriever::got_word(char const *,
int, int) <cycle 1> [4]
0.00 0.00 2886+132 Retriever::GetLocalUser(String const
&, StringList *) <cycle 1> [5]
0.00 0.00 209 Retriever::got_href(URL &, char const
*, int) <cycle 1> [6]
0.00 0.00 62 Retriever::GetLocal(String const &)
<cycle 1> [9]
0.00 0.00 45 lbnDiv_16 <cycle 1> [10]
0.00 0.00 16 Retriever::IsValidURL(String const &)
<cycle 1> [14]
0.00 0.00 10 strtoll <cycle 1> [17]
-----------------------------------------------
1 Retriever::got_word(char const *,
int, int) <cycle 1> [4]
65 Retriever::GetLocalUser(String const
&, StringList *) <cycle 1> [5]
4175 Retriever::got_href(URL &, char const
*, int) <cycle 1> [6]
0.00 0.00 20/330 Retriever::got_image(char const *)
[821]
[4] 0.0 0.00 0.00 4260+1 Retriever::got_word(char const *, int,
int) <cycle 1> [4]
2886 Retriever::GetLocalUser(String const
&, StringList *) <cycle 1> [5]
1 Retriever::got_word(char const *,
int, int) <cycle 1> [4]
-----------------------------------------------
132 Retriever::GetLocalUser(String const
&, StringList *) <cycle 1> [5]
2886 Retriever::got_word(char const *,
int, int) <cycle 1> [4]
[5] 0.0 0.00 0.00 2886+132 Retriever::GetLocalUser(String const &,
StringList *) <cycle 1> [5]
0.00 0.00 63/63 B_CreateAlgorithmObject [7]
0.00 0.00 63/63 B_DestroyAlgorithmObject [8]
65 Retriever::got_word(char const *,
int, int) <cycle 1> [4]
2 Retriever::GetLocal(String const &)
<cycle 1> [9]
1 strtoll <cycle 1> [17]
132 Retriever::GetLocalUser(String const
&, StringList *) <cycle 1> [5]
-----------------------------------------------
2 strtoll <cycle 1> [17]
7 lbnDiv_16 <cycle 1> [10]
0.00 0.00 2/330 b2d [23]
0.00 0.00 3/330 Retriever::ReportStatistics(String
const &) [349]
0.00 0.00 3/330 Server::Server(URL, StringList *)
[1760]
0.00 0.00 192/330 tzparse [20]
[6] 0.0 0.00 0.00 209 Retriever::got_href(URL &, char const *,
int) <cycle 1> [6]
0.00 0.00 3/14 parse_hes_list [15]
4175 Retriever::got_word(char const *,
int, int) <cycle 1> [4]
-----------------------------------------------
0.00 0.00 63/63 Retriever::GetLocalUser(String const
&, StringList *) <cycle 1> [5]
[7] 0.0 0.00 0.00 63 B_CreateAlgorithmObject [7]
-----------------------------------------------
0.00 0.00 63/63 Retriever::GetLocalUser(String const
&, StringList *) <cycle 1> [5]
[8] 0.0 0.00 0.00 63 B_DestroyAlgorithmObject [8]
-----------------------------------------------
2 Retriever::GetLocalUser(String const
&, StringList *) <cycle 1> [5]
2 lbnDiv_16 <cycle 1> [10]
0.00 0.00 7/330 __res_vinit [2062]
0.00 0.00 9/330 irs_dns_pw [992]
0.00 0.00 18/330 getpwcommon [792]
0.00 0.00 24/330 shaTransform [26]
[9] 0.0 0.00 0.00 62 Retriever::GetLocal(String const &)
<cycle 1> [9]
0.00 0.00 1/4 lbnDoubleExpMod_16 [19]
2 Retriever::IsValidURL(String const &)
<cycle 1> [14]
-----------------------------------------------
5 strtod <cycle 1> [22]
6 strtoll <cycle 1> [17]
0.00 0.00 1/330 Retriever::parse_url(URLRef &) [1268]
0.00 0.00 2/330 b2d [23]
0.00 0.00 6/330 __res_vinit [2062]
0.00 0.00 6/330 ulp [24]
0.00 0.00 19/330 getpwcommon [792]
[10] 0.0 0.00 0.00 45 lbnDiv_16 <cycle 1> [10]
0.00 0.00 21/21 lbnExpMod_16 [12]
7 Retriever::got_href(URL &, char const
*, int) <cycle 1> [6]
5 Retriever::IsValidURL(String const &)
<cycle 1> [14]
2 Retriever::GetLocal(String const &)
<cycle 1> [9]
-----------------------------------------------
0.00 0.00 1/26 Retriever::parse_url(URLRef &) [1268]
0.00 0.00 6/26 __res_vinit [2062]
0.00 0.00 19/26 getpwcommon [792]
[11] 0.0 0.00 0.00 26 lbnMontReduce_16 [11]
-----------------------------------------------
0.00 0.00 21/21 lbnDiv_16 <cycle 1> [10]
[12] 0.0 0.00 0.00 21 lbnExpMod_16 [12]
-----------------------------------------------
0.00 0.00 18/18 getpwcommon [792]
[13] 0.0 0.00 0.00 18 makeservent [13]
-----------------------------------------------
2 Retriever::GetLocal(String const &)
<cycle 1> [9]
2 strtod <cycle 1> [22]
5 lbnDiv_16 <cycle 1> [10]
0.00 0.00 7/330 s2b [1434]
[14] 0.0 0.00 0.00 16 Retriever::IsValidURL(String const &)
<cycle 1> [14]
1 strtod <cycle 1> [22]
-----------------------------------------------
0.00 0.00 3/14 Retriever::got_href(URL &, char const
*, int) <cycle 1> [6]
0.00 0.00 11/14 HTML::do_tag(Retriever &, String &)
[564]
[15] 0.0 0.00 0.00 14 parse_hes_list [15]
-----------------------------------------------
0.00 0.00 10/10 HTML::do_tag(Retriever &, String &)
[564]
[16] 0.0 0.00 0.00 10 Square [16]
-----------------------------------------------
1 Retriever::GetLocalUser(String const
&, StringList *) <cycle 1> [5]
0.00 0.00 2/330 getpwcommon [792]
0.00 0.00 7/330 __res_vinit [2062]
[17] 0.0 0.00 0.00 10 strtoll <cycle 1> [17]
6 lbnDiv_16 <cycle 1> [10]
2 Retriever::got_href(URL &, char const
*, int) <cycle 1> [6]
-----------------------------------------------
0.00 0.00 6/6 HTML::do_tag(Retriever &, String &)
[564]
[18] 0.0 0.00 0.00 6 pw_res_get [18]
-----------------------------------------------
0.00 0.00 1/4 Retriever::GetLocal(String const &)
<cycle 1> [9]
0.00 0.00 3/4 <locore> (28)
[19] 0.0 0.00 0.00 4 lbnDoubleExpMod_16 [19]
0.00 0.00 1/193 __svfscanf [1719]
-----------------------------------------------
0.00 0.00 1/4 HTML::do_tag(Retriever &, String &)
[564]
0.00 0.00 3/4 <locore> (28)
[20] 0.0 0.00 0.00 4 tzparse [20]
0.00 0.00 192/330 Retriever::got_href(URL &, char const
*, int) <cycle 1> [6]
0.00 0.00 192/193 __svfscanf [1719]
-----------------------------------------------
0.00 0.00 3/3 shaTransform [26]
[21] 0.0 0.00 0.00 3 Retriever::got_redirect(char const *,
DocumentRef *) [21]
-----------------------------------------------
4 strtod <cycle 1> [22]
1 Retriever::IsValidURL(String const &)
<cycle 1> [14]
0.00 0.00 2/330 __res_vinit [2062]
[22] 0.0 0.00 0.00 3+4 strtod <cycle 1> [22]
5 lbnDiv_16 <cycle 1> [10]
2 Retriever::IsValidURL(String const &)
<cycle 1> [14]
4 strtod <cycle 1> [22]
-----------------------------------------------
0.00 0.00 2/2 ulp [24]
[23] 0.0 0.00 0.00 2 b2d [23]
0.00 0.00 2/330 Retriever::got_href(URL &, char const
*, int) <cycle 1> [6]
0.00 0.00 2/330 lbnDiv_16 <cycle 1> [10]
-----------------------------------------------
0.00 0.00 2/2 s2b [1434]
[24] 0.0 0.00 0.00 2 ulp [24]
0.00 0.00 6/330 lbnDiv_16 <cycle 1> [10]
0.00 0.00 2/2 b2d [23]
-----------------------------------------------
0.00 0.00 1/1 <locore> (28)
[25] 0.0 0.00 0.00 1 nw_res_get [25]
-----------------------------------------------
1 shaTransform [26]
0.00 0.00 1/1 HTML::do_tag(Retriever &, String &)
[564]
[26] 0.0 0.00 0.00 1+1 shaTransform [26]
0.00 0.00 24/330 Retriever::GetLocal(String const &)
<cycle 1> [9]
0.00 0.00 3/3 Retriever::got_redirect(char const *,
DocumentRef *) [21]
1 shaTransform [26]
-----------------------------------------------
25 HTML::do_tag(Retriever &, String &)
[564]
[564] 0.0 0.00 0.00 0+25 HTML::do_tag(Retriever &, String &) [564]
0.00 0.00 11/14 parse_hes_list [15]
0.00 0.00 10/10 Square [16]
0.00 0.00 6/6 pw_res_get [18]
0.00 0.00 1/1 shaTransform [26]
0.00 0.00 1/4 tzparse [20]
25 HTML::do_tag(Retriever &, String &)
[564]
-----------------------------------------------
4 __svfscanf [1719]
0.00 0.00 1/193 lbnDoubleExpMod_16 [19]
0.00 0.00 192/193 tzparse [20]
[1719] 0.0 0.00 0.00 193+4 __svfscanf [1719]
4 __svfscanf [1719]
-----------------------------------------------
0.00 0.00 62/62 <locore> (28)
[1720] 0.0 0.00 0.00 62 String virtual table [1720]
-----------------------------------------------
0.00 0.00 7/7 isnan [1039]
[1721] 0.0 0.00 0.00 7 __tcf_2 [1721]
-----------------------------------------------
0.00 0.00 4/4 __res_vinit [2062]
[1722] 0.0 0.00 0.00 4 __sfseek [1722]
-----------------------------------------------
This table describes the call tree of the program, and was sorted by
the total amount of time spent in each function and its children.
Each entry in this table consists of several lines. The line with the
index number at the left hand margin lists the current function.
The lines above it list the functions that called this function,
and the lines below it list the functions this one called.
This line lists:
index A unique number given to each element of the table.
Index numbers are sorted numerically.
The index number is printed next to every function name so
it is easier to look up where the function in the table.
% time This is the percentage of the `total' time that was spent
in this function and its children. Note that due to
different viewpoints, functions excluded by options, etc,
these numbers will NOT add up to 100%.
self This is the total amount of time spent in this function.
children This is the total amount of time propagated into this
function by its children.
called This is the number of times the function was called.
If the function called itself recursively, the number
only includes non-recursive calls, and is followed by
a `+' and the number of recursive calls.
name The name of the current function. The index number is
printed after it. If the function is a member of a
cycle, the cycle number is printed between the
function's name and the index number.
For the function's parents, the fields have the following meanings:
self This is the amount of time that was propagated directly
from the function into this parent.
children This is the amount of time that was propagated from
the function's children into this parent.
called This is the number of times this parent called the
function `/' the total number of times the function
was called. Recursive calls to the function are not
included in the number after the `/'.
name This is the name of the parent. The parent's index
number is printed after it. If the parent is a
member of a cycle, the cycle number is printed between
the name and the index number.
If the parents of the function cannot be determined, the word
`<spontaneous>' is printed in the `name' field, and all the other
fields are blank.
For the function's children, the fields have the following meanings:
self This is the amount of time that was propagated directly
from the child into the function.
children This is the amount of time that was propagated from the
child's children to the function.
called This is the number of times the function called
this child `/' the total number of times the child
was called. Recursive calls by the child are not
listed in the number after the `/'.
name This is the name of the child. The child's index
number is printed after it. If the child is a
member of a cycle, the cycle number is printed
between the name and the index number.
If there are any cycles (circles) in the call graph, there is an
entry for the cycle-as-a-whole. This entry shows who called the
cycle (as parents) and the members of the cycle (as children.)
The `+' recursive calls entry shows the number of function calls that
were internal to the cycle, and the calls entry for each member shows,
for that member, how many times it was called from other members of
the cycle.
Index by function name
[7] B_CreateAlgorithmObject [1720] String virtual table [1] ng_minimize
[8] B_DestroyAlgorithmObject [23] b2d [25] nw_res_get
[5] Retriever::GetLocalUser(String const &, StringList *) [6]
Retriever::got_href(URL &, char const *, int) [15] parse_hes_list
[9] Retriever::GetLocal(String const &) [21] Retriever::got_redirect(char const *,
DocumentRef *) [18] pw_res_get
[14] Retriever::IsValidURL(String const &) [4] Retriever::got_word(char const *,
int, int) [26] shaTransform
[16] Square [10] lbnDiv_16 [22] strtod
[1722] __sfseek [19] lbnDoubleExpMod_16 [17] strtoll
[1719] __svfscanf [12] lbnExpMod_16 [20] tzparse
[2] __swsetup [11] lbnMontReduce_16 [24] ulp
[1721] __tcf_2 [13] makeservent [3] <cycle 1>
