Re: [AOLSERVER] What parts of the ns_* API do you use the most?
In a message dated 4/30/2001 8:58:36 PM Eastern Daylight Time,
[EMAIL PROTECTED] writes:
On 2001.04.30, Jim Davidson [EMAIL PROTECTED] wrote:
[ ... ns_stats, command traces, nsd/tclstats.c, etc. ... ]
I removed this from 4.x because it was being used much and requred an
undocumented, non-virtual server safe API no longer available.
Of course you had to burst my bubble, didn't you? ;-)
Did you remove it because it couldn't be fixed up in the current
state of AOLserver, or because you just didn't have time to fix it
up?
In other words, is it worthwhile for me to go and reimplement this
for 4.x? As I mentioned, I'd prefer to do it as a separately loadable
module like nsprofiler or nsstats (perhaps the latter is more
catchy) ... so that people could load the module, which could then
be toggled on or off at runtime.
I'm curious specifically which API you're referring to, as well.
Also, do you have any rough draft documentation regarding the
new virtual-server API? Is it both hardware and software virtual
server (both IP and name-based)? Do I just have to look through
the cvs logs to see what files you've been touching? ;-)
Hi,
If you look at the old nsd/tclstats.c code, you'll find this:
void
NsTclStatsInit(void)
{
Tcl_InitHashTable(statsTable, TCL_STRING_KEYS);
Ns_MutexSetName2(lock, ns, tclstats);
if (nsconf.tcl.statlevel 0) {
Ns_Log(Warning, tclstats: tracing to level %d,
nsconf.tcl.statlevel);
Ns_TclRegisterAtCreate(CreateTrace, NULL);
if (nsconf.tcl.statmaxbuf 0) {
Ns_TlsAlloc(tls, FreeBuf);
Ns_TclRegisterAtCleanup(FlushBufTrace, NULL);
}
}
}
Neither the Ns_TclRegisterAtCreate nor the Ns_TclRegisterAtCleanup API's
exist in AOLserver 4.x.
Ns_TclRegisterAtCleanup was removed because it's not virtual-server safe
(i.e., it doesn't take the server as a first argument) and because cleanup
is now generally handled via new command options in the ns_cleanup proc
(e.g., ns_db cleanup) so folks can fiddle with the cleanup behavior as they
wish.
Ns_TclRegisterAtCreate was removed because the new semantics of
Ns_TclInitInterps no longer require such a callback. This is a more subtle
change from 3.x which was also different in 2.x. Consider the following code
which is common in AOLserver modules:
static int
AddCmds(Tcl_Interp *interp, void *arg)
{
Tcl_CreateCommand(interp, mycmd, MyCmd, NULL, arg);
return TCL_OK;
}
int
Ns_ModuleInit(char *server, char *module)
{
...
Ns_TclInitInterps(server, AddCmds, mydata);
}
In AOLserver 2.0, this code would invoke AddCmds at startup for each
Tcl_Interp in the pool of interps for the virtual server.
In AOLserver 2.1 through 2.33, this code would invoke AddCmds once at startup
in the master interp of the virtual server, the command table and global
variables of which would be shared with all other per-thread half interps
in the virtual server.
In AOLserver 3.0 through 3.4, this code would invoke AddCmds once at startup
on the template interp, the commands and procs of which would be copied to
new per-thread interps when requested.
Finally, in AOLserver 4.0, this code will invoke AddCmds on each interp
created for the virtual server when requested, perhaps long after startup and
certainly long after the call stack of the function calling Ns_TclInitInterps
is gone (e.g., the Ns_ModuleInit).
So, sounds to me like ns_stats could come back into 4.x if you add an
ns_stats cleanup to flush the stats, add that to the ns_cleanup proc, and
write a custom Ns_TclInterpInitProc which creates the per-interp state and
command trace when enabled. You may want to store the state in Tcl assoc
data as is done for the NsInterp structure as well to ensure proper cleanup
at Tcl interp time and pass it via the StatsCmd ClientData as well, meaning
the Ns_Tls used in 3.x is probably not necessary.
-Jim
Re: [AOLSERVER] What parts of the ns_* API do you use the most?
On 2001.04.29, carl garland [EMAIL PROTECTED] wrote: Great presentation! Clearly, I haven't looked at the UrlTracking config. option -- it's not even in my config. file so I don't think I ever would have known about it. Thanks and oops, the UrlTracking config param was just another parameter that I added so as to add the ability to not have the overhead of running the filter on production systems that dont need this profiling. Just add the param in the main config file. I need to make clearer that it is not a normal option. Carl, What's strange, is that in my nsd.tcl I've always had these lines in and they don't seem to have any effect: ns_param globalstats on ns_param urlstats on ns_param maxurlstats 1000 I have a vague recollection of reading (on this mailing list, a long while back) that these settings are no longer used. I thought your presentation was referring to these in some way. Implementing profiling with two trace filters and an nsv memory array is just the idea I started forming -- good to see that you've proven that it can work, at least for dynamic analysis at the URL level! I guess I could start with what you've done, clean up the code, and start looking at what kind of hooks I'd need in order to get the Tcl interpreter to log the same kind of information about indivual proc calls (perhaps even including ALL proc calls) -- using an nsv is a great idea that I didn't think of, immediately. One concern: does the nsv implementation use mutexes? This will affect the way the trace filters must be written to avoid logging lock contention as part of a URL's performance, artificially bloating numbers in a non-linear way. - Dossy -- Dossy Shiobara mail: [EMAIL PROTECTED] Panoptic Computer Network web: http://www.panoptic.com/
Re: [AOLSERVER] What parts of the ns_* API do you use the most?
I guess I could start with what you've done, clean up the code, and start looking at what kind of hooks I'd need in order to get the Tcl interpreter to log the same kind of information about indivual proc calls (perhaps even including ALL proc calls) -- using an nsv is a great idea that I didn't think of, immediately. dossy, check out the C functions.. Tcl_CreateTrace() Tcl_DeleteTrace() they will allow you to trace tcl proc calls.. -mike One concern: does the nsv implementation use mutexes? This will affect the way the trace filters must be written to avoid logging lock contention as part of a URL's performance, artificially bloating numbers in a non-linear way. - Dossy -- Dossy Shiobara mail: [EMAIL PROTECTED] Panoptic Computer Network web: http://www.panoptic.com/
Re: [AOLSERVER] What parts of the ns_* API do you use the most?
At least on Linux, puts is thread-safe if the -nonewline option is used. I ran tests and used it for a URL logging feature we developed - worked great. So instead of puts $string, do puts -nonewline $string\n JIm I guess I could start with what you've done, clean up the code, and start looking at what kind of hooks I'd need in order to get the Tcl interpreter to log the same kind of information about indivual proc calls (perhaps even including ALL proc calls) -- using an nsv is a great idea that I didn't think of, immediately. dossy, check out the C functions.. Tcl_CreateTrace() Tcl_DeleteTrace() they will allow you to trace tcl proc calls.. -mike One concern: does the nsv implementation use mutexes? This will affect the way the trace filters must be written to avoid logging lock contention as part of a URL's performance, artificially bloating numbers in a non-linear way. - Dossy -- Dossy Shiobara mail: [EMAIL PROTECTED] Panoptic Computer Network web: http://www.panoptic.com/
Re: [AOLSERVER] What parts of the ns_* API do you use the most?
At 03:36 PM 4/28/01 -0400, you wrote: I'd like to conduct a quick informal poll. What are the top 10 most commonly used ns_* API calls that you use? Responses to the list, or directly to me, are fine. - Dossy -- Dossy Shiobara mail: [EMAIL PROTECTED] Panoptic Computer Network web: http://www.panoptic.com/ ns_db ns_set nsv_* ns_puts ... ? Jerry = Jerry Asher [EMAIL PROTECTED] 1678 Shattuck Avenue Suite 161Tel: (510) 549-2980 Berkeley, CA 94709Fax: (877) 311-8688
Re: [AOLSERVER] What parts of the ns_* API do you use the most?
- Original Message - From: Dossy [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Saturday, April 28, 2001 12:36 PM Subject: [AOLSERVER] What parts of the ns_* API do you use the most? I'd like to conduct a quick informal poll. What are the top 10 most commonly used ns_* API calls that you use? Responses to the list, or directly to me, are fine. ns_db ns_set ns_return -- Patrick Spence [EMAIL PROTECTED] Want weather reports delevered to your desktop? http://ww2.weatherbug.com/aff/default.asp?ZCode=z3345
Re: [AOLSERVER] What parts of the ns_* API do you use the most?
I'd like to conduct a quick informal poll. What are the top 10 most commonly used ns_* API calls that you use? Do you mean at authoring time or execution time?
Re: [AOLSERVER] What parts of the ns_* API do you use the most?
Here's a complete list of the ones we use. I should mention that we usually use ns_db via a wrapper; it's easily in the top 10. Not sure how useful this is, since it is a static analysis of source rather than a dynamic analysis of which routines are actually used the most. Is there a way to get a dynamic analysis? Guess we could redefine everything with a wrapper and count calls execution time. Yuk. This is from 584 .tcl files, 95,000 lines of TCL. -Jim Alphabetical: ns_chmod2 ns_config 20 ns_conn 353 ns_cp 5 ns_cpfp 11 ns_crypt7 ns_db 79 ns_dberrorcode 2 ns_dberrormsg 2 ns_dbquotevalue 513 ns_fmttime 219 ns_ftruncate7 ns_getcsv 9 ns_getform 107 ns_geturl 7 ns_guesstype3 ns_hostbyaddr 1 ns_http_gets1 ns_http_puts1 ns_httpget 25 ns_httptime 15 ns_info 504 ns_link 6 ns_localsqltimestamp11 ns_localtime35 ns_log 542 ns_mkdir10 ns_mktemp 7 ns_mutex103 ns_ns_http_readable 1 ns_openexcl 2 ns_paircmp 8 ns_parseheader 2 ns_parsehttptime11 ns_putscript2 ns_queryget 8 ns_quotehtml45 ns_register_filter 16 ns_register_proc41 ns_respond 2 ns_responds 1 ns_return 450 ns_returnbadrequest 1 ns_returnfile 17 ns_returnmoved 1 ns_returnnotfound 6 ns_returnredirect 80 ns_rwlock 8 ns_schedule_daily 8 ns_schedule_proc12 ns_sendmail 82 ns_sendmail_inject 2 ns_sendmail_smtpdown4 ns_server 5 ns_set 3550 ns_share843 ns_shutdown 2 ns_sleep47 ns_smtp_recv14 ns_smtp_send14 ns_sockcheck1 ns_socknread6 ns_sockopen 7 ns_sockreadwait 1 ns_sockselect 5 ns_sourceproc 6 ns_state38 ns_state_delete 4 ns_state_purge 3 ns_state_restore1 ns_state_save 4 ns_state_view 1 ns_striphtml115 ns_stripthtml 1 ns_symlink 5 ns_thread 46 ns_time 519 ns_tmpnam 3 ns_truncate 3 ns_unlink 65 ns_url2file 1 ns_urldecode15 ns_urlencode444 ns_user 4 ns_write6 ns_writefp 3 By frequency of source references: ns_hostbyaddr 1 ns_http_gets1 ns_http_puts1 ns_ns_http_readable 1 ns_responds 1 ns_returnbadrequest 1 ns_returnmoved 1 ns_sockcheck1 ns_sockreadwait 1 ns_state_restore1 ns_state_view 1 ns_stripthtml 1 ns_url2file 1 ns_chmod2 ns_dberrorcode 2 ns_dberrormsg 2 ns_openexcl 2 ns_parseheader 2 ns_putscript2 ns_respond 2 ns_sendmail_inject 2 ns_shutdown 2 ns_guesstype3 ns_state_purge 3 ns_tmpnam 3 ns_truncate 3 ns_writefp 3 ns_sendmail_smtpdown4 ns_state_delete 4 ns_state_save 4 ns_user 4 ns_cp 5 ns_server 5 ns_sockselect 5 ns_symlink 5 ns_link 6 ns_returnnotfound 6 ns_socknread6 ns_sourceproc 6 ns_write6 ns_crypt7 ns_ftruncate7 ns_geturl 7 ns_mktemp 7 ns_sockopen 7 ns_paircmp 8 ns_queryget 8 ns_rwlock 8 ns_schedule_daily 8 ns_getcsv 9 ns_mkdir10 ns_cpfp 11 ns_localsqltimestamp11 ns_parsehttptime11 ns_schedule_proc12 ns_smtp_recv14 ns_smtp_send14 ns_httptime 15 ns_urldecode15 ns_register_filter 16 ns_returnfile 17 ns_config 20 ns_httpget 25 ns_localtime35 ns_state38 ns_register_proc41 ns_quotehtml45 ns_thread 46 ns_sleep47 ns_unlink 65 ns_db 79 ns_returnredirect 80 ns_sendmail 82 ns_mutex103 ns_getform 107 ns_striphtml115 ns_fmttime 219 ns_conn 353 ns_urlencode444 ns_return 450 ns_info 504 ns_dbquotevalue 513 ns_time 519 ns_log 542 ns_share843 ns_set 3550 I'd like to conduct a quick informal poll. What are the top 10 most commonly used ns_* API calls that you use? Responses to the list, or directly to me, are fine. - Dossy -- Dossy Shiobara mail: [EMAIL PROTECTED] Panoptic Computer Network web: http://www.panoptic.com/
Re: [AOLSERVER] What parts of the ns_* API do you use the most?
On 2001.04.28, Rob Mayoff [EMAIL PROTECTED] wrote: I'd like to conduct a quick informal poll. What are the top 10 most commonly used ns_* API calls that you use? Do you mean at authoring time or execution time? Execution time, I guess. Pretend nsd had a profiling mode that kept track of what API calls were executed, how many times, and min/avg/max time spent executing each. What do you imagine would be the top 10 in the list, by count? (I think I might start working on this so that this profiling information could be reported via something like nstelemetry.adp -- wonder if anyone else thinks this could be useful.) - Dossy -- Dossy Shiobara mail: [EMAIL PROTECTED] Panoptic Computer Network web: http://www.panoptic.com/
Re: [AOLSERVER] What parts of the ns_* API do you use the most?
The script I used is below. It's a crappy TCL reader we wrote for a
specific purpose and doesn't work quite right - doesn't handle strings
correctly, probably other stuff.
We don't use nsv_ routines - still on 2.3.3. Personally, I don't like
the nsv_ interface. With ns_share, a variable is a variable except
for its scope; with nsv_ vars, the access semantics are different and
you need to always know if you are dealing with an nsv_ when writing
code. So it's impossible to write generalized routines. Not cool,
IMO. In reading the TCL bible, it seems like linked vars (link TCL
vars to C vars) would have been an interesting avenue to investigate
for implementing ns_share in standard TCL. If we did use nsv_
routines, they would dwarf anything on here by an order of magnitude,
both in static and dynamic references, i.e., we do *a lot* of shared
variable stuff.
I did a quick check of the ns_info references: most are to hostname or
pageroot, pageroot being the more common. When we need to do an
internal redirect, we source a TCL file and [ns_info pageroot] is
referenced. Lots of our TCL scripts set some variables and then
source another script. Not great, but that's what got established 4
years ago for whatever reason. This is a case where there are lots of
static references, but the frequency of reference at runtime is not
that high.
Jim
Here's the script I used to count references, like:
tcl scantcl.tcl `find modules/tcl pages -name *.tcl`nscounts
sort -n +1 -2nscountsnscounts.freq
proc gettclline {fd _tclline _linenr} {
upvar $_tclline tclline
upvar $_linenr linenr
set tclline
set tcllinelen 0
while {1} {
incr linenr
if {[gets $fd line] == -1} {
return -1
}
set tclline [string range $tclline 0 [expr $tcllinelen-2]]$line
set tcllinelen [string length $tclline]
if {[string index $tclline [expr $tcllinelen-1]] != \\} {
return $tclline
}
}
}
foreach f $argv {
set fd [open $f]
set linenr 0
while {[gettclline $fd line linenr] != -1} {
while {[regexp -nocase {(^|[^a-z])(ns_[-a-z0-9_]+)(.*)} $line match dummy command
line]} {
iincr comcount($command)
}
}
close $fd
}
foreach ix [lsort [array names comcount]] {
puts $ix $comcount($ix)
}
Jim,
This is absolutely awesome. Yes, static analysis is just fine for
what I'm trying to answer, and I'm thinking perhaps having some
profiling in nsd (which can be turned on and off) could be really
useful for the dynamic analysis.
I'm presuming you didn't count these all by hand, that you did
it via some script. Would you mind sharing that with the list so
others can report the same information?
Otherwise, I guess I could write one, but I'd prefer to avoid
re-inventing the wheel if I know it exists. ;-)
Interestingly, these are your top 10:
ns_set 3550
ns_share843
ns_log 542
ns_time 519
ns_dbquotevalue 513
ns_info 504
ns_return 450
ns_urlencode444
ns_conn 353
ns_fmttime 219
ns_striphtml115
I'm not surprised to see ns_set be on top, and since you didn't
include any nsv_* counts, I don't know if ns_share really should
be #2 or would nsv_set and nsv_get be up there ... I didn't
expect to see ns_log but now that I do, it makes perfect sense.
I'm really surprised to see ns_info up there, though. What
are you using it for, if I might ask?
- Dossy
Re: [AOLSERVER] What parts of the ns_* API do you use the most?
Here are my results ... you know you can get a list of the actual runtime tcl commands ... including ns_ if you have tclstats on in your config dir. ns_browsermatch 1 ns_cp 1 ns_dateentrywidget 1 ns_findrowbyid 1 ns_http_read 1 ns_localsqldate 1 ns_logroll 1 ns_mktemp 1 ns_openexcl 1 ns_parseheader 1 ns_returnnotfound 1 ns_setexpires 1 ns_set_precision 1 ns_shutdown 1 ns_sockopen 1 ns_table3 1 ns_timeentrywidget 1 ns_unschedule_proc 1 ns_updatebutton 1 ns_updateheader 1 ns_url2file 1 ns_writes 1 ns_buildsqltimestamp 2 ns_configsections 2 ns_findset 2 ns_htmlselect 2 ns_paircmp 2 ns_paren 2 ns_perm 2 ns_perm_authorization_p 2 ns_pg 2 ns_putscript 2 ns_quotehtml 2 ns_returnbadrequest 2 ns_rmdir 2 ns_set_vars 2 ns_stats 2 ns_striphtml 2 ns_table 2 ns_tagelementset 2 ns_uudecode 2 ns_uuencode 2 ns_buildsqldate 3 ns_buildsqltime 3 ns_configsection 3 ns_dbquotename 3 ns_issmallint 3 ns_localtime 3 ns_atclose 4 ns_cache_size 4 ns_dbformvalue 4 ns_dbformvalueput 4 ns_fastpath 4 ns_httpopen 4 ns_parsehttptime 4 ns_parsesqltime 4 ns_returnunauthorized 4 ns_sourceproc 4 ns_atshutdown 5 ns_cache_names 5 ns_register_proc 5 ns_urldecode 5 ns_adp_puts 6 ns_localsqltimestamp 6 ns_parsesqltimestamp 6 ns_thread 6 ns_cache_stats 7 ns_parsesqldate 7 ns_tagelement 7 ns_share 8 ns_crypt 9 ns_returnerror 9 ns_httpget 10 ns_perm_group_added_for_this_forum 10 ns_schedule_proc 10 ns_config 11 ns_column 12 ns_dbquotevalue 12 ns_parsetime 12 ns_server 14 ns_adp_return 17 ns_formvalueput 17 ns_getform 22 ns_rand 23 ns_unlink 23 ns_register_filter 26 ns_adp_include 27 ns_info 48 ns_httptime 49 ns_time 68 ns_fmttime 71 ns_urlencode 73 ns_adp_break 75 ns_return 77 ns_write 114 ns_log 129 ns_returnredirect 144 ns_queryget 163 ns_conn 186 ns_set 1771 ns_db 2186 ns_puts 5948 nsv_id 1 nsv_vars 2 NSV_Global 3 nsv_names 14 nsv_incr 34 nsv_unset 40 nsv_exists 69 nsv_array 114 nsv_append 143 nsv_set 260 nsv_get 484 _ Get your FREE download of MSN Explorer at http://explorer.msn.com
