Re: debug statements to debugs statements
lör 2007-04-28 klockan 00:14 +0300 skrev Tsantilas Christos: Here is the patch which converts the debug statements to debugs statements: http://www.chtsanti.net/others/CONVER_DEBUGS.diff It is not excellent but it is not so bad... Applied with a bit of whitespace cleanup - Less aggressive folding of lines - Indent fixes little here and there. Maninly after if, or in switch statemetnts. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: debug statements to debugs statements
Here is the patch which converts the debug statements to debugs statements: http://www.chtsanti.net/others/CONVER_DEBUGS.diff It is not excellent but it is not so bad... Regards, Christos
Re: Debug-Sections for Release
Amos Jeffries wrote: This is a cleanup of the published Debug Sections in Squid 3.0. Source files are only altered in the commented DEBUG: areas. All changes are to better prepare these sections for automatic processing. Some are to bring code sections marked with unknown (ie section ??) up to date by marking with the actual section that is already being used in the code. Have also updated the mk-debugs.sh patterns to include more inclusive filepath expansions actually used for locating and sorting these DEBUG: tags. So has anyone with approval ability looked at auditing this for inclusion in HEAD yet? Amos
Re: debug statements to debugs statements
I do a first conversion from debug statements to debugs statements.
The patch is big (~750K) to post it to mailing list so I put the result
here:
http://www.chtsanti.net/others/CONVER_DEBUGS.diff
I hope that it is something needed for squid3 because the conversion
takes a lot of my time (specially when you have to merge with many
changes in main squid3 head)..
My Comments:
1) Should I create a temporary branch? If squid developers decide that
this work needed, will be easier the merging with Duane works for large
objects and the merging with main squid code.
2) I did not format the result. I try to run the astyle
(version:1.20.2) and it changed too many things in code. Moreover the
formated code I took as result in some cases was really bad
Which version of the astyle used in squid3?
3) I did not convert the debugs statements which have formating strings
with field width or precision modifiers. The reason is that I am not so
sure that the result will be really better. For example a printf
statement like the following:
printf(Float:%.2f, Hex Integer:%x Dec Integer: %05d \n\n,2.345,
13,13);
using iostreams can be written as:
cout Float:
setprecision(3) 2.345
,Hex Integer:
hex 13 dec
Dec Integer:
setw(5) setfill('0') 13 endl;
Moreover a cout resetiosflags (..) must be included in debugs
statement to avoid a debugs statement change the look of consequently
debugs.
Of course in the other hand, you have not to check if we have float or
double or integer or long long integer.
Are conversions like the above OK?
4) I was not able to use ios manipulators like hex/dec or endl inside
some files eg wccp2.cc file. The compiler (g++ 4.1.2) always return the
following message:
wccp2.cc:1455: error: ‘hex’ was not declared in this scope
I try to include iostream and iomanim inside this file but the result
was the same. The strange is that happens only in some of the c++ files
not in all of them.
Any idea?
Regards,
Christos
Re: debug statements to debugs statements
On Sun, Apr 22, 2007, Tsantilas Christos wrote: Adrian Chadd wrote: There's now an implied \n (ie, eol) in the debug path; your patch has \n's in there. Oops! I really did not see it (@#$%^[EMAIL PROTECTED]@#)! I will try to remove it.. But I think It will take me some years :-( . Erk! Adrian
Re: debug statements to debugs statements
OK, It takes something less than a year... The patch at: http://www.chtsanti.net/others/CONVER_DEBUGS.diff does not contains \n any more. I hope that it is OK now Please try to do not find any such error ... in Sunday... you can find it on Monday morning :-) ! Regards, Christos Adrian Chadd wrote: There's now an implied \n (ie, eol) in the debug path; your patch has \n's in there. Oops! I really did not see it (@#$%^[EMAIL PROTECTED]@#)! I will try to remove it.. But I think It will take me some years :-( .
Re: Debug-Sections for Release
On Sun, Apr 22, 2007, Amos Jeffries wrote: Amos Jeffries wrote: This is a cleanup of the published Debug Sections in Squid 3.0. Source files are only altered in the commented DEBUG: areas. All changes are to better prepare these sections for automatic processing. Some are to bring code sections marked with unknown (ie section ??) up to date by marking with the actual section that is already being used in the code. Have also updated the mk-debugs.sh patterns to include more inclusive filepath expansions actually used for locating and sorting these DEBUG: tags. So has anyone with approval ability looked at auditing this for inclusion in HEAD yet? It superficially looks fine.. bug me on Wednesday if it hasn't been committed before then. Adrian
Re: debug statements to debugs statements
On Wed, Apr 18, 2007, Tsantilas Christos wrote: Hi all, Here is a perl script which can be used to convert debug statements to new debugs statements :-) . It mostly works or just do standard errors. It does not formats the output. I think the astyle can do the rest work. Does not convert the debug statements, if the format string contains formating flags like %2.1f. I think can be used as reference for other related convertions too. Should I proceed with the convertion? Should I post a patch to bugzila or open a temporary branch on sourceforge better? I'm going to be slightly greedy/annoying here and say: * Yes please! Because * It cuts down on the amount of work required when migrating the codebase from C strings (and the C-string-like String type that Squid-3 is currently using) as there's a lot of places where C strings are debug'ed and I wasn't looking forward to converting them all to %*.s or similar (which allows you to pass an int length specifier before a buffer, a pre-requisite for logging strings when your strings suddenly aren't NUL-terminated!), but * I'd just start submitting diffs against Squid-3-HEAD and include whatever can be done into the current codebase; this is something that can be done very incrementally and shouldn't have any actual operational impact, and * It makes my next set of work easier (ie, starting to get rid of the C strings in Squid-3 and replacing them with String as a temporary stepping stone towards refcounted buffers.) 2c, Adrian
Re: debug statements to debugs statements
On Wed, 2007-04-18 at 11:30 +0300, Tsantilas Christos wrote: Here is a perl script which can be used to convert debug statements to new debugs statements :-) . It mostly works or just do standard errors. It does not formats the output. I think the astyle can do the rest work. Does not convert the debug statements, if the format string contains formating flags like %2.1f. If you get a chance, please modify the script debugging/logging to use one line for both the conversion status and the debug statement: Found: $line\n Converted: $mod_statement\n LeftAsIs: $line\n This way it is much easier to grep/see debug statements that were _not_ converted, for example. A few minor suggestions, if I may: You may want to replace \d* in a line format with \w+ because the arguments are required and sometimes contain variable names rather than constants. You may want to replace space and tab characters in the format with \s. If you are not using a matched string, no need to use parens. For example, this may work: if ( $line =~ /\sdebug\(\s*\w+\s*,\s*\w+\s*\)/ ) Thank you, Alex.
Re: debug statements to debugs statements
Thanks Alex, Also I handled some other minor cases and run it in my sources. Some statistics: grep -n Converted CONVERT.log |wc -l - 2316 grep -n LeftAsIs CONVERT.log |wc -l - 137 Only 5-6 cases needed some work by me. Now compiles and the debug info I am getting looks OK. I will continue testing the results and when I am ready I am going to post the patch. Regards, Christos Alex Rousskov wrote: . If you get a chance, please modify the script debugging/logging to use one line for both the conversion status and the debug statement: . A few minor suggestions, if I may:
Re: DEBUG Section for IPAddress
On Wed, 2007-04-04 at 13:02 +1200, Amos Jeffries wrote: Gah! I diffed the files backwards. Here is the actual patch for doc/debug-sections.txt Amos, The doc/mk-debugs.sh script you were using (or equivalent) needs to be fixed. It needs to grep files in subdirectories of src/. For example, it currently misses: DEBUG: section 93 ICAP (RFC 3507) Client Using 'find src -name *.cc -o -name *.h' instead of a src/*.cc mask may work. Alex.
Re: DEBUG Section for IPAddress
Alex Rousskov wrote: On Wed, 2007-04-04 at 13:02 +1200, Amos Jeffries wrote: Gah! I diffed the files backwards. Here is the actual patch for doc/debug-sections.txt Amos, The doc/mk-debugs.sh script you were using (or equivalent) needs to be fixed. It needs to grep files in subdirectories of src/. For example, it currently misses: DEBUG: section 93 ICAP (RFC 3507) Client Using 'find src -name *.cc -o -name *.h' instead of a src/*.cc mask may work. Alex. You looking at my first copy of the patch? That was badly mangled. Its there in the amended patch I re-posted: +section 90HTTP Cache Control Header +section 90Storage Manager Client-Side Interface + +section 92Storage File System +section 93ICAP (RFC 3507) Client + + +section ? Common Removal policy +section --External DISKD process implementation. +section ?? FDE +section ? HEAP based removal policies Amos
Re: DEBUG Section for IPAddress
Alex Rousskov wrote: On Mon, 2007-03-26 at 23:59 +1200, Amos Jeffries wrote: I have so far not placed any permanent debug statements in the IPAddress object. It's come about time I should be making some more permanent ones and so need a section number. Is there some method I don't know yet for assigning a unique level or do I just pick an unused number and write it in? If you have not received better instructions yet, just use the next unused number for the debug section and add it to your file(s) prefix just like most other source files do. The stale doc/debug-sections.txt is (or should be) generated based on the source code. Do not update that file unless you are willing to freshen it up first. Cheers, Alex. Thank you Alex, No I have not received any further instructions and assumed that was the best action to take. I picked level 36, but that turns out to be cache dir processing and I expect its in use own another branch. If by 'freshen up' you mean rebuild the file from source using a grep of the source files headers for DEBUG section and a manual massage. I did that earlier before seeing the doc/ file. The update patch resulting is attached. FYI to all: IP Handling will join IP Cache in section 14, unless there are any objections by easter. Amos --- DEBUG_levels.txt 2007-04-04 12:56:03.0 +1200 +++ doc/debug-sections.txt 2007-03-27 20:12:49.0 +1200 @@ -3,14 +3,11 @@ section 0 Client Database section 0 Debug Routines section 0 DNS Resolver -section 0 Hash Tables -section 0 UFS Store Dump section 0 WWW Client section 1 Startup and Main Loop section 2 Unlink Daemon section 3 Configuration File Parsing section 4 Error Generation -section 5 Comms section 5 Socket Functions section 6 Disk I/O Routines section 7 Multicast @@ -21,35 +18,21 @@ section 12Internet Cache Protocol section 13High Level Memory Pool Management section 14IP Cache -section 14IP Storage and Handling section 15Neighbor Routines section 16Cache Manager Objects section 17Request Forwarding section 18Cache Manager Statistics section 19Store Memory Primitives section 20Storage Manager -section 20Storage Manager Heap-based replacement -section 20Storage Manager Logging Functions -section 20Storage Manager MD5 Cache Keys -section 20Storage Manager Swapfile Metadata -section 20Storage Manager Swapfile Unpacker -section 20Storage Manager Swapin Functions -section 20Storage Manager Swapout Functions -section 20Store Rebuild Routines -section 21Integer functions section 21Misc Functions -section 21Time Functions section 22Refresh Calculation section 23URL Parsing -section 23URL Scheme parsing - +section 24WAIS Relay section 25MIME Parsing section 26Secure Sockets Layer Proxy section 27Cache Announcer section 28Access Control section 29Authenticator -section 29Negotiate Authenticator -section 29NTLM Authenticator section 30Ident (RFC 931) section 31Hypertext Caching Protocol section 32Asynchronous Disk I/O @@ -61,26 +44,20 @@ section 38Network Measurement Database section 39Cache Array Routing Protocol section 40User-Agent and Referer logging -section 40User-Agent logging section 41Event Processing section 42ICMP Pinger program section 43AIOPS -section 43Windows AIOPS section 44Peer Selection Algorithm section 45Callback Data Registry section 46Access Log -section 47Store COSS Directory Routines section 47Store Directory Routines section 48Persistent Connections -section 49SNMP Interface -section 49SNMP support +section 49SNMP section 50Log file handling section 51Filedescriptor Functions section 52URN Parsing section 53AS Number handling -section 53 Radix tree data structure implementation section 54Interprocess Communication -section 54Windows Interprocess Communication section 55HTTP Header section 56HTTP Message Body section 57HTTP Status-line @@ -89,7 +66,7 @@ section 60Packer: A uniform interface to store-like modules section 61Redirector section 62Generic Histogram -section 63Low Level Memory Pool Management +section 63- section 64HTTP Range Header section 65HTTP Cache Control Header section 66HTTP Header Tools @@ -104,38 +81,18 @@ section 75WHOIS protocol section 76Internal Squid Object handling section 77Delay Pools -section 78DNS lookups; interacts with lib/rfc1035.c -section 79Disk IO Routines -section 79Squid-side DISKD I/O functions. -section 79Squid-side Disk I/O functions. -section 79Storage Manager COSS Interface -section 79Storage Manager UFS Interface -section 80WCCP Support -section 81aio_xxx() POSIX emulation on Windows -section 81 CPU Profiling
Re: DEBUG Section for IPAddress
Gah! I diffed the files backwards. Here is the actual patch for doc/debug-sections.txt Amos --- doc/debug-sections.txt 2007-03-27 20:12:49.0 +1200 +++ DEBUG_levels.txt 2007-04-04 12:56:03.0 +1200 @@ -3,11 +3,14 @@ section 0 Client Database section 0 Debug Routines section 0 DNS Resolver +section 0 Hash Tables +section 0 UFS Store Dump section 0 WWW Client section 1 Startup and Main Loop section 2 Unlink Daemon section 3 Configuration File Parsing section 4 Error Generation +section 5 Comms section 5 Socket Functions section 6 Disk I/O Routines section 7 Multicast @@ -18,21 +21,35 @@ section 12Internet Cache Protocol section 13High Level Memory Pool Management section 14IP Cache +section 14IP Storage and Handling section 15Neighbor Routines section 16Cache Manager Objects section 17Request Forwarding section 18Cache Manager Statistics section 19Store Memory Primitives section 20Storage Manager +section 20Storage Manager Heap-based replacement +section 20Storage Manager Logging Functions +section 20Storage Manager MD5 Cache Keys +section 20Storage Manager Swapfile Metadata +section 20Storage Manager Swapfile Unpacker +section 20Storage Manager Swapin Functions +section 20Storage Manager Swapout Functions +section 20Store Rebuild Routines +section 21Integer functions section 21Misc Functions +section 21Time Functions section 22Refresh Calculation section 23URL Parsing -section 24WAIS Relay +section 23URL Scheme parsing + section 25MIME Parsing section 26Secure Sockets Layer Proxy section 27Cache Announcer section 28Access Control section 29Authenticator +section 29Negotiate Authenticator +section 29NTLM Authenticator section 30Ident (RFC 931) section 31Hypertext Caching Protocol section 32Asynchronous Disk I/O @@ -44,20 +61,26 @@ section 38Network Measurement Database section 39Cache Array Routing Protocol section 40User-Agent and Referer logging +section 40User-Agent logging section 41Event Processing section 42ICMP Pinger program section 43AIOPS +section 43Windows AIOPS section 44Peer Selection Algorithm section 45Callback Data Registry section 46Access Log +section 47Store COSS Directory Routines section 47Store Directory Routines section 48Persistent Connections -section 49SNMP +section 49SNMP Interface +section 49SNMP support section 50Log file handling section 51Filedescriptor Functions section 52URN Parsing section 53AS Number handling +section 53 Radix tree data structure implementation section 54Interprocess Communication +section 54Windows Interprocess Communication section 55HTTP Header section 56HTTP Message Body section 57HTTP Status-line @@ -66,7 +89,7 @@ section 60Packer: A uniform interface to store-like modules section 61Redirector section 62Generic Histogram -section 63- +section 63Low Level Memory Pool Management section 64HTTP Range Header section 65HTTP Cache Control Header section 66HTTP Header Tools @@ -81,18 +104,38 @@ section 75WHOIS protocol section 76Internal Squid Object handling section 77Delay Pools -section 78DNS lookups -section 79Lowlevel store I/O -section 80WCCP -section 81Cache Store +section 78DNS lookups; interacts with lib/rfc1035.c +section 79Disk IO Routines +section 79Squid-side DISKD I/O functions. +section 79Squid-side Disk I/O functions. +section 79Storage Manager COSS Interface +section 79Storage Manager UFS Interface +section 80WCCP Support +section 81aio_xxx() POSIX emulation on Windows +section 81 CPU Profiling Routines section 82External ACL section 83SSL accelerator support section 84Helper process maintenance -section 85Client side request management - after parsing, before caching -section 86ESI processing -section 87client side stream management -section 88Client side reply management - from store to stream -section 89NAT / IP Interception -section 90Store Client -section 91Http Surrogate-Control Header -section 92Store File System +section 85Client-side Request Routines +section 86ESI Expressions +section 86ESI Processing +section 87Client-side Stream routines. +section 88Client-side Reply Routines +section 89NAT / IP Interception +section 90HTTP Cache Control Header +section 90Storage Manager Client-Side Interface + +section 92Storage File System +section 93ICAP (RFC 3507) Client + + +section ? Common Removal policy +section --External DISKD process implementation. +section ?? FDE +section ? HEAP based removal policies +section ??Linked list functions +section ? LRU Removal
Re: Debug
mån 2007-01-08 klockan 17:27 +1300 skrev Amos Jeffries: The problem I am tracking is a DNS timeout when a TCP_MISS:DIRECT begins but can't figure out how to get any action-by-action or function-by-function record of what calls are being made in the area. Well. it is not so easy. Squid is effectively one big state machine centered around the comm loop. While it's waiting for an external event (network I/O mainly) there is no code only state.. Assuming it's an address lookup failing then the two most important entry points to look as is ipcache_nbgethostbyname and idnsGrokReply. ipcache_nbgethostbyname is the entry point to the DNS code for looking up the address of a host name. idnsGrokReply is where the raw DNS responses is processed and results handed back to the module which asked for it.. ipcache_nbgethostbyname to the point where the DNS query is ready to be sent should be normal linear code. idnsGrokReply is all linear code up to an including the callback to the module which initiated the DNS query (i.e. all the way to the forwarding code when the query came from there.). But there is one or two callbacks before the forwarding code is reached so it's perhaps not entirely obvious if only reading code. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: Debug
Henrik Nordstrom wrote: mån 2007-01-08 klockan 17:27 +1300 skrev Amos Jeffries: The problem I am tracking is a DNS timeout when a TCP_MISS:DIRECT begins but can't figure out how to get any action-by-action or function-by-function record of what calls are being made in the area. Well. it is not so easy. Squid is effectively one big state machine centered around the comm loop. While it's waiting for an external event (network I/O mainly) there is no code only state.. I'm getting to see the picture as clearly an opaque problem. ;-) Over the last two days. The more I work on this the more I wish there was some developer documentation saying things like this clearly available in the source or on the website. So newbies like myself can quickly get a picture of where to look for the entry points to all the events without having to ask. Assuming it's an address lookup failing then the two most important entry points to look as is ipcache_nbgethostbyname and idnsGrokReply. ipcache_nbgethostbyname is the entry point to the DNS code for looking up the address of a host name. idnsGrokReply is where the raw DNS responses is processed and results handed back to the module which asked for it.. ipcache_nbgethostbyname to the point where the DNS query is ready to be sent should be normal linear code. idnsGrokReply is all linear code up to an including the callback to the module which initiated the DNS query (i.e. all the way to the forwarding code when the query came from there.). But there is one or two callbacks before the forwarding code is reached so it's perhaps not entirely obvious if only reading code. Regards Henrik Stranger, and stranger. Thank you for those entry points. I found the idnsGrokReply after much random code reading yesterday and found it doing something. Also, the error has moved further down the sequence now from DNS timeout to connection timeout (to localhost apache!). Without any code changes on my part. I'm going to have to trace the traffic in my environment I think to see exactly what its asking for before I get further into the code. :-( Amos
Re: Debug
tis 2007-01-09 klockan 13:30 +1300 skrev Amos Jeffries: Over the last two days. The more I work on this the more I wish there was some developer documentation saying things like this clearly available in the source or on the website. So newbies like myself can quickly get a picture of where to look for the entry points to all the events without having to ask. Some of this is actually in the programmers guide. ipcache_nbgethostbyname is at least.. but very very much is missing. The programmers guide got started many years after most of the code had been developed, and focus has been mostly on documenting the areas changed/redone.. Personally I'd like to see the little we have in terms of programmers guide translated to doxygen and moved down to the appropriate modules. Would hopefully make it more natural to write and maintain the code module documentation.. But that's a different project. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
