I have a tclsh script which gets data from AOLserver via an http GET.
As AOLserver writes out the data, the script reads it one row at a
time and does something with each row. If the script takes too long
for each row, the data gets silently truncated. In practice this
problem occurs rarely, but it's easy to replicate by putting a one
second sleep into the client script after it reads each row of data.
Now, obviously there are ways to avoid this problem by forcing the
client script to be a fast consumer of the data from the server. But,
what's really going? Is some socket timing out? How do I track this
down and guarantee that the data will NEVER be truncated?
I'm using Solaris (SunOS 5.8), AOLserver 3.3+ad13, and tclsh 8.3.3.
In my tclsh client script, doing 'fconfigure $fd -buffersize 1000000'
on the socket increased the amount of data I get back before the
truncation happens. But, the problem still occurs, just at a larger
number of bytes, and further increasing -buffersize makes no
difference.
I looked through lots of AOLserver and Tcl C code to try to better
understand what it's really doing with the underlying sockets, but
that didn't help much. Does the AOLserver keepalive stuff (parameters
keepalivetimeout and maxkeepalive, etc.) or the SO_KEEPALIVE and
SO_LINGER socket options have any bearing on this? My AOLserver's
maxkeepalive param is 0.
As an experiment I tried making the http get request several different
ways, in the hopes that they would show different behaviors and thus
help me narrow down what's really going on:
I have two AOLserver tcl pages, put.tcl and get.tcl. put.tcl simply
does an ns_returnfile of a file of test data. get.tcl opens an http
connection to put.tcl in one of three different ways and SLOWLY reads
each row of data and writes it to a file.
E.g, get.tcl is writing ou the data it gets from put.tcl like this:
set fd_out [open $file_out "w"]
while { [gets $fd line] >= 0 } {
puts $fd_out $line
if { $slow_p } {
ns_sleep 1
}
}
Above, $fs is the incoming socket handle, opened in one of three
different ways:
- wget (from a Tcl command pipeline)
- ns_httpopen (which uses ns_sockopen internally)
- tcl_socket (the Tcl socket command)
The actual code for each of those methods is further below.
My test data file is 334,323 bytes, 10,237 rows. In all test cases
the data gets truncated, but the details are different:
wget: Varies.
ns_httpopen: 86,052 bytes; 2,642 rows; same every time.
tcl_socket: 171,521 bytes; 5,266 rows; same every time.
The wget stuff is a bit weird, as wget is definitely doing some sort
of retry of its own. The wget output file often has MORE rows in it
then the source file, but it's a lot of duplicate rows, and always has
missing rows. The important thing there is that it also appears to
experience truncated data.
Any ideas on this?
Code for different ways of hitting put.tcl with http get request:
wget {
set fd [open [list |wget -O - -q $request] "r"]
}
ns_httpopen {
foreach [list fd fd_write conn_set_id] [ns_httpopen GET $request {}] { break }
}
tcl_socket {
##split request up into host, port, and the request 'file'.
if { ![regexp {^http://([A-Za-z]+):([0-9]+)(/.*)$} $request whole
Connection_Host Connection_Port r\eal_request] } {
error "bad request $request"
}
## open socket to host, configured binary so it doesn't mess with line endings.
set fd [socket $Connection_Host $Connection_Port]
fconfigure $fd -buffering line
fconfigure $fd -encoding binary
fconfigure $fd -translation {binary binary}
fconfigure $fd -buffersize 1000000
## Write http request
puts -nonewline $fd "GET $real_request HTTP/1.0\r\n"
puts -nonewline $fd "Fd: Keep-Alive\r\n"
puts -nonewline $fd "\r\n"
flush $fd
}
--
Andrew Piskorski <[EMAIL PROTECTED]>
http://www.piskorski.com
