Re: [AOLSERVER] Problem with file uploads larger than maxinput
Brian many thanks, i commited your change to the CVS on sourceforge. -gustaf neumann -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Dear Gustaf (and all) thanks Gustaf for the fixes - reading your code, it's interesting to see how AOLserver works internally. And it's all so elegant, meaning even I can follow it! I can confirm your fix has resolved the 413 problem. I attempted to take it a step further and add a 413 redirect to the config.tcl as follows: ns_section ns/server/${server}/redirects ns_param 413/global/file-too-large.html However, this caused AOLserver to crash when uploading large files. After some debugging, I figured out a solution to unmark the connection once the redirect has been called (patch to op.c attached). Obviously, please let me know if there is a better way. I also added a new TCL proc 'ns_returntoolarge' which maps to the Ns_ConnReturnEntityTooLarge C function. I find these TCL procs handy as an alternative to the config.tcl redirects, as they can be overloaded if needed. Patches to tclcmds.c and tclresp.c attached. I'd love to hear any feedback that people have on these patches. As it's my first time, I'd be very grateful if you could review them. Best wishes Brian From: Gustaf Neumann [neum...@wu-wien.ac.at] Sent: 09 July 2011 11:25 To: AOLserver Discussion Cc: Fenton, Brian Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Dear all, A few minutes ago, i have committed a change to the sourceforge aolserver repository that should should fix the problem finally. The primary problem was not that the driver replied directly to the client, but the fact that client (e.g. firefox) has not yet sent the full request while the server had already replied and closed the connection. Since the handling of the header happens in aolserver before the full request from the client is received, the server could close the connection before the client has sent the full request. So, although the client (e.g. firefox) has already received a proper reply from aolserver, it complained about the fact, that it could not sent the full request. This explains, why the error handling worked reliably with small upload files but not with large ones in current browsers; it explains as well, that simple clients (like nc) worked always reliable. Additionally, i have moved the reply handling to the connection threads, and made some more cosmetical changes. Brian, please check again. -gustaf neumann On 08.07.11 13:49, Fenton, Brian wrote: Dear Gustaf many thanks for the reply. Yes, I can confirm your results - if I set a limit of 10k and upload a file of 15k, it works perfectly for me. Would be great if you could look into this. If not, maybe you could give me a few pointers where to change the request handling thread. many thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neum...@wu-wien.ac.at] Sent: 08 July 2011 12:20 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Dear Brian, Thanks with the config files, they are fine. With some more testing, i found some more insights: The version in CVS head returns always the right result to a client (tested with nc -v localhost 8006 POST.txt), no matter of the size of the uploaded content, but works only reliably in browsers (firefox, chrome) for rather small requests (set your maxupload to 5k, upload a 10 k file), and does reliable NOT work with upload files 20MB (which is the interesting case). So, the problem looks like a timing/buffering problem to me. RFC 2616 says e.g. If the server chooses to close the connection immediately after sending the response, it SHOULD send a Connection header including the connection-token close. The reply of aolserver contains the connection: close, so this seems standard compliant. The driver has to close the connection immediately after sending the reply, so, the moral of the story seems to be that we have to get the response handling out of the driver and require a solution via the request handling thread - what's the best solution anyhow. So far, i don't understand, why the size of the upload request changes the behavior. maybe, i can look into this next week. -gustaf neumann On 06.07.11 13:08, Fenton, Brian wrote: Dear Gustaf thanks for the reply. I have tried it with the latest OpenACS config.tcl and also another older one I use (both attached). I just assumed I was doing something wrong, didn't want to sound as though I was criticising your work. kind regards Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neum...@wu-wien.ac.at] Sent: 05 July 2011 23:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Strange, it works for me. can you send me your config file? Concerning the right
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Dear all, A few minutes ago, i have committed a change to the sourceforge aolserver repository that should should fix the problem finally. The primary problem was not that the driver replied directly to the client, but the fact that client (e.g. firefox) has not yet sent the full request while the server had already replied and closed the connection. Since the handling of the header happens in aolserver before the full request from the client is received, the server could close the connection before the client has sent the full request. So, although the client (e.g. firefox) has already received a proper reply from aolserver, it complained about the fact, that it could not sent the full request. This explains, why the error handling worked reliably with small upload files but not with large ones in current browsers; it explains as well, that simple clients (like nc) worked always reliable. Additionally, i have moved the reply handling to the connection threads, and made some more cosmetical changes. Brian, please check again. -gustaf neumann On 08.07.11 13:49, Fenton, Brian wrote: Dear Gustaf many thanks for the reply. Yes, I can confirm your results - if I set a limit of 10k and upload a file of 15k, it works perfectly for me. Would be great if you could look into this. If not, maybe you could give me a few pointers where to change the request handling thread. many thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neum...@wu-wien.ac.at] Sent: 08 July 2011 12:20 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Dear Brian, Thanks with the config files, they are fine. With some more testing, i found some more insights: The version in CVS head returns always the right result to a client (tested with nc -v localhost 8006 POST.txt), no matter of the size of the uploaded content, but works only reliably in browsers (firefox, chrome) for rather small requests (set your maxupload to 5k, upload a 10 k file), and does reliable NOT work with upload files 20MB (which is the interesting case). So, the problem looks like a timing/buffering problem to me. RFC 2616 says e.g. If the server chooses to close the connection immediately after sending the response, it SHOULD send a Connection header including the connection-token close. The reply of aolserver contains the connection: close, so this seems standard compliant. The driver has to close the connection immediately after sending the reply, so, the moral of the story seems to be that we have to get the response handling out of the driver and require a solution via the request handling thread - what's the best solution anyhow. So far, i don't understand, why the size of the upload request changes the behavior. maybe, i can look into this next week. -gustaf neumann On 06.07.11 13:08, Fenton, Brian wrote: Dear Gustaf thanks for the reply. I have tried it with the latest OpenACS config.tcl and also another older one I use (both attached). I just assumed I was doing something wrong, didn't want to sound as though I was criticising your work. kind regards Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neum...@wu-wien.ac.at] Sent: 05 July 2011 23:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Strange, it works for me. can you send me your config file? Concerning the right place: As it was discussed here, it would be certainly better to move the reply-sending to a request handling thread (or a spooling thread like in naviserver), simply to be sure that the driver is never blocking. This should not be complicated, but i simply did not have to time, and i see no reason, why replying from the driver should not work. -gustaf -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Dear Brian, Thanks with the config files, they are fine. With some more testing, i found some more insights: The version in CVS head returns always the right result to a client (tested with nc -v localhost 8006 POST.txt), no matter of the size of the uploaded content, but works only reliably in browsers (firefox, chrome) for rather small requests (set your maxupload to 5k, upload a 10 k file), and does reliable NOT work with upload files 20MB (which is the interesting case). So, the problem looks like a timing/buffering problem to me. RFC 2616 says e.g. If the server chooses to close the connection immediately after sending the response, it SHOULD send a Connection header including the connection-token close. The reply of aolserver contains the connection: close, so this seems standard compliant. The driver has to close the connection immediately after sending the reply, so, the moral of the story seems to be that we have to get the response handling out of the driver and require a solution via the request handling thread - what's the best solution anyhow. So far, i don't understand, why the size of the upload request changes the behavior. maybe, i can look into this next week. -gustaf neumann On 06.07.11 13:08, Fenton, Brian wrote: Dear Gustaf thanks for the reply. I have tried it with the latest OpenACS config.tcl and also another older one I use (both attached). I just assumed I was doing something wrong, didn't want to sound as though I was criticising your work. kind regards Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neum...@wu-wien.ac.at] Sent: 05 July 2011 23:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Strange, it works for me. can you send me your config file? Concerning the right place: As it was discussed here, it would be certainly better to move the reply-sending to a request handling thread (or a spooling thread like in naviserver), simply to be sure that the driver is never blocking. This should not be complicated, but i simply did not have to time, and i see no reason, why replying from the driver should not work. -gustaf -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Dear Gustaf many thanks for the reply. Yes, I can confirm your results - if I set a limit of 10k and upload a file of 15k, it works perfectly for me. Would be great if you could look into this. If not, maybe you could give me a few pointers where to change the request handling thread. many thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neum...@wu-wien.ac.at] Sent: 08 July 2011 12:20 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Dear Brian, Thanks with the config files, they are fine. With some more testing, i found some more insights: The version in CVS head returns always the right result to a client (tested with nc -v localhost 8006 POST.txt), no matter of the size of the uploaded content, but works only reliably in browsers (firefox, chrome) for rather small requests (set your maxupload to 5k, upload a 10 k file), and does reliable NOT work with upload files 20MB (which is the interesting case). So, the problem looks like a timing/buffering problem to me. RFC 2616 says e.g. If the server chooses to close the connection immediately after sending the response, it SHOULD send a Connection header including the connection-token close. The reply of aolserver contains the connection: close, so this seems standard compliant. The driver has to close the connection immediately after sending the reply, so, the moral of the story seems to be that we have to get the response handling out of the driver and require a solution via the request handling thread - what's the best solution anyhow. So far, i don't understand, why the size of the upload request changes the behavior. maybe, i can look into this next week. -gustaf neumann On 06.07.11 13:08, Fenton, Brian wrote: Dear Gustaf thanks for the reply. I have tried it with the latest OpenACS config.tcl and also another older one I use (both attached). I just assumed I was doing something wrong, didn't want to sound as though I was criticising your work. kind regards Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neum...@wu-wien.ac.at] Sent: 05 July 2011 23:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Strange, it works for me. can you send me your config file? Concerning the right place: As it was discussed here, it would be certainly better to move the reply-sending to a request handling thread (or a spooling thread like in naviserver), simply to be sure that the driver is never blocking. This should not be complicated, but i simply did not have to time, and i see no reason, why replying from the driver should not work. -gustaf -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Hello I've been struggling a little with this change. I took the HEAD version including Gustaf's changes and compiled it. When I uploaded a file larger than the max, I could see that it was calling Gustaf's new function Ns_ConnReturnEntityToLarge (correctly triggered by the E_CRANGE) - however, the user still got the closed connection, so even though the function was running, it wasn't sending any message back to the user. So then I decided to do what Dossy/Enrique suggested: when the E_CRANGE condition was met, mark the socket state as SOCK_OVERFLOW, and then in driver.c in the Process sockets ready to run while loop, if socket state is SOCK_OVERFLOW then call Ns_ConnReturnEntityToLarge. However, this still isn't sending anything back to the user and the connection is still getting closed. To do a sanity check that the Ns_ConnReturnEntityToLarge function actually works, I created a TCL proc called ns_returntoolarge which mapped to Ns_ConnReturnEntityToLarge - and yes this works fine when called directly from a TCL program (correctly returning the The request entity (e.g. file to be uploaded) is too large. message to the browser). So knowing that the C function works fine, my conclusion then was that I must be calling it from the wrong place (or passing in the wrong conn pointer or whatever). So the question is - what's going on here and where is the right place to call Ns_ConnReturnEntityToLarge? I noticed that none of the similar Ns_ConnReturn* functions ever get called in driver.c, so I started to suspect that driver.c was the wrong place. The other Ns_ConnReturn* functions seem to mainly get called in op.c and queue.c, however neither of those seem to be involved in this particular situation. I don't see any other obvious place, so I'm a little bit stumped at the moment. Any suggestions? many thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neum...@wu-wien.ac.at] Sent: 24 June 2011 08:00 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Dear all, As a small contribution, i added a patch to sourceforge cvs. The patch returns the 413 error message via a new function Ns_ConnReturnEntityToLarge(), which is written in the same style as Ns_ConnReturnNotFound(). The patch is somewhat minimal and handles this issue entirely in the driver (also, forcing the log entry). I have paid no special attention to the custom response page. -gn On 23.06.11 19:14, Dossy Shiobara wrote: That was what I was thinking -- driver marks the request as exceeding the limit, and setting the response status code to 413. The benefits that I see (if implemented the way I'm imagining) -- 1) access logging of requests w/ 413 status code 2) custom response page via ns/server/${server}/redirects. If you and Brian could work up a patch, that'd be wonderful! Does anyone see any problems with this approach? Any reason not to do it? I don't suppose it can possibly break any kind of backward compatibility, as no code right now can even be written to handle such a scenario, anyway. Of course, once we decide on a behavior, and folks code against the implementation ... changing/fixing it becomes more expensive for everyone, so if there's any kind of worry about how this is going to work, lets iron out the details now ;) On 6/23/11 12:29 PM, Enrique Catalan wrote: IMHO, I agree with Dossy, to use the driver thread to check the hard limits and instead of dropping the connection, just mark the HTTP request and let the request handler to return the 413. I also think the template could be configured in the 'ns_section ns/server/${server}/redirects' , isn't it ? If you all agree with this, Brian and I can help to get a patch. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Strange, it works for me. can you send me your config file? Concerning the right place: As it was discussed here, it would be certainly better to move the reply-sending to a request handling thread (or a spooling thread like in naviserver), simply to be sure that the driver is never blocking. This should not be complicated, but i simply did not have to time, and i see no reason, why replying from the driver should not work. -gustaf -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Thanks for this Gustaf, I'm rolling up my sleeves, and have just put on my new C hat (never worn before). Let's see how I get on Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neum...@wu-wien.ac.at] Sent: 24 June 2011 08:00 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Dear all, As a small contribution, i added a patch to sourceforge cvs. The patch returns the 413 error message via a new function Ns_ConnReturnEntityToLarge(), which is written in the same style as Ns_ConnReturnNotFound(). The patch is somewhat minimal and handles this issue entirely in the driver (also, forcing the log entry). I have paid no special attention to the custom response page. -gn On 23.06.11 19:14, Dossy Shiobara wrote: That was what I was thinking -- driver marks the request as exceeding the limit, and setting the response status code to 413. The benefits that I see (if implemented the way I'm imagining) -- 1) access logging of requests w/ 413 status code 2) custom response page via ns/server/${server}/redirects. If you and Brian could work up a patch, that'd be wonderful! Does anyone see any problems with this approach? Any reason not to do it? I don't suppose it can possibly break any kind of backward compatibility, as no code right now can even be written to handle such a scenario, anyway. Of course, once we decide on a behavior, and folks code against the implementation ... changing/fixing it becomes more expensive for everyone, so if there's any kind of worry about how this is going to work, lets iron out the details now ;) On 6/23/11 12:29 PM, Enrique Catalan wrote: IMHO, I agree with Dossy, to use the driver thread to check the hard limits and instead of dropping the connection, just mark the HTTP request and let the request handler to return the 413. I also think the template could be configured in the 'ns_section ns/server/${server}/redirects' , isn't it ? If you all agree with this, Brian and I can help to get a patch. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
A thought for an alternate way to handle this - if the content-length is too large, the just don't read the content, but set an error flag on the conn. That way, an adventurous system could return a custom error page, or even handle reading the content through custom code. I think this could also allow handling of uploaded files larger than 2G as someone else was having. The default handling by returning a 413 error could be implemented as a pre-queue filter, which would still handle the error without needing a conn thread. If anyone's messing around in there, this might be a good time to also think about how to handle http/1.1's 'Expect: 100-continue', which should be most beneficial for large entity bodies anyway. -J Dossy Shiobara wrote: Yes, on E_CRANGE, the driver thread should return HTTP 413 Request too large, IMHO. And, we should be able to configure a custom response for status code 413 ... Patches welcome ;) On 6/23/11 11:28 AM, Fenton, Brian wrote: I'm using OpenACS hence the TCL. I just want to let the user know that their file is over the size limit. Could we, in driver.c instead of closing the connection socket, return a custom template (like Apache does) ? -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Jim Davidson wrote: I think today, in 2011, some of the flexibility we imagined back in 1995 isn't really so needed. This includes general purpose logging plugins and network drivers when there really is just the common log format, ordinary and SSL sockets. More in the core for the base HTTP processing could make things easier to maintain. I may be the only one, but I miss some of the flexibilty that has already been removed; in particular the nsunix driver that let me run multiple servers with different credentials. There are also arguments for leaving flexibility in other things. I suspect they're the same arguments you had in 1995, but I'll offer my thoughts too. Some people do want more than common log format written to a file (distributed logging, or writing to a database). There are extensions to HTTP on the horizon (websockets and SPDY come to mind). These things don't need to be directly supported, but a framework for letting someone add in support without rewriting major parts of the server is, IMHO, a good thing. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Dear all, Most of the times Aolserver is used with a reverse proxy like Pound or Nginx in front of it. I would just leave Aolserver as it is... and make sure that the reverse proxy checks the size of the objects to be uploaded and provides a proper error message. For Nginx: server { listen 80; client_max_body_size 12M; -- here the maximum size is 12M Then for the error message: To redirect to a simple 413 object too big page for a specific location: location /my_page { error_page413 = /my_page_object_too_big.html;} A site wide 413 page: server { listen 80; error_page 413 /website_object_too_big.html; ... Nothing to implement... everything is already done for us Ciao, Maurizio -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Dear all, As a small contribution, i added a patch to sourceforge cvs. The patch returns the 413 error message via a new function Ns_ConnReturnEntityToLarge(), which is written in the same style as Ns_ConnReturnNotFound(). The patch is somewhat minimal and handles this issue entirely in the driver (also, forcing the log entry). I have paid no special attention to the custom response page. -gn On 23.06.11 19:14, Dossy Shiobara wrote: That was what I was thinking -- driver marks the request as exceeding the limit, and setting the response status code to 413. The benefits that I see (if implemented the way I'm imagining) -- 1) access logging of requests w/ 413 status code 2) custom response page via ns/server/${server}/redirects. If you and Brian could work up a patch, that'd be wonderful! Does anyone see any problems with this approach? Any reason not to do it? I don't suppose it can possibly break any kind of backward compatibility, as no code right now can even be written to handle such a scenario, anyway. Of course, once we decide on a behavior, and folks code against the implementation ... changing/fixing it becomes more expensive for everyone, so if there's any kind of worry about how this is going to work, lets iron out the details now ;) On 6/23/11 12:29 PM, Enrique Catalan wrote: IMHO, I agree with Dossy, to use the driver thread to check the hard limits and instead of dropping the connection, just mark the HTTP request and let the request handler to return the 413. I also think the template could be configured in the 'ns_section ns/server/${server}/redirects' , isn't it ? If you all agree with this, Brian and I can help to get a patch. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Cool. The driver thread is technically a single threaded thing which shouldn't block and the connection return functions are blocking but as the response is small, it probably just works. Also, the connection code hasn't been verified to run in the driver (filters, traces, etc.) but again it likely just works unless there's a really goofy, blocking, involved filter lurking in some installation. A more involved, perhaps unnecessary solution would be to either move all the response writing back to the driver in non-blocking I/O mode for all requests and/or taking error condition connections and simply tightening them up and sending them to a connection thread to generate the response safely. Hmm The later is probably safer/easier and more backward compatible but the former concept always intrigued me. Anyway, this patch sounds like a smart first step and can stop the bleeding on the most common error response you'd want to know about, exceed content size. -Jim On Jun 24, 2011, at 1:00 AM, Gustaf Neumann wrote: Dear all, As a small contribution, i added a patch to sourceforge cvs. The patch returns the 413 error message via a new function Ns_ConnReturnEntityToLarge(), which is written in the same style as Ns_ConnReturnNotFound(). The patch is somewhat minimal and handles this issue entirely in the driver (also, forcing the log entry). I have paid no special attention to the custom response page. -gn On 23.06.11 19:14, Dossy Shiobara wrote: That was what I was thinking -- driver marks the request as exceeding the limit, and setting the response status code to 413. The benefits that I see (if implemented the way I'm imagining) -- 1) access logging of requests w/ 413 status code 2) custom response page via ns/server/${server}/redirects. If you and Brian could work up a patch, that'd be wonderful! Does anyone see any problems with this approach? Any reason not to do it? I don't suppose it can possibly break any kind of backward compatibility, as no code right now can even be written to handle such a scenario, anyway. Of course, once we decide on a behavior, and folks code against the implementation ... changing/fixing it becomes more expensive for everyone, so if there's any kind of worry about how this is going to work, lets iron out the details now ;) On 6/23/11 12:29 PM, Enrique Catalan wrote: IMHO, I agree with Dossy, to use the driver thread to check the hard limits and instead of dropping the connection, just mark the HTTP request and let the request handler to return the 413. I also think the template could be configured in the 'ns_section ns/server/${server}/redirects' , isn't it ? If you all agree with this, Brian and I can help to get a patch. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Hi, The short answer is no, there's no access log entry although there may be a server log message buried in the chatter. The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, max request exceeded. -Jim On Jun 23, 2011, at 6:04 AM, Fenton, Brian wrote: Hi When I upload a file larger than the maxinput I get the Firefox browser page with a the connection to the server was reset while the page was loading message. Looking in the access logs it seems to suggest that it doesn't even hit the website as there is no log entry. A file smaller than the maxinput uploads with no problems and everything is correctly logged. Increasing maxinput allows the file to upload and also is everything is correctly logged (so it's not a browser or network issue). I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it? Can anyone advise me on what to try? Ideally, I'd like to be able to capture the fact that the uploaded file is larger than our limit, and feedback to the user. Even more ideally, I'd like to be able to tell them BEFORE they upload. :-) I've got the following settings in my config.tcl (this is AOLserver 4.5.1 but also happens on 4.0.10 and on Windows version) set max_file_upload_mb10 set max_file_upload_min5 ns_section ns/server/${server}/module/nssock ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param maxpost[expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param recvwait [expr {$max_file_upload_min * 60}] ;# Maximum request time in minutes -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
IIRC, in AOLserver 4.5+, until the whole HTTP request is received by the server, it's being handled entirely by the driver thread -- and if the maxinput ns_limit is exceeded, the driver thread drops the connection before it's ever dispatched to a connection thread (which might be better referred to as a request handler thread). I could be wrong, it's been a few years since I looked at that code ... but that's what I seem to recall. On 6/23/11 8:04 AM, Fenton, Brian wrote: I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it? -- Dossy Shiobara | He realized the fastest way to change do...@panoptic.com | is to laugh at your own folly -- then you http://panoptic.com/ | can let go and quickly move on. (p. 70) * WordPress * jQuery * MySQL * Security * Business Continuity * -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Thanks Jim I should have mentioned that there was nothing in the error log either. So, how would you recommend I capture the fact that the uploaded file is larger than our limit, and feedback to the user? thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 13:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Hi, The short answer is no, there's no access log entry although there may be a server log message buried in the chatter. The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, max request exceeded. -Jim On Jun 23, 2011, at 6:04 AM, Fenton, Brian wrote: Hi When I upload a file larger than the maxinput I get the Firefox browser page with a the connection to the server was reset while the page was loading message. Looking in the access logs it seems to suggest that it doesn't even hit the website as there is no log entry. A file smaller than the maxinput uploads with no problems and everything is correctly logged. Increasing maxinput allows the file to upload and also is everything is correctly logged (so it's not a browser or network issue). I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it? Can anyone advise me on what to try? Ideally, I'd like to be able to capture the fact that the uploaded file is larger than our limit, and feedback to the user. Even more ideally, I'd like to be able to tell them BEFORE they upload. :-) I've got the following settings in my config.tcl (this is AOLserver 4.5.1 but also happens on 4.0.10 and on Windows version) set max_file_upload_mb10 set max_file_upload_min5 ns_section ns/server/${server}/module/nssock ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param maxpost[expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param recvwait [expr {$max_file_upload_min * 60}] ;# Maximum request time in minutes -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Alternatively, instead of the driver thread tossing the connection on the floor, it could mark the HTTP request as being truncated in some way, and passing up to maxinput bytes through to the connection thread. Then, the connection thread can decide how to handle the request -- but, the driver thread still makes sure that the hard limits are enforced on the request that will be passed on to the conn thread ... ? On 6/23/11 8:47 AM, Jim Davidson wrote: The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. -- Dossy Shiobara | He realized the fastest way to change do...@panoptic.com | is to laugh at your own folly -- then you http://panoptic.com/ | can let go and quickly move on. (p. 70) * WordPress * jQuery * MySQL * Security * Business Continuity * -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
I think if its possible in your case use the javascript and calculate the size of the upload file and return it right away if its exceeds the limit this way you dont' need to worry about sending the request to the server, waiting for a while and ended up with nothing returned. Regards, Majid. On Thu, Jun 23, 2011 at 5:16 PM, Fenton, Brian brian.fen...@quest.iewrote: Thanks Jim I should have mentioned that there was nothing in the error log either. So, how would you recommend I capture the fact that the uploaded file is larger than our limit, and feedback to the user? thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 13:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Hi, The short answer is no, there's no access log entry although there may be a server log message buried in the chatter. The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, max request exceeded. -Jim On Jun 23, 2011, at 6:04 AM, Fenton, Brian wrote: Hi When I upload a file larger than the maxinput I get the Firefox browser page with a the connection to the server was reset while the page was loading message. Looking in the access logs it seems to suggest that it doesn't even hit the website as there is no log entry. A file smaller than the maxinput uploads with no problems and everything is correctly logged. Increasing maxinput allows the file to upload and also is everything is correctly logged (so it's not a browser or network issue). I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it? Can anyone advise me on what to try? Ideally, I'd like to be able to capture the fact that the uploaded file is larger than our limit, and feedback to the user. Even more ideally, I'd like to be able to tell them BEFORE they upload. :-) I've got the following settings in my config.tcl (this is AOLserver 4.5.1 but also happens on 4.0.10 and on Windows version) set max_file_upload_mb10 set max_file_upload_min5 ns_section ns/server/${server}/module/nssock ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param maxpost[expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param recvwait [expr {$max_file_upload_min * 60}] ;# Maximum request time in minutes -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Howdy, Appears you need to set driver debug mode for the driver for the given socket module thing: ns_section ns/server/server1/module/nssock ns_param debug 1 Hopefully that's not too much muck in the server log. -Jim On Jun 23, 2011, at 7:16 AM, Fenton, Brian wrote: Thanks Jim I should have mentioned that there was nothing in the error log either. So, how would you recommend I capture the fact that the uploaded file is larger than our limit, and feedback to the user? thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 13:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Hi, The short answer is no, there's no access log entry although there may be a server log message buried in the chatter. The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, max request exceeded. -Jim On Jun 23, 2011, at 6:04 AM, Fenton, Brian wrote: Hi When I upload a file larger than the maxinput I get the Firefox browser page with a the connection to the server was reset while the page was loading message. Looking in the access logs it seems to suggest that it doesn't even hit the website as there is no log entry. A file smaller than the maxinput uploads with no problems and everything is correctly logged. Increasing maxinput allows the file to upload and also is everything is correctly logged (so it's not a browser or network issue). I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it? Can anyone advise me on what to try? Ideally, I'd like to be able to capture the fact that the uploaded file is larger than our limit, and feedback to the user. Even more ideally, I'd like to be able to tell them BEFORE they upload. :-) I've got the following settings in my config.tcl (this is AOLserver 4.5.1 but also happens on 4.0.10 and on Windows version) set max_file_upload_mb10 set max_file_upload_min5 ns_section ns/server/${server}/module/nssock ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param maxpost[expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param recvwait [expr {$max_file_upload_min * 60}] ;# Maximum request time in minutes -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
I think this could work, i.e., dummy up a runt or truncated connection to exercise the rest of the code. But, it's possible something would go uninitialized or assume goodness, i.e., just run off the end of the truncated content. Hmm... I think today, in 2011, some of the flexibility we imagined back in 1995 isn't really so needed. This includes general purpose logging plugins and network drivers when there really is just the common log format, ordinary and SSL sockets. More in the core for the base HTTP processing could make things easier to maintain. -Jim On Jun 23, 2011, at 7:27 AM, Dossy Shiobara wrote: Alternatively, instead of the driver thread tossing the connection on the floor, it could mark the HTTP request as being truncated in some way, and passing up to maxinput bytes through to the connection thread. Then, the connection thread can decide how to handle the request -- but, the driver thread still makes sure that the hard limits are enforced on the request that will be passed on to the conn thread ... ? On 6/23/11 8:47 AM, Jim Davidson wrote: The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. -- Dossy Shiobara | He realized the fastest way to change do...@panoptic.com | is to laugh at your own folly -- then you http://panoptic.com/ | can let go and quickly move on. (p. 70) * WordPress * jQuery * MySQL * Security * Business Continuity * -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Thanks Majid well there are Flash based on front end solutions e.g. SWFUpload, but it's not possible in Javascript. I think it's still important to have a server solution as well. I do agree though that it's kind of crazy to find out AFTER you upload a file, that the file was too big! Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Majid Khan [majidkha...@gmail.com] Sent: 23 June 2011 14:34 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput I think if its possible in your case use the javascript and calculate the size of the upload file and return it right away if its exceeds the limit this way you dont' need to worry about sending the request to the server, waiting for a while and ended up with nothing returned. Regards, Majid. On Thu, Jun 23, 2011 at 5:16 PM, Fenton, Brian brian.fen...@quest.iemailto:brian.fen...@quest.ie wrote: Thanks Jim I should have mentioned that there was nothing in the error log either. So, how would you recommend I capture the fact that the uploaded file is larger than our limit, and feedback to the user? thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COMmailto:AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.commailto:jgdavid...@mac.com] Sent: 23 June 2011 13:47 To: AOLSERVER@LISTSERV.AOL.COMmailto:AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Hi, The short answer is no, there's no access log entry although there may be a server log message buried in the chatter. The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, max request exceeded. -Jim On Jun 23, 2011, at 6:04 AM, Fenton, Brian wrote: Hi When I upload a file larger than the maxinput I get the Firefox browser page with a the connection to the server was reset while the page was loading message. Looking in the access logs it seems to suggest that it doesn't even hit the website as there is no log entry. A file smaller than the maxinput uploads with no problems and everything is correctly logged. Increasing maxinput allows the file to upload and also is everything is correctly logged (so it's not a browser or network issue). I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it? Can anyone advise me on what to try? Ideally, I'd like to be able to capture the fact that the uploaded file is larger than our limit, and feedback to the user. Even more ideally, I'd like to be able to tell them BEFORE they upload. :-) I've got the following settings in my config.tcl (this is AOLserver 4.5.1 but also happens on 4.0.10 and on Windows version) set max_file_upload_mb10 set max_file_upload_min5 ns_section ns/server/${server}/module/nssock ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param maxpost[expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param recvwait [expr {$max_file_upload_min * 60}] ;# Maximum request time in minutes -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.commailto:lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.commailto:lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.commailto:lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Thanks Jim yes that now logs the fact that the file was too big (multiple times in fact), but how can I access this fact in TCL? [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[38]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[39]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[40]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[41]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[42]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[43]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[44]: max content exceeded Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 14:44 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Howdy, Appears you need to set driver debug mode for the driver for the given socket module thing: ns_section ns/server/server1/module/nssock ns_param debug 1 Hopefully that's not too much muck in the server log. -Jim On Jun 23, 2011, at 7:16 AM, Fenton, Brian wrote: Thanks Jim I should have mentioned that there was nothing in the error log either. So, how would you recommend I capture the fact that the uploaded file is larger than our limit, and feedback to the user? thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 13:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Hi, The short answer is no, there's no access log entry although there may be a server log message buried in the chatter. The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, max request exceeded. -Jim On Jun 23, 2011, at 6:04 AM, Fenton, Brian wrote: Hi When I upload a file larger than the maxinput I get the Firefox browser page with a the connection to the server was reset while the page was loading message. Looking in the access logs it seems to suggest that it doesn't even hit the website as there is no log entry. A file smaller than the maxinput uploads with no problems and everything is correctly logged. Increasing maxinput allows the file to upload and also is everything is correctly logged (so it's not a browser or network issue). I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it? Can anyone advise me on what to try? Ideally, I'd like to be able to capture the fact that the uploaded file is larger than our limit, and feedback to the user. Even more ideally, I'd like to be able to tell them BEFORE they upload. :-) I've got the following settings in my config.tcl (this is AOLserver 4.5.1 but also happens on 4.0.10 and on Windows version) set max_file_upload_mb10 set max_file_upload_min5 ns_section ns/server/${server}/module/nssock ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param maxpost[expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param recvwait [expr {$max_file_upload_min * 60}] ;# Maximum request time in minutes -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http
Re: [AOLSERVER] Problem with file uploads larger than maxinput
I think the short answer is there is no way. Checking the code and your error message, this is error condition E_CRANGE. It's returned from SockReadLine which is called repeatedly to read the request line (GET /url/ ...) and headers. As it reads lines, it parses them for some special conditions. If it sees a content-length header, it's parsed and verified the content to be sent isn't out of range. If it is, it returns E_CRANGE and the connection is aborted. So, there's no Tcl environment available to fiddle with. There is some Tcl framework in there for que wait callbacks but it's not exposed for error conditions. Good idea. Question: What are you trying to do with Tcl? Maybe you could hack the driver.c code to do something smart at CRANGE alone? I think it's the most common limit-related error. -Jim On Jun 23, 2011, at 8:22 AM, Fenton, Brian wrote: Thanks Jim yes that now logs the fact that the file was too big (multiple times in fact), but how can I access this fact in TCL? [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[38]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[39]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[40]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[41]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[42]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[43]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[44]: max content exceeded Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 14:44 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Howdy, Appears you need to set driver debug mode for the driver for the given socket module thing: ns_section ns/server/server1/module/nssock ns_param debug 1 Hopefully that's not too much muck in the server log. -Jim On Jun 23, 2011, at 7:16 AM, Fenton, Brian wrote: Thanks Jim I should have mentioned that there was nothing in the error log either. So, how would you recommend I capture the fact that the uploaded file is larger than our limit, and feedback to the user? thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 13:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Hi, The short answer is no, there's no access log entry although there may be a server log message buried in the chatter. The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, max request exceeded. -Jim On Jun 23, 2011, at 6:04 AM, Fenton, Brian wrote: Hi When I upload a file larger than the maxinput I get the Firefox browser page with a the connection to the server was reset while the page was loading message. Looking in the access logs it seems to suggest that it doesn't even hit the website as there is no log entry. A file smaller than the maxinput uploads with no problems and everything is correctly logged. Increasing maxinput allows the file to upload and also is everything is correctly logged (so it's not a browser or network issue). I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it? Can anyone advise me on what to try? Ideally, I'd like to be able to capture the fact that the uploaded file is larger than our limit, and feedback to the user. Even more ideally, I'd like to be able to tell them BEFORE they upload. :-) I've got the following settings in my config.tcl (this is AOLserver 4.5.1 but also happens on 4.0.10 and on Windows version) set max_file_upload_mb10 set max_file_upload_min5 ns_section ns/server/${server}/module/nssock ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes ns_param maxpost[expr
Re: [AOLSERVER] Problem with file uploads larger than maxinput
I'm using OpenACS hence the TCL. I just want to let the user know that their file is over the size limit. Could we, in driver.c instead of closing the connection socket, return a custom template (like Apache does) ? Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 15:59 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput I think the short answer is there is no way. Checking the code and your error message, this is error condition E_CRANGE. It's returned from SockReadLine which is called repeatedly to read the request line (GET /url/ ...) and headers. As it reads lines, it parses them for some special conditions. If it sees a content-length header, it's parsed and verified the content to be sent isn't out of range. If it is, it returns E_CRANGE and the connection is aborted. So, there's no Tcl environment available to fiddle with. There is some Tcl framework in there for que wait callbacks but it's not exposed for error conditions. Good idea. Question: What are you trying to do with Tcl? Maybe you could hack the driver.c code to do something smart at CRANGE alone? I think it's the most common limit-related error. -Jim On Jun 23, 2011, at 8:22 AM, Fenton, Brian wrote: Thanks Jim yes that now logs the fact that the file was too big (multiple times in fact), but how can I access this fact in TCL? [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[38]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[39]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[40]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[41]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[42]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[43]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[44]: max content exceeded Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 14:44 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Howdy, Appears you need to set driver debug mode for the driver for the given socket module thing: ns_section ns/server/server1/module/nssock ns_param debug 1 Hopefully that's not too much muck in the server log. -Jim On Jun 23, 2011, at 7:16 AM, Fenton, Brian wrote: Thanks Jim I should have mentioned that there was nothing in the error log either. So, how would you recommend I capture the fact that the uploaded file is larger than our limit, and feedback to the user? thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 13:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Hi, The short answer is no, there's no access log entry although there may be a server log message buried in the chatter. The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, max request exceeded. -Jim On Jun 23, 2011, at 6:04 AM, Fenton, Brian wrote: Hi When I upload a file larger than the maxinput I get the Firefox browser page with a the connection to the server was reset while the page was loading message. Looking in the access logs it seems to suggest that it doesn't even hit the website as there is no log entry. A file smaller than the maxinput uploads with no problems and everything is correctly logged. Increasing maxinput allows the file to upload and also is everything is correctly logged (so it's not a browser or network issue). I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it? Can anyone advise me on what to try? Ideally, I'd like to be able to capture the fact that the uploaded file is larger than our limit, and feedback to the user. Even more
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Yes, on E_CRANGE, the driver thread should return HTTP 413 Request too large, IMHO. And, we should be able to configure a custom response for status code 413 ... Patches welcome ;) On 6/23/11 11:28 AM, Fenton, Brian wrote: I'm using OpenACS hence the TCL. I just want to let the user know that their file is over the size limit. Could we, in driver.c instead of closing the connection socket, return a custom template (like Apache does) ? -- Dossy Shiobara | He realized the fastest way to change do...@panoptic.com | is to laugh at your own folly -- then you http://panoptic.com/ | can let go and quickly move on. (p. 70) * WordPress * jQuery * MySQL * Security * Business Continuity * -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Agreed. Should go on the bug/feature list. Is there a list ? :) Sent from a phone On Jun 23, 2011, at 9:58 AM, Dossy Shiobara do...@panoptic.com wrote: Yes, on E_CRANGE, the driver thread should return HTTP 413 Request too large, IMHO. And, we should be able to configure a custom response for status code 413 ... Patches welcome ;) On 6/23/11 11:28 AM, Fenton, Brian wrote: I'm using OpenACS hence the TCL. I just want to let the user know that their file is over the size limit. Could we, in driver.c instead of closing the connection socket, return a custom template (like Apache does) ? -- Dossy Shiobara | He realized the fastest way to change do...@panoptic.com | is to laugh at your own folly -- then you http://panoptic.com/ | can let go and quickly move on. (p. 70) * WordPress * jQuery * MySQL * Security * Business Continuity * -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
Hi, The final goal is to have a server-side validation and return a page template explaining what the problem is rather than just closing the connection. Having client-side validation ( JavaScript/Flash like gmail ) helps in terms of UI but I presume wouldn't be safe enough agains DoS. IMHO, I agree with Dossy, to use the driver thread to check the hard limits and instead of dropping the connection, just mark the HTTP request and let the request handler to return the 413. I also think the template could be configured in the 'ns_section ns/server/${server}/redirects' , isn't it ? If you all agree with this, Brian and I can help to get a patch. Thanks for the great feedback!, Enrique. On Thu, Jun 23, 2011 at 4:28 PM, Fenton, Brian brian.fen...@quest.ie wrote: I'm using OpenACS hence the TCL. I just want to let the user know that their file is over the size limit. Could we, in driver.c instead of closing the connection socket, return a custom template (like Apache does) ? Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 15:59 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput I think the short answer is there is no way. Checking the code and your error message, this is error condition E_CRANGE. It's returned from SockReadLine which is called repeatedly to read the request line (GET /url/ ...) and headers. As it reads lines, it parses them for some special conditions. If it sees a content-length header, it's parsed and verified the content to be sent isn't out of range. If it is, it returns E_CRANGE and the connection is aborted. So, there's no Tcl environment available to fiddle with. There is some Tcl framework in there for que wait callbacks but it's not exposed for error conditions. Good idea. Question: What are you trying to do with Tcl? Maybe you could hack the driver.c code to do something smart at CRANGE alone? I think it's the most common limit-related error. -Jim On Jun 23, 2011, at 8:22 AM, Fenton, Brian wrote: Thanks Jim yes that now logs the fact that the file was too big (multiple times in fact), but how can I access this fact in TCL? [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[38]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[39]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[40]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[41]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[42]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[43]: max content exceeded [23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[44]: max content exceeded Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 14:44 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Howdy, Appears you need to set driver debug mode for the driver for the given socket module thing: ns_section ns/server/server1/module/nssock ns_param debug 1 Hopefully that's not too much muck in the server log. -Jim On Jun 23, 2011, at 7:16 AM, Fenton, Brian wrote: Thanks Jim I should have mentioned that there was nothing in the error log either. So, how would you recommend I capture the fact that the uploaded file is larger than our limit, and feedback to the user? thanks Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdavid...@mac.com] Sent: 23 June 2011 13:47 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput Hi, The short answer is no, there's no access log entry although there may be a server log message buried in the chatter. The reason is the access log is a trace that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions. You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, max request exceeded. -Jim On Jun 23, 2011, at 6:04 AM, Fenton, Brian wrote: Hi When I upload a file larger than the maxinput I get the Firefox browser page with a the connection to the server was reset while the page was loading message. Looking in the access logs it seems
Re: [AOLSERVER] Problem with file uploads larger than maxinput
In theory, it could go on SourceForge's issue tracker ... or opened as a issue on GitHub ... or ... On 6/23/11 12:13 PM, Jim Davidson wrote: Agreed. Should go on the bug/feature list. Is there a list ? :) Sent from a phone On Jun 23, 2011, at 9:58 AM, Dossy Shiobarado...@panoptic.com wrote: Yes, on E_CRANGE, the driver thread should return HTTP 413 Request too large, IMHO. And, we should be able to configure a custom response for status code 413 ... Patches welcome ;) -- Dossy Shiobara | He realized the fastest way to change do...@panoptic.com | is to laugh at your own folly -- then you http://panoptic.com/ | can let go and quickly move on. (p. 70) * WordPress * jQuery * MySQL * Security * Business Continuity * -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
That was what I was thinking -- driver marks the request as exceeding the limit, and setting the response status code to 413. The benefits that I see (if implemented the way I'm imagining) -- 1) access logging of requests w/ 413 status code 2) custom response page via ns/server/${server}/redirects. If you and Brian could work up a patch, that'd be wonderful! Does anyone see any problems with this approach? Any reason not to do it? I don't suppose it can possibly break any kind of backward compatibility, as no code right now can even be written to handle such a scenario, anyway. Of course, once we decide on a behavior, and folks code against the implementation ... changing/fixing it becomes more expensive for everyone, so if there's any kind of worry about how this is going to work, lets iron out the details now ;) On 6/23/11 12:29 PM, Enrique Catalan wrote: IMHO, I agree with Dossy, to use the driver thread to check the hard limits and instead of dropping the connection, just mark the HTTP request and let the request handler to return the 413. I also think the template could be configured in the 'ns_section ns/server/${server}/redirects' , isn't it ? If you all agree with this, Brian and I can help to get a patch. -- Dossy Shiobara | He realized the fastest way to change do...@panoptic.com | is to laugh at your own folly -- then you http://panoptic.com/ | can let go and quickly move on. (p. 70) * WordPress * jQuery * MySQL * Security * Business Continuity * -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem with file uploads larger than maxinput
I think this one case can be handled this way because you could truncate the request (some content may have been read ahead) and then flag it for special handling in the normal connection processing code/threads. The reason is the server has a full request, you're just choosing to not read all the content, so if you do a little touch up of the conn structure it can flow through the connection threads without tripping up. I'm cautious here because I've spent a lot of time in the past debugging half-baked conn problems. So, this patch makes sense for the E_CRANGE error alone, the most annoying problem. Later we could kick around the idea of getting the code in the driver to handle much more detailed logging, dumping the access log module, etc. -Jim On Jun 23, 2011, at 11:14 AM, Dossy Shiobara wrote: That was what I was thinking -- driver marks the request as exceeding the limit, and setting the response status code to 413. The benefits that I see (if implemented the way I'm imagining) -- 1) access logging of requests w/ 413 status code 2) custom response page via ns/server/${server}/redirects. If you and Brian could work up a patch, that'd be wonderful! Does anyone see any problems with this approach? Any reason not to do it? I don't suppose it can possibly break any kind of backward compatibility, as no code right now can even be written to handle such a scenario, anyway. Of course, once we decide on a behavior, and folks code against the implementation ... changing/fixing it becomes more expensive for everyone, so if there's any kind of worry about how this is going to work, lets iron out the details now ;) On 6/23/11 12:29 PM, Enrique Catalan wrote: IMHO, I agree with Dossy, to use the driver thread to check the hard limits and instead of dropping the connection, just mark the HTTP request and let the request handler to return the 413. I also think the template could be configured in the 'ns_section ns/server/${server}/redirects' , isn't it ? If you all agree with this, Brian and I can help to get a patch. -- Dossy Shiobara | He realized the fastest way to change do...@panoptic.com | is to laugh at your own folly -- then you http://panoptic.com/ | can let go and quickly move on. (p. 70) * WordPress * jQuery * MySQL * Security * Business Continuity * -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.