Re: [PHP] How to disable PHP's POST caching?
On Tue, May 23, 2006 5:23 pm, Adam Zey wrote:
why is port 80 a requirement - HTTP can technically over any port.
It must be accessible to any client, no matter what sort of firewall
or
proxy they go through. The only way to absolutely assure that is, as
far
as I know, to use port 80. It is the only port that you can count on
with a fair degree of certainty, if the user is proxied or firewalled.
I'd just as soon write my own simple webserver, but then it'd run into
conflicts with existing webservers on port 80.
My current solution is to buffer data on the client-side, and send a
fresh POST request every so many milliseconds (Say, 250) over a
keepalive connection with the latest buffered data. The downside of
this
is that it introduces up to 250ms of latency on top of the existing
network latency, and it produces a lot of POST requests. I would
really
like to eliminate that by streaming the data rather than splitting it
up
like that.
You might maybe want to try running PHP as a CGI instead of Module,
just to see if the dirt-simple fopen('php://stdin','r') will just
work...
It's also remotely possible that you could check if there's a way to
implement custom methods other than GET/POST in PHP, which might
open up the possibility of getting partial data.
It's even remotely possible that PUT is not buffered, though I doubt it.
Ultimately, though, I think you've been painted into a corner from
which there is no real escape...
Might be time to re-negotiated.
You MIGHT be able to run a Proxy on the server side, so that your HTTP
port 80 for this particular URL gets forwarded to some other port/IP
behind the scenes -- and then PHP sockets may be viable.
This would meet most of the original requirements, and I think it
would work...
--
Like Music?
http://l-i-e.com/artists.htm
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Why not reconfigure the webserver to proxy a certain url subdirectory to your php script that can be running on any old port? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Adam Zey wrote: Tunelling arbitrary TCP packets. Similar idea to SSH port forwarding, except tunneling over HTTP instead of SSH. A good example might be encapsulating an IRC (or telnet, or pop3, or ssh, etc) connection inside of an HTTP connection such that incomming IRC traffic goes over a GET to the client, and outgoing IRC traffic goes over a POST request. So, the traffic is bounced: [mIRC] --- [client.php] -internet- [apache --- server.php] -internet- [irc server] And the same in reverse. The connection between client.php and server.php is taking the IRC traffic and encapsulating it inside an HTTP connection, where it is unpacked by server.php before being sent on to the final destination. The idea is to get TCP tunneling working, once you do that you can rely on other programs to use that TCP tunnel for more complex things, like SOCKS. You're trying to get a square peg through a round hole. The HTTP protocol was not designed to do anything like this, so the standard implementation by most web servers and PHP does not allow what you are trying to do. I'm curious about your 'lots of POSTs' solution. How are you keeping the connection open on the server-side? It's certainly not possible to maintain that connection between requests without using a process outside the web server that maintains the connections. I've implemented a system in the past to proxy IRC, MSN and AIM connections in this way, but it only worked because the requests that came into PHP got passed to this other process which held all the connections and managed the traffic. And yes, it did generate a huge amount of traffic even when it wasn't doing anything due to the need to poll the server for new incoming messages. This demonstrates a point at which you need to reconsider whether a shared hosting environment (which I assume you're using given the restrictions you've mentioned) is enough for your purposes. If you had a dedicated server you could add another IP and run a custom server on it that would be capable of doing exactly what you want. In fact there are lots of nice free proxies that will happily sit on port 80. However, it's worth nothing that a lot of firewalls block traffic that doesn't look like HTTP, in which case you'll need to use SSL on port 443 to get past those checks. Anyways, long story (sorry) short, your square peg won't go in the round hole without serious modification. Hope that helps. -Stut -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Stut wrote: Adam Zey wrote: Tunelling arbitrary TCP packets. Similar idea to SSH port forwarding, except tunneling over HTTP instead of SSH. A good example might be encapsulating an IRC (or telnet, or pop3, or ssh, etc) connection inside of an HTTP connection such that incomming IRC traffic goes over a GET to the client, and outgoing IRC traffic goes over a POST request. So, the traffic is bounced: [mIRC] --- [client.php] -internet- [apache --- server.php] -internet- [irc server] And the same in reverse. The connection between client.php and server.php is taking the IRC traffic and encapsulating it inside an HTTP connection, where it is unpacked by server.php before being sent on to the final destination. The idea is to get TCP tunneling working, once you do that you can rely on other programs to use that TCP tunnel for more complex things, like SOCKS. You're trying to get a square peg through a round hole. The HTTP protocol was not designed to do anything like this, so the standard implementation by most web servers and PHP does not allow what you are trying to do. That's the fun of it, making things like PHP and HTTP do things they weren't supposed to. I'm curious about your 'lots of POSTs' solution. How are you keeping the connection open on the server-side? It's certainly not possible to maintain that connection between requests without using a process outside the web server that maintains the connections. I've implemented a system in the past to proxy IRC, MSN and AIM connections in this way, but it only worked because the requests that came into PHP got passed to this other process which held all the connections and managed the traffic. And yes, it did generate a huge amount of traffic even when it wasn't doing anything due to the need to poll the server for new incoming messages. With the lots-of-posts, the connection is a regular keepalive, which any webserver happily keeps open. When this keepalive connection closes, you open a new one. At least this way, while I still need to send lots of posts (Say, one every 100ms, or 250ms, something like that), I can limit the new connections to once every minute or two. While 4 messages per second may seem like a lot, I would imagine that an application such as Google Maps would generate a LOT more than that while a user is scrolling around; google maps would have to load in dozens of images per second as the user scrolled. Polling for incomming messages isn't a problem, as there is no incomming data for the POSTs. A seperate GET request handles incomming data, and I can simply do something like select, or even something as mundane as polling the socket myself. But I don't need to poll the server. And, the 4-per-second POST transactions don't need to be sent unless there is actually data to be sent. As long as a keepalive request is sent to make sure the remote server doesn't sever connection (My tests show apache 2 with a 15 second timeout on a keepalive connection), there doesn't need to be any POSTs unless there is data waiting to be sent. Of course, this solution has high latency (up to 250ms delay), and generates a fair number of POST requests, so it still isn't ideal. But it should work, since it doesn't do anything out-of-spec as far as HTTP is concerned. This demonstrates a point at which you need to reconsider whether a shared hosting environment (which I assume you're using given the restrictions you've mentioned) is enough for your purposes. If you had a dedicated server you could add another IP and run a custom server on it that would be capable of doing exactly what you want. In fact there are lots of nice free proxies that will happily sit on port 80. However, it's worth nothing that a lot of firewalls block traffic that doesn't look like HTTP, in which case you'll need to use SSL on port 443 to get past those checks. I wasn't targetting shared hosting environments. I imagine most of them use safe mode anyhow. I was thinking more along the lines of somebody with a dedicated server, or perhaps just a linux box in their closet. The thing is, I'm not writing a web proxy. I'm writing a tunneling solution. And, the idea is that firewalls won't block the traffic, because it doesn't just look like HTTP traffic, it really IS HTTP traffic. Is a firewall really going to block a download because the data being downloaded doesn't look legitimate? As far as the firewall is concerned, it just sees regular HTTP traffic. And of course, a bit of obuscation of the data being sent wouldn't be too hard. The idea here is that no matter what sort of proxy or firewall the user is behind, they will be able to get a TCP/IP connection for any protocol out to the outside world. Even if the user is sitting on a LAN with no gateway, no connection to the internet except a single proxy server, they should still be able to make a TCP/IP connection by tunneling it
Re: [PHP] How to disable PHP's POST caching?
On Tue, May 23, 2006 at 06:37:27PM -0400, Adam Zey wrote: The data going from client-server needs to be sent over an HTTP connection, which seems to limit me to PUT and POST requests, since they're the only ones that allow significant quantities of data to be sent by the client. Ideally, there should be no delay between the client wanting to send data and the data being sent over the connection; it should be as simple as wrapping the data and sending. So, I need some way to send data to a PHP script that lives on a webserver without any buffering going on. My backup approach, as I described in another mail, involves client-side buffering and multiple POST requests. But that induces quite a bit of latency, which is quite undesirable. How much data are you sending? A POST shouldn't cause that much delay unless your talking about a lot of POST data Curt. -- cat .signature: No such file or directory -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Curt Zirzow wrote: On Tue, May 23, 2006 at 06:37:27PM -0400, Adam Zey wrote: The data going from client-server needs to be sent over an HTTP connection, which seems to limit me to PUT and POST requests, since they're the only ones that allow significant quantities of data to be sent by the client. Ideally, there should be no delay between the client wanting to send data and the data being sent over the connection; it should be as simple as wrapping the data and sending. So, I need some way to send data to a PHP script that lives on a webserver without any buffering going on. My backup approach, as I described in another mail, involves client-side buffering and multiple POST requests. But that induces quite a bit of latency, which is quite undesirable. How much data are you sending? A POST shouldn't cause that much delay unless your talking about a lot of POST data Curt. Please see my more recent messages on the subject for the reasoning behind this. It's interactive data being sent that may require an immediate response. If a user is tunneling a telnet session, they expect a response within a matter of milliseconds, not seconds. POST holds onto the data until the client is done uploading, which with a persistant POST request never happens, which is why I spoke of multiple POST requests above and in more recent messages. Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
On Tue, May 23, 2006 at 03:51:51PM -0400, Adam Zey wrote: PHP seems to cache POST data, and waits for the entire POST to finish sending before it makes it available to php://input. I'd like to be able to read the post data from php://input while the client is still uploading it. How can I cause PHP to make the POST data available right away instead of after the client finishes sending? One thing you can do is limit the size of the post_max_size and upload_max_filesize (if using an upload) to someting like 1. php will issue a warning and not read the POST data. You then can read the contents of what was sent via php://input or php://stdin, i forget wich one it is. Of course you will need to read the data and parse the raw data as it was sent from the browser. HTH, Curt. -- cat .signature: No such file or directory -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
On Wed, May 24, 2006 at 05:44:56PM -0400, Adam Zey wrote: Curt Zirzow wrote: On Tue, May 23, 2006 at 06:37:27PM -0400, Adam Zey wrote: The data going from client-server needs to be sent over an HTTP connection, which seems to limit me to PUT and POST requests, since they're the only ones that allow significant quantities of data to be sent by the client. Ideally, there should be no delay between the client wanting to send data and the data being sent over the connection; it should be as simple as wrapping the data and sending. So, I need some way to send data to a PHP script that lives on a webserver without any buffering going on. My backup approach, as I described in another mail, involves client-side buffering and multiple POST requests. But that induces quite a bit of latency, which is quite undesirable. How much data are you sending? A POST shouldn't cause that much delay unless your talking about a lot of POST data Curt. Please see my more recent messages on the subject for the reasoning behind this. It's interactive data being sent that may require an I didn't ask why... I was asking how much. immediate response. If a user is tunneling a telnet session, they expect a response within a matter of milliseconds, not seconds. POST holds onto the data until the client is done uploading, which with a persistant POST request never happens, which is why I spoke of multiple POST requests above and in more recent messages. The only way you will be able to read data as it comes in is by removing php's reading and parsing of data from the HTTP posted data. I posted how to do this on a different thread. Curt. -- cat .signature: No such file or directory -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Adam Zey wrote: PHP seems to cache POST data, and waits for the entire POST to finish sending before it makes it available to php://input. I'd like to be able to read the post data from php://input while the client is still uploading it. How can I cause PHP to make the POST data available right away instead of after the client finishes sending? Regards, Adam Zey. PS: As far as I can tell, PHP caches the entire POST in memory as it is being sent, but just doesn't make it available to php://input until after the client is done. Since PHP already has it in memory, why isn't it accessible? are you sure it's php that is holding it in memory? $_POST is fully filled on the first line of the script which gives me the impression that the webserver is caching the response until it all been uploaded before even starting up php. or am I missing something here? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Jochem Maas wrote: Adam Zey wrote: PHP seems to cache POST data, and waits for the entire POST to finish sending before it makes it available to php://input. I'd like to be able to read the post data from php://input while the client is still uploading it. How can I cause PHP to make the POST data available right away instead of after the client finishes sending? Regards, Adam Zey. PS: As far as I can tell, PHP caches the entire POST in memory as it is being sent, but just doesn't make it available to php://input until after the client is done. Since PHP already has it in memory, why isn't it accessible? are you sure it's php that is holding it in memory? $_POST is fully filled on the first line of the script which gives me the impression that the webserver is caching the response until it all been uploaded before even starting up php. or am I missing something here? You're correct, of course. I was a bit confused, it seems. PUT behaves as I described above. If you do a PUT request for the PHP script directly, the PHP script is called right away, and any attempt to fread from php://input blocks until either the client is done sending, or some sort of buffer is filled (the fread occasionally returns with lots of data if you keep sending). So, might I amend my question to, is it possible to disable PHP buffering PUTs, or Apache buffering POST? (Or is it PHP that just buffers the POST before actually executing the script, but post Apache?) Essentially what I want is a persistant HTTP connection over which I can stream data and have the server-side PHP script process the data as it arrives, rather than when all the data is sent. The only other approach I can figure out is to send periodic POST requests with the latest data, the downside of which is a huge increase in latency between data production and consumption. Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
On Tue, May 23, 2006 4:39 pm, Adam Zey wrote: The only other approach I can figure out is to send periodic POST requests with the latest data, the downside of which is a huge increase in latency between data production and consumption. Sounds like you maybe want to run your own server... http://php.net/sockets -- Like Music? http://l-i-e.com/artists.htm -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Richard Lynch wrote: On Tue, May 23, 2006 4:39 pm, Adam Zey wrote: The only other approach I can figure out is to send periodic POST requests with the latest data, the downside of which is a huge increase in latency between data production and consumption. Sounds like you maybe want to run your own server... http://php.net/sockets Unfortunately, the requirement is that the script be able to accept data on port 80, and co-exist with an existing webserver at the same time. As far as I can tell, this requirement can only be fulfilled by running the script through a webserver. Also, sockets are not compiled into PHP by default, and stream sockets aren't included in versions below PHP 5.0.0, as far as I can tell. That sort of puts a hamper on socket usage. Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Adam Zey wrote: Jochem Maas wrote: ... Essentially what I want is a persistant HTTP connection over which I can stream data and have the server-side PHP script process the data as it arrives, rather than when all the data is sent. The only other approach I can figure out is to send periodic POST requests with the latest data, the downside of which is a huge increase in latency between data production and consumption. Richard's suggestion is most likely the best option (assuming you want to use php) otherwise you'll probably end up hacking webserver and/or php sources (painful, time consuming and a probable maintainance nightmare) ... which also comes with the risk of breaking lots http protocols 'rules' while your at it. Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Jochem Maas wrote:. ... Richard's suggestion is most likely the best option (assuming you want to use php) otherwise you'll probably end up hacking webserver and/or php sources (painful, time consuming and a probable maintainance nightmare) ... which also comes with the risk of breaking lots http protocols 'rules' while your at it. Regards, Adam Zey. As I mentioned in my more recent mail, this unfortunately isn't an option since I need to run on port 80 without disturbing the existing webserver, which requirse that the script be running through the webserver :( I've considered the possibility of looking into Perl or C (via CGI) to try to get the desired functionality, but they have their own issues (namely more complicated installation than just sticking a script anywhere in the web tree as you can with a default PHP installation). Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Adam Zey wrote: Jochem Maas wrote:. ... Richard's suggestion is most likely the best option (assuming you want to use php) otherwise you'll probably end up hacking webserver and/or php sources (painful, time consuming and a probable maintainance nightmare) ... which also comes with the risk of breaking lots http protocols 'rules' while your at it. Regards, Adam Zey. As I mentioned in my more recent mail, this unfortunately isn't an option since I need to run on port 80 without disturbing the existing why is port 80 a requirement - HTTP can technically over any port. webserver, which requirse that the script be running through the webserver :( I've considered the possibility of looking into Perl or C (via CGI) to try to get the desired functionality, but they have their own issues (namely more complicated installation than just sticking a script anywhere in the web tree as you can with a default PHP installation). sounds to me your requirement rules out the use of bog standard setups, that said I doubt using CGI (with any language you choose) will fix the problem because your still at thew mercy of the webserver that is running the CGI. Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Jochem Maas wrote: why is port 80 a requirement - HTTP can technically over any port. It must be accessible to any client, no matter what sort of firewall or proxy they go through. The only way to absolutely assure that is, as far as I know, to use port 80. It is the only port that you can count on with a fair degree of certainty, if the user is proxied or firewalled. I'd just as soon write my own simple webserver, but then it'd run into conflicts with existing webservers on port 80. My current solution is to buffer data on the client-side, and send a fresh POST request every so many milliseconds (Say, 250) over a keepalive connection with the latest buffered data. The downside of this is that it introduces up to 250ms of latency on top of the existing network latency, and it produces a lot of POST requests. I would really like to eliminate that by streaming the data rather than splitting it up like that. Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] How to disable PHP's POST caching?
[snip] As I mentioned in my more recent mail, this unfortunately isn't an option since I need to run on port 80 without disturbing the existing webserver, which requirse that the script be running through the webserver :( [/snip] I have been reading this thread with much interest and think that perhaps a different approach may be needed. What, exactly, are you trying to accomplish? Skip past the persistency, etc. and describe the problem for which you are seeking a solution. I have the feeling that there may be a way to do what you want with PHP if you will describe the process. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Jay Blanchard wrote: [snip] As I mentioned in my more recent mail, this unfortunately isn't an option since I need to run on port 80 without disturbing the existing webserver, which requirse that the script be running through the webserver :( [/snip] I have been reading this thread with much interest and think that perhaps a different approach may be needed. What, exactly, are you trying to accomplish? Skip past the persistency, etc. and describe the problem for which you are seeking a solution. I have the feeling that there may be a way to do what you want with PHP if you will describe the process. Essentially, I'm looking to write something in the same vein as GNU httptunnel, but in PHP, and running on port 80 serverside. The server-client part is easy, since a never-ending GET request can stream the data and be consumed by the client instantly. The thing I'm having trouble with is the other direction. Getting data from the client to the server. The data going from client-server needs to be sent over an HTTP connection, which seems to limit me to PUT and POST requests, since they're the only ones that allow significant quantities of data to be sent by the client. Ideally, there should be no delay between the client wanting to send data and the data being sent over the connection; it should be as simple as wrapping the data and sending. So, I need some way to send data to a PHP script that lives on a webserver without any buffering going on. My backup approach, as I described in another mail, involves client-side buffering and multiple POST requests. But that induces quite a bit of latency, which is quite undesirable. Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] How to disable PHP's POST caching?
[snip] Essentially, I'm looking to write something in the same vein as GNU httptunnel, but in PHP, and running on port 80 serverside. [/snip] All of that was nice, but still does not explain what you are trying to accomplish other than maintaining a connection state between client and server. What kind of process are you running that would require this? That is what I am looking for. What is the real issue? What would you do that would require a stated connection? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Jay Blanchard wrote: [snip] Essentially, I'm looking to write something in the same vein as GNU httptunnel, but in PHP, and running on port 80 serverside. [/snip] All of that was nice, but still does not explain what you are trying to accomplish other than maintaining a connection state between client and server. What kind of process are you running that would require this? That is what I am looking for. What is the real issue? What would you do that would require a stated connection? Tunelling arbitrary TCP packets. Similar idea to SSH port forwarding, except tunneling over HTTP instead of SSH. A good example might be encapsulating an IRC (or telnet, or pop3, or ssh, etc) connection inside of an HTTP connection such that incomming IRC traffic goes over a GET to the client, and outgoing IRC traffic goes over a POST request. So, the traffic is bounced: [mIRC] --- [client.php] -internet- [apache --- server.php] -internet- [irc server] And the same in reverse. The connection between client.php and server.php is taking the IRC traffic and encapsulating it inside an HTTP connection, where it is unpacked by server.php before being sent on to the final destination. The idea is to get TCP tunneling working, once you do that you can rely on other programs to use that TCP tunnel for more complex things, like SOCKS. Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
Mindaugas L wrote: I'm still new in php:) what about using cookies? nobody mentioned anything? store info in client cookie, and read it from server the same time? :)) On 5/24/06, *Adam Zey* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: *snip* Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- Mindaugas Cookie data is sent to the server as an HTTP header. That sort of puts the kibosh on that. Regards, Adam Zey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to disable PHP's POST caching?
jekillen wrote: On May 23, 2006, at 3:37 PM, Adam Zey wrote: Essentially, I'm looking to write something in the same vein as GNU httptunnel, but in PHP, and running on port 80 serverside. The server-client part is easy, since a never-ending GET request can stream the data and be consumed by the client instantly. The thing I'm having trouble with is the other direction. Getting data from the client to the server. Allow me to interject a suggestion/question. As far as I understand it AJAX or asyincronous connections sound like what youmr afterno(?) JK AJAX implies javascript, which means a browser. My situation doesn't involve a browser. Unless I'm mistaken, AJAX makes many GET requests to send data, which would have the same problem as sending many POST requests, except you can send less data. Regards, Adam ey. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
