Re: [PHP] php File upload
Jim Lucas wrote: What Apache starts, it reads the PHP memory limits in to the running Apache process. When you try and upload a file, it is a straight HTTP upload. PHP plays no part in the actual upload, except for the upload limits set in place by the php settings found in the php.ini or other Apache config files. Correct. Now, Apache actually handles the Upload. Once the file has been received completely, Apache then passes the file and process running to PHP. Also correct. I have never seen a case where Apache has stored the file on the file system. In my past experience it has always held the file in memory, therefor limited to the max physical memory that was install, minus a little for other things. Well, I can easily show you such a case - I can upload a 1Gb file without apache memory usage changing one bit. Even if Apache did upload into memory, why would that make the file limited to the max amount of physical memory?? How about if I upload a 1Gb file to a webserver on a machine that only has 256Mb memory - would you say that's impossible? /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
I think the sentiment is that you can't fit all of the file in the memory at once Luke Slater On 8 Aug 2008, at 07:59, Per Jessen [EMAIL PROTECTED] wrote: Jim Lucas wrote: What Apache starts, it reads the PHP memory limits in to the running Apache process. When you try and upload a file, it is a straight HTTP upload. PHP plays no part in the actual upload, except for the upload limits set in place by the php settings found in the php.ini or other Apache config files. Correct. Now, Apache actually handles the Upload. Once the file has been received completely, Apache then passes the file and process running to PHP. Also correct. I have never seen a case where Apache has stored the file on the file system. In my past experience it has always held the file in memory, therefor limited to the max physical memory that was install, minus a little for other things. Well, I can easily show you such a case - I can upload a 1Gb file without apache memory usage changing one bit. Even if Apache did upload into memory, why would that make the file limited to the max amount of physical memory?? How about if I upload a 1Gb file to a webserver on a machine that only has 256Mb memory - would you say that's impossible? /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Luke wrote: I think the sentiment is that you can't fit all of the file in the memory at once Which would be absolutely true, but irrelevant - the question is whether that would prevent me from uploading it? That is the problem Tom seems to be having. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Hello on a linux system (Suese 10.2) with 1 GB memory its not possible to upload via http a 1 Gb File. Thats no limit problem on my php config. i can look the mem stats when uploading and the growing tmp file. If the temp file has 900 MB, Main Memory free is 0 and the script aborts and php deletes the tmp file. Why don't php use swap memory ? no need for swap my system OpenSuSE 10.3 512MB RAM my uploaded file -rw-r--r-- 1 wwwrun www 10 8. Aug 09:39 mytestfile.out memory befor upload and during upload are nearly the same and no swap total used free sharedbuffers cached Mem:510488 504728 5760 0 60728 301476 -/+ buffers/cache: 142524 367964 Swap: 2104432 287562075676 a du shows that the tmp file is greater as the memory du -h /tmp/phpMmAGdN 659M/tmp/phpMmAGdN and grows du -h /tmp/phpMmAGdN 663M/tmp/phpMmAGdN So i think your script is wrong maybe you trie to read the hole contend from the upload file in a variable so you reach the memory post your script ro see if its is correct. BR/Torsten -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Maybe there is some configuration in the server somewhere causing it to incorrectly use the tmp? Luke Slater On 8 Aug 2008, at 08:48, Torsten Rosenberger [EMAIL PROTECTED] wrote: Hello on a linux system (Suese 10.2) with 1 GB memory its not possible to upload via http a 1 Gb File. Thats no limit problem on my php config. i can look the mem stats when uploading and the growing tmp file. If the temp file has 900 MB, Main Memory free is 0 and the script aborts and php deletes the tmp file. Why don't php use swap memory ? no need for swap my system OpenSuSE 10.3 512MB RAM my uploaded file -rw-r--r-- 1 wwwrun www 10 8. Aug 09:39 mytestfile.out memory befor upload and during upload are nearly the same and no swap total used free sharedbuffers cached Mem:510488 504728 5760 0 60728 301476 -/+ buffers/cache: 142524 367964 Swap: 2104432 287562075676 a du shows that the tmp file is greater as the memory du -h /tmp/phpMmAGdN 659M/tmp/phpMmAGdN and grows du -h /tmp/phpMmAGdN 663M/tmp/phpMmAGdN So i think your script is wrong maybe you trie to read the hole contend from the upload file in a variable so you reach the memory post your script ro see if its is correct. BR/Torsten -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Am Freitag, den 08.08.2008, 09:00 +0100 schrieb Luke: Maybe there is some configuration in the server somewhere causing it to incorrectly use the tmp? normla config php.ini ; Whether to allow HTTP file uploads. file_uploads = On ; Temporary directory for HTTP uploaded files (will use system default if not ; specified). upload_tmp_dir = /tmp ; Maximum allowed size for uploaded files. upload_max_filesize = 1500M -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom wrote: Hi Per, your result is on a suse 8.2 ? hmm, i don't know the reason why my suse 10.2 machines do that failure. My limits for for post_max_size and upload_max_size is both 1500M. Greets thanx, Tom Tom, check your maximum PHP execution time. I've just done some testing where the upload was interrupted after 60seconds with this error: PHP Fatal error: Maximum execution time of 60 seconds exceeded in Unknown on line 0, referer: http://tintin/uploadbig.php I increased time to 240seconds, and had no problem uploading a 1Gb file. This was on a openSUSE 10.3 system with only 256M RAM. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Per Jessen wrote: Jim Lucas wrote: I have never seen a case where Apache has stored the file on the file system. In my past experience it has always held the file in memory, therefor limited to the max physical memory that was install, minus a little for other things. Well, I can easily show you such a case - I can upload a 1Gb file without apache memory usage changing one bit. Even if Apache did upload into memory, why would that make the file limited to the max amount of physical memory?? How about if I upload a 1Gb file to a webserver on a machine that only has 256Mb memory - would you say that's impossible? Just did that (1Gb file uploaded to machine with 256M RAM) - Apache definitely does _not_ store the uploaded file in memory, I also straced the upload, and it's 1) read from socket followed by 2) write to disk. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Hi Per, Execution Time ist set to 72000, i learned to set it not to low for tests :-) I learned also the last days many things about the php.ini. Many changes affect later, its mostly for me a trial and error thing and results in much phenomenon. At moment its okay, i can upload 1.2 Gb, no problem. I test to upload a 1.9 GB file (reading the max is 2GB a browser can handle), but this failes. I don't know is this the overhead or what else. Now i search the max value Browseres can handle for input type=hidden name=MAX_FILE_SIZE value= /. Know you this value? Per Jessen [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] Tom wrote: Hi Per, your result is on a suse 8.2 ? hmm, i don't know the reason why my suse 10.2 machines do that failure. My limits for for post_max_size and upload_max_size is both 1500M. Greets thanx, Tom Tom, check your maximum PHP execution time. I've just done some testing where the upload was interrupted after 60seconds with this error: PHP Fatal error: Maximum execution time of 60 seconds exceeded in Unknown on line 0, referer: http://tintin/uploadbig.php I increased time to 240seconds, and had no problem uploading a 1Gb file. This was on a openSUSE 10.3 system with only 256M RAM. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom wrote: Hi Per, Execution Time ist set to 72000, i learned to set it not to low for tests :-) I learned also the last days many things about the php.ini. Many changes affect later, its mostly for me a trial and error thing and results in much phenomenon. At moment its okay, i can upload 1.2 Gb, no problem. I test to upload a 1.9 GB file (reading the max is 2GB a browser can handle), but this failes. I don't know is this the overhead or what else. Well, it seems to me that you've achieved what you need, right? You don't need one big upload, you need many smaller but concurrent uploads, yeah? I'll try a bigger file later today and see what happens. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Practical i implement a robust filebase in my new gamer portal and go to max. at upload values. If the users make a big upload, it should be stable. I think, later (after release) i will enhance it with a ftp port. But not yet. Here you can see, what i have in filebase, but a 1.9 GB upload fails. I don't know if this is the overhead. http://www.guildmeets.de/index.php?onlydirid=78 (in Folder Neue Game Demos) Per Jessen [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] Tom wrote: Hi Per, Execution Time ist set to 72000, i learned to set it not to low for tests :-) I learned also the last days many things about the php.ini. Many changes affect later, its mostly for me a trial and error thing and results in much phenomenon. At moment its okay, i can upload 1.2 Gb, no problem. I test to upload a 1.9 GB file (reading the max is 2GB a browser can handle), but this failes. I don't know is this the overhead or what else. Well, it seems to me that you've achieved what you need, right? You don't need one big upload, you need many smaller but concurrent uploads, yeah? I'll try a bigger file later today and see what happens. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Is a 1.9 gb file upload even sustainable on even a fairly small scale web application? Maybe you could implement FTP if you trust the people that want to upload the file. Luke Slater Lead Developer NuVoo On 9 Aug 2008, at 14:52, Tom [EMAIL PROTECTED] wrote: Practical i implement a robust filebase in my new gamer portal and go to max. at upload values. If the users make a big upload, it should be stable. I think, later (after release) i will enhance it with a ftp port. But not yet. Here you can see, what i have in filebase, but a 1.9 GB upload fails. I don't know if this is the overhead. http://www.guildmeets.de/index.php?onlydirid=78 (in Folder Neue Game Demos) Per Jessen [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] Tom wrote: Hi Per, Execution Time ist set to 72000, i learned to set it not to low for tests :-) I learned also the last days many things about the php.ini. Many changes affect later, its mostly for me a trial and error thing and results in much phenomenon. At moment its okay, i can upload 1.2 Gb, no problem. I test to upload a 1.9 GB file (reading the max is 2GB a browser can handle), but this failes. I don't know is this the overhead or what else. Well, it seems to me that you've achieved what you need, right? You don't need one big upload, you need many smaller but concurrent uploads, yeah? I'll try a bigger file later today and see what happens. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Per Jessen wrote: How about if I upload a 1Gb file to a webserver on a machine that only has 256Mb memory - would you say that's impossible? /Per Jessen, Zürich In my experience that would the correct. -- Jim Lucas Some men are born to greatness, some achieve greatness, and some have greatness thrust upon them. Twelfth Night, Act II, Scene V by William Shakespeare -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom wrote: Practical i implement a robust filebase in my new gamer portal and go to max. at upload values. If the users make a big upload, it should be stable. I think, later (after release) i will enhance it with a ftp port. But not yet. Here you can see, what i have in filebase, but a 1.9 GB upload fails. I don't know if this is the overhead. http://www.guildmeets.de/index.php?onlydirid=78 OK, I've also just tried uploading bigger files - 1280Mb,1536Mb,180MB are fine, but 1900Mb fails. I'll do that again with strace and see if I can spot something obvious - I didn't see any errors etc. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Per Jessen wrote: Tom wrote: Practical i implement a robust filebase in my new gamer portal and go to max. at upload values. If the users make a big upload, it should be stable. I think, later (after release) i will enhance it with a ftp port. But not yet. Here you can see, what i have in filebase, but a 1.9 GB upload fails. I don't know if this is the overhead. http://www.guildmeets.de/index.php?onlydirid=78 OK, I've also just tried uploading bigger files - 1280Mb,1536Mb,180MB are fine, but 1900Mb fails. I'll do that again with strace and see if I can spot something obvious - I didn't see any errors etc. This time I had set the upload_file_size and post_max_size to 3000Mb each, which probably didn't work. When I tried the 1900Mb file again I got this error: [Fri Aug 08 16:57:28 2008] [error] [client 192.168.2.113] PHP Warning: POST Content-Length of 1992294868 bytes exceeds the limit of -1149239296 bytes in Unknown on line 0, referer /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
On Fri, Aug 8, 2008 at 11:14 AM, Per Jessen [EMAIL PROTECTED] wrote: Per Jessen wrote: Tom wrote: Practical i implement a robust filebase in my new gamer portal and go to max. at upload values. If the users make a big upload, it should be stable. I think, later (after release) i will enhance it with a ftp port. But not yet. Here you can see, what i have in filebase, but a 1.9 GB upload fails. I don't know if this is the overhead. http://www.guildmeets.de/index.php?onlydirid=78 OK, I've also just tried uploading bigger files - 1280Mb,1536Mb,180MB are fine, but 1900Mb fails. I'll do that again with strace and see if I can spot something obvious - I didn't see any errors etc. This time I had set the upload_file_size and post_max_size to 3000Mb each, which probably didn't work. When I tried the 1900Mb file again I got this error: [Fri Aug 08 16:57:28 2008] [error] [client 192.168.2.113] PHP Warning: POST Content-Length of 1992294868 bytes exceeds the limit of -1149239296 bytes in Unknown on line 0, referer /Per Jessen, Zürich Just a guess 3000MB exceeds the value of a 4-byte integer when converted to bytes. -1149239296 is the signed equivalent of 3145728000, which is 3000 * 1024 * 1024. I'm guessing that since the upload size is greater than a (really large) negative number, PHP is throwing an error. That's probably where the 2GB figure comes from. Andrew
Re: [PHP] php File upload
Andrew Ballard wrote: This time I had set the upload_file_size and post_max_size to 3000Mb each, which probably didn't work. When I tried the 1900Mb file again I got this error: [Fri Aug 08 16:57:28 2008] [error] [client 192.168.2.113] PHP [Warning: POST Content-Length of 1992294868 bytes exceeds the limit of -1149239296 bytes in Unknown on line 0, referer /Per Jessen, Zürich Just a guess 3000MB exceeds the value of a 4-byte integer when converted to bytes. -1149239296 is the signed equivalent of 3145728000, which is 3000 * 1024 * 1024. Ah, well spotted. So the max request size is stored as a signed 32bit number. I'm guessing that since the upload size is greater than a (really large) negative number, PHP is throwing an error. That's probably where the 2GB figure comes from. That's certainly one place, but the 2Gb limitation is from HTTP or from apache. It's mentioned in the apache manual under LimitRequestBody'. This directive specifies the number of bytes from 0 (meaning unlimited) to 2147483647 (2GB) that are allowed in a request body. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Andrew write the answer: -1149239296 Bytes is indiz for to big. This time I had set the upload_file_size and post_max_size to 3000Mb each, which probably didn't work. When I tried the 1900Mb file again I got this error: [Fri Aug 08 16:57:28 2008] [error] [client 192.168.2.113] PHP Warning: POST Content-Length of 1992294868 bytes exceeds the limit of -1149239296 bytes in Unknown on line 0, referer /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
On 8/8/08, Luke [EMAIL PROTECTED] wrote: Is a 1.9 gb file upload even sustainable on even a fairly small scale web application? Maybe you could implement FTP if you trust the people that want to upload the file. This is why I am pushing for people to use PUT. Still over HTTP, uploaders can be coded to be 'smart' and resume, re-transmit on failure, etc. It does however require applet (java, flash, etc) or thick client support on the client, and a server that understands the DAV request. I have it working in nginx right now. Need to test large files and watch PHP's memory consumption, since I am spoonfeeding the file to PHP. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
On Fri, Aug 8, 2008 at 1:08 PM, mike [EMAIL PROTECTED] wrote: On 8/8/08, Luke [EMAIL PROTECTED] wrote: Is a 1.9 gb file upload even sustainable on even a fairly small scale web application? Maybe you could implement FTP if you trust the people that want to upload the file. This is why I am pushing for people to use PUT. Still over HTTP, uploaders can be coded to be 'smart' and resume, re-transmit on failure, etc. It does however require applet (java, flash, etc) or thick client support on the client, and a server that understands the DAV request. I have it working in nginx right now. Need to test large files and watch PHP's memory consumption, since I am spoonfeeding the file to PHP. I've not had to upload such large files over HTTP, so forgive my ignorance, but on the request end isn't the only difference between PUT and POST the verb used in the request (and the intent of the operation)? What can you do with PUT that cannot also be handled the same in POST? I don't see any implementations doing it, but from what I can tell the spec allows you to use Content-Range in the request headers that your client sends to PUT/POST the same way the server sends them in the response headers when serving a GET request. Andrew -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
On 8/8/08, Andrew Ballard [EMAIL PROTECTED] wrote: I've not had to upload such large files over HTTP, so forgive my ignorance, but on the request end isn't the only difference between PUT and POST the verb used in the request (and the intent of the operation)? What can you do with PUT that cannot also be handled the same in POST? I don't see any implementations doing it, but from what I can tell the spec allows you to use Content-Range in the request headers that your client sends to PUT/POST the same way the server sends them in the response headers when serving a GET request. POST sends mime-encoded PUT is raw (AFAIK) It's basically file exchange friendly. People usually say HTTP isn't for files, FTP is but that's a whole other can of worms especially when you're accepting uploads from -anyone- you have to implement a wacky two-step process, ensure people know how to use FTP, blahblah. Using PUT you can do pure file uploads, have smart clients that tell the server where to resume, it's not mime encoded (so not extra bytes), it can be processed as a web request via PHP, it can be done over SSL for security ... -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
mike wrote: On 8/8/08, Andrew Ballard [EMAIL PROTECTED] wrote: I've not had to upload such large files over HTTP, so forgive my ignorance, but on the request end isn't the only difference between PUT and POST the verb used in the request (and the intent of the operation)? What can you do with PUT that cannot also be handled the same in POST? I don't see any implementations doing it, but from what I can tell the spec allows you to use Content-Range in the request headers that your client sends to PUT/POST the same way the server sends them in the response headers when serving a GET request. POST sends mime-encoded PUT is raw (AFAIK) That's not necessarily a big difference between the two - mime-encoded can also be plain 8-bit. Using PUT you can do pure file uploads, have smart clients that tell the server where to resume, it's not mime encoded (so not extra bytes), it can be processed as a web request via PHP, it can be done over SSL for security ... Same as POST then :-) (except for the resume bit). I still don't see much of a difference. It doesn't matter much to me, I'd just like to understand what the real difference is. Maybe I need to go and read RFC2616. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
On 8/8/08, Per Jessen [EMAIL PROTECTED] wrote: Same as POST then :-) (except for the resume bit). I still don't see much of a difference. It doesn't matter much to me, I'd just like to understand what the real difference is. Maybe I need to go and read RFC2616. I'm all for using existing POST but it seems like a lot of configuration needs to occur, and PHP is expecting a single stream of data. If the stream is broken, how do you resume it? If you accepted the data raw you could dump the stream to a file until it ends, and then fseek($fp, $offset) when you get a resume request... PUT seems to be expected to be straight old file $x goes here ... I actually am not allowing the file to be put on the filesystem directly but fed to a PHP script. I am totally down for POST since POST is supported by browsers, flash, everything, and PUT has some limitations (java applets aplenty, flash can do it i think but you have to do actual raw socket communication and I haven't found an applet for that yet) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
PUT is raw (AFAIK) That sounds good. Have you any link to a basicly methode description? This is a really new methode for me but sounds better than http upload btw. resume function is not bad. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom wrote: Hi, on a linux system (Suese 10.2) with 1 GB memory its not possible to upload via http a 1 Gb File. Thats no limit problem on my php config. i can look the mem stats when uploading and the growing tmp file. If the temp file has 900 MB, Main Memory free is 0 and the script aborts and php deletes the tmp file. Why don't php use swap memory ? It doesn't need to - as you've noticed, the uploaded file is being written to disk, it's not being kept in memory. This sounds like a php limit problem to me. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
No, ist'not a php limit. The upload is written in main memory, if i look with vmstat, free is going to 0 and the php upload breaks at 0 bytes free. Nothing swap used. Any other ideas? Per Jessen [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] Tom wrote: Hi, on a linux system (Suese 10.2) with 1 GB memory its not possible to upload via http a 1 Gb File. Thats no limit problem on my php config. i can look the mem stats when uploading and the growing tmp file. If the temp file has 900 MB, Main Memory free is 0 and the script aborts and php deletes the tmp file. Why don't php use swap memory ? It doesn't need to - as you've noticed, the uploaded file is being written to disk, it's not being kept in memory. This sounds like a php limit problem to me. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom wrote: No, ist'not a php limit. The upload is written in main memory, if i look with vmstat, free is going to 0 and the php upload breaks at 0 bytes free. Nothing swap used. Any other ideas? Interesting problem - maybe an apache limit? Lack of diskspace? Permissions? Which error do you get? /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
No Apache limit, enough diskspace, no permission problem. Fact is: During upload looking at main memory, goes to 0 and if 0 uploadet tmp File was deleted and no error occur. It was great, if anybody can test such a upload. The uploaded File musst be greater as main memory. While teh uload is running, take a look at memory with vmstat. free is goin to 0, no swap is used. I don't know, that is a suse 10.2 effcet only. Per Jessen [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] Tom wrote: No, ist'not a php limit. The upload is written in main memory, if i look with vmstat, free is going to 0 and the php upload breaks at 0 bytes free. Nothing swap used. Any other ideas? Interesting problem - maybe an apache limit? Lack of diskspace? Permissions? Which error do you get? /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
No Apache limit, enough diskspace, no permission problem. Fact is: During upload looking at main memory, goes to 0 and if 0 uploadet tmp File was deleted and no error occur. It was great, if anybody can test such a upload. The uploaded File musst be greater as main memory. While teh uload is running, take a look at memory with vmstat. free is goin to 0, no swap is used. I don't know, that is a suse 10.2 effcet only. What happens with a smaller file? Say 500Mb? You could gradually increase the file size until you get the error. -- Richard Heyes http://www.phpguru.org -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
smaller files is no problem. The error occoured, if the uploaded File is bigger than the installed main memory. Ok, i know that http ist not designed for big uploads. But on a big website with upload enabled and 30 users upload simultan a 20 MB File, they lost the upload while php hold all in memory/Cache and swap is not used. I don't understand why php write the upload chunck for chunck (i can look this) in upload temp file AND holds the upload in main memory. Richard Heyes [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] No Apache limit, enough diskspace, no permission problem. Fact is: During upload looking at main memory, goes to 0 and if 0 uploadet tmp File was deleted and no error occur. It was great, if anybody can test such a upload. The uploaded File musst be greater as main memory. While teh uload is running, take a look at memory with vmstat. free is goin to 0, no swap is used. I don't know, that is a suse 10.2 effcet only. What happens with a smaller file? Say 500Mb? You could gradually increase the file size until you get the error. -- Richard Heyes http://www.phpguru.org -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Well, you lost me right about... Well when you started. But memory is cheap...:-/ -- Richard Heyes http://www.phpguru.org -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
hmm, memory buying is okay, not the final solution. I think php design fault on this function. It's not comprehensible why php use such a lot main memory for an upload. Richard Heyes [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] Well, you lost me right about... Well when you started. But memory is cheap...:-/ -- Richard Heyes http://www.phpguru.org -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom wrote: But on a big website with upload enabled and 30 users upload simultan a 20 MB File, they lost the upload while php hold all in memory/Cache and swap is not used. I don't understand why php write the upload chunck for chunck (i can look this) in upload temp file AND holds the upload in main memory. Well, I can't reproduce the problem - I uploaded a 1Gb file, and I never saw apache memory usage go beyond 15Mb. Might this problem be in your code processing the uploaded file(s)? In this test I just uploaded the file, I never tried to process it. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom wrote: hmm, memory buying is okay, not the final solution. I think php design fault on this function. It's not comprehensible why php use such a lot main memory for an upload. Well, PHP doesn't - the upload is apaches job. Once the file is uploaded, PHP is invoked to process it. You do your move_uploaded_file() etc. What do you do to the uploaded file? /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
With what linux BS have you tested? No, it isn't the apaches job/problem. I do nothing with the file. Looking for $_FILES Varables and then move_uploaded_file. But i said it before, the problem is not at this point. If i upload the file and watch memory with vmstat, free is going 0, cache is going to top and if free is 0 = temp file was deleted and script witout an error comes back. At the crash point, the File is not complete uploaded, so my $_FILES have no entry about the file specs. Im hanging some days with many tests on differnet machines (alls suse) on this Problem and can't solve it :-( Per Jessen [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] Tom wrote: hmm, memory buying is okay, not the final solution. I think php design fault on this function. It's not comprehensible why php use such a lot main memory for an upload. Well, PHP doesn't - the upload is apaches job. Once the file is uploaded, PHP is invoked to process it. You do your move_uploaded_file() etc. What do you do to the uploaded file? /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom wrote: With what linux BS have you tested? It was an older SUSE Linux 8.2. But the OS shouldn't matter - maybe the PHP release and maybe the apache ditto. No, it isn't the apaches job/problem. I do nothing with the file. Looking for $_FILES Varables and then move_uploaded_file. That's exactly what I just did. But i said it before, the problem is not at this point. If i upload the file and watch memory with vmstat, free is going 0, cache is going to top and if free is 0 = temp file was deleted and script witout an error comes back. At the crash point, the File is not complete uploaded, so my $_FILES have no entry about the file specs. Ah, so it's not PHP and it's not apache using up your memory - it's your filesystem cache. Check how many files you have in the tmp directory and the destination directory. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Ah, so it's not PHP and it's not apache using up your memory - it's your filesystem cache. Check how many files you have in the tmp directory and the destination directory. Okay, thats light in the darkness. But i don't have any idea what i can do here. tmp is empty. Look at the first entry, its after the upload start. The last entry is after free = 0 and break. I really have no idea whats going on here. procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 0 677076 28020 216052001758 370 166 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 0 0 667404 28028 225536001758 371 167 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 1 0 660088 28036 231456001760 371 168 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 1 0 656616 28040 235820001760 372 168 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 0 653640 28052 239364001760 372 168 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 0 650044 28056 242948001760 372 168 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 0 0 642480 28060 250432001760 372 169 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 0 635536 28068 257244001760 373 170 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 0 0 630080 28072 262440001760 373 170 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 0 624500 28080 267892001760 374 170 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 2 0 620532 28080 270440001762 374 171 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 2 0 620904 28080 270440001762 374 171 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 2 0 0 618672 28092 273756001762 374 171 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 0 613960 28096 278432001762 374 171 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 2 0 114984 28668 768276001788 406 209 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 0 105932 28684 778608001788 407 210 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 0 96756 28692 787448001788 407 211 1 0 98 1 linuxserver:/tmp # vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us
Re: [PHP] php File upload
Tom wrote: Ah, so it's not PHP and it's not apache using up your memory - it's your filesystem cache. Check how many files you have in the tmp directory and the destination directory. Okay, thats light in the darkness. But i don't have any idea what i can do here. tmp is empty. What about the destination directory? Which filesystem are you using? I'll try the 1Gb upload again and see how vmstat reacts. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
i use ext3 Im realy on no limit. destination ist /tmp and is fairly empty. The question is now, if cache full = must hypothetical swap used? ok, im glad to see your result. Per Jessen [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] Tom wrote: Ah, so it's not PHP and it's not apache using up your memory - it's your filesystem cache. Check how many files you have in the tmp directory and the destination directory. Okay, thats light in the darkness. But i don't have any idea what i can do here. tmp is empty. What about the destination directory? Which filesystem are you using? I'll try the 1Gb upload again and see how vmstat reacts. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom wrote: i use ext3 Im realy on no limit. destination ist /tmp and is fairly empty. The question is now, if cache full = must hypothetical swap used? ok, im glad to see your result. No, high utilization of file system cache will not cause any swapping - file system caching uses whatever is spare. If there's nothing, no caching. AFAIU. I can confirm what you're seeing about the cache being used up - I saw that too. My cache-number only went to about 485000, and free only down to 6000-7000. My webserver also has 1Gb RAM, but it does a few more things. Does your free memory actually go as low as 0? I don't think this is about the file system cache - I know you said no limits, but what do you have for post_max_size and upload_max_size ? I'm using 1200M for both for this test. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Hi Per, your result is on a suse 8.2 ? hmm, i don't know the reason why my suse 10.2 machines do that failure. My limits for for post_max_size and upload_max_size is both 1500M. Greets thanx, Tom Per Jessen [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] Tom wrote: i use ext3 Im realy on no limit. destination ist /tmp and is fairly empty. The question is now, if cache full = must hypothetical swap used? ok, im glad to see your result. No, high utilization of file system cache will not cause any swapping - file system caching uses whatever is spare. If there's nothing, no caching. AFAIU. I can confirm what you're seeing about the cache being used up - I saw that too. My cache-number only went to about 485000, and free only down to 6000-7000. My webserver also has 1Gb RAM, but it does a few more things. Does your free memory actually go as low as 0? I don't think this is about the file system cache - I know you said no limits, but what do you have for post_max_size and upload_max_size ? I'm using 1200M for both for this test. /Per Jessen, Zürich -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom wrote: Hi, on a linux system (Suese 10.2) with 1 GB memory its not possible to upload via http a 1 Gb File. Thats no limit problem on my php config. i can look the mem stats when uploading and the growing tmp file. If the temp file has 900 MB, Main Memory free is 0 and the script aborts and php deletes the tmp file. Why don't php use swap memory ? Greets Tom After reading this thread, I would like to add my thoughts. What Apache starts, it reads the PHP memory limits in to the running Apache process. When you try and upload a file, it is a straight HTTP upload. PHP plays no part in the actual upload, except for the upload limits set in place by the php settings found in the php.ini or other Apache config files. Now, Apache actually handles the Upload. Once the file has been received completely, Apache then passes the file and process running to PHP. I have never seen a case where Apache has stored the file on the file system. In my past experience it has always held the file in memory, therefor limited to the max physical memory that was install, minus a little for other things. I have worked on Debian (potato|woody|sarge), Redhat 5.0 - 9, Fedora 1 - 7, OpenBSD 3.6-current, and older versions of NetBSD and FreeBSD. I have also frequented a number of friends installs and I have never seen uploads happen any other way. Nor have I heard of uploads happening any other way. Just my 2cents :) -- Jim Lucas Some men are born to greatness, some achieve greatness, and some have greatness thrust upon them. Twelfth Night, Act II, Scene V by William Shakespeare -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php File upload
Tom's machine only has 1 GB of physical memory doesn't it? Could that be the problem? 2008/8/7 Jim Lucas [EMAIL PROTECTED] Tom wrote: Hi, on a linux system (Suese 10.2) with 1 GB memory its not possible to upload via http a 1 Gb File. Thats no limit problem on my php config. i can look the mem stats when uploading and the growing tmp file. If the temp file has 900 MB, Main Memory free is 0 and the script aborts and php deletes the tmp file. Why don't php use swap memory ? Greets Tom After reading this thread, I would like to add my thoughts. What Apache starts, it reads the PHP memory limits in to the running Apache process. When you try and upload a file, it is a straight HTTP upload. PHP plays no part in the actual upload, except for the upload limits set in place by the php settings found in the php.ini or other Apache config files. Now, Apache actually handles the Upload. Once the file has been received completely, Apache then passes the file and process running to PHP. I have never seen a case where Apache has stored the file on the file system. In my past experience it has always held the file in memory, therefor limited to the max physical memory that was install, minus a little for other things. I have worked on Debian (potato|woody|sarge), Redhat 5.0 - 9, Fedora 1 - 7, OpenBSD 3.6-current, and older versions of NetBSD and FreeBSD. I have also frequented a number of friends installs and I have never seen uploads happen any other way. Nor have I heard of uploads happening any other way. Just my 2cents :) -- Jim Lucas Some men are born to greatness, some achieve greatness, and some have greatness thrust upon them. Twelfth Night, Act II, Scene V by William Shakespeare -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- Luke Slater
Re: [PHP] php File upload
Simple answer is yes. But others might argue this answer. Luke wrote: Tom's machine only has 1 GB of physical memory doesn't it? Could that be the problem? 2008/8/7 Jim Lucas [EMAIL PROTECTED] Tom wrote: Hi, on a linux system (Suese 10.2) with 1 GB memory its not possible to upload via http a 1 Gb File. Thats no limit problem on my php config. i can look the mem stats when uploading and the growing tmp file. If the temp file has 900 MB, Main Memory free is 0 and the script aborts and php deletes the tmp file. Why don't php use swap memory ? Greets Tom After reading this thread, I would like to add my thoughts. What Apache starts, it reads the PHP memory limits in to the running Apache process. When you try and upload a file, it is a straight HTTP upload. PHP plays no part in the actual upload, except for the upload limits set in place by the php settings found in the php.ini or other Apache config files. Now, Apache actually handles the Upload. Once the file has been received completely, Apache then passes the file and process running to PHP. I have never seen a case where Apache has stored the file on the file system. In my past experience it has always held the file in memory, therefor limited to the max physical memory that was install, minus a little for other things. I have worked on Debian (potato|woody|sarge), Redhat 5.0 - 9, Fedora 1 - 7, OpenBSD 3.6-current, and older versions of NetBSD and FreeBSD. I have also frequented a number of friends installs and I have never seen uploads happen any other way. Nor have I heard of uploads happening any other way. Just my 2cents :) -- Jim Lucas Some men are born to greatness, some achieve greatness, and some have greatness thrust upon them. Twelfth Night, Act II, Scene V by William Shakespeare -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- Jim Lucas Some men are born to greatness, some achieve greatness, and some have greatness thrust upon them. Twelfth Night, Act II, Scene V by William Shakespeare -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php file upload permission query
Angelo Zanetti wrote: thanks, but Im sure there is something that is messing up the file permissions that I can change before the upload, instead of trying to cure the problem by setting the chmod of the file after its uplaoded Check the following things: 1. umask settings on the directory 2. Apache's default mask Also, please don't top post. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php file upload permission query
thanks, but Im sure there is something that is messing up the file permissions that I can change before the upload, instead of trying to cure the problem by setting the chmod of the file after its uplaoded Hristo Yankov wrote: Use chmod() on the uploaded file. === Hristo Yankov, Developer at Portellus, Inc. ICQ - 191445567 Yahoo! - yankov_hristo __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php file upload permission query
Use chmod() on the uploaded file. === Hristo Yankov, Developer at Portellus, Inc. ICQ - 191445567 Yahoo! - yankov_hristo __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php file upload permission query
The remote PHP machine maybe forces the files to change their permission when uploaded. This would be security measure. And I think that if this is the case, you won't be able to change the permissions. === Hristo Yankov, Developer at Portellus, Inc. ICQ - 191445567 Yahoo! - yankov_hristo __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php: file upload program limitation..
max_execution_time does not affect file uploads. Php script and thus also the timer starts after the upload is finished. max_input_time might influence it as it is the time allowed for the browser to post the data. Justin French wrote: See these three directives in php.ini: max_execution_time = 30 max_input_time = 60 memory_limit = 8M and this one you've already played with: post_max_size = 8M Obviously, for you to upload a 5 meg file over to a server is going to take more than 30 seconds, and may *possibly* consume more than 8M of memory (not sure there), so these settings need to be carefully modified to suit your needs. Justin French -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] php: file upload program limitation..
On Thursday, October 30, 2003, at 01:40 PM, Louie Miranda wrote: Im made a file upload program. I limit my php.ini to accept only 5mb but i told on my website that it is 2mb only. Now here's my problem. I only upload a 1.5mb and a 1.7mb file when ever i submit it the browser displays the page cannot be displayed but when ever i upload a file lower than 1mb it uploads it. Where the problem anyway? there are a few factors... more than likely it's the maximum execution time of either PHP (set in php.ini) or perhaps apache. See these three directives in php.ini: max_execution_time = 30 max_input_time = 60 memory_limit = 8M and this one you've already played with: post_max_size = 8M Obviously, for you to upload a 5 meg file over to a server is going to take more than 30 seconds, and may *possibly* consume more than 8M of memory (not sure there), so these settings need to be carefully modified to suit your needs. Justin French -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
