Re: [PHP] handling large files w/readfile
A little late but: Robin Getz wrote: If this runs for awhile things go very bad. This seems to be related to a specific download manager called NetAnts that seems to be popular in China. http://www.netants.com/ Which attempts to open the same url for downloading 10-15 times at the same instant. Download managers use Content-Range header [1] to request parts of the downloaded file in multiple instances. The client above is likely broken and does not recognize that your script does not support content-range requests, and keeps requesting and disconnecting instead of giving up. Someone suggested apache_child_terminate - but this function doesn't seem to be available to me. If the function is not available you can still use connection_aborted(). [1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.16 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] handling large files w/readfile
On Sunday 02 January 2005 16:43, Robin Getz wrote:
Rasmus Lerdorf wrote:
$buff = 0;
while (!feof($fp)) {
$buff = fread($fp, 4096);
print $buff;
}
unset($buff);
fclose ($fp);
Well, the above code does not use more than 4K of ram plus a bit of
overhead. So if something is causing your processes to grow to 450M you
need to look elsewhere because this code is definitely not the cause.
Well, the test case is:
1) above with big files = big apache processes - machine crashes
Are you using the above code on its own (ie not within some other code that
may affect the memory usage)?
--
Jason Wong - Gremlins Associates - www.gremlins.biz
Open Source Software Systems Integrators
* Web Design Hosting * Internet Intranet Applications Development *
--
Search the list archives before you post
http://marc.theaimsgroup.com/?l=php-general
--
New Year Resolution: Ignore top posted posts
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] handling large files w/readfile
Jason Wong wrote:
Are you using the above code on its own (ie not within some other code
that may affect the memory usage)?
Well, herethe entire file (It is pretty short - only a 2 pages, but sorry
in advance if anyone considers this bad form).
site is called with something like
http://blackfin.uclinux.org/frs/download.php/123/STAMP.jpg
Files are stored in:
$sys_upload_dir.$group_name.'/'.$filename
-- frs/download.php -
?php
/**
* GForge FRS Facility
*
* Copyright 1999-2001 (c) VA Linux Systems
* The rest Copyright 2002-2004 (c) GForge Team
* http://gforge.org/
*
* @version $Id: download.php,v 1.6 2004/10/08 23:05:29 gsmet Exp $
*
* This file is part of GForge.
*
* GForge is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* GForge is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with GForge; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston,
MA 02111-1307 USA */
$no_gz_buffer=true;
require_once('pre.php');
$arr=explode('/',$REQUEST_URI);
$file_id=$arr[3];
$res=db_query(SELECT frs_file.filename,frs_package.is_public,
frs_file.file_id,groups.unix_group_name,groups.group_id
FROM frs_package,frs_release,frs_file,groups
WHERE frs_release.release_id=frs_file.release_id
AND groups.group_id=frs_package.group_id
AND frs_release.package_id=frs_package.package_id
AND frs_file.file_id='$file_id');
if (db_numrows($res) 1) {
Header(Status: 404);
exit;
}
$is_public =db_result($res,0,'is_public');
$group_name=db_result($res,0,'unix_group_name');
$filename = db_result($res,0,'filename');
$release_id=db_result($res,0,'release_id');
$group_id = db_result($res,0,'group_id');
$Group = group_get_object($group_id);
if (!$Group || !is_object($Group) || $Group-isError()) {
exit_no_group();
}
if(!$Group-isPublic()) {
session_require(array('group' = $group_id)); }
// Members of projects can see all packages // Non-members can only see
public packages
if(!$is_public) {
if (!session_loggedin() || (!user_ismember($group_id)
!user_ismember(1,'A'))) {
exit_permission_denied();
}
}
/*
echo $group_name.'|'.$filename.'|'.$sys_upload_dir.$group_name.'/'.$filename;
if (file_exists($sys_upload_dir.$group_name.'/'.$filename)) {
echo 'br /file exists';
passthru($sys_upload_dir.$group_name.'/'.$filename);
}
*/
if (file_exists($sys_upload_dir.$group_name.'/'.$filename)) {
Header('Content-disposition: filename='.str_replace('', '',
$filename).'');
Header(Content-type: application/binary);
length = filesize($sys_upload_dir.$group_name.'/'.$filename);
Header(Content-length: $length);
# Here is where all the problems start
readfile($sys_upload_dir.$group_name.'/'.$filename);
if (session_loggedin()) {
s = session_get_user();
us=$s-getID();
} else {
us=100;
}
res=db_query(INSERT INTO frs_dlstats_file
(ip_address,file_id,month,day,user_id)
VALUES
('$REMOTE_ADDR','$file_id','.date('Ym').','.date('d').','$us'));
} else {
Header(Status: 404);
}
?
=
If this runs for awhile things go very bad. This seems to be related to a
specific download manager called NetAnts that seems to be popular in China.
http://www.netants.com/
Which attempts to open the same url for downloading 10-15 times at the same
instant.
If I replace things with:
snip =
if (file_exists($sys_upload_dir.$group_name.'/'.$filename)) {
# if the file is too big to download (10Meg) - use a different
method than php
$length = filesize($sys_upload_dir.$group_name.'/'.$filename);
Header('Content-disposition: filename='.str_replace('', '',
$filename).'');
Header(Content-type: application/binary);
Header(Content-length: $length);
fp = fopen($sys_upload_dir.$group_name.'/'.$filename,'rb');
buff=0;
while (!feof($fp)) {
buff = fread($fp, 4096);
print $buff;
}
unset($buff);
fclose ($fp);
=== snip - rest is the same =
I get exactly the same problem - I come back and there are 2-3-4 apache
processes that are consuming memory the size of the largest downloads.
The only way I can make things work with large downloads is to use this:
snip
if
Re: [PHP] handling large files w/readfile
On Tuesday 04 January 2005 22:04, Robin Getz wrote: Jason Wong wrote: Are you using the above code on its own (ie not within some other code that may affect the memory usage)? Well, herethe entire file (It is pretty short - only a 2 pages, but sorry in advance if anyone considers this bad form). I should have added that if you're not running that snippet of code on its own then try running it on its own, give it a fixed filename (so you don't have to go through the $_REQUEST and database business). Clouding the issue by running it in conjunction with other code (however innocuous) just complicates matters. -- Jason Wong - Gremlins Associates - www.gremlins.biz Open Source Software Systems Integrators * Web Design Hosting * Internet Intranet Applications Development * -- Search the list archives before you post http://marc.theaimsgroup.com/?l=php-general -- New Year Resolution: Ignore top posted posts -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] handling large files w/readfile
Sebastian wrote: yea. all the files aren't 100MB though.. some are 2mb (even less) while some files are over 300MB as well. so, does this need to be adjusted depending on the filesize? I believe that at a certain point, your setting there will be worse for the system if you make it too big instead of too small, regardless of the filesize to be written/read. This is because no matter how big your file is, there are buffers all along the chain of events between http://php.net/fread and the actual electrical signals re-aligning the magnetic spots on your hard drive disc. You're defining a buffer for fread. PHP may have an internal buffer in a layer deeper than that (probably not, but *could* have one) The C library may have a buffer in the next layer down. Again, probably not, but it *could* have one. The library inside the OS that the C library is calling maybe has a buffer. The OS File System that the OS library calls probably has a buffer. The physical hardware almost for sure has a buffer, probably several. If you have fancy hardware and/or RAID software, then there are even more buffers involved. And, of course, the RAM available to PHP at the time of doing this work will drastically affect performance -- If your server is loaded, and this fread with your (possibly over-sized) buffer forces PHP and/or the OS to swap something in/out to the hard drive, then you are going to KILL your system. At every step of the way, the data being buffered in a faster/cheaper storage medium will have an effect on your performance. Hopefully, for most uses, that is an overall Good effect. But nobody on Earth can tell you for sure for sure that it's all optimum for *YOUR* application's needs on *YOUR* hardware. So given your OS, and your hard drive and your file system, and your files, and the usual load on your server (RAM) there is some ideal number for that collection of hardware/software that you use. The only way to find your ideal number is to trial and error benchmark it. I'd try numbers that match up with: the cache size on the hard drive any internal buffers in PHP's fread() functions any buffer in your File System and/or OS of choice very large numbers very small numbers a random assortment of nice numbers (1Mb 2Mb 4Mb ...) Now, if your code has to run on LOTS of different hardware, you'd want to benchmark on as many different sets of hardware as you can to find the ideal number for *most* hardware. Or, you could, I suppose, try to write a system that detects certain specific categories of hardware/software and choose your buffer size based on that. Now, having said all that, the difference to the performance for MOST applications is going to be negligible. Find a number that works fast enough for your application (with reasonable testing/benchmarking) and move on with life. -- Like Music? http://l-i-e.com/artists.htm -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] handling large files w/readfile
Rasmus Lerdorf wrote:
$buff = 0;
while (!feof($fp)) {
$buff = fread($fp, 4096);
print $buff;
}
unset($buff);
fclose ($fp);
Well, the above code does not use more than 4K of ram plus a bit of
overhead. So if something is causing your processes to grow to 450M you
need to look elsewhere because this code is definitely not the cause.
Well, the test case is:
1) above with big files = big apache processes - machine crashes
2) download big files with:
Header(Location: .$html_pointer_to_file);
= small apache processes - works great
So, I don't know if it can be anything else but that - so I am open to
suggestions, or tests that anyone wants me to run.
-Robin
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] handling large files w/readfile
yea. all the files aren't 100MB though.. some are 2mb (even less) while some
files are over 300MB as well.
so, does this need to be adjusted depending on the filesize?
thanks.
- Original Message -
From: Rory Browne [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Cc: Sebastian [EMAIL PROTECTED]; php-general@lists.php.net
Sent: Friday, December 31, 2004 10:24 PM
Subject: Re: [PHP] handling large files w/readfile
I'd go with Richards Basic idea, but if you're outputting a 100Mb file
I'd use a hell of a lot bigger chunks than 4K. With the syscall and
loop overhead, i'd go with at least half a megabyte, or more likely
2Mb depending on your amount of memory.
To do this you'd change Richards
echo fread($fp, 4096); //4K chunks
to
echo fread($fp, 2048000); // ~2Mb chunks (actually 2000KB but couldn't
be bothered counting, and 2Mb is an arbitory figure anyway)
Just make sure you don't have output buffering on.
On Fri, 31 Dec 2004 15:17:51 -0800 (PST), Richard Lynch [EMAIL PROTECTED]
wrote:
Sebastian wrote:
i'm working on a app which output files with readfile() and some
headers..
i read a comment in the manual that says if your outputting a file php
will
use the same amount of memory as the size of the file. so, if the file
is
100MB php will use 100MB of memory.. is this true?
if it is, how can i work around this?
I don't know if it's STILL true (or ever was) that readfile() would suck
the whole file into RAM before spitting it out... Seems real unlikely,
but...
At any rate, you can simply do:
$file = 'whatever.mb';
$fp = @fopen($file, 'r') or trigger_error(Could not read $file,
E_USER_ERROR);
while (!feof($fp)){
echo fread($fp, 4096); //4K chunks
}
http://php.net/fopen http://php.net/fread http://php.net/feof
--
Like Music?
http://l-i-e.com/artists.htm
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] handling large files w/readfile
* Thus wrote Richard Lynch: Sebastian wrote: i'm working on a app which output files with readfile() and some headers.. i read a comment in the manual that says if your outputting a file php will use the same amount of memory as the size of the file. so, if the file is 100MB php will use 100MB of memory.. is this true? if it is, how can i work around this? I don't know if it's STILL true (or ever was) that readfile() would suck the whole file into RAM before spitting it out... Seems real unlikely, but... Never was and still isn't. using either readfile or fpassthru is the best route. Curt -- Quoth the Raven, Nevermore. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] handling large files w/readfile
Curt Zirzow wrote:
* Thus wrote Richard Lynch:
Sebastian wrote:
i'm working on a app which output files with readfile() and some
headers..
i read a comment in the manual that says if your outputting a file
php will use the same amount of memory as the size of the file. so,
if the file is 100MB php will use 100MB of memory.. is this true?
I don't know if it's STILL true (or ever was) that readfile() would
suck the whole file into RAM before spitting it out... Seems real
unlikely, but...
Never was and still isn't.
using either readfile or fpassthru is the best route.
All I know that I am hosting a GForge site, and if I leave the download.php
code as is, I will send up with apache processes that are 200+Meg. (the
size of my download files).
http://gforge.org/plugins/scmcvs/cvsweb.php/gforge/www/frs/download.php?rev=1.6;content-type=text%2Fplain;cvsroot=cvsroot%2Fgforge
(which uses readfile)
I have tried fpassthru - same thing.
I have even tried:
$fp = fopen($sys_upload_dir.$group_name.'/'.$filename,'rb');
while (!feof($fp)) {
$buff = fread($fp, 4096);
print $buff;
}
fclose ($fp);
and I get the same thing. The only thing that seems to work is:
Header(Location: .$html_pointer_to_fp);
which lets apache do the downloading.
I would do a apache_child_terminate, but the function does not seem to be
available to me (see my previous question about this).
Any thoughts, or suggestions, I am open to try.
My next experiment is:
var $buff;
while (!feof($fp)) {
$buff = fread($fp, 4096);
print $buff;
}
unset($buff);
fclose ($fp);
Hopefully that will make sure that the var $buff is only created once, and
that the memory is cleared after the function is done.
-Robin
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] handling large files w/readfile
Robin Getz wrote:
My next experiment is:
$buff = 0;
while (!feof($fp)) {
$buff = fread($fp, 4096);
print $buff;
}
unset($buff);
fclose ($fp);
Nope that doesn't work either - came back, and saw apache processes that
were +450Meg. Changed it back to apache redirection for now.
If anyone has __any__ suggestions, I am more than happy to try. I would
like to get this figured out.
Thanks
-Robin
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] handling large files w/readfile
Robin Getz wrote:
Robin Getz wrote:
My next experiment is:
$buff = 0;
while (!feof($fp)) {
$buff = fread($fp, 4096);
print $buff;
}
unset($buff);
fclose ($fp);
Nope that doesn't work either - came back, and saw apache processes that
were +450Meg. Changed it back to apache redirection for now.
If anyone has __any__ suggestions, I am more than happy to try. I would
like to get this figured out.
Well, the above code does not use more than 4K of ram plus a bit of
overhead. So if something is causing your processes to grow to 450M you
need to look elsewhere because this code is definitely not the cause.
-Rasmus
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] handling large files w/readfile
well, i really can't confirm what your seeing. but that is why i orginally
started this topic. i will do some tests..
are you setting headers before output? i just ran into another problem..
when downloading a .tar file it just returns an empty .tar file.. seems to
work fine with .exe, .zip, tar.gz, but not .tar
any ideas?
this is what im using:
header('Content-type: application/octet-stream');
header('Expires: ' . gmdate('D, d M Y H:i:s') . ' GMT');
header('Content-transfer-encoding: binary');
header('Content-Disposition: attachment; filename=' . $file['type'] . '');
header('Content-Length: ' . filesize($file['path'] . $file['type']));
header('Cache-Control: must-revalidate, post-check=0, pre-check=0');
- Original Message -
From: Robin Getz [EMAIL PROTECTED]
To: php-general@lists.php.net
Sent: Saturday, January 01, 2005 9:38 PM
Subject: RE: [PHP] handling large files w/readfile
Robin Getz wrote:
My next experiment is:
$buff = 0;
while (!feof($fp)) {
$buff = fread($fp, 4096);
print $buff;
}
unset($buff);
fclose ($fp);
Nope that doesn't work either - came back, and saw apache processes that
were +450Meg. Changed it back to apache redirection for now.
If anyone has __any__ suggestions, I am more than happy to try. I would
like to get this figured out.
Thanks
-Robin
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] handling large files w/readfile
Sebastian wrote:
i'm working on a app which output files with readfile() and some headers..
i read a comment in the manual that says if your outputting a file php
will
use the same amount of memory as the size of the file. so, if the file is
100MB php will use 100MB of memory.. is this true?
if it is, how can i work around this?
I don't know if it's STILL true (or ever was) that readfile() would suck
the whole file into RAM before spitting it out... Seems real unlikely,
but...
At any rate, you can simply do:
$file = 'whatever.mb';
$fp = @fopen($file, 'r') or trigger_error(Could not read $file,
E_USER_ERROR);
while (!feof($fp)){
echo fread($fp, 4096); //4K chunks
}
http://php.net/fopen http://php.net/fread http://php.net/feof
--
Like Music?
http://l-i-e.com/artists.htm
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] handling large files w/readfile
I'd go with Richards Basic idea, but if you're outputting a 100Mb file
I'd use a hell of a lot bigger chunks than 4K. With the syscall and
loop overhead, i'd go with at least half a megabyte, or more likely
2Mb depending on your amount of memory.
To do this you'd change Richards
echo fread($fp, 4096); //4K chunks
to
echo fread($fp, 2048000); // ~2Mb chunks (actually 2000KB but couldn't
be bothered counting, and 2Mb is an arbitory figure anyway)
Just make sure you don't have output buffering on.
On Fri, 31 Dec 2004 15:17:51 -0800 (PST), Richard Lynch [EMAIL PROTECTED]
wrote:
Sebastian wrote:
i'm working on a app which output files with readfile() and some headers..
i read a comment in the manual that says if your outputting a file php
will
use the same amount of memory as the size of the file. so, if the file is
100MB php will use 100MB of memory.. is this true?
if it is, how can i work around this?
I don't know if it's STILL true (or ever was) that readfile() would suck
the whole file into RAM before spitting it out... Seems real unlikely,
but...
At any rate, you can simply do:
$file = 'whatever.mb';
$fp = @fopen($file, 'r') or trigger_error(Could not read $file,
E_USER_ERROR);
while (!feof($fp)){
echo fread($fp, 4096); //4K chunks
}
http://php.net/fopen http://php.net/fread http://php.net/feof
--
Like Music?
http://l-i-e.com/artists.htm
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] handling large files w/readfile
i'm working on a app which output files with readfile() and some headers.. i read a comment in the manual that says if your outputting a file php will use the same amount of memory as the size of the file. so, if the file is 100MB php will use 100MB of memory.. is this true? if it is, how can i work around this? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] handling large files w/readfile
Hello! i'm working on a app which output files with readfile() and some headers.. i read a comment in the manual that says if your outputting a file php will use the same amount of memory as the size of the file. so, if the file is 100MB php will use 100MB of memory.. is this true? if it is, how can i work around this? Do not use readfile() or file_get_contents() to read the full file in one step. Read file in smaller chunks to a buffer variable whith fread() and output that immediately in the cycle. And again and again... You can read terrabytes of data and utput them, the amount of memory needed is based on your buffer size. -- = Heilig (Cece) Szabolcs - [EMAIL PROTECTED] - http://www.phphost.hu = -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] handling large files w/readfile
Sebastian wrote: i'm working on a app which output files with readfile() and some headers.. i read a comment in the manual that says if your outputting a file php will use the same amount of memory as the size of the file. so, if the file is 100MB php will use 100MB of memory.. is this true? I did a comparision study of readfile() and related functions you will see that results at http://www.raditha.com/wiki/Readfile_vs_include What I discovered was that require,include() and file_get_contents() would consume more memory than the file size. Not so with readfile or fpasstru if it is, how can i work around this? go ahead and use readfile. -- Raditha Dissanayake. -- http://www.radinks.com/print/card-designer/ | Card Designer Applet http://www.radinks.com/upload/ | Drag and Drop Upload -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
