Re: [PHP-DB] Splitting a CSV file
Chris, et al -- ...and then Chris Payne said... % % Hey there, Hiya! % % I just want to thank everyone for their time and help, you've all been so % valuable and solved my problems :-) I needed the hidden max_file_size bit Glad to hear it, and sorry it took me so long to get here. There's that other thread on the PHP list... *sigh* % in the form. But i'm also going to look at all your other suggestions Yep. I was going to suggest that you need post_max_size upload_max_filesize and perhaps memory_limit max_input_time bumped up from the defaults. A good way to check this sort of thing is to look at phpinfo() output and see what could be keeping you back. In my case, I didn't have _filesize and so I could watch the files spool up in /var/tmp and then suddenly disappear even before the upload finished! HTH & HAND :-D -- David T-G * There is too much animal courage in (play) [EMAIL PROTECTED] * society and not sufficient moral courage. (work) [EMAIL PROTECTED] -- Mary Baker Eddy, "Science and Health" http://justpickone.org/davidtg/ Shpx gur Pbzzhavpngvbaf Qrprapl Npg! pgp0.pgp Description: PGP signature
Re: [PHP-DB] Splitting a CSV file
Hey there, I just want to thank everyone for their time and help, you've all been so valuable and solved my problems :-) I needed the hidden max_file_size bit in the form. But i'm also going to look at all your other suggestions you've all made, as what you say makes perfect sense from a coding point of view :-) Again thank you everyone, I don't know what i'd do without you all. Regards Chris - Crazy, but slightly less so than this morning :-) > You may have hit the browser time-out limit, which you can not control (at > least for IE; don't know about others). > > To prevent this, arrange for the script send regularly some invisible > content to the browser > (eg send a HTML comment line every 100 lines of your CSV file) > > HTH > Ignatius > _ > - Original Message - > From: "Chris Payne" <[EMAIL PROTECTED]> > To: <[EMAIL PROTECTED]> > Sent: Tuesday, November 25, 2003 18:54 > Subject: Re: [PHP-DB] Splitting a CSV file > > > > Hi there, > > > > I did a test on my local machine, set the upload limit for PHP to 50Megs > for > > forms (WAAA more than I need, but then atleast I know that isn't the > > problem). > > > > Then, I select the file through my web form, put in the delimiter which > > works great on small files. Then, I try the larger file - exactly the > same > > format (As I just cut a portion of the file and pasted into another file > > about with just about 50 entries to test which worked) and then click > send, > > but when I try to echo the result to debug, the array is totally empty, > and > > even tells me it is empty, but only when the file is above a certain size > > (Not worked out what size that is yet though). > > > > The file itself is around 5.4 megs, and I know it's not a server timeout > > issue as i've increased everything I could find in my local Apache config, > > from a timeout of 300 to 3000 (Again, just to eliminate all possible > courses > > that I can think of). The only thing I can think of is that there is some > > illegal char somewhere in the file, but I have no idea what that could be. > > I'm using the straight line up as the delimiter to avoid any , issues etc > > . as I know that | isn't used in the file at all. > > > > It's just getting to me, there has to be a reason why :-( I'm using the > > following code (Minus the bit that enters the DB into the database, as > that > > works fine with small files so would with large too): > > > > if ($option == "importcsv") { > > > > if ($userfile == ""){ > > $failed = 'yes'; > > } else { > > > > $row = 1; > > $handle = fopen ("$userfile","r"); > > while ($data = fgetcsv ($handle, 10, "|")) { > > $num = count ($data); > > $row++; > > > > echo "$data[0];"; > > > > }; > > fclose ($handle); > > > > }}; > > > > Chris > > > > > I've never seen where the file size causes a problem that wasn't one of > > > the file size settings either in the upload form or php.ini. I'd focus > > > on figuring out why it's failing on the large files before trying to > > > split it. No errors at all? Just a blank screen? Are you uploading the > > > file through a browser? > > > > > > <>< Ryan > > > > > > -Original Message- > > > From: Matt Babineau [mailto:[EMAIL PROTECTED] > > > Sent: Monday, November 24, 2003 2:37 PM > > > To: [EMAIL PROTECTED] > > > Subject: Re: [PHP-DB] Splitting a CSV file > > > > > > Chris, > > > > > > If you are on a Redhat machine, you could try running a CLI command on > > > this. > > > > > > Try looking up the 'split' command, it may solve your problem if you > > > combine it with some PHP. > > > > > > -Matt > > > > > > On Mon, 2003-11-24 at 19:14, Chris Payne wrote: > > > > Hi there everyone, > > > > > > > > I'm writing an automation system to automatically insert data into a > > > mysql database from a CSV file, and it works great - until I try to > > > insert a large file, then it just doesn't do anything. > > > > > > > > I've set my PHP filesize to 10 Megs so that's not the issue, and a > > > server timeout isn't the issue either. So, is that a way that I can > > > split a CSV fi
Re: [PHP-DB] Splitting a CSV file
You may have hit the browser time-out limit, which you can not control (at least for IE; don't know about others). To prevent this, arrange for the script send regularly some invisible content to the browser (eg send a HTML comment line every 100 lines of your CSV file) HTH Ignatius _ - Original Message - From: "Chris Payne" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Tuesday, November 25, 2003 18:54 Subject: Re: [PHP-DB] Splitting a CSV file > Hi there, > > I did a test on my local machine, set the upload limit for PHP to 50Megs for > forms (WAAA more than I need, but then atleast I know that isn't the > problem). > > Then, I select the file through my web form, put in the delimiter which > works great on small files. Then, I try the larger file - exactly the same > format (As I just cut a portion of the file and pasted into another file > about with just about 50 entries to test which worked) and then click send, > but when I try to echo the result to debug, the array is totally empty, and > even tells me it is empty, but only when the file is above a certain size > (Not worked out what size that is yet though). > > The file itself is around 5.4 megs, and I know it's not a server timeout > issue as i've increased everything I could find in my local Apache config, > from a timeout of 300 to 3000 (Again, just to eliminate all possible courses > that I can think of). The only thing I can think of is that there is some > illegal char somewhere in the file, but I have no idea what that could be. > I'm using the straight line up as the delimiter to avoid any , issues etc > . as I know that | isn't used in the file at all. > > It's just getting to me, there has to be a reason why :-( I'm using the > following code (Minus the bit that enters the DB into the database, as that > works fine with small files so would with large too): > > if ($option == "importcsv") { > > if ($userfile == ""){ > $failed = 'yes'; > } else { > > $row = 1; > $handle = fopen ("$userfile","r"); > while ($data = fgetcsv ($handle, 10, "|")) { > $num = count ($data); > $row++; > > echo "$data[0];"; > > }; > fclose ($handle); > > }}; > > Chris > > > I've never seen where the file size causes a problem that wasn't one of > > the file size settings either in the upload form or php.ini. I'd focus > > on figuring out why it's failing on the large files before trying to > > split it. No errors at all? Just a blank screen? Are you uploading the > > file through a browser? > > > > <>< Ryan > > > > -Original Message- > > From: Matt Babineau [mailto:[EMAIL PROTECTED] > > Sent: Monday, November 24, 2003 2:37 PM > > To: [EMAIL PROTECTED] > > Subject: Re: [PHP-DB] Splitting a CSV file > > > > Chris, > > > > If you are on a Redhat machine, you could try running a CLI command on > > this. > > > > Try looking up the 'split' command, it may solve your problem if you > > combine it with some PHP. > > > > -Matt > > > > On Mon, 2003-11-24 at 19:14, Chris Payne wrote: > > > Hi there everyone, > > > > > > I'm writing an automation system to automatically insert data into a > > mysql database from a CSV file, and it works great - until I try to > > insert a large file, then it just doesn't do anything. > > > > > > I've set my PHP filesize to 10 Megs so that's not the issue, and a > > server timeout isn't the issue either. So, is that a way that I can > > split a CSV file into 2 files if it's larger than a certain size so that > > I can still use the automation I am working on? > > > > > > Actually, I won't see the files as it's for a company who just wants > > > to be able to select their CSV file (No matter what size) and it will > > > insert it automatically, and as I said it does work on small files but > > > > > not large :-( > > > > > > Any help would really be appreciated. > > > > > > Chris > > > > -- > > PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: > > http://www.php.net/unsub.php > > > > -- > > PHP Database Mailing List (http://www.php.net/) > > To unsubscribe, visit: http://www.php.net/unsub.php > > > > > > -- > PHP Database Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DB] Splitting a CSV file
try to parse the csv file line by line and you won't have memory problems ever: === $d = "\t"; // fields delimiter $fp = fopen ($file_name, "r"); while ($arr_row = fgetcsv ($fp, 4096, $d, "")) { do_something ($arr_row); } fclose ($fp); === (fgetcsv returns an array containing the fields read.) Tiberiu - Original Message - From: "Chris Payne" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Tuesday, November 25, 2003 1:14 AM Subject: [PHP-DB] Splitting a CSV file Hi there everyone, I'm writing an automation system to automatically insert data into a mysql database from a CSV file, and it works great - until I try to insert a large file, then it just doesn't do anything. I've set my PHP filesize to 10 Megs so that's not the issue, and a server timeout isn't the issue either. So, is that a way that I can split a CSV file into 2 files if it's larger than a certain size so that I can still use the automation I am working on? Actually, I won't see the files as it's for a company who just wants to be able to select their CSV file (No matter what size) and it will insert it automatically, and as I said it does work on small files but not large :-( Any help would really be appreciated. Chris -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DB] Splitting a CSV file
Hi there, I did a test on my local machine, set the upload limit for PHP to 50Megs for forms (WAAA more than I need, but then atleast I know that isn't the problem). Then, I select the file through my web form, put in the delimiter which works great on small files. Then, I try the larger file - exactly the same format (As I just cut a portion of the file and pasted into another file about with just about 50 entries to test which worked) and then click send, but when I try to echo the result to debug, the array is totally empty, and even tells me it is empty, but only when the file is above a certain size (Not worked out what size that is yet though). The file itself is around 5.4 megs, and I know it's not a server timeout issue as i've increased everything I could find in my local Apache config, from a timeout of 300 to 3000 (Again, just to eliminate all possible courses that I can think of). The only thing I can think of is that there is some illegal char somewhere in the file, but I have no idea what that could be. I'm using the straight line up as the delimiter to avoid any , issues etc . as I know that | isn't used in the file at all. It's just getting to me, there has to be a reason why :-( I'm using the following code (Minus the bit that enters the DB into the database, as that works fine with small files so would with large too): if ($option == "importcsv") { if ($userfile == ""){ $failed = 'yes'; } else { $row = 1; $handle = fopen ("$userfile","r"); while ($data = fgetcsv ($handle, 10, "|")) { $num = count ($data); $row++; echo "$data[0];"; }; fclose ($handle); }}; Chris > I've never seen where the file size causes a problem that wasn't one of > the file size settings either in the upload form or php.ini. I'd focus > on figuring out why it's failing on the large files before trying to > split it. No errors at all? Just a blank screen? Are you uploading the > file through a browser? > > <>< Ryan > > -----Original Message----- > From: Matt Babineau [mailto:[EMAIL PROTECTED] > Sent: Monday, November 24, 2003 2:37 PM > To: [EMAIL PROTECTED] > Subject: Re: [PHP-DB] Splitting a CSV file > > Chris, > > If you are on a Redhat machine, you could try running a CLI command on > this. > > Try looking up the 'split' command, it may solve your problem if you > combine it with some PHP. > > -Matt > > On Mon, 2003-11-24 at 19:14, Chris Payne wrote: > > Hi there everyone, > > > > I'm writing an automation system to automatically insert data into a > mysql database from a CSV file, and it works great - until I try to > insert a large file, then it just doesn't do anything. > > > > I've set my PHP filesize to 10 Megs so that's not the issue, and a > server timeout isn't the issue either. So, is that a way that I can > split a CSV file into 2 files if it's larger than a certain size so that > I can still use the automation I am working on? > > > > Actually, I won't see the files as it's for a company who just wants > > to be able to select their CSV file (No matter what size) and it will > > insert it automatically, and as I said it does work on small files but > > > not large :-( > > > > Any help would really be appreciated. > > > > Chris > > -- > PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: > http://www.php.net/unsub.php > > -- > PHP Database Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP-DB] Splitting a CSV file
I've never seen where the file size causes a problem that wasn't one of the file size settings either in the upload form or php.ini. I'd focus on figuring out why it's failing on the large files before trying to split it. No errors at all? Just a blank screen? Are you uploading the file through a browser? <>< Ryan -Original Message- From: Matt Babineau [mailto:[EMAIL PROTECTED] Sent: Monday, November 24, 2003 2:37 PM To: [EMAIL PROTECTED] Subject: Re: [PHP-DB] Splitting a CSV file Chris, If you are on a Redhat machine, you could try running a CLI command on this. Try looking up the 'split' command, it may solve your problem if you combine it with some PHP. -Matt On Mon, 2003-11-24 at 19:14, Chris Payne wrote: > Hi there everyone, > > I'm writing an automation system to automatically insert data into a mysql database from a CSV file, and it works great - until I try to insert a large file, then it just doesn't do anything. > > I've set my PHP filesize to 10 Megs so that's not the issue, and a server timeout isn't the issue either. So, is that a way that I can split a CSV file into 2 files if it's larger than a certain size so that I can still use the automation I am working on? > > Actually, I won't see the files as it's for a company who just wants > to be able to select their CSV file (No matter what size) and it will > insert it automatically, and as I said it does work on small files but > not large :-( > > Any help would really be appreciated. > > Chris -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DB] Splitting a CSV file
Chris, If you are on a Redhat machine, you could try running a CLI command on this. Try looking up the 'split' command, it may solve your problem if you combine it with some PHP. -Matt On Mon, 2003-11-24 at 19:14, Chris Payne wrote: > Hi there everyone, > > I'm writing an automation system to automatically insert data into a mysql database > from a CSV file, and it works great - until I try to insert a large file, then it > just doesn't do anything. > > I've set my PHP filesize to 10 Megs so that's not the issue, and a server timeout > isn't the issue either. So, is that a way that I can split a CSV file into 2 files > if it's larger than a certain size so that I can still use the automation I am > working on? > > Actually, I won't see the files as it's for a company who just wants to be able to > select their CSV file (No matter what size) and it will insert it automatically, and > as I said it does work on small files but not large :-( > > Any help would really be appreciated. > > Chris -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php