RE: [PHP-DB] Splitting a CSV file
I've never seen where the file size causes a problem that wasn't one of the file size settings either in the upload form or php.ini. I'd focus on figuring out why it's failing on the large files before trying to split it. No errors at all? Just a blank screen? Are you uploading the file through a browser? Ryan -Original Message- From: Matt Babineau [mailto:[EMAIL PROTECTED] Sent: Monday, November 24, 2003 2:37 PM To: [EMAIL PROTECTED] Subject: Re: [PHP-DB] Splitting a CSV file Chris, If you are on a Redhat machine, you could try running a CLI command on this. Try looking up the 'split' command, it may solve your problem if you combine it with some PHP. -Matt On Mon, 2003-11-24 at 19:14, Chris Payne wrote: Hi there everyone, I'm writing an automation system to automatically insert data into a mysql database from a CSV file, and it works great - until I try to insert a large file, then it just doesn't do anything. I've set my PHP filesize to 10 Megs so that's not the issue, and a server timeout isn't the issue either. So, is that a way that I can split a CSV file into 2 files if it's larger than a certain size so that I can still use the automation I am working on? Actually, I won't see the files as it's for a company who just wants to be able to select their CSV file (No matter what size) and it will insert it automatically, and as I said it does work on small files but not large :-( Any help would really be appreciated. Chris -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DB] Splitting a CSV file
try to parse the csv file line by line and you won't have memory problems
ever:
===
$d = \t; // fields delimiter
$fp = fopen ($file_name, r);
while ($arr_row = fgetcsv ($fp, 4096, $d, )) {
do_something ($arr_row);
}
fclose ($fp);
===
(fgetcsv returns an array containing the fields read.)
Tiberiu
- Original Message -
From: Chris Payne [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Sent: Tuesday, November 25, 2003 1:14 AM
Subject: [PHP-DB] Splitting a CSV file
Hi there everyone,
I'm writing an automation system to automatically insert data into a mysql
database from a CSV file, and it works great - until I try to insert a large
file, then it just doesn't do anything.
I've set my PHP filesize to 10 Megs so that's not the issue, and a server
timeout isn't the issue either. So, is that a way that I can split a CSV
file into 2 files if it's larger than a certain size so that I can still use
the automation I am working on?
Actually, I won't see the files as it's for a company who just wants to be
able to select their CSV file (No matter what size) and it will insert it
automatically, and as I said it does work on small files but not large :-(
Any help would really be appreciated.
Chris
--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DB] Splitting a CSV file
You may have hit the browser time-out limit, which you can not control (at
least for IE; don't know about others).
To prevent this, arrange for the script send regularly some invisible
content to the browser
(eg send a HTML comment line every 100 lines of your CSV file)
HTH
Ignatius
_
- Original Message -
From: Chris Payne [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Sent: Tuesday, November 25, 2003 18:54
Subject: Re: [PHP-DB] Splitting a CSV file
Hi there,
I did a test on my local machine, set the upload limit for PHP to 50Megs
for
forms (WAAA more than I need, but then atleast I know that isn't the
problem).
Then, I select the file through my web form, put in the delimiter which
works great on small files. Then, I try the larger file - exactly the
same
format (As I just cut a portion of the file and pasted into another file
about with just about 50 entries to test which worked) and then click
send,
but when I try to echo the result to debug, the array is totally empty,
and
even tells me it is empty, but only when the file is above a certain size
(Not worked out what size that is yet though).
The file itself is around 5.4 megs, and I know it's not a server timeout
issue as i've increased everything I could find in my local Apache config,
from a timeout of 300 to 3000 (Again, just to eliminate all possible
courses
that I can think of). The only thing I can think of is that there is some
illegal char somewhere in the file, but I have no idea what that could be.
I'm using the straight line up as the delimiter to avoid any , issues etc
. as I know that | isn't used in the file at all.
It's just getting to me, there has to be a reason why :-( I'm using the
following code (Minus the bit that enters the DB into the database, as
that
works fine with small files so would with large too):
if ($option == importcsv) {
if ($userfile == ){
$failed = 'yes';
} else {
$row = 1;
$handle = fopen ($userfile,r);
while ($data = fgetcsv ($handle, 10, |)) {
$num = count ($data);
$row++;
echo $data[0];;
};
fclose ($handle);
}};
Chris
I've never seen where the file size causes a problem that wasn't one of
the file size settings either in the upload form or php.ini. I'd focus
on figuring out why it's failing on the large files before trying to
split it. No errors at all? Just a blank screen? Are you uploading the
file through a browser?
Ryan
-Original Message-
From: Matt Babineau [mailto:[EMAIL PROTECTED]
Sent: Monday, November 24, 2003 2:37 PM
To: [EMAIL PROTECTED]
Subject: Re: [PHP-DB] Splitting a CSV file
Chris,
If you are on a Redhat machine, you could try running a CLI command on
this.
Try looking up the 'split' command, it may solve your problem if you
combine it with some PHP.
-Matt
On Mon, 2003-11-24 at 19:14, Chris Payne wrote:
Hi there everyone,
I'm writing an automation system to automatically insert data into a
mysql database from a CSV file, and it works great - until I try to
insert a large file, then it just doesn't do anything.
I've set my PHP filesize to 10 Megs so that's not the issue, and a
server timeout isn't the issue either. So, is that a way that I can
split a CSV file into 2 files if it's larger than a certain size so that
I can still use the automation I am working on?
Actually, I won't see the files as it's for a company who just wants
to be able to select their CSV file (No matter what size) and it will
insert it automatically, and as I said it does work on small files but
not large :-(
Any help would really be appreciated.
Chris
--
PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit:
http://www.php.net/unsub.php
--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DB] Splitting a CSV file
Hey there,
I just want to thank everyone for their time and help, you've all been so
valuable and solved my problems :-) I needed the hidden max_file_size bit
in the form. But i'm also going to look at all your other suggestions
you've all made, as what you say makes perfect sense from a coding point of
view :-)
Again thank you everyone, I don't know what i'd do without you all.
Regards
Chris - Crazy, but slightly less so than this morning :-)
You may have hit the browser time-out limit, which you can not control (at
least for IE; don't know about others).
To prevent this, arrange for the script send regularly some invisible
content to the browser
(eg send a HTML comment line every 100 lines of your CSV file)
HTH
Ignatius
_
- Original Message -
From: Chris Payne [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Sent: Tuesday, November 25, 2003 18:54
Subject: Re: [PHP-DB] Splitting a CSV file
Hi there,
I did a test on my local machine, set the upload limit for PHP to 50Megs
for
forms (WAAA more than I need, but then atleast I know that isn't the
problem).
Then, I select the file through my web form, put in the delimiter which
works great on small files. Then, I try the larger file - exactly the
same
format (As I just cut a portion of the file and pasted into another file
about with just about 50 entries to test which worked) and then click
send,
but when I try to echo the result to debug, the array is totally empty,
and
even tells me it is empty, but only when the file is above a certain
size
(Not worked out what size that is yet though).
The file itself is around 5.4 megs, and I know it's not a server timeout
issue as i've increased everything I could find in my local Apache
config,
from a timeout of 300 to 3000 (Again, just to eliminate all possible
courses
that I can think of). The only thing I can think of is that there is
some
illegal char somewhere in the file, but I have no idea what that could
be.
I'm using the straight line up as the delimiter to avoid any , issues
etc
. as I know that | isn't used in the file at all.
It's just getting to me, there has to be a reason why :-( I'm using the
following code (Minus the bit that enters the DB into the database, as
that
works fine with small files so would with large too):
if ($option == importcsv) {
if ($userfile == ){
$failed = 'yes';
} else {
$row = 1;
$handle = fopen ($userfile,r);
while ($data = fgetcsv ($handle, 10, |)) {
$num = count ($data);
$row++;
echo $data[0];;
};
fclose ($handle);
}};
Chris
I've never seen where the file size causes a problem that wasn't one
of
the file size settings either in the upload form or php.ini. I'd focus
on figuring out why it's failing on the large files before trying to
split it. No errors at all? Just a blank screen? Are you uploading the
file through a browser?
Ryan
-Original Message-
From: Matt Babineau [mailto:[EMAIL PROTECTED]
Sent: Monday, November 24, 2003 2:37 PM
To: [EMAIL PROTECTED]
Subject: Re: [PHP-DB] Splitting a CSV file
Chris,
If you are on a Redhat machine, you could try running a CLI command on
this.
Try looking up the 'split' command, it may solve your problem if you
combine it with some PHP.
-Matt
On Mon, 2003-11-24 at 19:14, Chris Payne wrote:
Hi there everyone,
I'm writing an automation system to automatically insert data into a
mysql database from a CSV file, and it works great - until I try to
insert a large file, then it just doesn't do anything.
I've set my PHP filesize to 10 Megs so that's not the issue, and a
server timeout isn't the issue either. So, is that a way that I can
split a CSV file into 2 files if it's larger than a certain size so
that
I can still use the automation I am working on?
Actually, I won't see the files as it's for a company who just wants
to be able to select their CSV file (No matter what size) and it
will
insert it automatically, and as I said it does work on small files
but
not large :-(
Any help would really be appreciated.
Chris
--
PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit:
http://www.php.net/unsub.php
--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DB] Splitting a CSV file
Chris, et al -- ...and then Chris Payne said... % % Hey there, Hiya! % % I just want to thank everyone for their time and help, you've all been so % valuable and solved my problems :-) I needed the hidden max_file_size bit Glad to hear it, and sorry it took me so long to get here. There's that other thread on the PHP list... *sigh* % in the form. But i'm also going to look at all your other suggestions Yep. I was going to suggest that you need post_max_size upload_max_filesize and perhaps memory_limit max_input_time bumped up from the defaults. A good way to check this sort of thing is to look at phpinfo() output and see what could be keeping you back. In my case, I didn't have _filesize and so I could watch the files spool up in /var/tmp and then suddenly disappear even before the upload finished! HTH HAND :-D -- David T-G * There is too much animal courage in (play) [EMAIL PROTECTED] * society and not sufficient moral courage. (work) [EMAIL PROTECTED] -- Mary Baker Eddy, Science and Health http://justpickone.org/davidtg/ Shpx gur Pbzzhavpngvbaf Qrprapl Npg! pgp0.pgp Description: PGP signature
[PHP-DB] Splitting a CSV file
Hi there everyone, I'm writing an automation system to automatically insert data into a mysql database from a CSV file, and it works great - until I try to insert a large file, then it just doesn't do anything. I've set my PHP filesize to 10 Megs so that's not the issue, and a server timeout isn't the issue either. So, is that a way that I can split a CSV file into 2 files if it's larger than a certain size so that I can still use the automation I am working on? Actually, I won't see the files as it's for a company who just wants to be able to select their CSV file (No matter what size) and it will insert it automatically, and as I said it does work on small files but not large :-( Any help would really be appreciated. Chris
Re: [PHP-DB] Splitting a CSV file
Chris, If you are on a Redhat machine, you could try running a CLI command on this. Try looking up the 'split' command, it may solve your problem if you combine it with some PHP. -Matt On Mon, 2003-11-24 at 19:14, Chris Payne wrote: Hi there everyone, I'm writing an automation system to automatically insert data into a mysql database from a CSV file, and it works great - until I try to insert a large file, then it just doesn't do anything. I've set my PHP filesize to 10 Megs so that's not the issue, and a server timeout isn't the issue either. So, is that a way that I can split a CSV file into 2 files if it's larger than a certain size so that I can still use the automation I am working on? Actually, I won't see the files as it's for a company who just wants to be able to select their CSV file (No matter what size) and it will insert it automatically, and as I said it does work on small files but not large :-( Any help would really be appreciated. Chris -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
