Re: [PHP] a better way to do a data import?

2008-01-22 Thread Richard Lynch
If you unset/NULL out *every* variable, then you should not run out of RAM... It might be a heck of a lot faster to LOAD DATA INFILE to a temp table, and then use SQL statements to compare the data and update/insert any altered/missing data... I do something similar with about the same number of

Re: [PHP] a better way to do a data import?

2008-01-21 Thread Eric Butera
On Jan 21, 2008 12:35 PM, blackwater dev [EMAIL PROTECTED] wrote: I have a text file that contains 200k rows. These rows are to be imported into our database. The majority of them will already exists while a few are new. Here are a few options I've tried: I've had php cycle through the

Re: [PHP] a better way to do a data import?

2008-01-21 Thread Robert Cummings
On Mon, 2008-01-21 at 12:35 -0500, blackwater dev wrote: I have a text file that contains 200k rows. These rows are to be imported into our database. The majority of them will already exists while a few are new. Here are a few options I've tried: I've had php cycle through the file row

Re: [PHP] a better way to do a data import?

2008-01-21 Thread blackwater dev
I think that's possible, so I'll give it a shot. For some reason, even with straight inserts my php script is dying around 180,000 rows. Basically, I took out all the compare/update code so now I grab the row from the db and if there isn't one, do an insert. I've wiped my db so should do

Re: [PHP] a better way to do a data import?

2008-01-21 Thread Eric Butera
On Jan 21, 2008 1:08 PM, blackwater dev [EMAIL PROTECTED] wrote: I think that's possible, so I'll give it a shot. For some reason, even with straight inserts my php script is dying around 180,000 rows. Basically, I took out all the compare/update code so now I grab the row from the db and if

RE: [PHP] a better way to do a data import?

2008-01-21 Thread Bastien Koert
what about uploading the entire file into a [semi]temp table..then doing cross table comparisons to load your main table with the data? bastien Date: Mon, 21 Jan 2008 13:08:27 -0500 From: [EMAIL PROTECTED] To: php-general@lists.php.net Subject: Re: [PHP] a better way to do a data import? I

Re: [PHP] a better way to do a data import?

2008-01-21 Thread Chris
blackwater dev wrote: I have a text file that contains 200k rows. These rows are to be imported into our database. The majority of them will already exists while a few are new. Here are a few options I've tried: I've had php cycle through the file row by row and if the row is there, delete