Re: [PHP-DB] Working with large datasets

2011-10-11 Thread Jim Giner
Exactly. That was my first guess - that his commandline request is first having to download 8M records which can take a long time. The OP's fear of overhead from apache... is not only unfounded, but would most definitely improve his response by simply running the query on the server and avoid

Re: [PHP-DB] Working with large datasets

2011-10-11 Thread user080701
-- From: Jack van Zanen j...@vanzanen.com Sent: Tuesday, October 11, 2011 12:09 PM To: Jason Pruim li...@pruimphotography.com Cc: Thompson, Jimi ji...@mail.cox.smu.edu; Bastien phps...@gmail.com; php-db@lists.php.net Subject: Re: [PHP-DB

RE: [PHP-DB] Working with large datasets

2011-10-10 Thread Toby Hart Dyke
It sounds as though you don't have an index on the right field. 8 million records should be no problem if you have the right indexes applied, and you're not trying to do anything too complicated. Toby -Original Message- From: Jason Pruim [mailto:li...@pruimphotography.com] Sent:

Re: [PHP-DB] Working with large datasets

2011-10-10 Thread Jim Giner
Toby Hart Dyke t...@hartdyke.com wrote in message news:00da01cc8768$ca9e9200$5fdbb600$@hartdyke.com... It sounds as though you don't have an index on the right field. 8 million records should be no problem if you have the right indexes applied, and you're not trying to do anything too

Re: [PHP-DB] Working with large datasets

2011-10-10 Thread Jason Pruim
Jason Pruim pru...@gmail.com On Oct 10, 2011, at 2:20 PM, Jim Giner wrote: Toby Hart Dyke t...@hartdyke.com wrote in message news:00da01cc8768$ca9e9200$5fdbb600$@hartdyke.com... It sounds as though you don't have an index on the right field. 8 million records should be no problem if

Re: [PHP-DB] Working with large datasets

2011-10-10 Thread Jim Giner
I don't do command line stuff so I may not be right in my thinking. If you are running a php query from a client, does the query get executed on the database server, or does all the data have to come down to you to be queried? -- PHP Database Mailing List (http://www.php.net/) To

Re: [PHP-DB] Working with large datasets

2011-10-10 Thread Jason Pruim
Jason Pruim pru...@gmail.com On Oct 10, 2011, at 2:42 PM, Jim Giner wrote: I don't do command line stuff so I may not be right in my thinking. If you are running a php query from a client, does the query get executed on the database server, or does all the data have to come down to you

Re: [PHP-DB] Working with large datasets

2011-10-10 Thread Jim Giner
When you say machine, do you mean the client that you're sitting at, or the machine that hosts the data? As for doing it thru a web server, the amount of time Apache, et al, would consume is miniscule. The web interface would not be involved in the reading of the data or the processing, just

RE: [PHP-DB] Working with large datasets

2011-10-10 Thread Toby Hart Dyke
with an index, it has to do a full table scan to retrieve the results. Toby -Original Message- From: Jason Pruim [mailto:pru...@gmail.com] Sent: Monday, October 10, 2011 2:37 PM To: Jim Giner Cc: php-db@lists.php.net Subject: Re: [PHP-DB] Working with large datasets RIght now though I only

Re: [PHP-DB] Working with large datasets

2011-10-10 Thread Bastien
On 2011-10-10, at 11:30 AM, Jason Pruim li...@pruimphotography.com wrote: Hey everyone, I am working with a database that has close to 8 million records in it and it will be growing. I have a state field in the data, and I am attempting to test some query's on it, all but 2 records

RE: [PHP-DB] Working with large datasets

2011-10-10 Thread Thompson, Jimi
: [PHP-DB] Working with large datasets On 2011-10-10, at 11:30 AM, Jason Pruim li...@pruimphotography.com wrote: Hey everyone, I am working with a database that has close to 8 million records in it and it will be growing. I have a state field in the data, and I am attempting to test some

Re: [PHP-DB] Working with large datasets

2011-10-10 Thread Karl DeSaulniers
PM To: Jason Pruim Cc: php-db@lists.php.net Subject: Re: [PHP-DB] Working with large datasets On 2011-10-10, at 11:30 AM, Jason Pruim li...@pruimphotography.com wrote: Hey everyone, I am working with a database that has close to 8 million records in it and it will be growing. I have

Re: [PHP-DB] Working with large datasets

2011-10-10 Thread Jason Pruim
Jason Pruim li...@pruimphotography.com On Oct 10, 2011, at 5:27 PM, Thompson, Jimi wrote: I really think that you should try running it from the command line and see what the issues are. Get both Apache and php out of the way. I've seen some PHP scripts use up all the file handles (OS

Re: [PHP-DB] Working with large datasets

2011-10-10 Thread Jack van Zanen
Hi You need to index the right fields. even on a laptop a select from 8 million rows with two rows returned should take a few seconds max only. The first time you run the query the data has to come from disk, second time you run same query you'd expect that data to sit in cache and be very