Re: [PHP] Re: My SQL speed.
On August 3, 2002 12:54 am, Jason Stechschulte wrote: On Sat, Aug 03, 2002 at 01:49:10AM -0300, Manuel Lemos wrote: Google has 1 billion pages and qurys in a few milliseconds... Real search engines do not use SQL databases. What do search engines use? Is there something out there that explains how they work? Generally search egines use various hash algorithms to store their data, such as B-trees, Hash Tables, etc... Even that, is not enough when dealing with an extremely large dataset, in which case expensive hardware is used to provide the necessary IO capacity to allow for fast look ups. If you are trully interested in how search engines do this, there are plenty of articles describing Google's setup in terms of hardware. As far as fetching data from large MySQL databases it is not impossible or slow as some people claim. I have a 4 million record database in MySQL that is routinely accessed and most queries on that table are completed in 0.01-0.03 seconds, speed mostly depending on the number of rows retrieved. Keep in mind that your system and database configuration will play a very large role in regard to performance once you go beyond a certain amount of data. -- Ilia Alshanetsky [EMAIL PROTECTED] http://fud.prohost.org/forum/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Re: My SQL speed.
Hello, On 08/03/2002 01:36 AM, Lord Loh. wrote: I have a database of mysql with 3 million records. No query is performed in less than 10 seconds!(With Index and all that) The index will not do magic by itself, especially if it is not unique. Any way to speed this up ? Sometimes your queries need to be rewritten How on earth does US Social Security dept. maintain so many records ? Most likely they use Oracle or something as sophisticated that among other things can do table partitioning. Google has 1 billion pages and qurys in a few milliseconds... Real search engines do not use SQL databases. I am not saying that they are using MySQL! Should this be asked in a mysql list ? Please suggest a NNTP server! Yes, MySQL is good but it is not magic. You can stretch it a bit with some optimization of your setup, but what often works better is rethinking your application. Does it really need to store 3 million records? -- Regards, Manuel Lemos -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: My SQL speed.
On Sat, Aug 03, 2002 at 01:49:10AM -0300, Manuel Lemos wrote: Google has 1 billion pages and qurys in a few milliseconds... Real search engines do not use SQL databases. What do search engines use? Is there something out there that explains how they work? -- Jason Stechschulte [EMAIL PROTECTED] http://www.ypisco.com -- We demand rigidly defined areas of doubt and uncertainty! -- Vroomfondel -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: My SQL speed.
Hello, On 08/03/2002 01:54 AM, Jason Stechschulte wrote: On Sat, Aug 03, 2002 at 01:49:10AM -0300, Manuel Lemos wrote: Google has 1 billion pages and qurys in a few milliseconds... Real search engines do not use SQL databases. What do search engines use? Is there something out there that explains how they work? They usually use read only DBM like databases that do not have the overhead or parsing SQL or doing joins to fetch the data. -- Regards, Manuel Lemos -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Re: My SQL speed.
Lord Loh. writes: How on earth does US Social Security dept. maintain so many records ? Google has 1 billion pages and qurys in a few milliseconds... Weel, I work as a consultant on development of a similar system for Italian Government. (instead of Social Security numbers it handles all territoric matters and related payments, ownershipd to it... etc ...). BTW: software is interfaced via PHP. In this database, we have over a few billions of records in some few thousands databases hosted on some few hundreds servers physically located in some few dozens of regions of Italy. We use Oracle for it. And, neigher Oracle does magic here. The system is deeply thought and organized in all its details. Nothing is left for a case and everything is very well monitored and backed up. (not mentioning the synchronization methods). In my experience, mySQL has always failed performance-wise (when not crashed complitely) while trying to keep on a database consisting of 1.000.000+ records. For large DBs I would reccomend you PostgreSQL as it has, in my own opinion (and not to start a new thread here) a better ralational mechanism, which is crucial (as Manuel mentioned) to design the right logic within your application. TIP: look for repeating data in your DB, and try to compress it somehow. Try to see if you can split and reuse the records. Add more supporting tables and so on Maxim Maletsky PHP Beginner (www.phpbeginner.com) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
