tor, 21,.04.2005 kl. 12.26 +0200, skrev Robert Klemme: > > I do need control over the import. The program I use can handle > > replication; it will add new records, update old and ignore unchanged. > > Ok, understood. Just wanted to make sure all options are on the table.
As for now, I'm using replication on small-volume tables, and text-file imports for medium and large-volume tables. The data-files are prepared by a tiny C++ program, and are read by both MySQL with InnoDB and MaxDB ;) I may choose to do this in production as well, if the performace is alot better than using a db-library. > > 1) There is a limit on 8 connections from one application instance(?) > > I will probably need hundreds of connections, as the servers > > I work on use threads and queued async IO with potantially tens > > of thousands concurrent users. > > With these dimensions you usually use connection pooling and probably want > to limit the number of concurrent connections to achieve optimal > performance. While 8 would be too low a limit 1:1 relationship between > clients of the app and db connections is likely no good either. I have written a C++ library with generic database-access trough connection-pooling. I plan to add support for maxDB. I just looked at the wrong option (the C++ wrapper in stead of the database client library). > > I know. That's why I mentioned it ;) It does however take _alot_ more > > time to insert records into a database with extra indexes. MaxDB use > > only 10 - 15 minutes to import the 135 million rows in my test-dataset > > with the loader, but then spends almost 100 minutes creating an extra > > index. With InnoDB (where the indexes are created prior to the import), > > each extra index adds about 80 - 100 minutes to the import. This is not > > all that much overhead, but when the row-numbers icrease to several > > billons, the extra time becomes significant. > > I'm a bit confused: the only difference I see here is the point in time when > you create the index. Did the numbers got mixed up somehow (100minutes vs > 80-100minutes doesn't sound that much difference to me)? Od did I miss > something here? Did you also do tests under same conditions (i.e. indexes > created prior to import and after import)? I have not timed the single steps, only the totals. I did a test yesterday, where MySQL with InnoDB loaded the 135 million records, with one primary key split over three cols, in 28 minutes. With a single row primary key and and an extra 3-col index the load took 93 minutes. I will do the same test with MaxDB later today. Jarle -- Jarle Aase email: [EMAIL PROTECTED] Author of freeware. http://www.jgaa.com news:alt.comp.jgaa War FTP Daemon: http://www.warftp.org War FTP Daemon FAQ: http://www.warftp.org/faq/warfaq.htm Jgaa's PGP key: http://war.jgaa.com/pgp NB: If you reply to this message, please include all relevant information from the conversation in your reply. Thanks. <<< no need to argue - just kill'em all! >>> -- MaxDB Discussion Mailing List For list archives: http://lists.mysql.com/maxdb To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
