Keywords: DataAccWG Hi Tim,
Do we really have to use ascii for ingest? This really ups the needed I/O bandwidth. Is decimal arithmetic next? :-)
Yes, we have to use ascii I'm afraid. It used to be a hot topic some time ago (on the internal slac-llnl mailing list I guess). The fastest way to do bulk ingest in MySQL is through ascii file (20x faster than any other method). I explicitly asked mysql developers about ingesting from pre-formatted binary files and they said it's not possible and they have no plans to implement that. Their argument is that the binary format is faster only if we ingest native types (e.g. float), and all non-native types would still have to be converted to mysql-specific internal format. Also, they claim their ascii-based ingest is way faster then other databases even if binary files are used. They claim the time spend on conversion is negligible comparing to other operations that have to be done. I am not sure I buy that argument 100% (converting billions of numbers to ascii and converting them back to binary can't be quite non-negligible). We decided to verify the performance of mysql and sql server (sql server: ascii and binary ingests), and then (maybe) raise this issue with mysql again. Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
