I like the feedback, I had forgotten all about LOAD DATA INFILE. I'd like to focus more that this is for any large file processing, not only just database. Keep the feedback coming. I just finished putting 79m apache log records into the system. I'll duplicate this using the load data setup and see how fast it is.
--Will On Wed, Sep 2, 2009 at 3:38 PM, Nicholas Leippe <[email protected]> wrote: > I totally agree, and have used this technique many times for data imports. > > Data scrubbing can be messy, but once it's in a table, SQL is so powerful > it > can make an initially daunting-in-appearance task very manageable. > > As a worst case scenario, you may need to use sed/awk/perl/(or gasp, maybe > php) to filter the file if it's not properly delimited. But it's often > _much_ > faster to filter the file on disk first and then use a LOAD DATA INFILE > than > to even batch it in via INSERT queries. > > I haven't tried this, but you might be able to save yourself the extra copy > if > you need to filter it this way by writing the filter output to a named pipe > and LOAD DATA INFILE on the pipe... > > > > /* > PLUG: http://plug.org, #utah on irc.freenode.net > Unsubscribe: http://plug.org/mailman/options/plug > Don't fear the penguin. > */ > -- Take care, William Attwood Idea Extraordinaire [email protected] Jonathan Swift<http://www.brainyquote.com/quotes/authors/j/jonathan_swift.html> - "May you live every day of your life." /* PLUG: http://plug.org, #utah on irc.freenode.net Unsubscribe: http://plug.org/mailman/options/plug Don't fear the penguin. */
