Henrik, Regarding parsing: The way LOAD DATA LOCAL INFILE works currently, it > sends the CSV file to server and there it is stored in /tmp/, read, > parsed and fed directly into the table. Possibly this is more > efficient than if the client would rewrite CSV data into SQL INSERTs, > then the server has to parse the SQL anyway, and SQL is quite wordy. > LOAD DATA is very efficient, I've managed to get 100k rows inserted by > second (small rows). I'm not sure you can easily get that with > INSERTs, (but maybe if you combine a lot of rows into one INSERT). > > It seems to me ideally you could just modify the current code so that > it will never read a file from the server, but instead it could > consume CSV data coming in from the client without saving it as a > temporary file first. >
Are you certain about any of this? I haven't looked at the code in a long time, but when I refactored the local infile code in the C API so many years ago, this is exactly what was happening. No temporarily file was created on the server at all. This is one of the biggest reasons to use it -- it doesn't require any extra space, permissions, or anything, on the server side. It's just a compact (if painful to escape) way to bulk-load data. The way it was actually implemented was: 1. Client sends server LOAD DATA LOCAL INFILE "string". 2. Server sends client response requesting data to be sent and passes the provided "string" back to the client. 3. Client by default opens file specified by "string". 4. Client sends server blocks of data until EOF. 5. Server parses stream and loads data into the table parsing each line and delimited row according to the parameters passed to initial command. The refactor that I did was to refactor the client-side code to use a callback function to supply the data blocks, and refactor file-reading code into a default callback function. The documentation for this is under mysql_set_local_infile_handler()<http://dev.mysql.com/doc/refman/5.0/en/mysql-set-local-infile-handler.html> . I would, however, be happy to see the server-side file reading code removed (along with select into outfile). Regards, Jeremy
_______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
