Ahh, I did it like this for the MySQL inserts - I had over 100mil records to put into the MySQL DB; doing them 8000 at a time was much faster than one at a time. True, there are faster ways to accomplish this, however, I had PHP available on this server, and didn't want to install un-needed software (it's bare bones MySQL with PHP for Crons). This is just a ref. for those trying to also accomplish this in PHP.
-Will On Tue, Sep 1, 2009 at 6:14 PM, Michael Torrie <[email protected]> wrote: > William Attwood wrote: > > this takes in each line from STDIN, adds it to an array, and when the > array > > hits 8000 (my memory limit at the time) it sends the array to a PHP > function > > that will process and input it into the DB I am using. > > > > Just in case anyone needs to process large files, stream them in > > > > # more file.log | php process.php > > I'm a little confused as to why you don't just process the file one line > at a time with little or no memory consumption (file reads are normally > buffered anyway, so reading until a line break is not a bottleneck). > Why the big buffer? I don't see any speed increases coming from that. > > Also sounds like Perl or Python would be a much better fit for your > little problem. PHP seems like a kludge in this particular case. In > Python, it's a matter of: > > for line in open(file): > dosomething_with(line) > > Maybe off-topic now, but if you need to do operations you'd normally do > in Bash with lots of pipes, you can use generators: > http://www.dabeaz.com/generators/ > > def my_grep(input_generator): > for line in input_generator: > if re.match(expression, line): > yield line > > > for line in my_grep(open(file)): > print 'pattern found in %s' % line > > If you string generators together intelligently you can probably match > the speed of Bash and friends, and it certainly is simple. > > > > > /* > PLUG: http://plug.org, #utah on irc.freenode.net > Unsubscribe: http://plug.org/mailman/options/plug > Don't fear the penguin. > */ > -- Take care, William Attwood Idea Extraordinaire [email protected] Charles de Gaulle<http://www.brainyquote.com/quotes/authors/c/charles_de_gaulle.html> - "The better I get to know men, the more I find myself loving dogs." /* PLUG: http://plug.org, #utah on irc.freenode.net Unsubscribe: http://plug.org/mailman/options/plug Don't fear the penguin. */
