> Date: Thu, 15 Sep 2011 09:26:41 -0700
> From: tigerpeng2...@yahoo.com
> Subject: Re: Tail Module + DBI Module, can\'t keep up!
> To: bphe...@gls.com
> CC: dbi-users@perl.org
>
> Separate the process into two steps and figure out which step should be fixed
> first.
>
> 1. Parse the log with Tail and dump the data to plain file.
> 2. Load the plain file with DBI (are you using batch insert?)
This would be my stratigy as well. you could also give the insert_array at try
but I do not think that will give you much as you are useing MySQL but at lease
you could use one script that
1) Bufferes up X-n records to insert in an array/arrays
2) at buffer X do an insert with execute_array
3) carry on burrering up the array.
one other thing to look out for is to make sure you are not 'prepareing' the
same SQL over and over agian. If you have only 1 query just prepare it once
and reuse tha handle. Also make sure you are not trying to reconnect each time
as both of these actions are rather heavy on resources.
Cheers
John
