When I import gedcom files into my genealogy software, I can read in
hundred of thousands of lines, do complex algorithms on each line,
create different objects on different variables and such, all in all
it takes a few seconds, perhaps like 30 doing everything.
Trausti
On Jan 16, 2007, at 7:18 PM, Mark Lubratt wrote:
On Jan 16, 2007, at 11:56 AM, Tim Jones wrote:
On Jan 16, 2007, at 10:38 AM, Kevin Windham wrote:
On Jan 16, 2007, at 11:32 AM, Tim Jones wrote:
On Jan 15, 2007, at 5:04 PM, Tim Jones wrote:
Hi Folks,
I've stumbled my way through the setup of a REALSQL database
and all seems to be working --- except that it's slow as
molasses in January (in the Northern hemisphere :) ) while
importing a large flat text file.
I have a text file that contains approximately 120K lines that
are separated by vertical bar symbols (pipes - | ). I parse
the data out of each line using NthField. The result is over
45 minutes to parse the flat file into two datatables on a G5
1.8GHz. Any thoughts to speed this up? The relevant code is
below:
No REALSQL Performance fans out there?
I've even taken this further worrying that the NthField
manipulation might be the issue and changed the line parsing to
use an array and the Split command, but the time is the same
implying that the issue is in the actual commit of the data to
the database.
Have you tried removing the DB code to see what the difference in
speed is? That sounds like unbelievably terrible performance.
Hi Kevin,
Yes, as a matter of fact, I have :). The difference is simply
parsing the file and splitting the line (or using NthField), the
read takes around 30 seconds on my PowerBook and 13 seconds on my
Mac Pro. Sorry I didn't mention that as it was one of the first
places I checked.
Are you doing each line as a separate transaction? Try
encapsulating about 10K lines within each transaction. I've found
that this trick speeds things up a lot.
HTH!
Mark
_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>
Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>
_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>
Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>
