> > From: Leyne, Sean
> > I need to extract the BI data as a true "snapshot" of data (ensuring FKs are
> valid), in as short a timeframe as possible.
> >
> > Because runtime is critical, I want to break the extract process into
> > logical pieces and run each piece is a separate process/thread (with its own
> connection) (aka run in parallel).
>
> Have you done some speed tests, to verify that the concept could really
> provide expected speed boost ?
Not directly.
I can tell you that a single database connection cannot extract data from the
engine fast enough to saturate even a slow SDD, never mind a PCIe SSD or RAID
of SSDs.
I see this today at our client installs, which have 4-5 HDDs in RAID 5.
You can see this for yourself using the following SQL:
SELECT COUNT(*) FROM (SELECT * FROM HugeTable)
If you look at the CPU usage for the FB process and find that the process is
using the equivalent of a single CPU/core, then FB is running as fast as can.
A single query/select runs in a single process/thread, so can never use more
than 1 core) and has not saturated the storage (whether HHD, SSD or PCIe).
Only when the FB CPU is lower is FB waiting for data from storage.
Also, remember in real world applications, the server prepares a batch (aka
buffer) of rows and sends them to the clients, which the application then walks
and then asks for the next batch. So, a single connection will always have
'idle' periods while the client application is working. So, only multiple
connections can truly saturate a storage system.
Sean
------------------------------------------------------------------------------
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
