Dennis's insight is apt. Just as you would eliminate "common sub
expressions" in your programs (or have an optimizer do it for you) it is
a good idea to do the same with systems.
Writing numbers in an endian agnostic manner can easily be done on your
flat file.
Dennis Cote wrote:
Ken wrote:
I do have the option of either putting the data into sqlite at
the start (when its read) or putting it into a flat file and then
Later loading it into a sqlite db via a downstream job.
A great deal of the data columns are simple numeric values and
thats where sqlite really shines in that is portable between systems
having differing endianness.
Here is a summary of the entire processing where a1/b1 are different
processes and probably differnt host platforms.
a1, read from producing system, generate output data (be it flat
file or sqlite).
a2. Compress data file.
a3. Transfer compressed datafile to target system.
a4. Goto a1
b1. Receive datafile and uncompress.
b2. Read datafile and load into Master DB. (This might just be a
simple attach).
b3. Massage data in Master db ???
b4. Read and process data from MasterDb. Delete or mark as deleteable.
b5. Delete processed data from MasterDb. (this could be in a
seperate thread).
b6. Goto step b1.
The nice thing about simply attaching as a DB in step b2 is that
when all data is processed from step b4 then step b5 to purge is a
simple detach and operating system unlink for the underlying datafile.
Which I suspect will be infinately faster than a sql delete command.
Ken,
If you are always going to move the data into an sqlite database at some
point, then I would think you would want to do that on system a when the
data is read. Unless system a is so much slower than system b that the
extra work is a burden on system a, whereas it is insignificant on
system b.
In either case your timing of the exchange file write function in system
a only is incomplete. You are neglecting the time it takes system b to
open and read the exchange file. If you sum those two times then you can
see which solution provides the fastest overall delivery of the data to
system b in the required format.
Dennis Cote
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
