Did you try using a ZFS file system? On my system freads()/fwrites() to
a unix file are significantly faster than QSAM/BSAM.
On 5/06/2013 9:33 AM, Bernd Oppolzer wrote:
Two weeks ago, I told you about tests with our table system,
which holds read-only data for our insurance math package.
The data was stored in DB2 tables until now, and we tried to get better
CPU usage etc. by moving the data to our file based table system, which
we have in the Windows and Unix environments.
First tests showed a reduction in CPU time, but an increased elapsed
time,
due to I/O waits during fseek / fread calls. See below.
Now we examined this more deeply, and, as it turned out, the main storage
cache simply didn't work due to configuration errors. This was the
reason for
the many file I/Os that occured and for the massive I/O waits.
We fixed this, and we set the cache size to 9 MB (the table sizes are
some
hundred MB). Furthermore, the file attribute have to be RECFM = FBS,
to support the fseek operations in an optimal way.
Doing this, we got the following results:
CPU time and elapsed time is reduced to about 50 % of the original value;
and that is not only the time for the table access, but for the whole
computation !!
That means, that the reduction in table access times must me still
much higher.
This can be explained by the following things:
a) because the table system and the cache is in the same address space as
the application, we have no address space switching traffic, as we
have with
the DB2 solution
b) DB2 is used by other requesters, too
c) because the tables are read-only, the table system does nothing with
respect to transaction control etc, no locking and logging, which makes
it faster than DB2
Of course, the drawback is:
you have to spend 9 MB cache storage more in every region where the
application
runs. But that's so little, that overall system control has no problem
with it.
We do a mass test tomorrow to check out the behaviour of the system,
if there
are many computations in a large number of parallel jobs. If the
results are the same,
we will migrate the system to the file based solution in the near future.
Kind regards
Bernd
Am 21.05.2013 16:44, schrieb Bernd Oppolzer:
BTW:
for another customer (also z/OS based), we put all the data in a data
space,
using a 3rd party system that the other customer provided. No problem
there.
For Windows server, the application runs multi-threaded, and the
cache data
of the table system needs to be serialized; this is done by setting
semaphores
when entering the cache handling routines. Works without problems. So
one thread
uses the cache data that another thread did read.
For Linux server, we first tried the same approach. But the semaphore
calls in
pthread turned out to be so expensive (much more expensive than the
code to
examine the cache), that the application didn't run well with Linux.
So in this case
we mapped the whole containers to memory using mmap at the beginning of
the process and simply didn't use file I/O at all.
The z/OS applications are batch regions, running all day long. With
DB2, the
cache is the database DBM1 area. But there is some overhead doing the
communication from the batch regions to DB2. With the file I/O, there
is no
DB2 usage and no overhead, but now the cache is in the batch regions.
It counts
for some MB only, so this is no big problem. The problem is, that the
CPU reduction
(which is significant) leads to an increase in elapsed time, because
of the
file I/O waits.
What we will try next:
- of course: look for optimizations in file I/O
- increase cache size to reduce file I/O (can be set using
environment variables)
- experiment with data space solutions (like above)
Kind regards
Bernd
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO IBM-MAIN
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO IBM-MAIN
