I'm sure the Store API will need to change some as we figure this
out. I was planning on (at least) three implementations: a memory
store; a jdbc store; and a journal store. The first two I checked
in and the latter I have on my laptop waiting to check in. The
memory store will be very fast but non-durable and non-reliable.
The jdbc store will be dog slow ;-) but very reliable since it
persists each instance in its own transaction without any
boxcaring, batching or SQL-reordering. We could implement a JPA-
based store which will give us batching and SQL re-ordering but
then we loose some reliability since writes are not necessarily
flushed (i.e. forced) and reported to the client. At that point,
it seems like the overhead of using a database is superfluous
given we would have guaranteed "ACI" but not "D".
Would a RDB DAS-based store make sense? If disconnected SDO's
(static or dynamic) could provide any benefit then you might
consider this.
Hi Kevin,
I thought about a disconnected solution but I'm not sure it is what
we need. I would characterize our main persistence requirement as
maximizing throughput in the following scenarios:
- No durability (and hence no reliability) for high-volume scenarios
that involve non-mission critical operations
- Durability with absolute reliability and full recovery (given the
right hardware), i.e. no chance of data loss.
- Durability with no reliability. For this to be more useful than #1,
there should be a very limited window where data loss is possible.
I'm not sold on whether this is really useful since generally
applications require reliability or they don't, and it typically is
not the case that "kind of reliable" is acceptable.
For situations where data loss cannot be tolerated we need to either
commit with a database transaction or do our own forced disk write,
bypassing any OS caches (e.g. through HOWL) as data loss could occur
during hardware failure. The jdbc store uses a database and performs
each write within its own transaction, which is really slow. The
journal store writes to a binary log and only does a forced write for
the final record block (all other blocks are written asynchronously),
providing high throughput. I did some preliminary performance tests
on a dual-core laptop using 1,000 threads running inside my IDE. For
writing 42byte records consisting of raw binary information, the
store performed a 1,000 forced writes in 337ms. I then ran a similar
test but included serializing 163 byte POJO with header information
and got 1,000 writes in 469ms. I would expect this could be tuned and
we would get a lot better performance, particularly if we did it on
real hardware. I also thought about adding a capability to the jdbc
store where it boxcared writes within a single transaction but at
that point I think the best solution is to just use a journal and
avoid the overhead of a database entirely.
In other scenarios potential data loss may be tolerable, e.g. the
client performs writes to the persistence provider but the provider
doesn't flush until after the write request returns (e.g. JPA, or
Hibernate). This may be because the persistence provider can batch
the updates and do some SQL reordering to optimize throughput. In
that case, though, the trade-off is the possibility that the runtime
or machine crashes before the disk write occurs, in which case the
data will be lost. Thinking about this more, I'm not sure such a JPA-
based store would provide that much of a throughput boost unless we
allowed it to batch operations from multiple clients. In user
applications, I've seen Hibernate and JPA boost performance by
batching and reordering SQL from one client performing many writes.
Data loss vulnerability is limited since the client (application)
will receive an error when it commits the transaction (if the machine
crashed before the commit or flush, then I guess data would be lost).
With the store, however, there are typically a very limited number of
writes performed by a large number of clients. Reordering and
batching per client probably would not yield much. If we allowed
batching across clients, I think the risk of data loss may be too
great. With a completely disconnected model, I think the risk of data
loss would likely be greater.
Jim
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
