Comments on original "out of memory problems"
- Running out of disk space for logs can also be a problem as well as
transaction size
- The max # of locks needed will grow with the log of the DB size and
linearly with the number of concurrent transactions. You can run
db_stat to see what your resource usage is.
- When bulk loading be sure to pull the data from a file into memory
before running the transaction if there might be any concurrent
transactions on that DB. (think about what happens when you have to
restart the transaction and the DB state is reset but the stream &
loop variables in memory are not).
but not for larger logical transactions where they'd
really be called for. Raising the limits at best slightly lessens the
pain, but is not really a solution.
If raising the limits isn't a solution, then your application is
probably mismatched with Elephant/BDB. There might be some artificial
reasons you are seeing these problems.
1) If you have a large DB with deep indices and highly concurrent
transactions, then you may start hit lock limits as described above.
You should be able to set these limits high enough that you'll never
hit problems. Fortunately increasing locks is a good solution. max-
concurrent-sessions * max-objects-per-session * 2 + 20% is a good
number for max locks. You can use db_stat to see what kind of locks-
per-session numbers you're seeing and an analysis of your application
should tell you the max-concurrent-sessions.
Max concurrent transactions is upper-bounded by your machine's
processing capacity and max # of locks is proportional to the
logarithm of the number of objects which will grow very slowly so can
be given practical upper bounds.
(Set lock sizes in config.sexp - :berkeley-db-max-locks & :berkeley-db-
max-objects)
2) If you really are blowing out the BDB data cache in normal
operation (vs. bulk loads) then your application is either not using
transactions efficiently or is highly mismatched to any transactional
DB whether that be BDB or some future lisp implementation.
If a logical transaction results in writing or modifying 10's of
megabytes of data, then no transactional DB will ever work for you.
This is just physics. To get ACID properties you have to be able to
stage all side effects in memory to ensure that they are written as an
atomic unit. If you don't have enough memory to perform this staging,
then you can't implement a transaction.
There is a special case where you are manipulating large data objects
such as full-res images or uploaded files. I use a hybrid solution
for these where BDB keeps pointers to disk files which store just the
large items. If you can handle potential inconsistency between the on-
disk storage and the DB, then you can write the file to disk, keep a
pointer to it and do the bookkeeping in the DB inside a single
transaction. You'll run into serializer limits before you run into
cache size limits for large data items anyway.
The following may be obvious, but it's easy to get complacent and
forget that Elephant is backed up by a transactional DB and that you
can't really ignore those implications for real-world applications.
- The concept behind the transaction is to combine small DB operations
into a single atomic chunk. Sometimes you can wrap a small set of
such chunks for performance. You shouldn't be wrapping lots of
unrelated operations into a single transaction. This is better for
avoiding concurrency conflicts as well.
For example, all composite operations inside Elephant use transactions
to ensure consistency of slot and index values. Slot writes to
indexed slots wrap both the write and the index updates into a single
transaction. Otherwise every sub-operation ends up in its own little
transaction and you can get inconsistent indices if you have an error
in between the slot write and index update.
- One downside of a transactional store on BDB is every little
transaction results in a set of pages being flushed to disk (any Btree
index pages and the leaf pages with data). Flushing pages to disk
waits for a write-done from the driver; this can be very expensive!
(10's of ms if I remember correctly). Flushing one page takes about
the same time as flushing a few dozen pages so you can save
significant run time by wrapping your leaf operations in a transaction
- plus all the usual ACID properties.
In my web application (using weblocks) I wrap every ajax callback on
the server side in one big transaction since I know apriori that
operations within these callbacks aren't going to blow out 20M of
cache or 1000's of locks. Typically I update a couple of objects and
indices. This may be a few dozen 4k or 8k pages.
Cheers,
Ian
On Sep 23, 2008, at 9:42 AM, Marc wrote:
Ian Eslick wrote:
You could be running out of cache or locks. I believe there are now
parameters in config.sexp you can set to raise the default limits.
The lack of robustness to allocation failures is a problem with
Berkeley DB.
Unfortunately, running recovery isn't a trivial process. You have to
guarantee that all other threads have released any Berkeley DB
resources (abort all active transactions) and don't try to request
any
more (meaning no persistent slot reads/writes for Elephant) so you
essentially need to get inside the state of each process, abort any
transactions, and then suspend each thread. This isn't something
that
you can canonicalize inside the Elephant library.
Chalk this up as another reason to someday implement a lisp-only
version of the library!
Indeed, that'd be a dream. For us, at least, this prevents us from
seriously using transactions at all in elephant. We do use them to
speed
up some bulk inserts when we know that the number of inserts in one go
won't be too big, but not for larger logical transactions where they'd
really be called for. Raising the limits at best slightly lessens the
pain, but is not really a solution.
Best regards,
Marc
On Sep 21, 2008, at 10:25 PM, Red Daly wrote:
I have recently run into "Cannot allocate memory" problems with
elephant on a production server. Unfortunately when one transaction
is too large, it seems to blow the database until a manual recover
is
done.
The occasional failure is slightly worrisome but it the whole
database requiring a manual recover from one extra-large transaction
is a scary thought for a live application with thousands of users.
Why does the memory allocation failure sour the whole database
instead of aborting a single transaction? I think that elephant
should try to encapsulate this failure, recovering the database or
whatever is necessary to make the store usable for the next
transaction.
Best,
Red Daly
On Sat, Jan 5, 2008 at 5:02 PM, Victor Kryukov
<[EMAIL PROTECTED]> wrote:
On Jan 4, 2008 2:54 AM, Ian Eslick <[EMAIL PROTECTED]> wrote:
Hi Victor,
Sounds like your transaction is blowing out the shared memory
allocated by Berkeley DB to store dirty pages. This is caused by
transactions that are too large; putting an entire file of data
could
well accomplish this. (We really should change the error message
to
be more informative in these cases).
Try pushing with-transaction into the loop in import-movie as
follows:
Thanks for your suggestion, Ian - the problem was solved once I've
moved with-transaction inside the collect-rating-info.
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
