Hi Victor,
Sounds like your transaction is blowing out the shared memory
allocated by Berkeley DB to store dirty pages. This is caused by
transactions that are too large; putting an entire file of data could
well accomplish this. (We really should change the error message to
be more informative in these cases).
Try pushing with-transaction into the loop in import-movie as follows:
(defun import-movie (n)
"Import ratings information for movie number N."
(iter (for line in-file (movie-filename n) using 'read-line)
(unless (first-time-p) ; skip first line
(with-transaction ()
(collect-rating-info n line)))))
I expect that will fix it.
Ian
PS - Your data representation sounds fine as a first start. You may
want to rethink it when you start doing queries and in fact may want
to build a custom data structure in static memory for performance
reasons once your algorithms and related queries are clear to you.
PS - You may get some additional mileage by increasing the total
memory cache size (you should lookup whether the transaction size is
correlated with the overall cache size) using the my-config.sexp
option (:berkeley-db-cachesize . 20971520)
On Jan 4, 2008, at 3:15 AM, Leslie P. Polzer wrote:
I've decided to put elephant 0.9.1 under some heavy load test, and
play with Netflix data set a little bit. The attached program that
tries to import everything in BerkeleyDB fails when trying to import
movie file number 8 with the following traceback. Do you have any
idea
what could be the problem?
No, but we might if you tell us
- OS
- RAM
- Swap limits
- Kernel limits on memory (e.g. /etc/security/limits.conf for GNU/
Linux)
- size of the data you're trying to pull
Leslie
--
My personal blog: http://blog.viridian-project.de/
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
