Hello everyone,
I wrote two concurreny test cases, added Bordeaux Threads and revised
the Postmodern error and transaction handling (for example, deadlocks
are no longer a problem).
Also included is the patch to close dangling connections I sent about
two weeks ago.
Please test them if you can spare a minute.
Thanks!
Leslie
New patches:
[DB-POSTMODERN: reap old connections when a new one is requested.
[EMAIL PROTECTED] {
hunk ./src/db-postmodern/pm-controller.lisp 71
+
+(defun reap-orphaned-connections (sc)
+ (let ((n-reaped 0))
+ #+sbcl(maphash (lambda (thread bookkeeper)
+ (let ((alive-p (sb-thread:thread-alive-p thread)))
+ (unless alive-p
+ (cl-postgres:close-database (car bookkeeper))
+ (incf n-reaped)
+ (remhash thread (controller-db-table sc)))))
+ (controller-db-table sc))
+ n-reaped))
+
hunk ./src/db-postmodern/pm-controller.lisp 90
+ (reap-orphaned-connections sc)
}
[added concurrency test cases.
[EMAIL PROTECTED] {
hunk ./elephant-tests.asd 63
+ (:file "testconcurrency")
addfile ./tests/testconcurrency.lisp
hunk ./tests/testconcurrency.lisp 1
+(in-package :ele-tests)
+
+;; not enabled by default
+;(in-suite* testthreads :in elephant-tests)
+
+;;;
+;;; Leslie P. Polzer <[EMAIL PROTECTED]> 2008
+;;;
+;;; These tests will (as of March 2008) fail horribly on
+;;; * BDB without deadlock detection enabled.
+;;; * CLSQL/SQLite
+;;; * Probably other CLSQL backends
+;;;
+;;; The Postmodern backend should handle them correctly,
+;;; and BDB as well (although I noticed a major slowdown).
+;;;
+
+
+(defpclass zork ()
+ ((slot1 :accessor slot1 :initarg :slot1 :initform nil :index t)
+ (slot2 :accessor slot2 :initarg :slot2 :initform nil :index t)))
+
+
+; A basic simulation of a web application using Elephant
+; This is also a test showing whether database connections get cleaned up
+; correctly.
+
+(test threaded-idx-access
+ (dotimes (i 10)
+ (make-instance 'zork :slot1 i :slot2 i))
+
+ (dotimes (batch 20)
+ (dotimes (i 5)
+ (bt:make-thread (lambda ()
+ (dotimes (i 5)
+ (format t "thread ~A: batch ~A, run ~A~%" (bt:current-thread) batch i)
+ (dolist (obj (elephant::get-instances-by-class 'zork))
+ (format t "now handling obj ~A~%" obj)
+ (setf (slot-value obj 'slot1) i)
+ (setf (slot-value obj 'slot2) (slot-value obj 'slot1)))))))
+ #+sbcl(dolist (thr (bt:all-threads))
+ (format t "waiting for thread ~A to finish...~%" thr)
+ (unless (eq thr (bt:current-thread))
+ (sb-thread:join-thread thr)))
+ (format t "batch finished!~%"))
+
+ (drop-instances (get-instances-by-class 'zork))
+ (format t "test finished!~%"))
+
+(test provoke-deadlock ;; sometimes throws a 23505 (primary key constraint violation)
+ ;; I have not tracked this down, yet.
+ (dotimes (i 10)
+ (make-instance 'zork :slot1 i :slot2 i))
+
+ (dotimes (i 30)
+ (bt:make-thread
+ (lambda ()
+ (format t "thread no ~A starting~%" i)
+ (let ((obj (car (get-instances-by-class 'zork))))
+ (setf (slot-value obj 'slot1) i)) ;; this only provokes deadlocks when
+ ;; the slot in question is indexed.
+ (format t "thread finished.~%"))))
+
+ (drop-instances (get-instances-by-class 'zork)))
+
}
[added BORDEAUX-THREADS dependency and changed PM controller to use it instead of SB-THREAD stuff.
[EMAIL PROTECTED] {
hunk ./elephant.asd 327
- :depends-on (:uffi :cl-base64))
+ :depends-on (:uffi :cl-base64 :bordeaux-threads))
hunk ./src/db-postmodern/package.lisp 8
- #+sbcl :sb-thread)
+ :bordeaux-threads)
hunk ./src/db-postmodern/pm-controller.lisp 74
- #+sbcl(maphash (lambda (thread bookkeeper)
- (let ((alive-p (sb-thread:thread-alive-p thread)))
+ (maphash (lambda (thread bookkeeper)
+ (let ((alive-p (thread-alive-p thread)))
hunk ./tests/elet-package.lisp 22
- (:use :common-lisp :elephant :5am)
+ (:use :common-lisp :elephant :5am :bordeaux-threads)
hunk ./tests/elet-package.lisp 24
- (:use :common-lisp :elephant :regression-test)
+ (:use :common-lisp :elephant :regression-test :bordeaux-threads)
hunk ./tests/elet-package.lisp 60
+
}
[DB-POSTMODERN: support transaction retries; handle deadlock; add savepoint utility functions; add warnings to help debugging problematic conditions.
[EMAIL PROTECTED] {
hunk ./src/db-postmodern/pm-sql.lisp 3
+(define-condition dbpm-error (error)
+ ((errno :reader errno :initarg :errno))
+ (:report (lambda (cond s)
+ (format s "DB error ~S"
+ (errno cond)))))
+
hunk ./src/db-postmodern/pm-sql.lisp 155
+(defun set-savepoint (con savepoint)
+ (ignore-errors
+ (cl-postgres:exec-query con (concatenate 'string "SAVEPOINT " savepoint))))
+
+(defun rollback-to-savepoint (con savepoint)
+ (ignore-errors
+ (cl-postgres:exec-query con (concatenate 'string "ROLLBACK TO " savepoint))))
+
hunk ./src/db-postmodern/pm-sql.lisp 181
+ (warn "42P05: prepared statement already exists!")
+ 'ignoring-this-error)
hunk ./src/db-postmodern/pm-sql.lisp 190
- 'ignoring-this-error)
hunk ./src/db-postmodern/pm-sql.lisp 201
- (handler-case
- (exec-prepared name-string)
- (cl-postgres:database-error (e)
- ;; Sometimes the prepared statement might hold references to old oids,
- ;; which might be have been dropped after a rollback. For safety, try
- ;; to remove the prepared statement and prepare it again
- (cond
- ((string= (cl-postgres:database-error-code e)
- "42P01")
- ;; It seems that this error automatically drops the transaction! Postgresql bug?
- (cl-postgres:exec-query (active-connection) (concatenate 'string "DEALLOCATE " name-string))
- (cl-postgres:prepare-query (active-connection) name-string sql)
- (exec-prepared name-string))
- (t (error e)))))))))
+ (let ((savepoint (princ-to-string (gensym))))
+ ;(set-savepoint (active-connection) savepoint)
+ (handler-case
+ (progn
+ ;(format t "Executing prepared query ~A~%" name-string)
+ (exec-prepared name-string))
+ (cl-postgres:database-error (e)
+ ;; Sometimes the prepared statement might hold references to old oids,
+ ;; which might be have been dropped after a rollback. For safety, try
+ ;; to remove the prepared statement and prepare it again
+ (warn "Error while executing prepared statement ~S (params: ~A).~%"
+ name-string params)
+ (cond
+ ((string= (cl-postgres:database-error-code e)
+ "42P01")
+ ;; It seems that this error automatically drops the transaction! Postgresql bug?
+ (warn "42P01: Prepared statement already exists; trying to remove it.")
+ ;(rollback-to-savepoint (active-connection) savepoint)
+ (cl-postgres:exec-query (active-connection) (concatenate 'string "DEALLOCATE " name-string))
+ (cl-postgres:prepare-query (active-connection) name-string sql)
+ (exec-prepared name-string))
+
+ ((string= (cl-postgres:database-error-code e)
+ "40P01") ; deadlock, defer to txn handler
+ (error 'dbpm-error :errno "40P01"))
+
+ ((string= (cl-postgres:database-error-code e)
+ "25P02")
+ (warn "25P02: Transaction aborted; something wasn't handled correctly!")
+ 'ignoring-this-error)
+
+ (t (error e))))))))))
+
hunk ./src/db-postmodern/pm-sql.lisp 330
+
hunk ./src/db-postmodern/pm-transaction.lisp 30
- &key (always-rollback nil) &allow-other-keys)
- ;; SQL doesn't support nested transaction
+ &key (retries 50) (always-rollback nil) &allow-other-keys)
hunk ./src/db-postmodern/pm-transaction.lisp 32
- (if (> (tran-count-of sc) 0)
- (funcall txn-fn)
- (let (tran
- commited
- (*txn-value-cache* (make-value-cache sc)))
- (incf (tran-count-of sc))
- (unwind-protect
- (prog2
- (setf tran (controller-start-transaction sc))
- (funcall txn-fn) ;;this gets returned
- (unless always-rollback ;;automatically commit unless always rollback
- (controller-commit-transaction sc tran)
- (setf commited t)))
- (unless commited (controller-abort-transaction sc tran))
- (decf (tran-count-of sc)))))))
+ (let (savepoint (try 0))
+ (tagbody
+ retry (incf try)
+ (when (>= try retries)
+ (cerror "Retry transaction again?"
+ 'transaction-retry-count-exceeded
+ :format-control "Transaction exceeded the limit of ~A retries"
+ :format-arguments (list retries)
+ :count retries))
+ ;(format t "txn-mgr (thr ~A): try ~A~%" sb-thread::*current-thread* try)
+ ;; XXX honor max retries
+ (if (> (tran-count-of sc) 0) ;; SQL doesn't support nested transaction
+ (progn
+ ;(setf savepoint (princ-to-string (gensym)))
+ ;(set-savepoint (active-connection) savepoint)
+ ;(setf savepoint nil)
+ ;(format t "detected nested transaction~%")
+ (return-from execute-transaction (funcall txn-fn)))
+ (let (tran)
+ (handler-case
+ (let (commited (*txn-value-cache* (make-value-cache sc)))
+ (incf (tran-count-of sc))
+ (unwind-protect
+ (return-from execute-transaction
+ (prog2
+ (setf tran (controller-start-transaction sc))
+ (funcall txn-fn) ;; this gets returned
+ (unless always-rollback ;; automatically commit unless always rollback
+ (controller-commit-transaction sc tran)
+ (setf commited t))))
+ (unless commited (controller-abort-transaction sc tran))
+ (decf (tran-count-of sc))))
+ (dbpm-error (e)
+ (warn "dbpm txn manager: caught error ~A~%" (errno e))
+ (cond
+ ((string= (errno e) "40P01")
+ ;(if savepoint
+ ;(rollback-to-savepoint (active-connection) savepoint)
+ (controller-abort-transaction sc tran)
+ (go retry)))))))))))
}
Context:
[Check for unbound slot; potential fix for a compiler error
[EMAIL PROTECTED]
[Fix test dependence for ff-index-test
[EMAIL PROTECTED]
[Improve berkeley DB version agnostic code
[EMAIL PROTECTED]
Added an error message to configure my-config.sexp and made sure we load
it directly from my-config.sexp so that we get it right at load time.
Prior patch didn't override default until after load time.
]
[Support for multiple BDB versions
[EMAIL PROTECTED]
[db-bdb updated to BerkeleyDB 4.6
[EMAIL PROTECTED]
Changed only BDB constants as upgrade 4.5 -> 4.6 they were
changed.
A kind of hack perhaps. But it works. The testing was not excessive,
but it works well for my project.
]
[add test for STRING types (as opposed to SIMPLE-STRING types)
[EMAIL PROTECTED]
[Refactor UTF{16,32}LE serializers.
[EMAIL PROTECTED]
[Enable multiple process connections to a BDB data-store via DB_REGISTER option
[EMAIL PROTECTED]
[Enable multi-store indexed classes
[EMAIL PROTECTED]
[Change semantics of transaction :retry-count from tries to retries
[EMAIL PROTECTED]
[Minor edits, fixed a comment, fixed a db-lisp out of date error
[EMAIL PROTECTED]
[Alex's patch for 8.3
[EMAIL PROTECTED]
I entered here the patch from Alex of 2088/02/16
which apparently makes this compatible with Postgres 8.3.
It is green for me on all tests on Posgres 8.1, so
I am committing it.
]
[mtype change in dcm
[EMAIL PROTECTED]
[moved cache-instance into initial-persistent-setup
[EMAIL PROTECTED]
because it was bypassed by recreate-instance otherwise
]
[accessor name in tests change
[EMAIL PROTECTED]
[db-postmodern: pm-btree initialization fixed
[EMAIL PROTECTED]
[recreate-instance stuff improved
[EMAIL PROTECTED]
[db-postmodern: removed specialized map-index
[EMAIL PROTECTED]
because pure cursor version works fine and is more robust
]
[cursor-duplicate removed from db-postmodern
Henrik Hjelte<[EMAIL PROTECTED]>*-20071124163701]
[db-postmodern fix map-index optimization bug
Henrik Hjelte <[EMAIL PROTECTED]>**20080104151644]
[db-postmodern: cursors re-implemented
[EMAIL PROTECTED]
[controller-doc-improvement
[EMAIL PROTECTED]
[tutorial
[EMAIL PROTECTED]
[non-keyword-accessors
[EMAIL PROTECTED]
allows lispworks to run tests.
]
[function-call-key-form
[EMAIL PROTECTED]
[documentation type fix
[EMAIL PROTECTED]
[Fix the use of internal symbol of sb-kernel in memutils
Leonardo Varuzza <[EMAIL PROTECTED]>**20071230000120
memutil.lisp use the functions sb-kernel::copy-*-from-system-area, which
aren't exported in the latest version of sbcl.
Fix it adding the :: when appropriate
]
[db-bdb bugfix: when bdb key comparison compared only the first half of utf16 strings
[EMAIL PROTECTED]
[Fix instance deserialization to bypass initialization protocol
[EMAIL PROTECTED]
[db-postmodern: optimized map-index for -by-value case
[EMAIL PROTECTED]
[db-postmodern: optimized form-slot-key for persistent-slot-reader
[EMAIL PROTECTED]
it uses SBCL internal function now, for other implementation it's less optimized.
]
[db-postmodern: small example update
[EMAIL PROTECTED]
[added sh script for flushing logs sample
[EMAIL PROTECTED]
[db-postmodern removed possiblity of using NIL as a key in btrees
Henrik Hjelte<[EMAIL PROTECTED]>**20071124163828]
[cursor-duplicate removed from db-postmodern
Henrik Hjelte<[EMAIL PROTECTED]>**20071124163701]
[removed a little compiler warning (typo)
Henrik Hjelte<[EMAIL PROTECTED]>**20071122151929]
[remove kind-hints parameter from add-index
Henrik Hjelte<[EMAIL PROTECTED]>**20071122151046
Probably a coming feature from Ian, but
right now it breaks the generic function add-index
and thus postmodern, so I removed it for now.
]
[Ensure set-db-synch is defined before pset is loaded
[EMAIL PROTECTED]
[Fix instance deserialization to bypass initialization protocol
[EMAIL PROTECTED]
[Fix to from-end traversal of new map-index
[EMAIL PROTECTED]
[New map-index implementation
[EMAIL PROTECTED]
[Cheaper get-instance-by-value
[EMAIL PROTECTED]
[TAG ELEPHANT-0-9-1
[EMAIL PROTECTED]
Patch bundle hash:
3ce64eb4f1020f3d96cab54935d3e7cbb9ae06fd_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
