This looks good. I pushed it to elephant-unstable.
I wonder if the same problem will occur with the other backends and
whether there is a common patch we might apply? Has anyone tried
upgrading from 0.9.1 or an earlier release to elephant-unstable?
Thanks,
Ian
On Jan 2, 2009, at 8:15 AM, quan hu wrote:
> Hi Ian,
>
> The 0.9.1 upgrade fix patch is attached.
> I continuously run the upgrade test 3 times. All passed.
>
> In the fix, if the DB version is <= 0.9.1, (make-instance 'bdb-
> indexed-btree :from-oid -3 :sc sc :indices (make-hash-table) is
> always called, regardless of the indices exist before.
> In my previous fix, I only call it when the indices does not
> exist in DB file, but find that approach does not work. A
> unserialized error is triggered when try to open the source 0.9.1 DB
> file for the second time. This is caused by that the oid->schema-id
> has an assumption on that instance-table is only created on a brand
> new DB file.
>
> If this is not a suitable solution, please let me know.
>
> Thanks
> Quan
>
>
>
>
>
> 2009/1/2 Ian Eslick <esl...@media.mit.edu>
> pushed. -Ian
>
> On Jan 1, 2009, at 8:31 AM, quan hu wrote:
>
> > Hi Ian,
> >
> > For the pathname fix, I attached the patch.
> >
> > For the 0.9.1 upgrade fix, I got one failure and one
> > successful result.
> > I need to double check if the failure is caused by my test
> > execution step or other issue.
> > I'll send the result later.
> >
> > Thanks
> > Quan
> >
> >
> > 2008/12/31 Ian Eslick <esl...@media.mit.edu>
> > I don't think we've tried the upgrade procedure in awhile, so it's
> > possible an incompatibility slipped in. It may be that all you need
> > to do is make sure that defaults are properly created for slot
> values
> > that are unbound during the store controller opening phase.
> >
> > Adding a case for pathname is easy enough. I take it one of the
> slots
> > of the pathname structure object is a function; migrating this is
> > causing the error.
> >
> > With a little work you should be able to do both of these fixes,
> > certainly the pathname fix. I'd be happy to review and commit
> them to
> > the repository, but unfortunately I won't have any more time to work
> > on elephant for a little while.
> >
> > Ian
> >
> > On Dec 31, 2008, at 8:37 AM, quan hu wrote:
> >
> > > Hi Ian,
> > >
> > > I downloaded the patches and have more testing today. The
> lock
> > > issue is gone!
> > > Thanks a lot.
> > >
> > > 1. But, another issue is found. migrate pathname fails. I
> test
> > > it under SBCL 1.0.21
> > > e.g.
> > > > (migrate *store-controller* #p"/tmp) => Error: Cannot
> > > migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-
> > > ENOUGH>)
> > >
> > > It is caused by that (defmethod migrate ((dst store-
> > > controller) (src structure-object) is called for this case, as
> > > pathname is taken as subtype of structure-object.
> > > > (subtypep 'pathname 'structure-object) => t t
> > > Maybe, one more migrate method is needed for pathname case.
> > >
> > > 2. I also want to know if upgrading database file from 0.9.1
> > > to unstable release is supported.
> > > When opening the DB file generated in 0.9.1 in unstable
> > > release, I get error message
> > > " The slot DB-BDB::INDICES is unbound in the object #<BDB-
> > > INDEXED-BTREE oid:-3>.
> > > [Condition of type UNBOUND-SLOT]"
> > >
> > > Thanks
> > > Quan
> > >
> > >
> > >
> > >
> > > 2008/12/31 Ian Eslick <esl...@media.mit.edu>
> > > Hi Quan,
> > >
> > > Thank you again for the information on this bug. The short
> story is
> > > that the way we are keeping track of transactions assumed that
> there
> > > were only two stores in the nested transaction stack (outer and
> > > inner). Migrating a nested object inside a with-transaction would
> > not
> > > pass the parent transaction to the backend so it wouldn't know a
> > > transaction in progress and attempted to start a new one, which
> > would
> > > block until the old one was completed.
> > >
> > > I just checked in a patch that made transaction tracking a bit
> more
> > > intelligent (it depends on the backends taking care of handling
> > nested
> > > transactions - which I believe they all do fine). I also checked
> > in a
> > > test which implements Quan's test case.
> > >
> > > This patch and the prior patches of the past few days pass all
> tests
> > > on SBCL/Mac/BDB.
> > >
> > > Ian
> > >
> > > PS - I'd love to know the current status of the elephant-unstable
> > tree
> > > with the postmodern backend.
> > >
> > >
> > > On Dec 30, 2008, at 9:34 AM, quan hu wrote:
> > >
> > > > Hello,
> > > >
> > > > After more investigation, here are some suspects:
> > > >
> > > > In the current elephant codes, the macro my-current-
> > > > transaction will check if the current transaction belongs to the
> > sc
> > > > parameter. If not, just return +NULL-CHAR+. That's exactly the
> > > > case in the following nested transaction where the store-
> > controller
> > > > is different.
> > > > The situation becomes:
> > > > Transaction 1 begins
> > > > One data is put into DB with Transaction 1
> > > > ...
> > > > Another data is put into the same DB with
> NULL
> > > > transaction parameter(i.e. no transaction??)
> > > >
> > > > It looks like this scenario is not supported by BDB.
> > > > I simulate above case with a C program to access BDB and
> > get
> > > > the similar failure symptom.
> > > > ...
> > > > txn1 = NULL;
> > > > ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > > > if (ret != 0) {
> > > > printf ("Transaction begin failed.\n");
> > > > return 1;
> > > > }
> > > > put_record(FOO,FOO,txn1, dbp1);
> > > > printf("foo\n");
> > > > put_record(FOO,FOO,NULL, dbp1);
> > > > printf("bar\n");
> > > > txn1->commit(txn1,0);
> > > > printf("baz\n");
> > > > ...
> > > > After "foo" is printed, the C program is blocked. If I change
> > the
> > > > NULL to txn1, everything is fine.
> > > >
> > > >
> > > > Regarding to the solution, my gut feeling is that it would be
> > > > difficult to avoid this kind of nested transaction in migrate
> > > > process, because of the recursive function call. Maybe,
> > > > *transaction-stack* can be added to track the active
> transaction
> > > > stack. Get it pushed and poped in right place of execute-
> > > > transaction. Then, in my-current-transaction, check
> *transaction-
> > > > stack* to see if there is an active transaction belongs to the
> > same
> > > > store-controller. If it is found, use it. My simple test shows
> > it
> > > > works. If this is the right direction, I can do more test.
> > > >
> > > > Thanks for your help.
> > > > Quan
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 2008/12/30 quan hu <ihuq...@gmail.com>
> > > > Hi Ian, Yarek,
> > > >
> > > > Thanks for providing the information.
> > > > I downloaded the elephant-unstable and test it with BDB4.7.
> > > > The failure symptom is the same: target store controller
> get
> > > > locked.
> > > >
> > > > After I make following changes in migrate.lisp; The test
> > > passed.
> > > >
> > > > (defmethod copy-btree-contents ((sc store-controller) dst
> > src)
> > > > (let (to-be-migrated)
> > > > (map-btree (lambda (key value)
> > > > (let ((newval (migrate sc
> > > value))
> > > > (newkey (migrate sc
> > > key)))
> > > > (push (list newkey
> > > > newval) to-be-migrated)))
> > > > ;;;; (setf (get-
> > value
> > > > newkey dst) newval))) ;;
> > > > src)
> > > > (loop for (k v) in to-be-migrated
> > > > do
> > > > (setf (get-value k dst) v))))
> > > >
> > > > Above debug change just avoids the nested transaction of
> > > > different store controller for the specific test case.
> > > > However, it is not the solution, as the nested transaction
> > > > still could appear in other data pattern.
> > > >
> > > > What confuses me is that the nested transaction of the same
> > > > store controller seem not bring any block.
> > > > Following test passed:
> > > > (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > > (add-to-root 'foo 'foo :sc *migrate-dst*)
> > > > (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > > (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > > >
> > > > But, the nested transaction of different store controller
> > always
> > > > results in lock.
> > > >
> > > > (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > > (add-to-root 'foo 'foo :sc *migrate-dst*)
> > > > (with-transaction (:store-controller *migrate-
> > > > src*) ;; trigger database lock
> > > > (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > > >
> > > > Thanks
> > > > Quan
> > > >
> > > >
> > > >
> > > > 2008/12/30 Yarek Kowalik <yarek.kowa...@gmail.com>
> > > >
> > > > Quan,
> > > >
> > > > Unstable is here:
> > > >
> > > > http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> > > >
> > > > Yarek
> > > >
> > > >
> > > > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <esl...@media.mit.edu
> >
> > > > wrote:
> > > > That would be elephant-unstable
> > > >
> > > > Sent from my iPhone
> > > >
> > > > On Dec 29, 2008, at 7:36 AM, Ian Eslick <esl...@media.mit.edu>
> > > wrote:
> > > >
> > > > > Hi Quan,
> > > > >
> > > > > Can you try the latest darcs with BDB 4.6 or 4.7? I made some
> > > > changes
> > > > > to migration, but I can't recall if they were in the latest
> > > darcs or
> > > > > not - I think the latest darcs doesn't play well with BDB
> 4.5 so
> > > > > please make sure you can reproduce under the above
> configuration
> > > if
> > > > > you can.
> > > > >
> > > > > Thank you,
> > > > > Ian
> > > > >
> > > > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > > > >
> > > > >> Hello,
> > > > >>
> > > > >> I run into a problem when doing the garbage collection
> via
> > > data
> > > > >> migration.
> > > > >> The environment is elephant 0.9.1 and BDB 4.5. I also
> > tried
> > > > >> the latest elephant in darc and get the same result.
> > > > >>
> > > > >> 1. Test case to reproduce the problem.
> > > > >>
> > > > >> (defpclass user-profile()
> > > > >> ((id :initform nil :index t)
> > > > >> (sms-inbox :initform (make-pset))))
> > > > >>
> > > > >> ;; Create a new db for test purpose
> > > > >> (setf *migrate-src* (open-store '(:bdb "/tmp/db/
> > > src/")))
> > > > >> (setf *test-obj* (make-instance 'user-profile :sc
> > > > >> *migrate-src*))
> > > > >> (insert-item 'foo (slot-value *test-obj* 'sms-
> > inbox))
> > > > >> (setf *migrate-dst* (open-store '(:bdb "/tmp/db/
> > > dst/")))
> > > > >>
> > > > >> TEST> (migrate *migrate-dst* *migrate-src*)
> > > > >> Migrating class indexes for: USER-PROFILE
> > > > >> ; => Input get blocked, not response anymore
> > > > >>
> > > > >> 2. Using "db_stat -C A" get following output:
> > > > >> 1 Lock requests not available due to
> conflicts,
> > > for
> > > > >> which we waited
> > > > >>
> > > > >> 3. I do some debug work and found the problem may be
> > > caused
> > > > >> by nested transaction of different store controller.
> > > > >> I can reproduce the issue with following code
> > segment
> > > > >>
> > > > >> TEST> (ensure-transaction (:store-controller
> *migrate-
> > > > dst*)
> > > > >> (add-to-root 'foo 'foo :sc
> *migrate-
> > > > dst*)
> > > > >> (ensure-transaction (:store-
> > controller
> > > > >> *migrate-src*)
> > > > >> (add-to-root 'bar 'bar :sc
> > > > *migrate-
> > > > >> dst*)))
> > > > >>
> > > > >> This kind of nested transaction scenario can
> appear
> > > in
> > > > >> the migrate process, because it uses source/destination store
> > > > >> controller at the same time.
> > > > >>
> > > > >> I do not understand why different store
> controller's
> > > > >> transaction can result in the lock conflict and want to know
> > > how to
> > > > >> fix it.
> > > > >>
> > > > >> Is there anyone meet the similar problem before?
> > > > >>
> > > > >> Thanks
> > > > >> Quan
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> _______________________________________________
> > > > >> elephant-devel site list
> > > > >> elephant-devel@common-lisp.net
> > > > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > elephant-devel site list
> > > > > elephant-devel@common-lisp.net
> > > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel@common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel@common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel@common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel@common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel@common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel@common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > <migrate-pathname-
> > patch.gz>_______________________________________________
> > elephant-devel site list
> > elephant-devel@common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel@common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> <upgrade-0.9.1-fix-
> patch>_______________________________________________
> elephant-devel site list
> elephant-devel@common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
