Hi Aaron
On Wed, 2009-10-28 at 16:36 +0100, Aaron Birkland wrote:
the implications of these
fine-granied operations on datastream versions is unclear.
You are quite right. I have simply not thought of the number of
versions
that would be created from doing these operations in a series.
I think that executing multiple operations in series as part of a
single
logical change that would otherwise be preserved as a unit might be
an
anti-pattern in general (some of these preserved versions would
necessarily be inconsistent with the desired end state).
Yes, that would be an anti-pattern.
Each operation needs to do versioning, as if it was singular, as it
can
have no knowledge of the logical change, and thus the unit to
preserve.
I have an idea for a transactional system for Fedora, that I would
like
to hear your opinion on.
There will be 3 new methods, like StartTransaction, CommitTransaction
and DeleteTransaction. StartTransaction gives you a token, that
identifies the transaction. All the normal API methods will take that
token as a parameter, and execute the operation within the
transaction.
Normal operation of Fedora will work as normal.
When an object is modified as part of a transaction, the normal
procedure for an API call is not followed.
1. First, the Fedora system attempt to get a write-lock on the object.
The object is being written as part of this transaction, and does not
allow other processes to edit it.
2. The object is parsed into memory, and stored as part of the
transaction.
3. The change is executed on the in-memory object, preserving
information about which new datastream is created.
4. Return
Normal reads of the object will see the unmodified object. Reads from
within the transaction will see the modified object.
Further modifications will either lock other objects, or work on one
of
the already locked objects. The interesting case is when an object is
modified twice in the same transaction.
1. In case the change involves a unversioned property (unversioned
datastream, object property), the change should just overwrite the
previous value, even if that was set as part of this transaction.
2. In case the change involves something versioned, but the unit has
not
been modified in this transaction: Make a new version as normal, with
the change.
3. In case the change involves something versioned, and this thing has
already been modified as part of this transaction: Find the new
version
created, and replace the values in that.
4. In case the change involves something deleted in the same
transaction: This cause an error, and the change is not carried out.
The procedure above would ensure that changes to the same logical unit
would be made part of the same storage version.
When the change is committed:
First, the system upgrades all write locks on modified objects to read
locks. The locked objects are parsed into memory, and used to service
read requests while the transaction is written to storage.
The objects are written to the store, one at a time (as that is the
only
way to do so). If there is a problem with writing one of the objects,
the transaction is aborted, and all objects written are replaced with
their previously parsed counterparts.
(This is the risky step. If, and only if, the fedora system goes down
while committing a transaction, will the repository be left in an
inconsistent state.)
When all the modifications have been written, the old data objects are
cleared from memory and the locks on the objects released.
When a transaction is deleted:
Remove the transaction object, and all the modified objects. These
exist
only in memory at that moment, so this change will be invisible to the
storage system.
Risks at different stages
1. The client goes down during a transaction, thus locking some
objects:
Timeout on the transaction remedies this problem
2. The server goes down during a transaction, before commit is called:
The transaction is stored only in memory (or similar non-persistent
storage) so all recollection of the transaction is only relevant while
the server is running. Reboot removes all transactions.
3. The server goes down during a commit: The repo is left
inconsistent.
Easiest way to mitigate: Write the old versions of the objects to some
more permanent store before changing the versions in the repository.
When the server starts up, restore any objects from this store, so
that
the finished repo is consistent again.
Gotchas in this approach:
1. The triple store will not be transactional. It will reflect the
current contents of the repo. We can, as the last part of a
transaction,
ingest all the rdf statements from the changed objects into the triple
store, so that it gets them all at the same time. Still, there will
be a
tiny desynchronisation between fedora and triple store.
2. Content from remote locations (ie. URLs) will not be downloaded
until
the transaction is committed. That is the only part that cannot be
reasonably validated, before the attempt is made.
However, in my
own use case, I would say that 90-99% of all datastream
modifications do
involve only *one* change - and in that case, this API proposal fits
very well by providing an additional, lightweight, easy to use tool.
I am glad to hear that. For a moment, I thought that the problem you
pointed out would be the death of this API.
Some of my most common use cases are:
1) updating datastream content, keeping all properties the same
2) changing datastream state, keeping content and other properties
the
same
3) fixing datastream MIME type, keeping content and other
properties the
same
A less common (but important) use case for me involving changes to
both
content and properties is updating content, label, and mime type. I
would want to to perform that change one atomic operation.
I do not propose to change things without creating an Audit entry.
What
made you think I meant that?
I was not sure if the granularity of these operations had any
implications on auditing.
I do have a problem with the Audit system at the moment, as it does
not
store the old values. This is not good when changing properties that
are
not part of the versioned bit of a datastream.
If a client intends to modify
both datastream content as well as datastream properties, does this
imply that it MUST first change datastream content, then change
properties?
Why should the order matter?
You're right - it does not matter. It may have implications on what
the inconsistent intermediate versions would look like, but does not
present a fundamental difference.
This has been a delightful topic, Asger. Thanks!
Thanks!
-Aaron
------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart
your
developing skills, take BlackBerry mobile applications to market and
stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Fedora-commons-developers mailing list
Fedora-commons-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers
