Re: [Sugar-devel] Datastore redesign
On Thu, Jul 9, 2009 at 8:00 AM, Tomeu Vizoso wrote: > On Mon, Jul 6, 2009 at 16:33, Eben Eliason wrote: >> On Mon, Jul 6, 2009 at 10:02 AM, Sascha >> Silbe wrote: >>> On Mon, Jul 06, 2009 at 12:29:53PM +0200, Tomeu Vizoso wrote: >>> Agreed. I have to say that your proposal is excellent, congratulations! >>> >>> Thanks, I'm flattered. :) >>> >>> > Is the asynchronous API design useful enough to warrant more complex > implementation? I'm not sure, but I think that whatever decision we take should be made based on actual usage of the DS. What about proposing an example of how an existing activity would be modified to use the new API? >>> >>> OK, will work on one. >>> >>> > - For save() calls activity needs to wait for result (containing new > version_id) before it can invoke save() again for the same object > which can take quite some time if save() is sync - especially if other > activities are saving at the same time. What about having a separate call that returns synchronously a new tree_id and/or version_id? >>> >>> Interesting idea, need to think about it. As we're going to use UUIDs not >>> using requested versions shouldn't be an issue (for other version number >>> schemes like the one you propose below "holes" in the numbering could be >>> troublesome). >> >> I think "holes" can be expected, so we should definitely be prepared for >> them. >> >>> > Making the API fully asynchronous is the cause for much of the complexity > of > my proposal, but if we eliminate the queueing the response times for > write > accesses and checkout() can be very long even for unrelated operations. Why for unrelated operations? >>> >>> Because we're serializing VCS operations. They are IO bound (more >>> specifically: disk bound) and parallelisation would only lead to IO >>> starvation, especially for HDDs. >>> >>> # do we want an optimized way to determine (only) the branch HEADs of a given tree_id? This depends on the intended UI. My opinion is that if we branch at every interesting modification (triggered by the activity detecting an interesting change or by the user clicking on the Keep button), we >> >> I don't think we need to branch in these instances. These events >> should create new versions, but not necessarily new branches. > > In my proposal, branch is not a concept directly exposed to the user. > Is just an artefact that allows the journal to display the relevant > info to an user. If we branch at the following events, displaying only > HEADs of branches in the journal list view makes sense for the user: > > - new tree_id, > - resume entry, > - keep button, > - after a big change in the activity model (user deletes the whole > drawing, etc). > > There are other ways to manage what we want, but this approach made it > very easy to implement. > > Just to make sure I'm understood, I see why using branches this way > may seem conceptually wrong, it's not how we would work in a VCS or > CMS. But by creating new branches at those points and displaying only > HEADs of branches in the list view is the simplest way I found of > displaying the relevant entries in a robust way (resisting activity > crashes). That's fine, assuming these are actually the entries we want to show, but I'm not sure that's always the case. For instance, we might show only the most recent version in the object view, while showing each version within the action view. If we store "action-objects" as Ben proposed, we may have an entirely different way of querying what should be shown in the actions view anyway. We've also had some ideas on how to expose the branching structure within the version popup, in which case branching as one would in a VCS would make more sense. Eben would like to display in the object list all the HEADs of each branch in each tree_id. In that case yes, we need a way to retrieve that list that is fast on both the client and the server side. >>> >>> My imagined usage of branches was to create them automatically upon altering >>> a non-HEAD version. >> >> This makes sense to me, personally. >> >>> A user basing off an old version could mean the newer version is "broken" >>> (in that case promoting the new version to the HEAD of the current branch >>> makes more sense) or that (s)he uses the older version as a kind of template >>> to create derivates (so creating a branch would make most sense). >>> But I'm open to alternative suggestions. We'd most likely need a way to >>> explicitly create branches then. >>> >>> # using symlink instead of hardlink for "incoming" queue since we want to support directory trees, not just files >>> What justifies this new requirement? >>> >>> That it's >>> a) of use to activities (IIRC some of them use ZIP files right instead now), >> >> This has its own merits, though. Encapsulating the related files as a >> single archive make
Re: [Sugar-devel] Datastore redesign
On Mon, Jul 6, 2009 at 16:33, Eben Eliason wrote: > On Mon, Jul 6, 2009 at 10:02 AM, Sascha > Silbe wrote: >> On Mon, Jul 06, 2009 at 12:29:53PM +0200, Tomeu Vizoso wrote: >> >>> Agreed. I have to say that your proposal is excellent, congratulations! >> >> Thanks, I'm flattered. :) >> >> Is the asynchronous API design useful enough to warrant more complex implementation? >>> >>> I'm not sure, but I think that whatever decision we take should be >>> made based on actual usage of the DS. What about proposing an example >>> of how an existing activity would be modified to use the new API? >> >> OK, will work on one. >> >> - For save() calls activity needs to wait for result (containing new version_id) before it can invoke save() again for the same object which can take quite some time if save() is sync - especially if other activities are saving at the same time. >>> >>> What about having a separate call that returns synchronously a new >>> tree_id and/or version_id? >> >> Interesting idea, need to think about it. As we're going to use UUIDs not >> using requested versions shouldn't be an issue (for other version number >> schemes like the one you propose below "holes" in the numbering could be >> troublesome). > > I think "holes" can be expected, so we should definitely be prepared for them. > >> Making the API fully asynchronous is the cause for much of the complexity of my proposal, but if we eliminate the queueing the response times for write accesses and checkout() can be very long even for unrelated operations. >>> >>> Why for unrelated operations? >> >> Because we're serializing VCS operations. They are IO bound (more >> specifically: disk bound) and parallelisation would only lead to IO >> starvation, especially for HDDs. >> >> >>> # do we want an optimized way to determine (only) the branch HEADs of >>> a given tree_id? >>> >>> This depends on the intended UI. My opinion is that if we branch at >>> every interesting modification (triggered by the activity detecting an >>> interesting change or by the user clicking on the Keep button), we > > I don't think we need to branch in these instances. These events > should create new versions, but not necessarily new branches. In my proposal, branch is not a concept directly exposed to the user. Is just an artefact that allows the journal to display the relevant info to an user. If we branch at the following events, displaying only HEADs of branches in the journal list view makes sense for the user: - new tree_id, - resume entry, - keep button, - after a big change in the activity model (user deletes the whole drawing, etc). There are other ways to manage what we want, but this approach made it very easy to implement. Just to make sure I'm understood, I see why using branches this way may seem conceptually wrong, it's not how we would work in a VCS or CMS. But by creating new branches at those points and displaying only HEADs of branches in the list view is the simplest way I found of displaying the relevant entries in a robust way (resisting activity crashes). >>> would like to display in the object list all the HEADs of each branch >>> in each tree_id. In that case yes, we need a way to retrieve that list >>> that is fast on both the client and the server side. >> >> My imagined usage of branches was to create them automatically upon altering >> a non-HEAD version. > > This makes sense to me, personally. > >> A user basing off an old version could mean the newer version is "broken" >> (in that case promoting the new version to the HEAD of the current branch >> makes more sense) or that (s)he uses the older version as a kind of template >> to create derivates (so creating a branch would make most sense). >> But I'm open to alternative suggestions. We'd most likely need a way to >> explicitly create branches then. >> >> >>> # using symlink instead of hardlink for "incoming" queue since we want >>> to support directory trees, not just files >> >>> What justifies this new requirement? >> >> That it's >> a) of use to activities (IIRC some of them use ZIP files right instead now), > > This has its own merits, though. Encapsulating the related files as a > single archive makes transporting the file around, or sending it to a > friend, trivial. It's good for the same reason that activity bundles > are good. But this may be considered an internal implementation detail of the DS, unless we want to support users directly browsing the DS backend. >> b) easy enough to achieve with the new design and >> c) leads to better delta compression and thus disk space effiency. >> >> >>> # since an index rebuild can take a lot of time we need to provide UI >>> feedback while doing that >>> >>> Any I/O operation can potentially take a lot of time, but with the >>> current version of the DS rebuilding an index with a few thousands of >>> entries is not so slow on the XO. We should never need to rebuild the >>>
Re: [Sugar-devel] Datastore redesign
On Mon, Jul 6, 2009 at 16:02, Sascha Silbe wrote: > On Mon, Jul 06, 2009 at 12:29:53PM +0200, Tomeu Vizoso wrote: > >> Agreed. I have to say that your proposal is excellent, congratulations! > > Thanks, I'm flattered. :) > > >>> Is the asynchronous API design useful enough to warrant more complex >>> implementation? >> >> I'm not sure, but I think that whatever decision we take should be >> made based on actual usage of the DS. What about proposing an example >> of how an existing activity would be modified to use the new API? > > OK, will work on one. > > >>> - For save() calls activity needs to wait for result (containing new >>> version_id) before it can invoke save() again for the same object >>> which can take quite some time if save() is sync - especially if other >>> activities are saving at the same time. >> >> What about having a separate call that returns synchronously a new >> tree_id and/or version_id? > > Interesting idea, need to think about it. As we're going to use UUIDs not > using requested versions shouldn't be an issue (for other version number > schemes like the one you propose below "holes" in the numbering could be > troublesome). The issue I see with this is the actual creation or update operation failing, thus further operations on that uuid would be invalid. I guess activities will need to be able to notice that the creation failed and do something accordingly. >>> Making the API fully asynchronous is the cause for much of the complexity >>> of >>> my proposal, but if we eliminate the queueing the response times for >>> write >>> accesses and checkout() can be very long even for unrelated operations. >> >> Why for unrelated operations? > > Because we're serializing VCS operations. They are IO bound (more > specifically: disk bound) and parallelisation would only lead to IO > starvation, especially for HDDs. What's the scenario in which starvation would happen and with which consequences? >> # do we want an optimized way to determine (only) the branch HEADs of >> a given tree_id? >> >> This depends on the intended UI. My opinion is that if we branch at >> every interesting modification (triggered by the activity detecting an >> interesting change or by the user clicking on the Keep button), we >> would like to display in the object list all the HEADs of each branch >> in each tree_id. In that case yes, we need a way to retrieve that list >> that is fast on both the client and the server side. > > My imagined usage of branches was to create them automatically upon altering > a non-HEAD version. > A user basing off an old version could mean the newer version is "broken" > (in that case promoting the new version to the HEAD of the current branch > makes more sense) or that (s)he uses the older version as a kind of template > to create derivates (so creating a branch would make most sense). > But I'm open to alternative suggestions. We'd most likely need a way to > explicitly create branches then. I actually would do a branch at every resume, even if it's the current HEAD. >> # using symlink instead of hardlink for "incoming" queue since we want >> to support directory trees, not just files > >> What justifies this new requirement? > > That it's > a) of use to activities (IIRC some of them use ZIP files right instead now), > b) easy enough to achieve with the new design and > c) leads to better delta compression and thus disk space effiency. That's fine, but I think we should first address present needs, once we are done we can try to anticipate new ones. We are already planning to do very big changes in one single go and would be very unfortunate to have to back up those because of some peripheral requirements that were added because they seemed easy enough at the time. I tell you this because of my experience in releasing broken datastores. Would be good if we could learn from the past. >> # since an index rebuild can take a lot of time we need to provide UI >> feedback while doing that >> >> Any I/O operation can potentially take a lot of time, but with the >> current version of the DS rebuilding an index with a few thousands of >> entries is not so slow on the XO. We should never need to rebuild the >> index, so this new requirement might not be justified (given the >> current resources, all the other work we need to do, etc). > > OK, good to know index rebuilding is fast. So the simple, boolean API I > proposed (check_ready() / Ready()) suffices. Well, that's the index as implemented in the current DS, may not hold for the future. >> # detecting identical files across objects isn't as important since >> duplicates are mostly expected to occur as versions of the same object > >> Based on how current activities are using the DS, this isn't like >> that. >> The most common case I have heard from the field are children >> downloading a PDF for reading several times. > > Oh, didn't know that, so it's a new requirement. > >> An alternative to the current method for d
Re: [Sugar-devel] Datastore redesign
On Mon, Jul 6, 2009 at 10:02 AM, Sascha Silbe wrote: > On Mon, Jul 06, 2009 at 12:29:53PM +0200, Tomeu Vizoso wrote: > >> Agreed. I have to say that your proposal is excellent, congratulations! > > Thanks, I'm flattered. :) > > >>> Is the asynchronous API design useful enough to warrant more complex >>> implementation? >> >> I'm not sure, but I think that whatever decision we take should be >> made based on actual usage of the DS. What about proposing an example >> of how an existing activity would be modified to use the new API? > > OK, will work on one. > > >>> - For save() calls activity needs to wait for result (containing new >>> version_id) before it can invoke save() again for the same object >>> which can take quite some time if save() is sync - especially if other >>> activities are saving at the same time. >> >> What about having a separate call that returns synchronously a new >> tree_id and/or version_id? > > Interesting idea, need to think about it. As we're going to use UUIDs not > using requested versions shouldn't be an issue (for other version number > schemes like the one you propose below "holes" in the numbering could be > troublesome). I think "holes" can be expected, so we should definitely be prepared for them. > >>> Making the API fully asynchronous is the cause for much of the complexity >>> of >>> my proposal, but if we eliminate the queueing the response times for >>> write >>> accesses and checkout() can be very long even for unrelated operations. >> >> Why for unrelated operations? > > Because we're serializing VCS operations. They are IO bound (more > specifically: disk bound) and parallelisation would only lead to IO > starvation, especially for HDDs. > > >> # do we want an optimized way to determine (only) the branch HEADs of >> a given tree_id? >> >> This depends on the intended UI. My opinion is that if we branch at >> every interesting modification (triggered by the activity detecting an >> interesting change or by the user clicking on the Keep button), we I don't think we need to branch in these instances. These events should create new versions, but not necessarily new branches. >> would like to display in the object list all the HEADs of each branch >> in each tree_id. In that case yes, we need a way to retrieve that list >> that is fast on both the client and the server side. > > My imagined usage of branches was to create them automatically upon altering > a non-HEAD version. This makes sense to me, personally. > A user basing off an old version could mean the newer version is "broken" > (in that case promoting the new version to the HEAD of the current branch > makes more sense) or that (s)he uses the older version as a kind of template > to create derivates (so creating a branch would make most sense). > But I'm open to alternative suggestions. We'd most likely need a way to > explicitly create branches then. > > >> # using symlink instead of hardlink for "incoming" queue since we want >> to support directory trees, not just files > >> What justifies this new requirement? > > That it's > a) of use to activities (IIRC some of them use ZIP files right instead now), This has its own merits, though. Encapsulating the related files as a single archive makes transporting the file around, or sending it to a friend, trivial. It's good for the same reason that activity bundles are good. > b) easy enough to achieve with the new design and > c) leads to better delta compression and thus disk space effiency. > > >> # since an index rebuild can take a lot of time we need to provide UI >> feedback while doing that >> >> Any I/O operation can potentially take a lot of time, but with the >> current version of the DS rebuilding an index with a few thousands of >> entries is not so slow on the XO. We should never need to rebuild the >> index, so this new requirement might not be justified (given the >> current resources, all the other work we need to do, etc). > > OK, good to know index rebuilding is fast. So the simple, boolean API I > proposed (check_ready() / Ready()) suffices. > > >> # detecting identical files across objects isn't as important since >> duplicates are mostly expected to occur as versions of the same object > >> Based on how current activities are using the DS, this isn't like >> that. >> The most common case I have heard from the field are children >> downloading a PDF for reading several times. > > Oh, didn't know that, so it's a new requirement. > >> An alternative to the current method for detecting duplicates is moving >> this task to >> activities, is that what you suggest? > > I'm ambivalent about it. On one hand it's not so easy to achieve in > datastore (for various backends) and more indicative of UI deficiencies (why This might be the case, indeed. On other operating systems/browsers, this (downloading multiple copies if the link is clicked multiple times) is expected behavior. Perhaps we can work out some ways to make the UI clearer
Re: [Sugar-devel] Datastore redesign
On Mon, Jul 06, 2009 at 12:29:53PM +0200, Tomeu Vizoso wrote: Agreed. I have to say that your proposal is excellent, congratulations! Thanks, I'm flattered. :) Is the asynchronous API design useful enough to warrant more complex implementation? I'm not sure, but I think that whatever decision we take should be made based on actual usage of the DS. What about proposing an example of how an existing activity would be modified to use the new API? OK, will work on one. - For save() calls activity needs to wait for result (containing new version_id) before it can invoke save() again for the same object which can take quite some time if save() is sync - especially if other activities are saving at the same time. What about having a separate call that returns synchronously a new tree_id and/or version_id? Interesting idea, need to think about it. As we're going to use UUIDs not using requested versions shouldn't be an issue (for other version number schemes like the one you propose below "holes" in the numbering could be troublesome). Making the API fully asynchronous is the cause for much of the complexity of my proposal, but if we eliminate the queueing the response times for write accesses and checkout() can be very long even for unrelated operations. Why for unrelated operations? Because we're serializing VCS operations. They are IO bound (more specifically: disk bound) and parallelisation would only lead to IO starvation, especially for HDDs. # do we want an optimized way to determine (only) the branch HEADs of a given tree_id? This depends on the intended UI. My opinion is that if we branch at every interesting modification (triggered by the activity detecting an interesting change or by the user clicking on the Keep button), we would like to display in the object list all the HEADs of each branch in each tree_id. In that case yes, we need a way to retrieve that list that is fast on both the client and the server side. My imagined usage of branches was to create them automatically upon altering a non-HEAD version. A user basing off an old version could mean the newer version is "broken" (in that case promoting the new version to the HEAD of the current branch makes more sense) or that (s)he uses the older version as a kind of template to create derivates (so creating a branch would make most sense). But I'm open to alternative suggestions. We'd most likely need a way to explicitly create branches then. # using symlink instead of hardlink for "incoming" queue since we want to support directory trees, not just files What justifies this new requirement? That it's a) of use to activities (IIRC some of them use ZIP files right instead now), b) easy enough to achieve with the new design and c) leads to better delta compression and thus disk space effiency. # since an index rebuild can take a lot of time we need to provide UI feedback while doing that Any I/O operation can potentially take a lot of time, but with the current version of the DS rebuilding an index with a few thousands of entries is not so slow on the XO. We should never need to rebuild the index, so this new requirement might not be justified (given the current resources, all the other work we need to do, etc). OK, good to know index rebuilding is fast. So the simple, boolean API I proposed (check_ready() / Ready()) suffices. # detecting identical files across objects isn't as important since duplicates are mostly expected to occur as versions of the same object Based on how current activities are using the DS, this isn't like that. The most common case I have heard from the field are children downloading a PDF for reading several times. Oh, didn't know that, so it's a new requirement. An alternative to the current method for detecting duplicates is moving this task to activities, is that what you suggest? I'm ambivalent about it. On one hand it's not so easy to achieve in datastore (for various backends) and more indicative of UI deficiencies (why did the children download the file several times in the first place; it's bandwidth wastage as well), on the other hand it might not be easy to do in Browse, too. But maybe storing the URL as metadata and looking for that is enough for most cases? I guess it happens during a single session so the URL (even if including a session ID or whatever) should be stable enough? About the benefits of differential compression I would like to note that if you analize a real world journal, the biggest files are videos, mp3, pdfs, etc., so files in formats not easily editable with the activities we currently have. Which is neither an argument pro nor contra delta compression as storage requirements should be about the same either way. OTOH most activities that do support modification currently save in a text based format, so for the large number of versions I expect (remember we're autosaving on activity switch) it could be a huge
Re: [Sugar-devel] Datastore redesign
On Sun, Jul 5, 2009 at 22:32, Sascha Silbe wrote: > Hi! > > After reading the recent threads about Journal / data store and the IDs used > by them the big picture is much more clear now, thanks! > I've adjusted by datastore redesign proposal [1] accordingly and would like > to submit it for review now. > It's not clear a full redesign is The Right Thing to do now, but I'd rather > like to specify what it should look like, not what steps to take next > towards it. Agreed. I have to say that your proposal is excellent, congratulations! > There's an important design decision that's still open: > > Is the asynchronous API design useful enough to warrant more complex > implementation? I'm not sure, but I think that whatever decision we take should be made based on actual usage of the DS. What about proposing an example of how an existing activity would be modified to use the new API? > - DBus operations can be run asynchronously so UI responsiveness shouldn't > be > an issue > - For save() calls activity needs to wait for result (containing new > version_id) before it can invoke save() again for the same object > which can take quite some time if save() is sync - especially if other > activities are saving at the same time. What about having a separate call that returns synchronously a new tree_id and/or version_id? > Making the API fully asynchronous is the cause for much of the complexity of > my proposal, but if we eliminate the queueing the response times for write > accesses and checkout() can be very long even for unrelated operations. Why for unrelated operations? > [1] > http://git.sugarlabs.org/projects/versionsupport-project/repos/mainline/blobs/master/datastore-redesign.html Some comments and questions: # do we want an optimized way to determine (only) the branch HEADs of a given tree_id? This depends on the intended UI. My opinion is that if we branch at every interesting modification (triggered by the activity detecting an interesting change or by the user clicking on the Keep button), we would like to display in the object list all the HEADs of each branch in each tree_id. In that case yes, we need a way to retrieve that list that is fast on both the client and the server side. # using symlink instead of hardlink for "incoming" queue since we want to support directory trees, not just files What justifies this new requirement? I agree it would be nice to have, but I would prefer to start with the basics then grow step by step. We have had (several) cases before of overpromising datastores that failed to provide the bare minimum. # since an index rebuild can take a lot of time we need to provide UI feedback while doing that Any I/O operation can potentially take a lot of time, but with the current version of the DS rebuilding an index with a few thousands of entries is not so slow on the XO. We should never need to rebuild the index, so this new requirement might not be justified (given the current resources, all the other work we need to do, etc). # detecting identical files across objects isn't as important since duplicates are mostly expected to occur as versions of the same object Based on how current activities are using the DS, this isn't like that. The most common case I have heard from the field are children downloading a PDF for reading several times. An alternative to the current method for detecting duplicates is moving this task to activities, is that what you suggest? About the benefits of differential compression I would like to note that if you analize a real world journal, the biggest files are videos, mp3, pdfs, etc., so files in formats not easily editable with the activities we currently have. With that I don't mean is not an interesting challenge or something that we won't need in the future, just that it has a relatively low impact as of today. # activities should not submit new entries while the previously submitted one hasn't been fully committed yet Why so? # version_id and parent_id Have you thought about version_id being of the form of '2.1.4'? That would make parent_id unneeded because we could refer to the parent as (tree_id, 2.1.3). And would also allow us to identify the HEAD of each branch. # creator What is it for? # activity saves data to a disk, ensuring it has been committed (sync) and proper access rights for data store By sync you mean written to disk? Why activities need to worry about this? #Changes the (unversioned/version-specific) metadata of the given object to match metadata. Fully synchronous, no return value. How do we know which properties are version-specific and which aren't? # Remove (all versions of) given object from data store. Fully synchronous. Doesn't return anything, emits signal Deleted(tree_id). Do we have any operation in the UI that matches this? # Get/Got Maybe should we make it a bit more verbose? Like GetData? # Prefixing a key name with '!' inverts the sense of matching Is this us
