Knapp Martin wrote:

> Thanks for the in-depth reply. Are there known issues with saving LC
> stacks to Dropbox? I was thinking of Roland's issue with corruption
> and Phil's suggested fix of watching the file with the tilde character
> before allowing a quit to complete. I have an app where users create
> documents (stacks) and saving them to Dropbox is a built-in option.

My hunch (read, "I'm likely wrong, wait for something more definitive from someone on the core team or Brian Milby, since he reads the engine source <g>) is that LC files are less prone to a wide range of disk-related issues by virtue of being written in a single pass.

HyperCard, FileMaker, SQLite, and others are read and written in pages, which works more reliably in some (SQLite) than others (no one misses HC's 504 errors <g>). Moving stuff in and out of portions of a file that may not change much otherwise isn't necessarily brittle in itself (SQLite as a reputation for being enormously robust), but we can envision how larger SQLite files may confuse a syncing system that can only move chunks of data over a network so quickly.

Since LC files are, near as I recall, written fresh from beginning to end with each save command, the result is usually a cleaner file than with some paging systems.

And it's also faster to write everything in one pass than writing the same amount of data by moving back and forth throughout a file, so my hunch would be that LC stack files would likely be less prone to error when synced over Dropbox than SQLite files.

But by no means should this be taken as gospel, and certainly not as any indication that we should drop SQLite and use stack files instead. Different formats for different purposes.

And as Mark Wieder noted, some of the biggest challenges with syncing occur with multiple users accessing a given resource simultaneously.

Those are very real challenges, which may be why so many developers over the years keep writing their own sync mechanisms, with no single solution having emerged as the one-size-fits-all answer.

Consider this scenario with stack files, for example:

Mark writes a stack, then I open it, then you open it. You and I are both making changes, and I save mine a few seconds before you save yours. In that scenario, what's on disk?

When I get back to work on the file tomorrow, I'll likely be disappointed to find that everything I'd done is now gone. I can see only your changes, since yours were the last saved and when you saved you hadn't yet updated to have a copy with my changes. So after Dropbox faithfully performs everything we ask it to do, my changes are gone forever.

Write-locking can help with things like that, and that's a subject in itself. At what level of granularity should we lock? And before editing begins do we first compare what's in memory with what's in the data store to ensure what we're about to edit is the most recent copy? How do we handle conflicts, when one inevitably occurs?

So many ways to handle sync....

> I have one customer who stores their stacks on a network server and
> a few times they've ended up with a corrupted stack.

Corruption is very interesting to me. Lost data is usually explainable through simple multiuser workflow weaknesses, such as inadequate locking mechanisms. But corruption in LC means something interrupted the write, or overwrote portions before the write was completed. And since LC stacks files are written in a single pass, whatever caused it to corrupt was very efficient at it, as it had little time to do it in.

I can conceive of a possible scenario in which neither Dropbox nor LC has a bug per se at all, but which may account for corruption:

1. Your app issues a save command on a large stack.
2. Dropbox begins syncing the file to the server.
3. Your friend has the same file open on his machine.
4. On his machine, Dropbox notices the server has a newer version, and prepares to update his copy.
5. He starts a save in his stack.
6. Dropbox has written a portion of the stack from the server to his disk, but stopped when it noticed the file was being updated. 7. After your friend's save is done, the corrupted stack is then copied to the server, where it later propagates to your machine.
RESULT: everyone now has the same file, and it's broken.

I haven't studied the file system triggers and server monitoring that trigger reads and writes in Dropbox. And if it's proprietary, chances are no one outside that company has either.

There are many ways that imagined scenario might for all I know be accounted for in the way Dropbox is written. But there may also be other scenarios that can produce the same corrupted result that I haven't thought of.

 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web

use-livecode mailing list
Please visit this url to subscribe, unsubscribe and manage your subscription 

Reply via email to