Richard Newman wrote:
Concurrent modifications happen (both from user input and due to user
activity), and a system not specifically designed to address this (and races,
and offlining, and other kinds of timing/conflict issues) will *at sufficient
usage* suffer from a background hum of user dissatisfaction.
This is simply mathematically incorrect. I asked you for specific,
realistic, relevant use cases, not overly general statements that are
simply not backed by any facts. Events with infinitely small probability
are irrelevant, even in sizable populations ("sufficient usage"). Until
you are willing to discuss concrete cases you are merely adding noise here.
You're specifically proposing that we build and ship a system we *know* to be
less reliable than Firefox Sync, just to get to market sooner and move some
complexity around. I don't think avoiding domain-specific reconciling buys us
anything worth a constant hiss of avoidable data loss.
Existing sync is atrociously unreliable, actually. You are conflating
reliability and "conflict resolution rules". The relative quality of
conflict resolution of current sync is entirely irrelevant. We must
resolve conflicts well enough to solve our given use cases. As long we
resolve conflicts well enough for our use cases, its irrelevant if we
resolve many fewer corner cases than current sync. Remember that the
cases we discussed are black swans. It doesn't matter how many black
swans you support, because you won't encounter them often enough to matter.
The same scenario applies for passwords. They very rarely change, and they tend
to change in a linear fashion. The chances that I independently change a
password to two different values while not syncing just with the right timing
is simply irrelevant.
I think you're assuming that sync records contain only user-entered data, and
only change when users act on them. That's at odds with the data we record to
provide our user experience, which is part of the data we sync. That's what I
was alluding to when I mentioned timestamps, which are eminently reconcilable
and automatically change with some frequency.
Ok, now I guess your stale argument is back. Please help me understand
in what precise scenario it matters to precisely reconcile timestamps
instead of in a sufficiently approximate matter? Or counters for that
matter.
Perhaps a good example of user-entered data is bookmark structure. If you dump
something in your toolbar on your tablet, and do *anything* to your toolbar on
your desktop, you have a conflict in whatever record or records defines the
toolbar structure.
Yes, this is an important case to design right. Current sync expresses
the bookmark tree as individual bookmarks, and structure is built upon
that. That is very difficult to sync with couchdb's model. I raised this
concern earlier with Lloyd. One way to resolve this is to sync entire
folders at once. If you make concurrent changes to the same folder,
someone loses. By using the right key (I proposed full folder path name
as the key) you always will have a consistent structure. Again, between
desktop and mobile there won't be a lot of conflicts, since those use
different roots, and as Doug pointed out very well, we want to push for
near-instant sync. Other models are plausible, each with slightly
different characteristics. I have yet to see one that doesn't satisfy MVP.
(We especially cannot do this if there's structure involved, because
consistency spans individual records; conflict resolution requires an
understanding of the data model. Achieving consistency is a big motivator for
the *Sync proposals.)
Consistency by definition cannot be achieved if you have non-zero latency and
disconnected operation.
I think you and I are talking about different definitions of consistency.
I think this is correct. You are talking about abstract consistency. I
am talking about engineering: solving the use cases we were given, on a
time scale we were given.
For the kind of consistency we care about here -- stable modification of sets
of records -- you need a shared store, an atomic write primitive, the right
data model, for the server to not alter the contents of content-addressable
records, and for clients to retry when the atomic write fails due to a conflict.
What you can't have is one client saying "folder X contains items A, B, C", and another
client having just uploaded "folder X contains items A, D", and for the server's magic
reconciling approach to silently orphan B and C. A client (probably the first) needs to know so
that it can do something sane -- merge the items, for example.
You should read up a bit on eventual consistency. Raising atomic write
primitives in a distributed data model is a no-go.
Furthermore, a sync implementation already needs to do client-side
content-based data merging if it wants to support setup without blowing away
your local data. So it's not like this is a significant additional requirement.
There is no clear user stories on this for the MVP. Without such a hard requirement a
"Setting up sync with an existing account will delete all local bookmarks.
Yes/No." is a perfectly fine solution for the first shipping version when we detect
a 2nd device being linked.
If it's not in the list, I suspect that's because nobody considered that scenario in
drafting the MVP. It's not in any "not-MVP" list I've seen.
I thought I was clear in the meeting, but let me fix that for you:
anything thats not on the list, is explicitly out of scope, until we
re-open use cases. Anything that is not on that list is by definition on
the "not-MVP" list.
As a user, I expect that setting up Sync on my phone doesn't delete the data on
my desktop or phone, instead merging everything together in a way that makes
sense.
Yep, thats exactly what we will do, except for black swan events, which
don't matter.
And at the extreme, where a data type *does* admit to a "don't bother"
reconciling technique, that's trivial to implement on the client: a client downloading a
server record that 'conflicts' will just apply it over their local record. No conflict.
We managed to not have an acceptable sync solution for many years. If these
things were trivial, we wouldn't be at this juncture today.
From my perspective, the reasons we don't have an acceptable sync solution are
(a) we didn't rewrite Weave before it shipped in Fx4, and thus were baked into
a fundamentally limited protocol and were stuck maintaining a Labs codebase,
(b) we were never adequately resourced to build something new while still
keeping the old one from cracking (and adding features to it, and porting it to
Android!), and (c) each abortive attempt at a redesign included backward
compatibility or extensive code reuse requirements. Until it was too late.
In short: we don't have an acceptable Sync solution because we were too busy
working on the unacceptable one to build anything else.
That's water under the bridge, but I think it's important context.
Now that's we've organizationally said "let it burn and build something else", I very much want
PiCL to avoid sliding down the same slope of expedient decision making, which will ultimately result in us
shipping something that's more "labsy" and less "producty". That'll kill us in the long
run.
PiCL is our one shot at building the replacement for Sync. We should make sure
it's not Weave.
I understand that you're very focused on getting to market. I don't see much
value in getting to market with a solution that's worse than what we have, and
doesn't let us do better without another rewrite. If we do that, we'll stay DFL.
I've spent the last two and a half years fielding bugs and user reports that
shine very bright lights on edge case assumptions, not to mention building a
brand new Sync 1.1 client from the ground up and seeing all the design flaws
anew. My focus is on making sure we ship something that won't make us bleed to
death from paper cuts or axe wounds.
There is so much wrong with your analysis that I don't know where to
start. Fortunately, its also so far out of scope that I don't have to.
Your perspective is appreciated, but we have made certain decisions at
this point, and we will not revisit them. Until we get different
guidance, the use cases are what they are, and the goal of getting an
MVP to market is set. If you disagree with that strategy, we will have
to agree to disagree, but I will still need your full support with
implementing it. If you would like to help with resolving remaining
technical detail issues, please do so by listing concrete, relevant,
realistic use cases. General statements of disagreement are not useful.
Thanks,
Andreas
-R
_______________________________________________
Sync-dev mailing list
[email protected]
https://mail.mozilla.org/listinfo/sync-dev
