Re: Best practices for data types to be synced

[Luke Chang]

Hi Richard, Nicholas and syn-dev,

We've come up with our first version of the form autofill schema and would
like to share with you (see [1]).

Also, since JS-implementation on top of SQLite is not that reusable for
other devices, we will simply use JSON-based storage and leverage the
existing module used in Password Manager [2].

Any feedback are welcome.

Thanks,
Luke

[1] https://docs.google.com/document/d/1HsJCn5XtWKD3kWupZQ6oXgHeEp3Dq
1Z88xWSCO3weLY/edit?usp=sharing
[2]
https://dxr.mozilla.org/mozilla-central/source/toolkit/components/passwordmgr/LoginStore.jsm


2016-09-03 9:27 GMT+08:00 Richard Newman :

> I guess it depends on how we would want to handle a conflict where two
>> clients changed their phone numbers (added or removed) before a sync
>> happened. Some methods for handling that would require more than just an
>> array of strings containing phone numbers e.g. timestamps.
>>
>
> You can make the record reconciler as complicated as you wish; indeed, if
> you store all local changes, the shared parent, and buffer the incoming
> record you could essentially write your own one-big-JSON-blob synchronizer.
>
> The rule for Sync is: if two things are related (that is: changes to one
> must be seen at the same time as another), then they should be in the same
> record. That can mean lots of separate records, or several big records, or
> even one huge record (if you can fit it in 256K).
>
>
> ​I said I'd consider using SQLite for desktop if we can share
>> implementation code. I guess I was hoping for a shared JS-implemented
>> version but didn't think that would be usable on iOS, for example (at least
>> I don't think storage-mozStorage.js was used for passwords on iOS).
>
>
> Your expectations are correct: storage on iOS is a wholly different beast.
>
>
> ​It seems like these same motivations would apply to form autofill
>> profiles and therefore I likely wouldn't be able to share a JS-implemented
>> storage implementation, right​?
>
>
> If you use JSON, it needs to be owned by Java. If you use SQLite, it can
> be reused, but it's a little painful. Our experience is that it's easier to
> just write the whole thing again in Java than it is to use Gecko code.
>
>
___
Sync-dev mailing list
Sync-dev@mozilla.org
https://mail.mozilla.org/listinfo/sync-dev

Re: Best practices for data types to be synced

[Richard Newman]

>
> As far as form data, I think we can include it with passwords. In our
> recent card sort, 42% of users grouped together Form Autofill Options and
> Saved Logins & Passwords.
>

To drill down on this:

Existing form history is actually already controlled by the *History*
checkbox — it's the history of things you typed into web forms, if you
squint. I would be surprised if users expected their addresses etc. to do
the same thing, so this might need to be carefully revisited.

(And clumping the user's shipping address in with their saved passwords
might be too aggressive — I might be okay with my shipping stuff syncing to
my work machine, but not my history or my passwords.)
___
Sync-dev mailing list
Sync-dev@mozilla.org
https://mail.mozilla.org/listinfo/sync-dev

Re: Best practices for data types to be synced

[Ryan Feeley]

>
> * There are currently 6 checkboxes for sync "engines".
>

> * Thom's got a bug where we want to sync search engine preferences - which
> ends up wanting to sync the engines themselves to do a sane thing. We're
> thinking that maybe we can maybe abuse the "preferences" collection for
> that, but it still leaves the UX - do we need to have a UI that
> differentiates "preferences" from "search engines and related preferences"?
>

I think "Preferences" is a fine place for this, does not require extra
explanation. The UI for it is definitely within the Preferences section.


> * Light-weight themes already abuse the preferences engine - they carry
> the pref name, but not the themes themselves. There's already a hack to
> special case the setting of the pref name using the LWTheme API, but doing
> this right also looks like the search engines above WRT also syncing the
> themes.
>

Oh, I assumed all themes were synced under "Add-ons". Perhaps I'm not
understanding something.


> auto-fill seems a similar "scope-creep" problem - and one we will keep
> hitting as we improve what can be synced. I doubt we want 10+ checkboxes
> and I'm not sure we can group all this stuff under a generic "Preferences".
> Maybe call it "Firefox Settings" and stick with about:config for advanced
> users? Maybe re-think the prefs UI entirely? Something else?
>
> rfeeley, WDYT?


You are correct. I don't want to add addtional checkboxes.

I think the approach to add little-understood items like add-ons data to
about:config makes total sense.

As far as form data, I think we can include it with passwords. In our
recent card sort, 42% of users grouped together Form Autofill Options and Saved
Logins & Passwords.

https://www.optimalworkshop.com/optimalsort/x4jm4zoj/winning/shared-results/s63v8ih8h5euj07yjf3n2k4k6648by47#/t/analysisTools/similarityMatrix

The UI we're going to soon be testing for prefs will look something like
this.

[image: Inline image 1]

I modified it from the current prototype which we'll be polishing and
testing with users. It's based on some great research we did that seems to
make users 30% more successful at finding specific preferences (with 8
popular tasks). We tested our current prefs, then ran a card sort
experiment to see how users would group all the prefs, and then we tested
again, and now we're finishing and testing this prototype.

Take a peek (knowing it is a WIP)
http://people.mozilla.org/~mliang/Preferences/#general

Ryan Feeley
Sync & Accounts UX
Mozilla Toronto
___
Sync-dev mailing list
Sync-dev@mozilla.org
https://mail.mozilla.org/listinfo/sync-dev

Re: Best practices for data types to be synced

[Mark Hammond]

On 3/09/2016 9:41 AM, Matthew N. wrote:

Hello sync-dev,

Is there documentation on best practices for making a new data type
that will eventually be synced? I'm trying to figure out a schema and
storage backend for Form Autofill profiles (on desktop to start) and
though we're not implementing the sync engine initially, I would like
the data to be ready for sync without significant re-implemention of
the storage.


Sorry for the delay, but in addition to the "sync engine" semantics 
already discussed, this also raises UX we are already starting to hit in 
other ways. How do we expose a UI for syncing additional data without 
overwhelming the user?


eg:

* There are currently 6 checkboxes for sync "engines".

* There's the new chrome.storage.sync facility (bug 1290440). Instead of 
adding a 7th checkbox we took a short-term decision to tie it to the 
"addons" engine in the UI while offering an about:config pref to allow 
one without the other.


* Thom's got a bug where we want to sync search engine preferences - 
which ends up wanting to sync the engines themselves to do a sane thing. 
We're thinking that maybe we can maybe abuse the "preferences" 
collection for that, but it still leaves the UX - do we need to have a 
UI that differentiates "preferences" from "search engines and related 
preferences"?


* Light-weight themes already abuse the preferences engine - they carry 
the pref name, but not the themes themselves. There's already a hack to 
special case the setting of the pref name using the LWTheme API, but 
doing this right also looks like the search engines above WRT also 
syncing the themes.


auto-fill seems a similar "scope-creep" problem - and one we will keep 
hitting as we improve what can be synced. I doubt we want 10+ checkboxes 
and I'm not sure we can group all this stuff under a generic 
"Preferences". Maybe call it "Firefox Settings" and stick with 
about:config for advanced users? Maybe re-think the prefs UI entirely? 
Something else?


rfeeley, WDYT?

Mark
___
Sync-dev mailing list
Sync-dev@mozilla.org
https://mail.mozilla.org/listinfo/sync-dev

Re: Best practices for data types to be synced

[Richard Newman]

Done. The main syncing observations are around dates; I think storing
creation and modified times is fine, so long as you apply some careful
thought when syncing those values (and we can cross that bridge when we
come to it).

I left some pointers to the two relevant "Falsehoods programmers believe
about…", which should be enough encouragement to store an arbitrarily large
unconstrained "name" field rather than thinking about concatenation, and
detail fields (a concession to bad data models on the web) instead of
trying to split. I also left a test case for you.

Similarly, try to make the address field(s) as unconstrained as possible:
addresses can even have multiple countries, so once again the world and
ecommerce sites collide :)

On Thu, Sep 22, 2016 at 12:56 AM, Luke Chang  wrote:

> Hi Richard, Nicholas and syn-dev,
>
> We've come up with our first version of the form autofill schema and would
> like to share with you (see [1]).
>
> Also, since JS-implementation on top of SQLite is not that reusable for
> other devices, we will simply use JSON-based storage and leverage the
> existing module used in Password Manager [2].
>
> Any feedback are welcome.
>
> Thanks,
> Luke
>
> [1] https://docs.google.com/document/d/1HsJCn5XtWKD3kWupZQ6o
> XgHeEp3Dq1Z88xWSCO3weLY/edit?usp=sharing
> [2] https://dxr.mozilla.org/mozilla-central/source/toolkit/components/
> passwordmgr/LoginStore.jsm
>
>
> 2016-09-03 9:27 GMT+08:00 Richard Newman :
>
>> I guess it depends on how we would want to handle a conflict where two
>>> clients changed their phone numbers (added or removed) before a sync
>>> happened. Some methods for handling that would require more than just an
>>> array of strings containing phone numbers e.g. timestamps.
>>>
>>
>> You can make the record reconciler as complicated as you wish; indeed, if
>> you store all local changes, the shared parent, and buffer the incoming
>> record you could essentially write your own one-big-JSON-blob synchronizer.
>>
>> The rule for Sync is: if two things are related (that is: changes to one
>> must be seen at the same time as another), then they should be in the same
>> record. That can mean lots of separate records, or several big records, or
>> even one huge record (if you can fit it in 256K).
>>
>>
>> ​I said I'd consider using SQLite for desktop if we can share
>>> implementation code. I guess I was hoping for a shared JS-implemented
>>> version but didn't think that would be usable on iOS, for example (at least
>>> I don't think storage-mozStorage.js was used for passwords on iOS).
>>
>>
>> Your expectations are correct: storage on iOS is a wholly different beast.
>>
>>
>> ​It seems like these same motivations would apply to form autofill
>>> profiles and therefore I likely wouldn't be able to share a JS-implemented
>>> storage implementation, right​?
>>
>>
>> If you use JSON, it needs to be owned by Java. If you use SQLite, it can
>> be reused, but it's a little painful. Our experience is that it's easier to
>> just write the whole thing again in Java than it is to use Gecko code.
>>
>>
>
___
Sync-dev mailing list
Sync-dev@mozilla.org
https://mail.mozilla.org/listinfo/sync-dev

Re: Best practices for data types to be synced

[Richard Newman]

>
> I guess it depends on how we would want to handle a conflict where two
> clients changed their phone numbers (added or removed) before a sync
> happened. Some methods for handling that would require more than just an
> array of strings containing phone numbers e.g. timestamps.
>

You can make the record reconciler as complicated as you wish; indeed, if
you store all local changes, the shared parent, and buffer the incoming
record you could essentially write your own one-big-JSON-blob synchronizer.

The rule for Sync is: if two things are related (that is: changes to one
must be seen at the same time as another), then they should be in the same
record. That can mean lots of separate records, or several big records, or
even one huge record (if you can fit it in 256K).


​I said I'd consider using SQLite for desktop if we can share
> implementation code. I guess I was hoping for a shared JS-implemented
> version but didn't think that would be usable on iOS, for example (at least
> I don't think storage-mozStorage.js was used for passwords on iOS).


Your expectations are correct: storage on iOS is a wholly different beast.


​It seems like these same motivations would apply to form autofill profiles
> and therefore I likely wouldn't be able to share a JS-implemented storage
> implementation, right​?


If you use JSON, it needs to be owned by Java. If you use SQLite, it can be
reused, but it's a little painful. Our experience is that it's easier to
just write the whole thing again in Java than it is to use Gecko code.
___
Sync-dev mailing list
Sync-dev@mozilla.org
https://mail.mozilla.org/listinfo/sync-dev

Re: Best practices for data types to be synced

[Matthew N.]

Thanks Richard, I'll dig into this more next week.

On Fri, Sep 2, 2016 at 6:03 PM, Richard Newman  wrote:

> Is there documentation on best practices for making a new data type that
>>> will eventually be synced? I'm trying to figure out a schema and storage
>>> backend for Form Autofill profiles (on desktop to start) and though we're
>>> not implementing the sync engine initially, I would like the data to be
>>> ready for sync without significant re-implemention of the storage.
>>>
>>
>> I'm not aware of a link to such documentation.
>>
>
> Me neither, sadly. There's a lot of accumulated knowledge in mailing
> lists, wikis, and bugs… but digging it up takes more time than just writing
> it here!
>
>
> You're correct.  A GUID as the primary key is preferred, since you'll need
>> to track a (remote) GUID anyway.
>>
>
> To expand on this:
>
> Sync records have an identifier. This is preferably a short (12-char)
> Places-style GUID, but some older engines use UUIDs instead. The identifier
> is its name on the server, so it's cleartext — for this reason we don't
> just use URLs or straightforward hashes of URLs.
>
> In some cases you might wish to store a salt somewhere and hash URLs — no
> need to store a GUID.
>
> In some circumstances you'll need to replace your local GUID with one from
> the server.
>
> In addition to Nick's notes about best practices/thoughts, here are some
> (not necessarily aimed at autofill):
>
>
>- You'll need a way to decide at the moment of syncing which local
>records have changed. The desktop engines typically watch observer
>notifications, but this has drawbacks — it's hard to spot exactly what
>changed (move? rename?), and we tend to do crazy things like not send all
>the notifications if too many items change. On Android we use timestamps,
>which is wrong, but so it goes. On iOS (and soon on desktop for bookmarks!)
>we directly keep track of Sync-related changes as storage operations hit
>the disk, which is ideal. I've discussed this at some length if you're
>interested
>.
>- You'll need to figure out how your feature works with Refresh
>Firefox, and what Sync is supposed to do in that case.
>- You'll have to decide whether your feature:
>   - Gets a checkbox of its own in Preferences and in the FxA setup
>   flow. Form History, for example, reuses History's checkbox.
>   - Is enabled by default. (See existing bugs
>   .)
>   - Syncs on desktop, iOS, and/or Android.
>- Document your object format
>. I can help
>with that.
>- You should give some thought to how the object format might change
>over time, and whether it's acceptable to lock out clients who don't
>support the new format. This is a topic of its own.
>- Consider writing TPS tests. The rest of the Sync team can help with
>that.
>- Involve the server and ops folks in your design process. Records
>that change very frequently, are very large, or both, are not friendly
>to sync .
>- Consider whether Firefox Sync itself is a good candidate for syncing
>your data type. Sync is good for:
>   - Strong crypto
>   - Relatively small and infrequent changes
>   - Non-realtime
>   - Independent records
>   - Less than 25MB total
>   - No sharing between users.
>
> so, for example, we use Kinto on desktop and Android for a bunch of stuff
> that doesn't require that collection of properties.
>
>
​Thanks for more details on best practices for the engine implementation
but, to be clear, there are no short term plans to implement the sync
engine, the focus is on the storage for the feature itself but I'm hoping
to make it friendly to writing the sync engine later.​ Regardless, these
notes will be useful for that future time when we're implementing the
engine.

As for best practices, the number one thing is to remember that Sync is an
>> object store with no good way to maintain references between objects.
>> Therefore, if you can keep records independent of each other, you will have
>> a *much* better time.  As an example, history records are independent, but
>> bookmark records are not: bookmark folders reference bookmark items (and
>> vice-versa -- parents!) and no end of pain ensues trying to keep references
>> coherent.
>>
>
> I can't agree with this enough. Reading the design notes for iOS bookmark
> sync , or talking
> to markh and Kit, should be enough to scare you away from interdependent
> records.
>
>
> Storage implementations don't have to be complicated on Android; neither
>> do Sync engine implementations.  (Once you've decided how to reconcile
>> conflicts, of course!)
>>
>
> Indeed: it's simpler t

Re: Best practices for data types to be synced

[Matthew N.]

On Fri, Sep 2, 2016 at 4:52 PM, Nicholas Alexander 
wrote:

> On Fri, Sep 2, 2016 at 4:41 PM, Matthew N.  wrote:
>
>> Hello sync-dev,
>>
>> Is there documentation on best practices for making a new data type that
>> will eventually be synced? I'm trying to figure out a schema and storage
>> backend for Form Autofill profiles (on desktop to start) and though we're
>> not implementing the sync engine initially, I would like the data to be
>> ready for sync without significant re-implemention of the storage.
>>
>
> I'm not aware of a link to such documentation.
>

​OK, I kinda suspected that since the harder part is the engine itself.

On the schema side, an obvious thing is to use a GUID (shorter ones are
>> preferred IIUC) instead of an incrementing ID for the primary key. It seems
>> things like tracking of deleted records and how to handle conflicts between
>> the server and client can be figured out when the engine is added but
>> correct me if I'm wrong?
>>
>
> You're correct.  A GUID as the primary key is preferred, since you'll need
> to track a (remote) GUID anyway.
>
> As for best practices, the number one thing is to remember that Sync is an
> object store with no good way to maintain references between objects.
> Therefore, if you can keep records independent of each other, you will have
> a *much* better time.  As an example, history records are independent, but
> bookmark records are not: bookmark folders reference bookmark items (and
> vice-versa -- parents!) and no end of pain ensues trying to keep references
> coherent.
>

​Right, I see Chromium is using multiple tables and one-to-many
​relationships for data like names, email addresses and phone numbers but I
have yet to figure out if that's something we will want. I would guess
their design is optimizing for the case where you add a phone number or
email address to an existing profile in which case it's just a matter of
adding a row with two fields (guid, number/address). I guess it depends on
how we would want to handle a conflict where two clients changed their
phone numbers (added or removed) before a sync happened. Some methods for
handling that would require more than just an array of strings containing
phone numbers e.g. timestamps.


> Concretely, that means you should prefer a JSON record per-profile rather
> than a profile record with sub-item records; or perhaps you would make each
> record contain a "profile name" and an item identifier, etc.
>

​Makes sense.​



> Regarding the storage method, my understanding is that SQLite is used on
>> mobile to provide access to the data from a separate non-Gecko sync process
>> (at least on Android, not sure how things work on iOS). For desktop I would
>> prefer to use a simple JSON file like logins.json since we don't need the
>> features of an RDBMS but it's possible to re-use a SQLite backend between
>> desktop and mobile then I'll consider it. I'm not sure if that's the case
>> though since it seems like each client would write it's own native wrapper
>> around the SQLite
>>
>
> You may be able to share storage implementation code, if you back your
> store with SQLite.  Since you don't want to (JSON on Desktop), you won't be
> able to.  So Android will require its own storage implementation, and its
> own Sync engine implementation.  iOS will require the same.
>

​I said I'd consider using SQLite for desktop if we can share
implementation code. I guess I was hoping for a shared JS-implemented
version but didn't think that would be usable on iOS, for example (at least
I don't think storage-mozStorage.js was used for passwords on iOS).


> Storage implementations don't have to be complicated on Android; neither
> do Sync engine implementations.  (Once you've decided how to reconcile
> conflicts, of course!)
>

Sure, but it adds to the testing matrix and I would like to avoid that if
possible.

(I saw Passwords was moving to a Java implementation on Android though I'm
>> not sure of the motivations for that TBH) in which case very little code
>> would be shared.
>>
>
> The storage layer exists, but it's not currently used.  The motivation is
> multi-fold.  First, we see many PW storage failures on Android.  That is,
> the DB is corrupt, crypto failed, whatever.  Second, we see concurrency
> problems, performance loss, and memory waste due to the required additional
> process for Gecko's SQLite compatibility.  And finally, we'd get closer to
> severing the Fennec front-end from Gecko if the store was independent of
> Gecko.  (In general, we've learned never to use Gecko if we want a good
> Android experience.)
>

​It seems like these same motivations would apply to form autofill profiles
and therefore I likely wouldn't be able to share a JS-implemented storage
implementation, right​?


> Happy to elaborate in more detail; it would help for you to post your
> suggested schema and Desktop storage proposal.
>

​Yeah, I plan to share those when I have them figured out. I wanted to hear
what the

Re: Best practices for data types to be synced

[Richard Newman]

>
> Is there documentation on best practices for making a new data type that
>> will eventually be synced? I'm trying to figure out a schema and storage
>> backend for Form Autofill profiles (on desktop to start) and though we're
>> not implementing the sync engine initially, I would like the data to be
>> ready for sync without significant re-implemention of the storage.
>>
>
> I'm not aware of a link to such documentation.
>

Me neither, sadly. There's a lot of accumulated knowledge in mailing lists,
wikis, and bugs… but digging it up takes more time than just writing it
here!


You're correct.  A GUID as the primary key is preferred, since you'll need
> to track a (remote) GUID anyway.
>

To expand on this:

Sync records have an identifier. This is preferably a short (12-char)
Places-style GUID, but some older engines use UUIDs instead. The identifier
is its name on the server, so it's cleartext — for this reason we don't
just use URLs or straightforward hashes of URLs.

In some cases you might wish to store a salt somewhere and hash URLs — no
need to store a GUID.

In some circumstances you'll need to replace your local GUID with one from
the server.

In addition to Nick's notes about best practices/thoughts, here are some
(not necessarily aimed at autofill):


   - You'll need a way to decide at the moment of syncing which local
   records have changed. The desktop engines typically watch observer
   notifications, but this has drawbacks — it's hard to spot exactly what
   changed (move? rename?), and we tend to do crazy things like not send all
   the notifications if too many items change. On Android we use timestamps,
   which is wrong, but so it goes. On iOS (and soon on desktop for bookmarks!)
   we directly keep track of Sync-related changes as storage operations hit
   the disk, which is ideal. I've discussed this at some length if you're
   interested
   .
   - You'll need to figure out how your feature works with Refresh Firefox,
   and what Sync is supposed to do in that case.
   - You'll have to decide whether your feature:
  - Gets a checkbox of its own in Preferences and in the FxA setup
  flow. Form History, for example, reuses History's checkbox.
  - Is enabled by default. (See existing bugs
  .)
  - Syncs on desktop, iOS, and/or Android.
   - Document your object format
   . I can help
   with that.
   - You should give some thought to how the object format might change
   over time, and whether it's acceptable to lock out clients who don't
   support the new format. This is a topic of its own.
   - Consider writing TPS tests. The rest of the Sync team can help with
   that.
   - Involve the server and ops folks in your design process. Records that
   change very frequently, are very large, or both, are not friendly to sync
   .
   - Consider whether Firefox Sync itself is a good candidate for syncing
   your data type. Sync is good for:
  - Strong crypto
  - Relatively small and infrequent changes
  - Non-realtime
  - Independent records
  - Less than 25MB total
  - No sharing between users.

so, for example, we use Kinto on desktop and Android for a bunch of stuff
that doesn't require that collection of properties.



>
> As for best practices, the number one thing is to remember that Sync is an
> object store with no good way to maintain references between objects.
> Therefore, if you can keep records independent of each other, you will have
> a *much* better time.  As an example, history records are independent, but
> bookmark records are not: bookmark folders reference bookmark items (and
> vice-versa -- parents!) and no end of pain ensues trying to keep references
> coherent.
>

I can't agree with this enough. Reading the design notes for iOS bookmark
sync , or talking to
markh and Kit, should be enough to scare you away from interdependent
records.


Storage implementations don't have to be complicated on Android; neither do
> Sync engine implementations.  (Once you've decided how to reconcile
> conflicts, of course!)
>

Indeed: it's simpler to write a safe Java JSON-backed store and make it
accessible to Sync and to Gecko than it is to access a Gecko-owned sqlite
database.


The storage layer exists, but it's not currently used.  The motivation is
> multi-fold.  First, we see many PW storage failures on Android.  That is,
> the DB is corrupt, crypto failed, whatever.  Second, we see concurrency
> problems, performance loss, and memory waste due to the required additional
> process for Gecko's SQLite compatibility.  And finally, we'd get closer to
> severing the Fennec front-end from Gecko if the store was independent of
> Gecko.  (In general, we've learned 

Re: Best practices for data types to be synced

[Nicholas Alexander]

On Fri, Sep 2, 2016 at 4:41 PM, Matthew N.  wrote:

> Hello sync-dev,
>
> Is there documentation on best practices for making a new data type that
> will eventually be synced? I'm trying to figure out a schema and storage
> backend for Form Autofill profiles (on desktop to start) and though we're
> not implementing the sync engine initially, I would like the data to be
> ready for sync without significant re-implemention of the storage.
>

I'm not aware of a link to such documentation.


> On the schema side, an obvious thing is to use a GUID (shorter ones are
> preferred IIUC) instead of an incrementing ID for the primary key. It seems
> things like tracking of deleted records and how to handle conflicts between
> the server and client can be figured out when the engine is added but
> correct me if I'm wrong?
>

You're correct.  A GUID as the primary key is preferred, since you'll need
to track a (remote) GUID anyway.

As for best practices, the number one thing is to remember that Sync is an
object store with no good way to maintain references between objects.
Therefore, if you can keep records independent of each other, you will have
a *much* better time.  As an example, history records are independent, but
bookmark records are not: bookmark folders reference bookmark items (and
vice-versa -- parents!) and no end of pain ensues trying to keep references
coherent.

Concretely, that means you should prefer a JSON record per-profile rather
than a profile record with sub-item records; or perhaps you would make each
record contain a "profile name" and an item identifier, etc.


> Regarding the storage method, my understanding is that SQLite is used on
> mobile to provide access to the data from a separate non-Gecko sync process
> (at least on Android, not sure how things work on iOS). For desktop I would
> prefer to use a simple JSON file like logins.json since we don't need the
> features of an RDBMS but it's possible to re-use a SQLite backend between
> desktop and mobile then I'll consider it. I'm not sure if that's the case
> though since it seems like each client would write it's own native wrapper
> around the SQLite
>

You may be able to share storage implementation code, if you back your
store with SQLite.  Since you don't want to (JSON on Desktop), you won't be
able to.  So Android will require its own storage implementation, and its
own Sync engine implementation.  iOS will require the same.

Storage implementations don't have to be complicated on Android; neither do
Sync engine implementations.  (Once you've decided how to reconcile
conflicts, of course!)


> (I saw Passwords was moving to a Java implementation on Android though I'm
> not sure of the motivations for that TBH) in which case very little code
> would be shared.
>

The storage layer exists, but it's not currently used.  The motivation is
multi-fold.  First, we see many PW storage failures on Android.  That is,
the DB is corrupt, crypto failed, whatever.  Second, we see concurrency
problems, performance loss, and memory waste due to the required additional
process for Gecko's SQLite compatibility.  And finally, we'd get closer to
severing the Fennec front-end from Gecko if the store was independent of
Gecko.  (In general, we've learned never to use Gecko if we want a good
Android experience.)

Happy to elaborate in more detail; it would help for you to post your
suggested schema and Desktop storage proposal.

Best,
Nick
___
Sync-dev mailing list
Sync-dev@mozilla.org
https://mail.mozilla.org/listinfo/sync-dev