[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal
On Tue, Oct 27, 2009 at 8:55 PM, Tim Steele t...@chromium.org wrote:
I can take a stab at more formal heuristics for bookmarks, at least. We
will have a better idea of actual limiting parameters for bookmarks (as in
how many operations in a certain time frame is reasonable) once the
ExtensionsActivityMonitor I just landed percolates and we can aggregate
across representative sample data it produces.
A couple thoughts:
a) I think it is overly clever to hash the changes to the bookmarks
and count per unique-params. This can be easily or accidentally
defeated by just doing something like update({title:foo}),
update({url:blech}), over and over, anyway. Instead, at least for
bookmarks, I think a simple per-item-count is very reasonable. It
doesn't make sense to me to update the same bookmark more than a few
times per minute. An easy heuristic could be that updating the same
bookmark more than twice a minute sustained over 10 minutes. For
creates it's a bit tricker. In that case, maybe the best we can do is
the same combination of properties.
b) What is wrong with solving this by rate limiting the communication
between the Chrome sync client and the Chrome sync server? It seems
like that would be more foolproof; if the client tries to send too
many updates for any reason, they are rate limited.
- a
--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---
[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal
I'm confused by the diagram. In step 5, why does F' get added to the model. Are you saying the 'extension cloud' service always creates a new bookmark, without verifying if the model already has a matching entry? -Scott On Wed, Oct 21, 2009 at 5:08 PM, Tim Steele t...@chromium.org wrote: [re-sending from correct email account] Hi, I wrote up a document that discusses some interesting unintentional relationships that can exist between independent extensions, and how this general problem also currently affects the browser sync engine. This issue was discovered from trying to explain the primary symptom of unusually high syncing traffic generated by Chrome clients. Please find it here: A Tale of Two (or more) Sync Engines You should read that before continuing! This led to me thinking about what we do long term, short term, or basically before Chrome Sync and extensions are running in parallel in a beta channel environment. You'll see a bit of this at the end of the first document, but after posing the problem as an extensions problem I ended up at a random idea that I think makes at least a little sense, though I admit I was having fun thinking and writing about it so maybe I missed some major roadblocks along the way. There are downsides, mainly revolving around the added hand-holding we would impose on extensions. Please read! Hoping for comments and feedback. Extensions API quotaserver In addition to that, Colin and Todd (cc'ed) brought up some sync specific ideas they have (I mention it a bit at the end of the first doc). We'll try to get a separate thread going about this soon! Thanks! Tim --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal
On Wed, Oct 28, 2009 at 8:46 AM, Scott Violet s...@chromium.org wrote: I'm confused by the diagram. In step 5, why does F' get added to the model. Are you saying the 'extension cloud' service always creates a new bookmark, without verifying if the model already has a matching entry? This is indeed the behavior we're seeing with one extensionhttp://www.uniformedopinion.com/ -- delete all old bookmarks, write a whole new copy. - nick -Scott On Wed, Oct 21, 2009 at 5:08 PM, Tim Steele t...@chromium.org wrote: [re-sending from correct email account] Hi, I wrote up a document that discusses some interesting unintentional relationships that can exist between independent extensions, and how this general problem also currently affects the browser sync engine. This issue was discovered from trying to explain the primary symptom of unusually high syncing traffic generated by Chrome clients. Please find it here: A Tale of Two (or more) Sync Engines You should read that before continuing! This led to me thinking about what we do long term, short term, or basically before Chrome Sync and extensions are running in parallel in a beta channel environment. You'll see a bit of this at the end of the first document, but after posing the problem as an extensions problem I ended up at a random idea that I think makes at least a little sense, though I admit I was having fun thinking and writing about it so maybe I missed some major roadblocks along the way. There are downsides, mainly revolving around the added hand-holding we would impose on extensions. Please read! Hoping for comments and feedback. Extensions API quotaserver In addition to that, Colin and Todd (cc'ed) brought up some sync specific ideas they have (I mention it a bit at the end of the first doc). We'll try to get a separate thread going about this soon! Thanks! Tim --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal
On Wed, Oct 28, 2009 at 10:42 AM, Nick Carter n...@chromium.org wrote: On Wed, Oct 28, 2009 at 8:46 AM, Scott Violet s...@chromium.org wrote: I'm confused by the diagram. In step 5, why does F' get added to the model. Are you saying the 'extension cloud' service always creates a new bookmark, without verifying if the model already has a matching entry? This is indeed the behavior we're seeing with one extension -- delete all old bookmarks, write a whole new copy. One thing I forgot I wanted to point out about that extension in particular. In Tim's proposal he says: ...in Chrome Sync, we try really hard to not get into feedback loops with ourselves, and not to make unnecessary / no-op changes, but we have seen at least one extension (GBX) periodically delete all bookmarks and then recreate all bookmarks with no changes to the nodes in question. That seems almost positively unnecessary. If we detect that they have removed the same tree (from the same root position) 10 times in one day, this does really seem fishy and enough to trigger limiting. But at that point the onus is on extension developers, not us. It sounds like Tim agrees here, but I just want to reiterate: we should not be the arbiters of quality or taste wrt extensions. There will be buggy and crappy extensions. We should try and design things such that you have to go out of your way to write crappy things, but at the end of the day we want this to be an open environment where people surprise us -- both in good and bad ways :). If we need to protect our own systems that is fine, but other than that we shouldn't prevent things in the extensions system, even if they don't seem to make much sense. -a --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal
On Wed, Oct 28, 2009 at 10:42 AM, Nick Carter n...@chromium.org wrote: On Wed, Oct 28, 2009 at 8:46 AM, Scott Violet s...@chromium.org wrote: I'm confused by the diagram. In step 5, why does F' get added to the model. Are you saying the 'extension cloud' service always creates a new bookmark, without verifying if the model already has a matching entry? This is indeed the behavior we're seeing with one extension -- delete all old bookmarks, write a whole new copy. Ugh! I don't think there is going to be a way to make it impossible to write poorly written extensions. Perhaps sync should have a way to detect lots of mutations from an extension and then disable either the extension or itself at some point with a suitable warning. It should certainly be possible to track number of mutations from an extension and to know which extension is the result of the mutations. -Scott - nick -Scott On Wed, Oct 21, 2009 at 5:08 PM, Tim Steele t...@chromium.org wrote: [re-sending from correct email account] Hi, I wrote up a document that discusses some interesting unintentional relationships that can exist between independent extensions, and how this general problem also currently affects the browser sync engine. This issue was discovered from trying to explain the primary symptom of unusually high syncing traffic generated by Chrome clients. Please find it here: A Tale of Two (or more) Sync Engines You should read that before continuing! This led to me thinking about what we do long term, short term, or basically before Chrome Sync and extensions are running in parallel in a beta channel environment. You'll see a bit of this at the end of the first document, but after posing the problem as an extensions problem I ended up at a random idea that I think makes at least a little sense, though I admit I was having fun thinking and writing about it so maybe I missed some major roadblocks along the way. There are downsides, mainly revolving around the added hand-holding we would impose on extensions. Please read! Hoping for comments and feedback. Extensions API quotaserver In addition to that, Colin and Todd (cc'ed) brought up some sync specific ideas they have (I mention it a bit at the end of the first doc). We'll try to get a separate thread going about this soon! Thanks! Tim --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal
On Tue, Oct 27, 2009 at 11:00 PM, Aaron Boodman a...@chromium.org wrote:
On Tue, Oct 27, 2009 at 8:55 PM, Tim Steele t...@chromium.org wrote:
I can take a stab at more formal heuristics for bookmarks, at least. We
will have a better idea of actual limiting parameters for bookmarks (as
in
how many operations in a certain time frame is reasonable) once the
ExtensionsActivityMonitor I just landed percolates and we can aggregate
across representative sample data it produces.
A couple thoughts:
a) I think it is overly clever to hash the changes to the bookmarks
and count per unique-params. This can be easily or accidentally
defeated by just doing something like update({title:foo}),
update({url:blech}), over and over, anyway. Instead, at least for
bookmarks, I think a simple per-item-count is very reasonable. It
doesn't make sense to me to update the same bookmark more than a few
times per minute. An easy heuristic could be that updating the same
bookmark more than twice a minute sustained over 10 minutes. For
creates it's a bit tricker. In that case, maybe the best we can do is
the same combination of properties.
The update{foo} update{blech} case is most likely a different kind of
failure, though, and I was thinking we could limit that with a generic cap
on just the number of updates in a period of time. From the data we have
seen so far, most common is 'update{foo}' in a loop type behavior. But if
the generic cap gets us far enough along, then I agree simpler is better.
b) What is wrong with solving this by rate limiting the communication
between the Chrome sync client and the Chrome sync server? It seems
like that would be more foolproof; if the client tries to send too
many updates for any reason, they are rate limited.
We have this. The problem is that the user doesn't even realize the
extension is spamming the server. Our server knows the client is producing a
lot of traffic, that's all. So what happens is we limit the client and the
user is left bewildered and helpless because a rogue extension is eating
away his quota. What I just landed was a way to correlate how many of our
sync commits originate from extensions, but we need to find a way to solve
the problem once we learn from the data. The reason I suggested this, is
because it dawned on me that this problem affects any extension author
trying to send updates from Chrome to their servers. If extension Bob and
Eve are installed, Eve can bring down Bob's service because of a silly bug.
I was proposing we just add some layer of protection in here, because we
can, to help our extensions developers out (and Chrome Sync in the process).
- a
--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---
[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal
Ugh! I don't think there is going to be a way to make it impossible to write poorly written extensions. Perhaps sync should have a way to detect lots of mutations from an extension and then disable either the extension or itself at some point with a suitable warning. It should certainly be possible to track number of mutations from an extension and to know which extension is the result of the mutations. Yes, I just added ExtensionsActivityMonitor to get this kind of data. Our quickfix idea is precisely to do this and kill either the extension or Chrome Sync if we detect that things get out of hand. But this feels like a problem that could be nipped at a lower level than Chrome Sync. If we have this kind of data available, we could make it available to the developers by reporting quota-limited extensions so they can realize that their extension is acting wrongly. Maybe they already know and/or won't do anything about it. Either way, we will likely need to disable that extension *anyway *if it is bringing down *our* service. So if we have to solve this problem for us, I don't see a great reason not to offer the help to them (e.g if Chrome Sync isn't installed) by reporting the data back and choking the traffic at its origin, instead of falling back and relying on our sync servers to differentiate traffic from different parts of the client (which would have to be done at the point when our actual Chrome sync protocol messages are being processed, which means all the server infrastructure treats these requests as valid until quite late in the chain eating up resources along the way). Granted, I'm not an extensions developer, but I have a hard time believing this wouldn't be a useful and friendly feature to offer. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal
On Wed, Oct 28, 2009 at 11:39 AM, Tim Steele t...@chromium.org wrote:
The update{foo} update{blech} case is most likely a different kind of
failure, though, and I was thinking we could limit that with a generic cap
on just the number of updates in a period of time. From the data we have
seen so far, most common is 'update{foo}' in a loop type behavior. But if
the generic cap gets us far enough along, then I agree simpler is better.
Cool. I hope and suspect we can get far enough with the simple approach.
We have this. The problem is that the user doesn't even realize the
extension is spamming the server. Our server knows the client is producing a
lot of traffic, that's all. So what happens is we limit the client and the
user is left bewildered and helpless because a rogue extension is eating
away his quota. What I just landed was a way to correlate how many of our
sync commits originate from extensions, but we need to find a way to solve
the problem once we learn from the data. The reason I suggested this, is
because it dawned on me that this problem affects any extension author
trying to send updates from Chrome to their servers. If extension Bob and
Eve are installed, Eve can bring down Bob's service because of a silly bug.
I was proposing we just add some layer of protection in here, because we
can, to help our extensions developers out (and Chrome Sync in the process).
Thanks for explaining. Makes sense to me.
- a
--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---
[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal
I can take a stab at more formal heuristics for bookmarks, at least. We
will have a better idea of actual limiting parameters for bookmarks (as in
how many operations in a certain time frame is reasonable) once the
ExtensionsActivityMonitor I just landed percolates and we can aggregate
across representative sample data it produces.
I guess the docs were too long for more folks to read through :)
For those who have read, what are thoughts on the proposal to maintain a
counter per-function, and per-item-id in some cases? So for detecting the
situation
bookmarks.update(123, title:Foo)
bookmarks.update(123, title:Foo)
bookmarks.update(123, title:Foo)
... 17 times for example
we would have a map somewhere with one entry for each bookmarks function,
and the 'update' entry would look like
( bookmarks.update, { (123, hashtitle:Foo, 20), ...} )
maintaining a history of length *at most* 1 for at most every bookmark. If
an 'update' comes in for '123' that tries to update it in some *other* way,
the hash would be different so we'd reset the count and allow the operation.
We could also either store a timestamp or use a timer task to
discard records that were 'old enough'.
For bookmarks.create, store (hashlast input, count), and limit quota to a
count of 10 per hour (mostly arbitrary numbers right now).
Same for bookmarks.remove. For starters, out of simplicity I wouldn't try
and associate removals with adds or do any cross-referencing to detect
things like 'you just added and removed the same item 100 times in a
minute', but hope that 'you just added the same item 100 times in a minute'
+ 'you just removed the same item 100 times in a minute' signals will help a
little on their own, independently from each other.
Do these seem like they would eat up too much memory?
As for requesting such a quota limitation be turned on, what are thoughts on
my proposal to request it in the manifest/permissions block? Is that doable?
We would need a way for browser/sync to toggle it too, but that's likely
just an implementation detail.
On Wed, Oct 21, 2009 at 5:55 PM, Aaron Boodman a...@chromium.org wrote:
Have not read all of it yet, but am in favor in general of having some
sort of quota for extension api calls to protect from meltdown.
We have a very convenient chokepoint to implement this in our system,
so it would just be a matter of a more detailed design of the
heuristics to use for each API type.
- a
On Wed, Oct 21, 2009 at 5:08 PM, Tim Steele t...@chromium.org wrote:
[re-sending from correct email account]
Hi,
I wrote up a document that discusses some interesting unintentional
relationships that can exist between independent extensions, and how this
general problem also currently affects the browser sync engine. This
issue
was discovered from trying to explain the primary symptom of unusually
high
syncing traffic generated by Chrome clients. Please find it here:
A Tale of Two (or more) Sync Engines
You should read that before continuing!
This led to me thinking about what we do long term, short term, or
basically
before Chrome Sync and extensions are running in parallel in a beta
channel
environment. You'll see a bit of this at the end of the first document,
but
after posing the problem as an extensions problem I ended up at a random
idea that I think makes at least a little sense, though I admit I was
having
fun thinking and writing about it so maybe I missed some major roadblocks
along the way. There are downsides, mainly revolving around the added
hand-holding we would impose on extensions. Please read! Hoping for
comments and feedback. Extensions API quotaserver
In addition to that, Colin and Todd (cc'ed) brought up some sync specific
ideas they have (I mention it a bit at the end of the first doc). We'll
try
to get a separate thread going about this soon!
Thanks!
Tim
--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---
[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal
Have not read all of it yet, but am in favor in general of having some sort of quota for extension api calls to protect from meltdown. We have a very convenient chokepoint to implement this in our system, so it would just be a matter of a more detailed design of the heuristics to use for each API type. - a On Wed, Oct 21, 2009 at 5:08 PM, Tim Steele t...@chromium.org wrote: [re-sending from correct email account] Hi, I wrote up a document that discusses some interesting unintentional relationships that can exist between independent extensions, and how this general problem also currently affects the browser sync engine. This issue was discovered from trying to explain the primary symptom of unusually high syncing traffic generated by Chrome clients. Please find it here: A Tale of Two (or more) Sync Engines You should read that before continuing! This led to me thinking about what we do long term, short term, or basically before Chrome Sync and extensions are running in parallel in a beta channel environment. You'll see a bit of this at the end of the first document, but after posing the problem as an extensions problem I ended up at a random idea that I think makes at least a little sense, though I admit I was having fun thinking and writing about it so maybe I missed some major roadblocks along the way. There are downsides, mainly revolving around the added hand-holding we would impose on extensions. Please read! Hoping for comments and feedback. Extensions API quotaserver In addition to that, Colin and Todd (cc'ed) brought up some sync specific ideas they have (I mention it a bit at the end of the first doc). We'll try to get a separate thread going about this soon! Thanks! Tim --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
