Re: Service Worker issues

[Ben Kelly]

On Thu, Jul 28, 2016 at 4:13 PM, Tab Atkins Jr. 
wrote:

> > caches.open("blog - 2016-06-10 14:14:23 -0700").then(c => c.keys())
> > Promise { : "pending" }
>
> Note that this test will *not* tell you whether or not c.keys()
> returns a promise; the .then callback is allowed to return a
> non-promise, and .then() always returns a promise regardless.  You
> have to log the return value of c.keys() directly.
>

caches.open("blog - 2016-06-10 14:14:23 -0700").then(c =>
console.log(c.keys()))
Promise { : "pending" }
Promise { : "pending" }

Re: Service Worker issues

[Ben Kelly]

On Wed, Jul 27, 2016 at 11:57 AM, Sam Ruby  wrote:

> The following is a mix of spec and implementation issues that I
> encountered in my as-of-yet unsuccessful attempt to make use of service
> workers in the ASF Board Agenda tool.
>

First, let me say thank you for the feedback!  Its very helpful to hear
from developers.


> 1) the "offline fallback" use case for Service Workers involves
> intercepting fetch requests, issuing the request, and then recovering from
> any errors that may occur.  Unfortunately, issuing the event.request will
> not faithfully reproduce the original navigate request, in particular,
> credentials are not included.
>

Yes, there have been a lot of bugs that fall into this category.  Its
re-creating the Response object from the browser native network stack is
particularly prone to missing information.

If you can be more specific about the type of request you are seeing this
with we can write a WPT test and make sure it doesn't regress.


> 3) event.request.headers.get("Authorization") is available on Firefox but
> not on Chrome.
>

Is this Authorization header on the original request being set by the
browser or manually in an XHR/fetch?

Looking at the spec the service worker should probably *not* see
Authorization headers added by the browser.  That's supposed to happen
after the service worker interception.


> 4) cache.keys() is supposed to return a promise.  On both Firefox and
> Chrome, cache.keys() returns an empty array instead.  cache.matchAll() can
> be used to get all of the responses, but not the keys.
>

Can you share the code you are using?  It returns a promise in our tests
and in my quick check in the console:

caches.open("blog - 2016-06-10 14:14:23 -0700").then(c => c.keys())
Promise { : "pending" }


> 5) calling registration.unregister() followed by
> navigator.serviceWorker.getRegistrations() will still return a list
> containing the now unregistered service worker in both Firefox and Chrome.
>

Are you waiting for the promise returned by unregister() to resolve?


> 6) EventSource is available to service workers in Chrome but not in
> Firefox.  Using importScripts in an attempt to load the Yaffle polyfill
> doesn't work.
>

We have a bug filed for this:

https://bugzilla.mozilla.org/show_bug.cgi?id=1267903

Thanks again.

Ben

Re: [Service Workers] meeting july/august?

[Ben Kelly]

On Thu, May 26, 2016 at 4:59 PM, Ben Kelly <bke...@mozilla.com> wrote:

> On Wed, May 25, 2016 at 9:19 PM, Chaals McCathie Nevile <
> cha...@yandex-team.ru> wrote:
>
>> Hi folks,
>>
>> at the last meeting people suggested another meeting in July/August.
>> Should we be trying to schedule one?
>>
>
> We'd actually already been discussing this between the last participants.
> Our current plan is:
>
>   July 28, 29
>   Mozilla office in Toronto
>
> The w3c/webplatformwg page for the meeting should be up soon.
>

FYI, the meeting page is available here:

https://github.com/w3c/WebPlatformWG/blob/gh-pages/meetings/16-07-28-29SW.md

Sorry I forgot to post the link once it went live.

Ben

Re: [Service Workers] meeting july/august?

[Ben Kelly]

On Wed, May 25, 2016 at 9:19 PM, Chaals McCathie Nevile <
cha...@yandex-team.ru> wrote:

> Hi folks,
>
> at the last meeting people suggested another meeting in July/August.
> Should we be trying to schedule one?
>

We'd actually already been discussing this between the last participants.
Our current plan is:

  July 28, 29
  Mozilla office in Toronto

The w3c/webplatformwg page for the meeting should be up soon.

The next meeting after this will likely be at TPAC.

Thanks.

Ben

Re: Service worker F2F meeting - 26th Jan - San Francisco

[Ben Kelly]

FYI, it turns out I am going to have to attend virtually.  This should not
be a problem since the meeting is taking place in a mozilla conference room
with our standard vidyo setup.  I am planning to be on vidyo for the
meeting all day.

I'm sorry for the change here, but some personal events require me to be in
western NY next week.  I will be mostly on PTO starting this Friday, but I
do plan to attend the SW meeting in full.  Someone will have to drink beer
for me, though.

Thanks.

Ben

On Tue, Jan 5, 2016 at 12:40 PM, Jatinder Mann  wrote:

> On Mon, Jan 4, 2016 at 9:12 AM, Anne van Kesteren 
> wrote:
> > We have three more seats at this point, allocated on the basis of
> > first come, first served.
> >
>
> Hi folks, I'm planning on attending as well from the Edge team. Thanks!
>
> Jatinder Mann
>
>
>

Re: Indexed DB + Promises

[Ben Kelly]

On Wed, Sep 30, 2015 at 3:13 AM, David Rajchenbach-Teller <
dtel...@mozilla.com> wrote:

> Joshua, I am trying to understand how your proposal relates to
> microtasks. Does the extension of lifetime mean that a transaction is
> alive 1/ until the end of the event (including pending microtasks) or 2/
> that it can be kept alive across several events?
>

Wouldn't it have to be 2/ here?  What in the proposal prevents code from
doing this?

  tx.waitUntil(new Promise(function(resolve) {
setTimeout(resolve, 5000);
  }));

That seems like a good thing to me, though.  It would be nice not to
require code to perform work synchronously on the main thread in order to
have a large transaction.  This tx.waitUntil() now lets code yield the main
thread for large transactions.  If I understand it correctly...

Ben

Re: Starting work on Indexed DB v2 spec - feedback wanted

[Ben Kelly]

On 4/17/2014 5:41 PM, Kyle Huey wrote:

On Thu, Apr 17, 2014 at 2:10 PM, Dale Harvey d...@arandomurl.com wrote:

No features that slow it down, as with Tim I also implemented the same thing
in node.js and see much better perfomance against straight leveldb, with
websql still being ~5x faster than idb


Do you have benchmarks for this?  When we've profiled IndexedDB
performance for Gaia apps in the past the issue is invariably that the
main thread event loop is busy and IndexedDB's responses have to go to
the end of a long line.


I would hazard a guess that some of SQL's more feature rich constructs 
allow you to do more in a single API call.  This could mean you need to 
hit the event loop less often to accomplish the same amount of work in 
many cases.


Just a theory.

Ben

Re: [IndexedDB] request feedback on IDBKeyRange.inList([]) enhancement

[Ben Kelly]

On Jun 11, 2013, at 2:21 PM, Arthur Barstow art.bars...@nokia.com wrote:
 Hi Ben - your comment is considered a Last Call comment and it was included 
 in the LC's comment tracking document [1].
 
 In [2], Joshua proposed this comment be addressed/resolved as a feature 
 request and as such, it was added to the IDB feature request list [3].
 
 For the purposes of tracking your comment, please indicate if this resolution 
 is acceptable or not.

Sounds good.  Thank you.

Ben

 -Thanks, ArtB
 
 [1] 
 https://dvcs.w3.org/hg/IndexedDB/raw-file/default/Comments-16-May-2013-LCWD.html
 [2] http://lists.w3.org/Archives/Public/public-webapps/2013AprJun/0817.html
 [3] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures
 
 
 On 5/17/13 5:37 PM, ext Ben Kelly wrote:
 Hello all,
 
 Recently I've been working on a mobile application that makes heavy use of 
 IndexedDB.  In particular, there are times when this app must query a 
 potentially large, non-consecutive list of keys.  Currently (to my 
 knowledge) the IndexedDB API requires that this be done via separate get() 
 calls.  Due to some performance issues I investigated enhancing the 
 IndexedDB API to allow the list of keys to be queried in a single request.  
 The resulting changes seem to show significant performance improvement on 
 the mozilla mobile platform.
 
 I would like to get your feedback and input on this API change.
 
 The enhancement essentially adds an inList() function to IDBKeyRange.  
 Similar to the other factory methods on IDBKeyRange, this returns an object 
 which can be used to query a matching set of keys.  The inList() function 
 takes an array of keys to match against.  In practice it would look like the 
 following:
 
   var keyRange = IDBKeyRange.inList(['key-1', 'key-2', 'key-3']);
   var request = index.openCursor(keyRange);
 
 Duplicate keys in the list are ignored.  The order of the results would be 
 controlled by the normal cursor ordering mechanisms.
 
 I've written a rough proof-of-concept for the mozilla platform here:
 
   https://bugzilla.mozilla.org/show_bug.cgi?id=872741
 
 I realize there has been some discussion of this topic in the past.  In 
 particular, Ben Turner referred me to:
 
   https://www.w3.org/Bugs/Public/show_bug.cgi?id=16595
   
 https://docs.google.com/a/mozilla.com/document/d/1vvC5tFZCZ9T8Cwd2DteUvw5WlU4YJa2NajdkHn6fu-I/edit
 
 From these links it sounds like there has been a lack of interest, but no 
 strong objection.  Since there appears to be some legitimate benefit from 
 the API enhancement I thought I would send it out to the list for feedback.  
 I have to admit I'm new to the standardization process, though.  I apologize 
 for the noise if this is essentially a non-starter.
 
 Any feedback is greatly appreciated.  Thank you!
 
 Ben Kelly
 
 

Re: [IndexedDB] request feedback on IDBKeyRange.inList([]) enhancement

[Ben Kelly]

On May 21, 2013, at 10:05 AM, Ben Kelly bke...@mozilla.com wrote:
 On May 20, 2013, at 3:18 PM, Joshua Bell jsb...@google.com wrote:
 Cool. Knowing what performance difference you see between multi-get and just 
 a bunch of gets in parallel (for time to delivery of the last value) will be 
 interesting. A multi-get of any sort should avoid a bunch of messaging 
 overhead and excursions into script to deliver individual values, so it will 
 almost certainly be faster, but I wonder how significantly the duration from 
 first-get to last-success will differ.
 
 Here are some rough wall-clock numbers for previous testing I've done.  In 
 all cases we are loading 1000 nsIDOMContact objects in batches.  Time is 
 essentially wall-clock to load the entire 1000 values in the data set.
 
  - 20 get() calls per txn:~2000ms
  - 1000 get() calls in single txn:~1600ms 
  - getAll + inList for 20 keys per txn:   ~1500ms
  - getAll + inList for 64 keys per txn:   ~1050ms
  - getAll + inList for 256 keys per txn:  ~950ms
  - getAll for all 1000 keys:  ~1200ms
 
 I've hesitated to post these because they are somewhat specific to my 
 workload, test case, and device.  I'll try to pull out a more isolated test 
 case while I work on the optimization for parallel get() calls.

I finally got some time to break out an isolated test case:

  http://blog.wanderview.com/idb-inlist-test/

Results for firefox nightly+inList() patch on my MBPr quad core:

  cursor:   329ms
  get:  88ms
  getAll:   71ms
  inList:   44ms

Code for each test case can be found here:

  https://github.com/wanderview/idb-inlist-test/tree/gh-pages/scripts/tests

I'm still new to the IDB API, so there is a good chance I made a mistake 
somewhere in the tests.  Pull requests or other feedback welcome.

Thanks!

Ben

 
 Ignoring deplicating keys is not a useful feature in query. In fact, I 
 will like result be respective ordering of given key list.
 
 Well, I would prefer to respect ordering as well.  I just assumed that 
 people would prefer not to impose that on all calls.  Perhaps the cases 
 could be separated:
 
  IDBKeyList.inList([]) // unordered
  IDBKeyList.inOrderedList([])  // ordered
 
 I would be happy to include duplicate keys as well.
 
 Thanks again.
 
 Ben
 
 
 
 
 
 

Re: [IndexedDB] request feedback on IDBKeyRange.inList([]) enhancement

[Ben Kelly]

On Jun 3, 2013, at 6:06 PM, Joshua Bell jsb...@google.com wrote:
 Cool, thanks for sharing!
 
   cursor:   329ms
   get:  88ms
 
 ^ Not surprising, given the async overhead
 
   get:  88ms
   getAll:   71ms
   inList:   44ms
 
 ^ Looking at the test, it seems like getAll is fetching all 500 records in 
 the store whereas get and inList are fetching only 250? Am I misreading?

That is correct.  I put getAll() in as a reference point since the inList test 
is also using getAll().

I've also added a cursor+inList test to the repo.  It performs slightly better 
than the cursor test, but worse than get.  On this debug/profiling build (so 
not comparable to above):

cursor  489
inListCursor415
get 277
getAll  135
inList  91

I'm going to spend more time in the profiler tomorrow to try to understand 
better where the differences are coming from.

Thanks.

Ben

Re: [IndexedDB] request feedback on IDBKeyRange.inList([]) enhancement

[Ben Kelly]

On May 21, 2013, at 10:01 PM, Kyaw Tun kyaw...@yathit.com wrote:
 Thank you.  
 
   - 1000 get() calls in single txn: ~1600ms
   - getAll for all 1000 keys:~1200ms
 
 I would expect getAll could have better than that. 
 
 It seems context switching between js and database is cheap. In that case, 
 cursor walk could even better perform.  

I agree the getAll() result was not as expected.  I suspect there was something 
going wrong in my test, but I don't have a good explanation right now.  The 
improvements in time with subsequently larger inList() requests did suggest to 
me that round trips are significant on this device.

I really need to extract an isolated test.  Unfortunately it doesn't look like 
I will have time to do that until the first week of June.  I have some other 
work I need to do this week and I will be at JSConf next week.  Sorry for the 
delay.

 nsIDOMContact objects is fine, it is a typical object found in web app. 
 
 
 On Tue, May 21, 2013 at 10:05 PM, Ben Kelly bke...@mozilla.com wrote:
 On May 20, 2013, at 3:18 PM, Joshua Bell jsb...@google.com wrote:
  Cool. Knowing what performance difference you see between multi-get and 
  just a bunch of gets in parallel (for time to delivery of the last value) 
  will be interesting. A multi-get of any sort should avoid a bunch of 
  messaging overhead and excursions into script to deliver individual values, 
  so it will almost certainly be faster, but I wonder how significantly the 
  duration from first-get to last-success will differ.
 
 Here are some rough wall-clock numbers for previous testing I've done.  In 
 all cases we are loading 1000 nsIDOMContact objects in batches.  Time is 
 essentially wall-clock to load the entire 1000 values in the data set.
 
   - 20 get() calls per txn: ~2000ms
   - 1000 get() calls in single txn: ~1600ms
   - getAll + inList for 20 keys per txn:~1500ms
   - getAll + inList for 64 keys per txn:~1050ms
   - getAll + inList for 256 keys per txn:   ~950ms
   - getAll for all 1000 keys:   ~1200ms
 
 I've hesitated to post these because they are somewhat specific to my 
 workload, test case, and device.  I'll try to pull out a more isolated test 
 case while I work on the optimization for parallel get() calls.
 
Ignoring deplicating keys is not a useful feature in query. In fact, I 
will like result be respective ordering of given key list.
  
   Well, I would prefer to respect ordering as well.  I just assumed that 
   people would prefer not to impose that on all calls.  Perhaps the cases 
   could be separated:
  
 IDBKeyList.inList([]) // unordered
 IDBKeyList.inOrderedList([])  // ordered
  
   I would be happy to include duplicate keys as well.
  
   Thanks again.
  
   Ben
  
  
  
 
 
 
 

Re: [IndexedDB] request feedback on IDBKeyRange.inList([]) enhancement

[Ben Kelly]

On May 20, 2013, at 3:18 PM, Joshua Bell jsb...@google.com wrote:
 Cool. Knowing what performance difference you see between multi-get and just 
 a bunch of gets in parallel (for time to delivery of the last value) will be 
 interesting. A multi-get of any sort should avoid a bunch of messaging 
 overhead and excursions into script to deliver individual values, so it will 
 almost certainly be faster, but I wonder how significantly the duration from 
 first-get to last-success will differ.

Here are some rough wall-clock numbers for previous testing I've done.  In all 
cases we are loading 1000 nsIDOMContact objects in batches.  Time is 
essentially wall-clock to load the entire 1000 values in the data set.

  - 20 get() calls per txn: ~2000ms
  - 1000 get() calls in single txn: ~1600ms 
  - getAll + inList for 20 keys per txn:~1500ms
  - getAll + inList for 64 keys per txn:~1050ms
  - getAll + inList for 256 keys per txn:   ~950ms
  - getAll for all 1000 keys:   ~1200ms

I've hesitated to post these because they are somewhat specific to my workload, 
test case, and device.  I'll try to pull out a more isolated test case while I 
work on the optimization for parallel get() calls.

   Ignoring deplicating keys is not a useful feature in query. In fact, I 
   will like result be respective ordering of given key list.
 
  Well, I would prefer to respect ordering as well.  I just assumed that 
  people would prefer not to impose that on all calls.  Perhaps the cases 
  could be separated:
 
IDBKeyList.inList([]) // unordered
IDBKeyList.inOrderedList([])  // ordered
 
  I would be happy to include duplicate keys as well.
 
  Thanks again.
 
  Ben
 
 
 
 
 

Re: [IndexedDB] request feedback on IDBKeyRange.inList([]) enhancement

[Ben Kelly]

Thanks for the feedback!

On May 19, 2013, at 9:25 PM, Kyaw Tun kyaw...@yathit.com wrote:
 IDBKeyRange.inList looks practically useful, but it can be achieve continue 
 (continuePrimary) cursor iteration. Performance will be comparable except 
 multiple round trip between js and database.

I'm sorry, but I don't understand this bit.  How do you envision getting the 
cursor in the first place here without a way to form a query based on an 
arbitrary key list?  I'm sure I'm just missing an obvious part of the API here.

 Querying by parallel multiple get in a single transaction should also be fast 
 as well. 

Yes, Jonas Sicking did recommend a possible optimization for the multiple get() 
within a transaction.  It would seem to me, however, that this will likely 
impose some cost on the general single get() case.  It would be nice if the 
client had the ability to be explicit about their use case vs using a heuristic 
to infer it.

In any case, I plan to prototype this in the next week or two.

 Additionally IDBKeyRange.inList violate contiguous key range nature of 
 IDBKeyRange. It is assumed in some use case, like checking a key in the key 
 range or not. If this feature are to be implemented, it should not mess with 
 IDBKeyRange, but directly handle by index batch request.

Good point.  I suppose an IDBKeySet or IDBKeyList type could be added.

 Ignoring deplicating keys is not a useful feature in query. In fact, I will 
 like result be respective ordering of given key list.

Well, I would prefer to respect ordering as well.  I just assumed that people 
would prefer not to impose that on all calls.  Perhaps the cases could be 
separated:

  IDBKeyList.inList([]) // unordered
  IDBKeyList.inOrderedList([])  // ordered

I would be happy to include duplicate keys as well.

Thanks again.

Ben

Re: [IndexedDB] request feedback on IDBKeyRange.inList([]) enhancement

[Ben Kelly]

On May 20, 2013, at 1:39 PM, Joshua Bell jsb...@google.com wrote:
 On Mon, May 20, 2013 at 6:37 AM, Ben Kelly bke...@mozilla.com wrote:
 On May 19, 2013, at 9:25 PM, Kyaw Tun kyaw...@yathit.com wrote:
  IDBKeyRange.inList looks practically useful, but it can be achieve continue 
  (continuePrimary) cursor iteration. Performance will be comparable except 
  multiple round trip between js and database.
 
 I'm sorry, but I don't understand this bit.  How do you envision getting the 
 cursor in the first place here without a way to form a query based on an 
 arbitrary key list?  I'm sure I'm just missing an obvious part of the API 
 here.
 
 
 Here's an example I whipped up:
 
 https://gist.github.com/inexorabletash/5613707

Thanks!  Yes, I totally missed you could pass the next desired key to 
continue().

Unfortunately I don't think this approach would help much with the use case I 
am looking at.  The round trips are significant and add up on this mobile 
platform.  Also, it appears this would lose any parallelism from issuing 
multiple get() requests simultaneously.

  Querying by parallel multiple get in a single transaction should also be 
  fast as well.
 
 Yes, Jonas Sicking did recommend a possible optimization for the multiple 
 get() within a transaction.  It would seem to me, however, that this will 
 likely impose some cost on the general single get() case.  It would be nice 
 if the client had the ability to be explicit about their use case vs using a 
 heuristic to infer it.
 
 In any case, I plan to prototype this in the next week or two.
 
 Thanks for taking this on - we'll be watching your implementation experience 
 closely. :)
 
 Some discussion here: https://www.w3.org/Bugs/Public/show_bug.cgi?id=16595
 
 (That also links to a very raw document with other IDB v2 thoughts c/o a 
 very informal google/moz brainstorming session.)
 
 One approach that adds generally useful primitives to IDB is (1) something 
 akin to a key range that is a list of keys (per above) and (2) batch cursors 
 that deliver up to N values at a time. Either of those is quite useful 
 independently.

The batch cursors do look useful.  I had not run into that need yet since I am 
actually working with our prefixed getAll() implementation.


  Additionally IDBKeyRange.inList violate contiguous key range nature of 
  IDBKeyRange. It is assumed in some use case, like checking a key in the key 
  range or not. If this feature are to be implemented, it should not mess 
  with IDBKeyRange, but directly handle by index batch request.
 
 Good point.  I suppose an IDBKeySet or IDBKeyList type could be added.
 
 I'm not entirely convinced that's necessary. I don't believe we expose is in 
 range in the platform currently, so exposing a new type to script seems 
 excessive. On the other hand, range is a pretty well defined concept in 
 general so it would be a shame to break it.

I don't have a preference one way or another.  I'm happy to implement a new 
type or not as long we can make non-consecutive key queries fast.

Thanks again.  I'll post back when I have the multi-get optimization prototyped 
out.

Ben

  
 
  Ignoring deplicating keys is not a useful feature in query. In fact, I will 
  like result be respective ordering of given key list.
 
 Well, I would prefer to respect ordering as well.  I just assumed that people 
 would prefer not to impose that on all calls.  Perhaps the cases could be 
 separated:
 
   IDBKeyList.inList([]) // unordered
   IDBKeyList.inOrderedList([])  // ordered
 
 I would be happy to include duplicate keys as well.
 
 Thanks again.
 
 Ben
 
 
 

[IndexedDB] request feedback on IDBKeyRange.inList([]) enhancement

[Ben Kelly]

Hello all,

Recently I've been working on a mobile application that makes heavy use of 
IndexedDB.  In particular, there are times when this app must query a 
potentially large, non-consecutive list of keys.  Currently (to my knowledge) 
the IndexedDB API requires that this be done via separate get() calls.  Due to 
some performance issues I investigated enhancing the IndexedDB API to allow the 
list of keys to be queried in a single request.  The resulting changes seem to 
show significant performance improvement on the mozilla mobile platform.

I would like to get your feedback and input on this API change.

The enhancement essentially adds an inList() function to IDBKeyRange.  Similar 
to the other factory methods on IDBKeyRange, this returns an object which can 
be used to query a matching set of keys.  The inList() function takes an array 
of keys to match against.  In practice it would look like the following:

  var keyRange = IDBKeyRange.inList(['key-1', 'key-2', 'key-3']);
  var request = index.openCursor(keyRange);

Duplicate keys in the list are ignored.  The order of the results would be 
controlled by the normal cursor ordering mechanisms.

I've written a rough proof-of-concept for the mozilla platform here:

  https://bugzilla.mozilla.org/show_bug.cgi?id=872741

I realize there has been some discussion of this topic in the past.  In 
particular, Ben Turner referred me to:

  https://www.w3.org/Bugs/Public/show_bug.cgi?id=16595
  
https://docs.google.com/a/mozilla.com/document/d/1vvC5tFZCZ9T8Cwd2DteUvw5WlU4YJa2NajdkHn6fu-I/edit

From these links it sounds like there has been a lack of interest, but no 
strong objection.  Since there appears to be some legitimate benefit from the 
API enhancement I thought I would send it out to the list for feedback.  I have 
to admit I'm new to the standardization process, though.  I apologize for the 
noise if this is essentially a non-starter.

Any feedback is greatly appreciated.  Thank you!

Ben Kelly