Re: Synchronous postMessage for Workers?

[David Herman]

On Feb 16, 2012, at 7:06 AM, Glenn Maynard wrote:

> On Wed, Feb 15, 2012 at 12:39 AM, Jonas Sicking  wrote:
> Put it another way, when you spin the event loop, not only does your
> code need to be prepared for anything happening. All functions up the
> call stack also has to. That makes it very hard to reason about any of
> your code, not just the code that calls yieldUntil.
> 
> FWIW, I've used similar mechanisms, such as Lua coroutines, and it worked 
> well.

It works well when you know that the function you're calling is blocking. As 
Boris, Mark, and Jonas have said, it doesn't work so well as your code starts 
growing and you lose track of who might be blocking when.

> The message polling/blocking can be done without spinning the event loop, of 
> course.  It would be nice to have capabilities like running two sync XHRs, 
> but with message blocking, you'd be able to do that yourself with a bit more 
> work (spin up two workers to do the XHRs, then block on response messages).

Using workers for this is heavyweight. It works very nicely in ES6 with task.js:

spawn(function*() {
try {
var [file1, file2] = yield join(xhr("file1.txt"), xhr("file2.txt"));
element1.innerHTML = file1;
element2.innerHTML = file2;
} catch (e) {
console.log("error: " + e);
}
});

The difference between task.js and continuations/coroutines/synchronous workers 
is that the blocking is always explicitly labelled with `yield`. So you can 
never accidentally call a function that might suspend your stack and allow 
other code to run in between.

I wrote about this recently:

http://calculist.org/blog/2011/12/14/why-coroutines-wont-work-on-the-web/

A commenter correctly pointed out that the worker approach makes it easier for 
engines to capture native frames, so maybe it's not so onerous for implementors 
as my post makes it out to be. But we made a conscious choice in TC39 *not* to 
add full coroutines to JS because of the fragility of implicit suspension.

Dave

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Wed, Feb 15, 2012 at 12:39 AM, Jonas Sicking  wrote:

> Put it another way, when you spin the event loop, not only does your
> code need to be prepared for anything happening. All functions up the
> call stack also has to. That makes it very hard to reason about any of
> your code, not just the code that calls yieldUntil.
>

FWIW, I've used similar mechanisms, such as Lua coroutines, and it worked
well.

The message polling/blocking can be done without spinning the event loop,
of course.  It would be nice to have capabilities like running two sync
XHRs, but with message blocking, you'd be able to do that yourself with a
bit more work (spin up two workers to do the XHRs, then block on response
messages).

(By the way, don't forget the "poll a MessageChannel without blocking"
case.  It'd help things like gracefully stopping a long-running operation
rather than having to terminate the worker, which is a pattern I've used a
lot in threaded algorithms in other platforms.)

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[Jonas Sicking]

On Wed, Feb 15, 2012 at 11:09 AM, John J Barton
 wrote:
> On Tue, Feb 14, 2012 at 10:39 PM, Jonas Sicking  wrote:
> ...
>> The problem is when you have functions which call yieldUntil. I.e.
>> when you have code like this:
>>
>> function doStuff() {
>>  yieldUntil("x");
>> };
>>
>> now what looks like perfectly safe innocent code:
>>
>> function myFunction() {
>>  ... code here ...
>>  doStuff();
>>  ... more code ...
>> }
>>
>> The myFunction code might look perfectly sane and safe. However since
>> the call to doStuff spins the event loop, the two code snippets can
>> see entirely different worlds.
>>
>> Put it another way, when you spin the event loop, not only does your
>> code need to be prepared for anything happening. All functions up the
>> call stack also has to. That makes it very hard to reason about any of
>> your code, not just the code that calls yieldUntil.
>
> This argument makes good sense, but can we make it more concrete and
> thus clearer?
>
> What I am fishing for is an example that clearer shows the
> yieldUntil() pattern is hard compared to a function call in to a large
> library or into the platform and compared to a function call that
> passes state fro myFunction() into a closure. Can a function on
> another event loop access the private (closure) state of myFunction()
> in a way that the other two patterns cannot?

Consider the following code

setInterval(function () {
  ... non-reentrant-safe code ...
  doStuff();
  ... non-reentrant-safe code ...
}, 100);

This code is perfectly safe and fine. Unless doStuff spins the event
loop. If doStuff spins the event loop you may or may not break the
invariants that the callback relies upon.

The same thing would happen if the function wasn't anonymous and
doStuff explicitly called it. But avoiding to explicitly call a
function is pretty easy. As long as the page isn't intentionally
trying to hack itself it's a pretty safe thing to rely on.

/ Jonas

Re: Synchronous postMessage for Workers?

[Mark S. Miller]

On Wed, Feb 15, 2012 at 9:37 AM, Mark S. Miller  wrote:

> On Wed, Feb 15, 2012 at 8:09 AM, John J Barton <
> johnjbar...@johnjbarton.com> wrote:
>
>> On Tue, Feb 14, 2012 at 10:39 PM, Jonas Sicking  wrote:
>>
> [...]
>
>> > function doStuff() {
>> >  yieldUntil("x");
>> > };
>> >
>> > now what looks like perfectly safe innocent code:
>> >
>> > function myFunction() {
>> >  ... code here ...
>> >  doStuff();
>> >  ... more code ...
>>
> [...]
>
>> What I am fishing for is an example [...] a function on
>>
>> another event loop access the private (closure) state of myFunction()
>> in a way that the other two patterns cannot?
>>
>
Oops. I forgot the most important part of my code ;).

"use strict";



>
>   function makeStatusHolder(status) {
> var listeners = [];
> return Object.freeze({
>   addListener: function(newListener) { listeners.add(newListener); },
>   getStatus: function() { return status; },
>   setStatus: function(newStatus) {
> status = newStatus;
> listeners.forEach(function(listener) {
>   listener.statusChanged(newStatus);
> });
>   }
> });
>   }
>
> This pattern is unsafe of any of the listeners might call back into the
> statusHolder during notification (see Fig 13.2 of my thesis). However, we
> may judge a particular usage ok if we know that none of the listeners has
> access, directly or indirectly, to this statusHolder itself.
> Under maintenance, one of these listeners may be revised to use  yieldUntil
> to wait from a result from a foreign worker that itself does not have
> access to this statusHolder, thereby not violating the assumption above.
> Nevertheless, if this event loop is receptive to external events while it
> is waiting, this pattern becomes unsafe.
>
> In the absence of yieldUntil, the revised listener can gain most of the
> same brevity and expressivity benefits by using <
> http://wiki.ecmascript.org/doku.php?id=strawman:async_functions> (Already
> supported by Kris' implementation using FF generators). Though this
> provides most of the benefits of yieldUntil, it does not cause these
> dangers. The notification loop runs to completion in the current turn. The
> listener proceeds from the yield point in a separate turn. Any
> interleavings that happen during the yield also happen in separate turns
> after the notification loop completes.
>
> --
> Cheers,
> --MarkM
>



-- 
Cheers,
--MarkM

Re: Synchronous postMessage for Workers?

[Mark S. Miller]

On Wed, Feb 15, 2012 at 9:37 AM, Mark S. Miller  wrote:

> On Wed, Feb 15, 2012 at 8:09 AM, John J Barton <
> johnjbar...@johnjbarton.com> wrote:
>
>> On Tue, Feb 14, 2012 at 10:39 PM, Jonas Sicking  wrote:
>>
> [...]
>
>> > function doStuff() {
>> >  yieldUntil("x");
>> > };
>> >
>> > now what looks like perfectly safe innocent code:
>> >
>> > function myFunction() {
>> >  ... code here ...
>> >  doStuff();
>> >  ... more code ...
>>
> [...]
>
>> What I am fishing for is an example [...] a function on
>>
>> another event loop access the private (closure) state of myFunction()
>> in a way that the other two patterns cannot?
>>
>
>   function makeStatusHolder(status) {
> var listeners = [];
> return Object.freeze({
>   addListener: function(newListener) { listeners.add(newListener); },
>   getStatus: function() { return status; },
>   setStatus: function(newStatus) {
> status = newStatus;
> listeners.forEach(function(listener) {
>   listener.statusChanged(newStatus);
> });
>   }
> });
>   }
>
> This pattern is unsafe of any of the listeners might call back into the
> statusHolder during notification (see Fig 13.2 of my thesis). However, we
> may judge a particular usage ok if we know that none of the listeners has
> access, directly or indirectly, to this statusHolder itself.
> Under maintenance, one of these listeners may be revised to use  yieldUntil
> to wait from a result from a foreign worker that itself does not have
> access to this statusHolder, thereby not violating the assumption above.
> Nevertheless, if this event loop is receptive to external events while it
> is waiting, this pattern becomes unsafe.
>
> In the absence of yieldUntil, the revised listener can gain most of the
> same brevity and expressivity benefits by using <
> http://wiki.ecmascript.org/doku.php?id=strawman:async_functions>
>

Concretely, combining the previous examples:

  var doStuff = Q.async(function*() {
yield until("x");
  });

  var myListener = Q.async(function*(newStatus) {
   ... code here ...
   yield doStuff();
   ... more code ...
  });

Of course, if we revise the notification loop itself to be an async
function yielding on each notification, then we've recreated all the
original hazards. But there is no reason to do so.



> (Already supported by Kris' implementation using FF generators). Though
> this provides most of the benefits of yieldUntil, it does not cause these
> dangers. The notification loop runs to completion in the current turn. The
> listener proceeds from the yield point in a separate turn. Any
> interleavings that happen during the yield also happen in separate turns
> after the notification loop completes.
>
> --
> Cheers,
> --MarkM
>



-- 
Cheers,
--MarkM

Re: Synchronous postMessage for Workers?

[Mark S. Miller]

On Wed, Feb 15, 2012 at 8:09 AM, John J Barton
wrote:

> On Tue, Feb 14, 2012 at 10:39 PM, Jonas Sicking  wrote:
>
[...]

> > function doStuff() {
> >  yieldUntil("x");
> > };
> >
> > now what looks like perfectly safe innocent code:
> >
> > function myFunction() {
> >  ... code here ...
> >  doStuff();
> >  ... more code ...
>
[...]

> What I am fishing for is an example [...] a function on
> another event loop access the private (closure) state of myFunction()
> in a way that the other two patterns cannot?


  function makeStatusHolder(status) {
var listeners = [];
return Object.freeze({
  addListener: function(newListener) { listeners.add(newListener); },
  getStatus: function() { return status; },
  setStatus: function(newStatus) {
status = newStatus;
listeners.forEach(function(listener) {
  listener.statusChanged(newStatus);
});
  }
});
  }

This pattern is unsafe of any of the listeners might call back into the
statusHolder during notification (see Fig 13.2 of my thesis). However, we
may judge a particular usage ok if we know that none of the listeners has
access, directly or indirectly, to this statusHolder itself.
Under maintenance, one of these listeners may be revised to use  yieldUntil
to wait from a result from a foreign worker that itself does not have
access to this statusHolder, thereby not violating the assumption above.
Nevertheless, if this event loop is receptive to external events while it
is waiting, this pattern becomes unsafe.

In the absence of yieldUntil, the revised listener can gain most of the
same brevity and expressivity benefits by using <
http://wiki.ecmascript.org/doku.php?id=strawman:async_functions> (Already
supported by Kris' implementation using FF generators). Though this
provides most of the benefits of yieldUntil, it does not cause these
dangers. The notification loop runs to completion in the current turn. The
listener proceeds from the yield point in a separate turn. Any
interleavings that happen during the yield also happen in separate turns
after the notification loop completes.

-- 
Cheers,
--MarkM

Re: Synchronous postMessage for Workers?

[John J Barton]

On Tue, Feb 14, 2012 at 10:39 PM, Jonas Sicking  wrote:
...
> The problem is when you have functions which call yieldUntil. I.e.
> when you have code like this:
>
> function doStuff() {
>  yieldUntil("x");
> };
>
> now what looks like perfectly safe innocent code:
>
> function myFunction() {
>  ... code here ...
>  doStuff();
>  ... more code ...
> }
>
> The myFunction code might look perfectly sane and safe. However since
> the call to doStuff spins the event loop, the two code snippets can
> see entirely different worlds.
>
> Put it another way, when you spin the event loop, not only does your
> code need to be prepared for anything happening. All functions up the
> call stack also has to. That makes it very hard to reason about any of
> your code, not just the code that calls yieldUntil.

This argument makes good sense, but can we make it more concrete and
thus clearer?

What I am fishing for is an example that clearer shows the
yieldUntil() pattern is hard compared to a function call in to a large
library or into the platform and compared to a function call that
passes state fro myFunction() into a closure. Can a function on
another event loop access the private (closure) state of myFunction()
in a way that the other two patterns cannot?

jjb

Re: Synchronous postMessage for Workers?

[Jonas Sicking]

On Tue, Feb 14, 2012 at 2:52 PM, Glenn Maynard  wrote:
> On Tue, Feb 14, 2012 at 10:07 AM, Jonas Sicking  wrote:
>>
>> Spinning the event loop is very bug prone. When calling the yieldUntil
>> function basically anything can happen, including re-entering the same
>> code. Protecting against the entire world changing under you is very
>> hard. Add to that that it's very racy. I.e. changes may or may not
>> happen under you depending on when exactly you call yieldUntil, and
>> which order events come in.
>
>
> I don't think this:
>
> function() {
>     f1();
>     yieldUntil(condition);
>     f2();
> }
>
> is any more racy than this:
>
> function() {
>     f1();
>     waitForCondition(function() {
>         f2();
>     });
> }

The problem is when you have functions which call yieldUntil. I.e.
when you have code like this:

function doStuff() {
  yieldUntil("x");
};

now what looks like perfectly safe innocent code:

function myFunction() {
  ... code here ...
  doStuff();
  ... more code ...
}

The myFunction code might look perfectly sane and safe. However since
the call to doStuff spins the event loop, the two code snippets can
see entirely different worlds.

Put it another way, when you spin the event loop, not only does your
code need to be prepared for anything happening. All functions up the
call stack also has to. That makes it very hard to reason about any of
your code, not just the code that calls yieldUntil.

/ Jonas

Re: Synchronous postMessage for Workers?

[Mark S. Miller]

On Tue, Feb 14, 2012 at 11:32 AM, John J Barton  wrote:

> On Tue, Feb 14, 2012 at 11:14 AM, David Bruant  wrote:
> > Le 14/02/2012 14:31, Arthur Barstow a écrit :
>
> > Another addition will be promises.
> > An already working example of promises can be found at
> > https://github.com/kriskowal/q
>
> Just to point out that promises are beyond the working example stage,
> they are deployed in the major JS frameworks, eg:
>
> http://dojotoolkit.org/reference-guide/dojo/Deferred.html
> http://api.jquery.com/category/deferred-object/
>
> The Q library is more like an exploration of implementation issues in
> promises, trying to push them further.


Relevant to the thread here is that the Q library uses promises both for
local asynchrony and for asynchronous distributed messaging. The other
promise libraries (even though they derive from (dojo -> mochikit ->
Twisted Python -> E) only support local asynchrony.


-- 
Cheers,
--MarkM

Re: Synchronous postMessage for Workers?

[Mark S. Miller]

On Mon, Feb 13, 2012 at 12:08 PM, John J Barton  wrote:

> On Mon, Feb 13, 2012 at 11:44 AM, Ian Hickson  wrote:
> > On Thu, 17 Nov 2011, Joshua Bell wrote:
> >>
> >> Wouldn't it be lovely if the Worker script could simply make a
> >> synchronous call to fetch data from the Window?
> >
> > It wouldn't be so much a synchronous call, so much as a blocking get.
> >..
> > Anyone object to me adding something like this? Are there any better
> > solutions? Should we just tell authors to get used to the async style?
>
> I guess the "Q" folks would say that remote promises provides another
> solution. If promises are adopted by the platform, then the async
> style gets much easier to work with.
> https://github.com/kriskowal/q
> (spec is somewhere on the es wiki)
>



This spec doesn't quite correspond to Kris' implementation. We're still
working that out. John's "then" below corresponds to "when" on the wiki
page.


>
> In the Q model you would fetch data like:
>  parentWindow.fetchData('myQueryString').then(  // block until reply.
>function(data) {...},
>function(err) {...}
>  );
> Q has functions to join promises; q_comm add remote promises.
>
> I believe this can be done today with q_comm in workers.
>
> Your signal/yieldUntil looks like what es-discuss calls "generators".
> I found them much harder to understand than promises, but then I come
> from JS not python.
>
> jjb
>
>


-- 
Cheers,
--MarkM

Re: Synchronous postMessage for Workers?

[Boris Zbarsky]

On 2/14/12 2:52 PM, Glenn Maynard wrote:

I don't think this:

function() {
 f1();
 yieldUntil(condition);
 f2();
}

is any more racy than this:

function() {
 f1();
 waitForCondition(function() {
 f2();
 });
}


Let's say the function above is called "f", and consider this code:

var inG = false;
function g() {
  inG = true;
  h(); /* This function does things differently depending on
  whether inG is true */
  f();
  inG = false;
}

"g" is fine with the second snippet above, but broken with the first one.

So while in practice the two are equally racy, writing correct code that 
calls into the second snippet is much easier than writing correct code 
that calls into the first one.  And in practice, a developer would 
perceive the first snippet as being less racy than the second one, 
because the race is not nearly as obvious.



I don't believe it's any more bug-prone than the equivalent
callback-based code, either.


See above.  This is not a hypothetical issue.  We have some experience 
with APIs that look like the yieldUntil one above, and in practice it 
causes serious code maintainability problems because everything up the 
callstack has to make sure that it's data structures are in a consistent 
state before calling into functions.



I believe continuations are less error
prone than callbacks, because it leads to code that's simpler and easier
to understand.


It leads to code that _looks_ simpler and easier to understand.  Whether 
it _is_ that way is an interesting question.


-Boris

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Tue, Feb 14, 2012 at 10:07 AM, Jonas Sicking  wrote:

> Spinning the event loop is very bug prone. When calling the yieldUntil
> function basically anything can happen, including re-entering the same
> code. Protecting against the entire world changing under you is very
> hard. Add to that that it's very racy. I.e. changes may or may not
> happen under you depending on when exactly you call yieldUntil, and
> which order events come in.
>

I don't think this:

function() {
f1();
yieldUntil(condition);
f2();
}

is any more racy than this:

function() {
f1();
waitForCondition(function() {
f2();
});
}

The order of events with yieldUntil depends on when you yield, but the same
is true when you return and continue what you're doing from a callback.  I
don't believe it's any more bug-prone than the equivalent callback-based
code, either.  I believe continuations are less error prone than callbacks,
because it leads to code that's simpler and easier to understand.

An alternative way of defining it, rather than spinning the event loop,
might be to store the JS call stack (eg. setjmp), then to return undefined
to the original native caller.  When the condition is signalled, resume the
call stack (from a new native caller).  This would mean that if you yield
from an event handler (or any other callback), the event dispatch would
continue immediately, as if the event handler returned; when your code is
resumed, you'd no longer be inside event dispatch (eg. you can't call
stopPropagation on an event after yielding).  That makes sense to me, since
dispatching events shouldn't block, though it would probably surprise
people.

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[John J Barton]

On Tue, Feb 14, 2012 at 11:14 AM, David Bruant  wrote:
> Le 14/02/2012 14:31, Arthur Barstow a écrit :

> Another addition will be promises.
> An already working example of promises can be found at
> https://github.com/kriskowal/q

Just to point out that promises are beyond the working example stage,
they are deployed in the major JS frameworks, eg:

http://dojotoolkit.org/reference-guide/dojo/Deferred.html
http://api.jquery.com/category/deferred-object/

The Q library is more like an exploration of implementation issues in
promises, trying to push them further.

jjb

Re: Synchronous postMessage for Workers?

[David Bruant]

Le 14/02/2012 14:31, Arthur Barstow a écrit :
> On 2/14/12 2:02 AM, ext David Bruant wrote:
>> Le 13/02/2012 20:44, Ian Hickson a écrit :
>>> Should we just tell authors to get used to the async style?
>> I think we should. More constructs are coming in ECMAScript. Things
>> related to language concurrency should probably be left to the core
>> language unless there is an extreme need (which there isn't as far as I
>> know).
>
> David - if you have some recommended reading(s) re concurrency
> constructs that are relevant to this discussion and are coming to
> ECMAScript, please let me know.
* http://wiki.ecmascript.org/doku.php?id=harmony:generators (with yield)
Yield has been for quite some time in SpiderMonkey. i'm not sure the
syntax in the spec (especially the "function*" functions) is
implemented, but there is a form of it anyway
Unless 2012 ends because of some Mayan calendar issues, generators will
be part of ECMAScript 6.
** Task.js by Dave Herman which uses generators. Built on top of
SpiderMonkey generators (so not the upcoming standard), but really worth
having a look at.
http://taskjs.org/
Related blog post:
http://blog.mozilla.com/dherman/2011/03/11/who-says-javascript-io-has-to-be-ugly/


* Concurrency: http://wiki.ecmascript.org/doku.php?id=strawman:concurrency
Probably one of the most ambitious and promising work. Did not cut it to
ECMAScript 6, but I'm hopeful will be part of ES7.
In a nutshell, this brings the event loop in ECMAScript itself. Even in
ES6, ECMAScript will have no concurrency mechanism whatsoever.
Concurrency is currently defined in HTML5 (event loop, setTimeout, etc.).
Another addition will be promises.
An already working example of promises can be found at
https://github.com/kriskowal/q
** This proposal is championned by Mark S. Miller (added in copy).
The strawman on concurrency seems largely inspired by what is done in
the E programming language. I highly recommend reading the Part III of
Mark's thesis on the topic: http://erights.org/talks/thesis/markm-thesis.pdf

> (I tend to agree if there isn't some real urgency here, we should be
> careful [with our specs]).
I've been participating on es-discuss for more than a year now and
following progress in implementations and things are moving. They are
moving in the right direction (browser vendors cooperate). Maybe slowly,
but they are moving.

I really think all topics that are low level (like concurrency) should
be left to ECMAScript now or addressed first on es-discuss before
thinking of a DOM/WebApp API. Low level things belong to ECMAScript and
any API we think of should be a library built on top of that. That's the
way I see it at least and I'd be happy to discuss if some disagree.

David

Re: Synchronous postMessage for Workers?

[Jonas Sicking]

On Mon, Feb 13, 2012 at 2:44 PM, Ian Hickson  wrote:
> On Thu, 17 Nov 2011, Joshua Bell wrote:
>>
>> Wouldn't it be lovely if the Worker script could simply make a
>> synchronous call to fetch data from the Window?
>
> It wouldn't be so much a synchronous call, so much as a blocking get.
>
>
> On Thu, 17 Nov 2011, Jonas Sicking wrote:
>>
>> We can only allow child workers to block on parent workers. Never the
>> other way around.
>
> Indeed. And it would have to be limited to the built-in ports that workers
> have, because regular ports can be shunted all over the place and you
> could end up with a deadlock situation.
>
> It would be easy enough to add a blocking get on DedicatedWorkerGlobalScope.
> Is this something for which there's a lot of demand?
>
>   // blocking get
>   // no self.onmessage handler needed
>   ...
>   var message = self.getMessage();
>   ...
>
> An alternative is to add continuations to the platform:
>
>   // continuation
>   // (this is not a formal proposal, just a illustration of the concept)
>   var message;
>   self.onmessage = function (event) {
>     message = event;
>     signal('got message'); // queues a task to resume from yieldUntil()
>   };
>   ...
>   yieldUntil('got message');
>   ...
>
> This would be a more general solution and would be applicable in many
> other parts of the platform. As we get more and more async APIs, I think
> it might be worth considering adding this.
>
> We could add it to HTML as an API rather than adding it to JS as a
> language construct. It would be relatively easy to define, if much harder
> to implement:
>
>   yieldUntil(id) - spin the event loop until the signal() API unblocks
>                    this script
>
>   signal(id)     - unblock the script with the oldest invokation of
>                    yieldUntil() called with the given ID, if any
>
> Given our definition of "spin the event loop", this doesn't even result,
> per the spec, in nested event loops or anything like that. Note that per
> this definition, "signal" just queues a task to resume the earlier script,
> it is not your typical coroutine. That is:
>
>   setTimeout(function () {
>     console.log('2');
>     signal('test');
>     console.log('3');
>   }, 1000);
>   console.log('1');
>   yieldUntil('test');
>   console.log('4');
>
> ...logs 1, 2, 3, 4, not 1, 2, 4, 3.
>
> Anyone object to me adding something like this? Are there any better
> solutions? Should we just tell authors to get used to the async style?

Spinning the event loop is very bug prone. When calling the yieldUntil
function basically anything can happen, including re-entering the same
code. Protecting against the entire world changing under you is very
hard. Add to that that it's very racy. I.e. changes may or may not
happen under you depending on when exactly you call yieldUntil, and
which order events come in.

This is something we've been fighting with in Gecko for a long time.
Code that spins the event loop has very often lead to terrible bugs
and so we've been trying to get rid of it as much as possible. And
that despite being used to very random things happening under us since
we often call out into unknown/hostile code (js-code in a webpage).

/ Jonas

Re: Synchronous postMessage for Workers?

[Charles Pritchard]

On 2/14/2012 5:31 AM, Arthur Barstow wrote:

On 2/14/12 2:02 AM, ext David Bruant wrote:

Le 13/02/2012 20:44, Ian Hickson a écrit :

Should we just tell authors to get used to the async style?

I think we should. More constructs are coming in ECMAScript. Things
related to language concurrency should probably be left to the core
language unless there is an extreme need (which there isn't as far as I
know).


David - if you have some recommended reading(s) re concurrency 
constructs that are relevant to this discussion and are coming to 
ECMAScript, please let me know.


(I tend to agree if there isn't some real urgency here, we should be 
careful [with our specs]).


We could still use some kind of synchronous semantic for passing 
gestures between frames.


This issue has popped up in various ways with Google Chrome extensions.

We can't have a user click on frame (a) and have frame (b) treat it as a 
gesture, for things like popup windows and the like.


This is of course intentional, for the purpose of security. But if both 
frames want it to happen, it's a side effect of the asynchronous nature 
of postMessage.

Re: Synchronous postMessage for Workers?

[Arthur Barstow]

On 2/14/12 2:02 AM, ext David Bruant wrote:

Le 13/02/2012 20:44, Ian Hickson a écrit :

Should we just tell authors to get used to the async style?

I think we should. More constructs are coming in ECMAScript. Things
related to language concurrency should probably be left to the core
language unless there is an extreme need (which there isn't as far as I
know).


David - if you have some recommended reading(s) re concurrency 
constructs that are relevant to this discussion and are coming to 
ECMAScript, please let me know.


(I tend to agree if there isn't some real urgency here, we should be 
careful [with our specs]).


-Thanks, ArtB

Re: Synchronous postMessage for Workers?

[David Bruant]

Le 13/02/2012 20:44, Ian Hickson a écrit :
> On Thu, 17 Nov 2011, Joshua Bell wrote:
>> Wouldn't it be lovely if the Worker script could simply make a 
>> synchronous call to fetch data from the Window?
> It wouldn't be so much a synchronous call, so much as a blocking get.
>
>
> On Thu, 17 Nov 2011, Jonas Sicking wrote:
>> We can only allow child workers to block on parent workers. Never the 
>> other way around.
> Indeed. And it would have to be limited to the built-in ports that workers 
> have, because regular ports can be shunted all over the place and you 
> could end up with a deadlock situation.
>
> It would be easy enough to add a blocking get on DedicatedWorkerGlobalScope.
> Is this something for which there's a lot of demand?
>
>// blocking get
>// no self.onmessage handler needed
>...
>var message = self.getMessage();
>...
>
> An alternative is to add continuations to the platform:
>
>// continuation
>// (this is not a formal proposal, just a illustration of the concept)
>var message;
>self.onmessage = function (event) {
>  message = event;
>  signal('got message'); // queues a task to resume from yieldUntil()
>};
>...
>yieldUntil('got message');
>...
>
> This would be a more general solution and would be applicable in many 
> other parts of the platform. As we get more and more async APIs, I think 
> it might be worth considering adding this.
>
> We could add it to HTML as an API rather than adding it to JS as a 
> language construct. It would be relatively easy to define, if much harder 
> to implement:
>
>yieldUntil(id) - spin the event loop until the signal() API unblocks 
> this script
>
>signal(id) - unblock the script with the oldest invokation of
> yieldUntil() called with the given ID, if any
>
> Given our definition of "spin the event loop", this doesn't even result, 
> per the spec, in nested event loops or anything like that. Note that per 
> this definition, "signal" just queues a task to resume the earlier script, 
> it is not your typical coroutine. That is:
>
>setTimeout(function () {
>  console.log('2');
>  signal('test');
>  console.log('3');
>}, 1000);
>console.log('1');
>yieldUntil('test');
>console.log('4');
>
> ...logs 1, 2, 3, 4, not 1, 2, 4, 3.
>
> Anyone object to me adding something like this?
If a web author does a typo (and we all know that especially with a
string-based API, it will happen or people will generate dynamically the
strings), what happens? Are you just blocked can can't resume after the
yieldUntil?

> Are there any better solutions?
"yield" as a keyword is considered being added to ECMAScript 6:
http://wiki.ecmascript.org/doku.php?id=harmony:generators#generator_functions
Web browsers will start implementing this one alongside with other new
ECMAScript features.
The yieldUntil/signal API would be probably a duplication of this.

I also recommand having a look at Dave Herman's Task.js http://taskjs.org/

> Should we just tell authors to get used to the async style?
I think we should. More constructs are coming in ECMAScript. Things
related to language concurrency should probably be left to the core
language unless there is an extreme need (which there isn't as far as I
know).

David

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Mon, Feb 13, 2012 at 3:10 PM, Glenn Maynard  wrote:

> alert("hello", finishedMessage);
> yieldUntil(finishedMessage);
> finishedFunc();
>


> which would send a message on port1 when it completes.  Waiting on
> multiple ports would be straightforward, eg. msg = yield([port1, port2,
> port3]), though MessageEvent would need a pointer back to the MessagePort
> so you can tell which it came from.
>

A use case tying these together is synchronously running two parallel XHR
requests.  Currently, if you're writing code in a Worker, your only option
to make two XHR calls simultaneously is to use the async API, which means
your own API must be async.  If you want to write a synchronous API, you
have to use the sync API, which means you can only make a single request at
a time.

With this mechanism, a hypothetical XHR API could allow this:

xhr1.open("GET", ..., port1a);
xhr1.send();
xhr2.open("GET", ..., port2a);
xhr2.send();

yieldUntil([port1b, port2b]); // wait for one of the requests to complete
yieldUntil([port1b, port2b]); // wait for the other request to complete

The same could be applied to use any combination of APIs concurrently, in a
function that behaves synchronously.  (These could have used the same
message channel.)

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Mon, Feb 13, 2012 at 1:44 PM, Ian Hickson  wrote:

> An alternative is to add continuations to the platform:
>
>   // continuation
>   // (this is not a formal proposal, just a illustration of the concept)
>   var message;
>   self.onmessage = function (event) {
> message = event;
> signal('got message'); // queues a task to resume from yieldUntil()
>   };
>   ...
>   yieldUntil('got message');
>   ...
>
> This would be a more general solution and would be applicable in many
> other parts of the platform. As we get more and more async APIs, I think
> it might be worth considering adding this.
>

Even better: we could design blocking APIs without having to have separate,
redundant sync and async APIs.  For example, instead of having
(hypothetically)

alert("hello"); finishedFunc(); // sync
alertAsync("hello", finishedFunc); // async

we could have a single async API,

alert("hello", finishedMessage);
yieldUntil(finishedMessage);
finishedFunc();

I suppose it's a little less pretty, and it wouldn't help with existing
APIs, but it's worth thinking about.

Anyone object to me adding something like this? Are there any better
> solutions?


This would definitely be nice to see, if it can be implemented.

This approach would also allow blocking for a message on any message port,
and not just the builtin worker ports, since it wouldn't result in
deadlock.  The failure mode would be permanently yielded, "stuck" call
stacks, where a function is waiting for a message that it'll never
receive.  But, that's inherent in this feature, since you could always
yield and then never signal.

> Should we just tell authors to get used to the async style?

One of the biggest gains of Workers, in my view, is specifically that it
allows writing code in a synchronous style.  Writing code in an async way
is fine when you really need something to run in the UI thread, but it
leads to more complex, harder to write code.  We should definitely prefer
to give robust ways to write linear, synchronous code.

If this can even be done in the UI thread, as I think what you're
suggesting would allow, that's even better.


For what it's worth, it would be nice to see this implemented in terms of
message ports.  For example, something like this:

port2.postMessage("message");
msg = yield(port1);
// msg.data == "message";

Other APIs could then signal wakeups; as a simple example (not a proposal),

alert("hello", port1);

which would send a message on port1 when it completes.  Waiting on multiple
ports would be straightforward, eg. msg = yield([port1, port2, port3]),
though MessageEvent would need a pointer back to the MessagePort so you can
tell which it came from.

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[John J Barton]

On Mon, Feb 13, 2012 at 11:44 AM, Ian Hickson  wrote:
> On Thu, 17 Nov 2011, Joshua Bell wrote:
>>
>> Wouldn't it be lovely if the Worker script could simply make a
>> synchronous call to fetch data from the Window?
>
> It wouldn't be so much a synchronous call, so much as a blocking get.
>..
> Anyone object to me adding something like this? Are there any better
> solutions? Should we just tell authors to get used to the async style?

I guess the "Q" folks would say that remote promises provides another
solution. If promises are adopted by the platform, then the async
style gets much easier to work with.
https://github.com/kriskowal/q
(spec is somewhere on the es wiki)

In the Q model you would fetch data like:
  parentWindow.fetchData('myQueryString').then(  // block until reply.
function(data) {...},
function(err) {...}
  );
Q has functions to join promises; q_comm add remote promises.

I believe this can be done today with q_comm in workers.

Your signal/yieldUntil looks like what es-discuss calls "generators".
I found them much harder to understand than promises, but then I come
from JS not python.

jjb

Re: Synchronous postMessage for Workers?

[Ian Hickson]

On Thu, 17 Nov 2011, Joshua Bell wrote:
>
> Wouldn't it be lovely if the Worker script could simply make a 
> synchronous call to fetch data from the Window?

It wouldn't be so much a synchronous call, so much as a blocking get.


On Thu, 17 Nov 2011, Jonas Sicking wrote:
> 
> We can only allow child workers to block on parent workers. Never the 
> other way around.

Indeed. And it would have to be limited to the built-in ports that workers 
have, because regular ports can be shunted all over the place and you 
could end up with a deadlock situation.

It would be easy enough to add a blocking get on DedicatedWorkerGlobalScope.
Is this something for which there's a lot of demand?

   // blocking get
   // no self.onmessage handler needed
   ...
   var message = self.getMessage();
   ...

An alternative is to add continuations to the platform:

   // continuation
   // (this is not a formal proposal, just a illustration of the concept)
   var message;
   self.onmessage = function (event) {
 message = event;
 signal('got message'); // queues a task to resume from yieldUntil()
   };
   ...
   yieldUntil('got message');
   ...

This would be a more general solution and would be applicable in many 
other parts of the platform. As we get more and more async APIs, I think 
it might be worth considering adding this.

We could add it to HTML as an API rather than adding it to JS as a 
language construct. It would be relatively easy to define, if much harder 
to implement:

   yieldUntil(id) - spin the event loop until the signal() API unblocks 
this script

   signal(id) - unblock the script with the oldest invokation of
yieldUntil() called with the given ID, if any

Given our definition of "spin the event loop", this doesn't even result, 
per the spec, in nested event loops or anything like that. Note that per 
this definition, "signal" just queues a task to resume the earlier script, 
it is not your typical coroutine. That is:

   setTimeout(function () {
 console.log('2');
 signal('test');
 console.log('3');
   }, 1000);
   console.log('1');
   yieldUntil('test');
   console.log('4');

...logs 1, 2, 3, 4, not 1, 2, 4, 3.

Anyone object to me adding something like this? Are there any better 
solutions? Should we just tell authors to get used to the async style?

-- 
Ian Hickson   U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

Re: Synchronous postMessage for Workers?

[Bjoern Hoehrmann]

* Joshua Bell wrote:
>Jonas and I were having an offline discussing regarding the synchronous
>Indexed Database API and noting how clean and straightforward it will allow
>Worker scripts to be. One general Worker issue we noted - independent of
>IDB - was that there are cases where Worker scripts may need to fetch data
>from the Window. This can be done today using bidirectional postMessage,
>but of course this requires the Worker to then be coded in now common
>asynchronous JavaScript fashion, with either a tangled mess of callbacks or
>some sort of Promises/Futures library, which removes some of the benefits
>of introducing sync APIs to Workers in the first place.

I note that you can overcome manually coded continuation passing style
by exposing the continuations. In JavaScript 1.7 "Generators" do that in
some limited fashion: a generator can yield control to the calling code,
which can then resume the generator at its discretion, allowing you to
implement Wait functions that wait for whatever you want them waiting
for, for however long you might want them to wait for it. Some years ago
I made http://lists.w3.org/Archives/Public/www-archive/2008Jul/0009.html
as a quick example of using generators in this manner.

The generator there essentially schedules resource loads and yields con-
trol until the resource finishes loading in which case the event handler
would resume the generator. A more realistic example would schedule all
the resources needed to proceed and watch out for load failures and so
on, but it demonstrates that cooperative multitasking is quite feasible.

Obviously, even if "ES.next" provides better support for this, it is a
good bit more complicated to use directly, and easier to get wrong, than
a dedicated solution with limited scope as you propose for this special
case, but adding blocking APIs here and there to address a few use cases
also doesn't strike me as the best approach.
-- 
Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Fri, Nov 18, 2011 at 1:33 PM, Rick Waldron wrote:

> My initial concerns aside and with no trolling intentions... The above
> literal expressions are all syntax errors, = should be :
>

A typo I make regularly; it's a side-effect of using having too many
languages rattling around in my head at once.  (Lua uses "=".)

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[Rick Waldron]

On Nov 18, 2011, at 12:56 PM, Glenn Maynard  wrote:

> On Fri, Nov 18, 2011 at 12:18 PM, David Levin  wrote:
> The primary use case is one in which messages sent to 
> DedicatedWorkerGlobalScope are reserved for synchronous messaging.
> 
> Mostly, yes, or otherwise dealing with unrelated messages coming in while 
> you're waiting for a response.
> 
> It's possible to safely use async and sync message receipt on the same 
> channel.  For example,
> 
> onmessage = function(e)
> {
> if(e.data.message == "pause")
> {
> msg = waitForMessage();
> // assert(msg.message == "continue");

The name "waitForMessage()" definitely has a "promise/futures" connotation... 


> }
> }
> 
> and in the main thread,
> 
> worker.postMessage({message = "pause"});
> alert("The worker is currently paused and waiting to be continued");
> worker.postMessage({message = "continue"});


My initial concerns aside and with no trolling intentions... The above literal 
expressions are all syntax errors, = should be :
> 
> Once you send "pause", events and timers in the worker will be deferred until 
> you resume it.  Since messages are delivered in order, this is guaranteed to 
> work.
> 
> (By the way, the above isn't why I prefer this API; I prefer it because it 
> avoids adding more messaging mechanisms and it creates a simpler API.  The 
> above is just an interesting way it might be used.)
> 
> -- 
> Glenn Maynard
> 

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Fri, Nov 18, 2011 at 12:18 PM, David Levin  wrote:

> The primary use case is one in which messages sent to
> DedicatedWorkerGlobalScope are reserved for synchronous messaging.
>

Mostly, yes, or otherwise dealing with unrelated messages coming in while
you're waiting for a response.

It's possible to safely use async and sync message receipt on the same
channel.  For example,

onmessage = function(e)
{
if(e.data.message == "pause")
{
msg = waitForMessage();
// assert(msg.message == "continue");
}
}

and in the main thread,

worker.postMessage({message = "pause"});
alert("The worker is currently paused and waiting to be continued");
worker.postMessage({message = "continue"});

Once you send "pause", events and timers in the worker will be deferred
until you resume it.  Since messages are delivered in order, this is
guaranteed to work.

(By the way, the above isn't why I prefer this API; I prefer it because it
avoids adding more messaging mechanisms and it creates a simpler API.  The
above is just an interesting way it might be used.)

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[David Levin]

On Fri, Nov 18, 2011 at 9:07 AM, Glenn Maynard  wrote:

> On Fri, Nov 18, 2011 at 11:50 AM, David Levin  wrote:
>
>> So the primary use case is code in the worker which has no other (async)
>> messages coming in?
>>
>
> No--you can always create another message channel for other types of
> messages, if you need to.
>

I'll try again. :)

The primary use case is one in which messages sent to
DedicatedWorkerGlobalScope are reserved for synchronous messaging.

dave

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Fri, Nov 18, 2011 at 11:50 AM, David Levin  wrote:

> So the primary use case is code in the worker which has no other (async)
> messages coming in?
>

No--you can always create another message channel for other types of
messages, if you need to.

(Of course, you shouldn't receive messages on other ports *while* blocking;
that is, the event loop isn't spun while waiting.  That's just the same as
any other synchronous worker API.)

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[David Levin]

On Fri, Nov 18, 2011 at 8:16 AM, Glenn Maynard  wrote:

> On Thu, Nov 17, 2011 at 10:33 PM, David Levin  wrote:
>
>> Ah so the proposal is really only adding a new method only
>> on DedicatedWorkerGlobalScope which send a synchronous message and
>> something corresponding on Worker which can respond to this.
>>
>
> There's no need for a new sending method; only a receiving method.  To
> reuse the original example:
>
>   postMessage({action: "prompt_user", prompt: "How about a nice game of
> chess?"});
>   var msg = waitForMessage();
>   if(msg && msg.data) { chess_game.begin(); }
>
> The other side is as usual:
>
>   worker.onmessage = function(e) { worker.postMessage(true); }
>
> without caring which API the worker is using to receive the response.
>

So the primary use case is code in the worker which has no other (async)
messages coming in?

dave

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Thu, Nov 17, 2011 at 10:33 PM, David Levin  wrote:

> Ah so the proposal is really only adding a new method only
> on DedicatedWorkerGlobalScope which send a synchronous message and
> something corresponding on Worker which can respond to this.
>

There's no need for a new sending method; only a receiving method.  To
reuse the original example:

  postMessage({action: "prompt_user", prompt: "How about a nice game of
chess?"});
  var msg = waitForMessage();
  if(msg && msg.data) { chess_game.begin(); }

The other side is as usual:

  worker.onmessage = function(e) { worker.postMessage(true); }

without caring which API the worker is using to receive the response.

Note that the nonblocking case should definitely work on all ports.
There's no risk of deadlocks when not blocking.

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[David Levin]

On Thu, Nov 17, 2011 at 6:41 PM, Jonas Sicking  wrote:

> On Thu, Nov 17, 2011 at 6:05 PM, David Levin  wrote:
> >
> >
> > On Thu, Nov 17, 2011 at 5:05 PM, Jonas Sicking  wrote:
> >>
> >> On Thu, Nov 17, 2011 at 2:07 PM, David Levin 
> wrote:
> >> > It seems like this mechanism would deadlock a worker if two workers
> send
> >> > each other a synchronous message.
> >>
> >> Indeed. We can only allow child workers to block on parent workers.
> >> Never the other way around.
> >
> > So the api would have to know who is listening to the other end of the
> port
> > and throw if it isn't a parent?
>
> I'm not convinced that we can do this with ports in a sane manner.
> Only on dedicated workers. There you always know who is listening on
> the other end. I.e. only on the DedicatedWorkerGlobalScope/Worker
> interfaces.


Ah so the proposal is really only adding a new method only
on DedicatedWorkerGlobalScope which send a synchronous message and
something corresponding on Worker which can respond to this.

This proposal as you see it does nothing for port or shared workers.

That seems to make more sense now.

dave

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Thu, Nov 17, 2011 at 8:05 PM, Jonas Sicking  wrote:

> On Thu, Nov 17, 2011 at 2:07 PM, David Levin  wrote:
> > It seems like this mechanism would deadlock a worker if two workers send
> > each other a synchronous message.
>
> Indeed. We can only allow child workers to block on parent workers.
> Never the other way around.
>

Note that the port entangled with the port you're waiting on might be
transferred between workers while you're waiting on it, changing hands from
a parent worker to a child worker.  It might be a valid port to block on
when you make the call, and stop being valid later.

I'm still thinking about this, but one note is that this isn't needed for
the polling case (where timeout is 0).

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[Charles Pritchard]

On Nov 17, 2011, at 6:41 PM, Jonas Sicking  wrote:

> On Thu, Nov 17, 2011 at 6:05 PM, David Levin  wrote:
>> 
>> 
>> On Thu, Nov 17, 2011 at 5:05 PM, Jonas Sicking  wrote:
>>> 
>>> On Thu, Nov 17, 2011 at 2:07 PM, David Levin  wrote:
 It seems like this mechanism would deadlock a worker if two workers send
 each other a synchronous message.
>>> 
>>> Indeed. We can only allow child workers to block on parent workers.
>>> Never the other way around.
>> 
>> So the api would have to know who is listening to the other end of the port
>> and throw if it isn't a parent?
> 
> I'm not convinced that we can do this with ports in a sane manner.
> Only on dedicated workers. There you always know who is listening on
> the other end. I.e. only on the DedicatedWorkerGlobalScope/Worker
> interfaces.

I agree on this one. A big no-thanks to synchronous messaging over ports.

Still, dedicated workers to a parent seems doable.

I've proposed a different worker scope for lightweight threads. Dedicated 
workers are relatively heavy and so a few workers blocked on the parent is not 
outside of acceptability.

-Charles

Re: Synchronous postMessage for Workers?

[Jonas Sicking]

On Thu, Nov 17, 2011 at 6:05 PM, David Levin  wrote:
>
>
> On Thu, Nov 17, 2011 at 5:05 PM, Jonas Sicking  wrote:
>>
>> On Thu, Nov 17, 2011 at 2:07 PM, David Levin  wrote:
>> > It seems like this mechanism would deadlock a worker if two workers send
>> > each other a synchronous message.
>>
>> Indeed. We can only allow child workers to block on parent workers.
>> Never the other way around.
>
> So the api would have to know who is listening to the other end of the port
> and throw if it isn't a parent?

I'm not convinced that we can do this with ports in a sane manner.
Only on dedicated workers. There you always know who is listening on
the other end. I.e. only on the DedicatedWorkerGlobalScope/Worker
interfaces.

/ Jonas

Re: Synchronous postMessage for Workers?

[David Levin]

On Thu, Nov 17, 2011 at 5:05 PM, Jonas Sicking  wrote:

> On Thu, Nov 17, 2011 at 2:07 PM, David Levin  wrote:
> > It seems like this mechanism would deadlock a worker if two workers send
> > each other a synchronous message.
>
> Indeed. We can only allow child workers to block on parent workers.
> Never the other way around.
>

So the api would have to know who is listening to the other end of the port
and throw if it isn't a parent?

dave

Re: Synchronous postMessage for Workers?

[Jonas Sicking]

On Thu, Nov 17, 2011 at 2:07 PM, David Levin  wrote:
> It seems like this mechanism would deadlock a worker if two workers send
> each other a synchronous message.

Indeed. We can only allow child workers to block on parent workers.
Never the other way around.

I think in theory it would be possible to reverse the direction for
given parent/child pairs, but I'm not yet sure that that couldn't lead
to deadlocks. And I'm definitely not sure it's worth the complexity.

/ Jonas

Re: Synchronous postMessage for Workers?

[Rick Waldron]

On Thu, Nov 17, 2011 at 6:50 PM, Glenn Maynard  wrote:

> On Thu, Nov 17, 2011 at 6:17 PM, Rick Waldron wrote:
>
>> No, it's not. Messaging should not block either process.
>>
>
> No, it's perfectly fine to block a worker, as long as the worker
> explicitly requests it and is expecting it.
>
> I don't know what that means.  Synchronous APIs are one of the major
>> reasons to have Workers in the first place.
>>
>> Considering the current messaging API and the allowed host APIs, I
>> strongly disagree.
>>
>
> I'm not quite sure what you mean by "allowed host APIs", but there are
> lots of synchronous APIs in Workers: File API, Filesystem API, IndexedDB.
> These are all new APIs, not historical accidents (like sync XHR in the UI
> thread).  Synchronous (blocking) APIs in workers is entirely by design, in
> order to allow writing clean, linear code instead of code that has to yield
> to the calling environment all the time.
>

Sorry, I should've been clearer - I was referring to the renderer-to-worker
messaging API - all event based, all async.

TBH, at this point I'd be more interested in an actual prototype
implementation to see the effects it would have on Worker to Worker
messaging, SharedWorker etc.

Rick




> --
> Glenn Maynard
>
>

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Thu, Nov 17, 2011 at 6:17 PM, Rick Waldron wrote:

> No, it's not. Messaging should not block either process.
>

No, it's perfectly fine to block a worker, as long as the worker explicitly
requests it and is expecting it.

I don't know what that means.  Synchronous APIs are one of the major
> reasons to have Workers in the first place.
>
> Considering the current messaging API and the allowed host APIs, I
> strongly disagree.
>

I'm not quite sure what you mean by "allowed host APIs", but there are lots
of synchronous APIs in Workers: File API, Filesystem API, IndexedDB.  These
are all new APIs, not historical accidents (like sync XHR in the UI
thread).  Synchronous (blocking) APIs in workers is entirely by design, in
order to allow writing clean, linear code instead of code that has to yield
to the calling environment all the time.

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[Rick Waldron]

On Nov 17, 2011, at 2:37 PM, Glenn Maynard  wrote:

> On Thu, Nov 17, 2011 at 2:16 PM, Rick Waldron  wrote:
> This is counter to the whole premise of Workers, which should be independent 
> of their renderer process and certainly not block themselves while waiting 
> for responses from the renderer (which inherently describes an async 
> behaviour). 
> 
> That's backwards.  The premise is that the renderer (UI) will not be blocked 
> by workers, not that workers won't be blocked by the renderer.

No, it's not. Messaging should not block either process. 

> 
> > Sync apis are sore thumb in GlobalWorkerScope's world.
> 
> I don't know what that means.  Synchronous APIs are one of the major reasons 
> to have Workers in the first place.

Considering the current messaging API and the allowed host APIs, I strongly 
disagree. 



> 
> -- 
> Glenn Maynard
> 

Re: Synchronous postMessage for Workers?

[David Levin]

It seems like this mechanism would deadlock a worker if two workers send
each other a synchronous message.

dave


On Thu, Nov 17, 2011 at 10:37 AM, Joshua Bell  wrote:

> Jonas and I were having an offline discussing regarding the synchronous
> Indexed Database API and noting how clean and straightforward it will allow
> Worker scripts to be. One general Worker issue we noted - independent of
> IDB - was that there are cases where Worker scripts may need to fetch data
> from the Window. This can be done today using bidirectional postMessage,
> but of course this requires the Worker to then be coded in now common
> asynchronous JavaScript fashion, with either a tangled mess of callbacks or
> some sort of Promises/Futures library, which removes some of the benefits
> of introducing sync APIs to Workers in the first place.
>
> Wouldn't it be lovely if the Worker script could simply make a synchronous
> call to fetch data from the Window?
>
> GTNW.prototype.end = function () {
> var result = self.sendMessage({action: "prompt_user", prompt: "How
> about a nice game of chess?"});
> if (result) { chess_game.begin(); }
> }
>
> The requirement would be that the Window side is asynchronous (of course).
> Continuing the silly example above, the Window script responds to the
> message by fetching some new HTML UI via async XHR, adding it to the DOM,
> and only after user input and validation events is a response sent back to
> the Worker, which proceeds merrily on its way.
>
> I don't have a specific API suggestion in mind. On the Worker side it
> should take the form of a single blocking call taking the data to be passed
> and possibly a timeout, and allowing a return value (on
> timeout return undefined or throw?). On the Window side it could be a new
> event on Worker which delivers a "Promise" type object which the Window
> script can later fulfill (or break). Behavior on multiple event listeners
> would need to be defined (all get the same "Promise", first "fulfill" wins,
> others throw?).
>
>

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Thu, Nov 17, 2011 at 3:47 PM, Joshua Bell  wrote:

> ^^^ Nit: That would revert back to being postMessage(), no new API on the
> Worker side.
>

Right, I just copied your example.  (You mean "no new API for sending
messages"--the "get the next event" method would be new API for workers.)

One concern with this overall approach is that any pending message would be
> grabbed, not just one intended as a response.
>

You just create separate MessagePorts to segregate different types of
messages.  messagePort.getMessageIfExists would only retrieve messages on
messagePort.

"Retrieve the first message from a list of message ports", akin to
poll/select, may also be useful, eg. getFirstMessage([port1, port2],
timeout).  I'm not sure if there are use cases for this, but it's worth
mentioning.

I agree that "return" makes more sense than "dispatch", since we're
> introducing this to support a linear programming style. On the other hand,
> you could emulate "return" with "dispatch" via helper functions that swap
> in a temporary onmessage handler (a variant on your
> getPendingMessageWithoutDelivery example in the older thread)
>

Right, you can emulate either using the other.  Alternatively, do both:
fire the event synchronously, then return it.

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[Joshua Bell]

On Thu, Nov 17, 2011 at 11:28 AM, Glenn Maynard  wrote:

>
> We discussed a very similar thing about a year ago; I've been meaning to
> bring that up again, so this is probably as good a time as any.
> http://lists.w3.org/Archives/Public/public-webapps/2010OctDec/1075.html
>

Ah, thanks - I should have gone digging.


> The proposal is to allow polling a MessagePort, causing the first queued
> message, if any, to be dispatched (or alternatively, to be returned).  This
> could be easily extended to handle the above, by adding a "blocking
> duration" parameter.
>
> For example, working from Jonas's getMessageIfExists proposal:
>
>
>   self.sendMessage({action: "prompt_user", prompt: "How about a nice game
> of chess?"});
>
^^^ Nit: That would revert back to being postMessage(), no new API on the
Worker side.


>   var msg = messagePort.getMessageIfExists(5.0);
>   if(msg && msg.data) { chess_game.begin(); }
>
> Here, 5.0 means to block for five seconds (with a sentinel like -1 would
> mean "block forever"), and the return value is the MessageEvent, returning
> null if no message is received.
>

One concern with this overall approach is that any pending message would be
grabbed, not just one intended as a response. But we're talking about
workers that are intending to run long-lived functions here anyway so they
must explicitly block/poll to receive any communication. It's a more
complicated model for developers, but the complexity is opt-in. So I
like it.

I preferred the "dispatch the first message" approach before, but your
> blocking use case may make the "return the first message" approach better.
>

I agree that "return" makes more sense than "dispatch", since we're
introducing this to support a linear programming style. On the other hand,
you could emulate "return" with "dispatch" via helper functions that swap
in a temporary onmessage handler (a variant on your
getPendingMessageWithoutDelivery example in the older thread)

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Thu, Nov 17, 2011 at 2:16 PM, Rick Waldron wrote:

> This is counter to the whole premise of Workers, which should be
> independent of their renderer process and certainly not block themselves
> while waiting for responses from the renderer (which inherently describes
> an async behaviour).
>

That's backwards.  The premise is that the renderer (UI) will not be
blocked by workers, not that workers won't be blocked by the renderer.

> Sync apis are sore thumb in GlobalWorkerScope's world.

I don't know what that means.  Synchronous APIs are one of the major
reasons to have Workers in the first place.

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[Glenn Maynard]

On Thu, Nov 17, 2011 at 1:37 PM, Joshua Bell  wrote:

> Jonas and I were having an offline discussing regarding the synchronous
> Indexed Database API and noting how clean and straightforward it will allow
> Worker scripts to be. One general Worker issue we noted - independent of
> IDB - was that there are cases where Worker scripts may need to fetch data
> from the Window. This can be done today using bidirectional postMessage,
> but of course this requires the Worker to then be coded in now common
> asynchronous JavaScript fashion, with either a tangled mess of callbacks or
> some sort of Promises/Futures library, which removes some of the benefits
> of introducing sync APIs to Workers in the first place.
>
> Wouldn't it be lovely if the Worker script could simply make a synchronous
> call to fetch data from the Window?
>
> GTNW.prototype.end = function () {
> var result = self.sendMessage({action: "prompt_user", prompt: "How
> about a nice game of chess?"});
> if (result) { chess_game.begin(); }
> }
>

We discussed a very similar thing about a year ago; I've been meaning to
bring that up again, so this is probably as good a time as any.
http://lists.w3.org/Archives/Public/public-webapps/2010OctDec/1075.html

The proposal is to allow polling a MessagePort, causing the first queued
message, if any, to be dispatched (or alternatively, to be returned).  This
could be easily extended to handle the above, by adding a "blocking
duration" parameter.

For example, working from Jonas's getMessageIfExists proposal:

  self.sendMessage({action: "prompt_user", prompt: "How about a nice game
of chess?"});
  var msg = messagePort.getMessageIfExists(5.0);
  if(msg && msg.data) { chess_game.begin(); }

Here, 5.0 means to block for five seconds (with a sentinel like -1 would
mean "block forever"), and the return value is the MessageEvent, returning
null if no message is received.

I preferred the "dispatch the first message" approach before, but your
blocking use case may make the "return the first message" approach better.

I think this is a fundamental missing piece to worker communication.  A
basic reason for having Workers in the first place is so you can write
linear code, instead of having to structure code to be able to return
regularly (often awkward and inconvenient), but currently in order to
receive messages in workers you still have to do that.

The use case that lead me to this initially was wanting to cancel
operations cleanly, without having to terminate and reinitialize the worker
entirely, eg. for input completion where processing restarts on each
keypress.  (That was an issue because the completion was initialized from
localStorage, making startup expensive; IndexedDB will help that particular
example, but this applies to any worker with nontrivial startup times.)

-- 
Glenn Maynard

Re: Synchronous postMessage for Workers?

[Rick Waldron]

On Thu, Nov 17, 2011 at 1:37 PM, Joshua Bell  wrote:

> Jonas and I were having an offline discussing regarding the synchronous
> Indexed Database API and noting how clean and straightforward it will allow
> Worker scripts to be. One general Worker issue we noted - independent of
> IDB - was that there are cases where Worker scripts may need to fetch data
> from the Window. This can be done today using bidirectional postMessage,
> but of course this requires the Worker to then be coded in now common
> asynchronous JavaScript fashion, with either a tangled mess of callbacks or
> some sort of Promises/Futures library, which removes some of the benefits
> of introducing sync APIs to Workers in the first place.


Workers are suited quite well to Promises/Futures apis. Sync apis are sore
thumb in GlobalWorkerScope's world.


>
> Wouldn't it be lovely if the Worker script could simply make a synchronous
> call to fetch data from the Window?
>
> GTNW.prototype.end = function () {
> var result = self.sendMessage({action: "prompt_user", prompt: "How
> about a nice game of chess?"});
> if (result) { chess_game.begin(); }
> }
>
> The requirement would be that the Window side is asynchronous (of course).
> Continuing the silly example above, the Window script responds to the
> message by fetching some new HTML UI via async XHR, adding it to the DOM,
> and only after user input and validation events is a response sent back to
> the Worker, which proceeds merrily on its way.
>
> I don't have a specific API suggestion in mind. On the Worker side it
> should take the form of a single blocking call taking the data to be passed
> and possibly a timeout, and allowing a return value (on
> timeout return undefined or throw?).
>

This is counter to the whole premise of Workers, which should be
independent of their renderer process and certainly not block themselves
while waiting for responses from the renderer (which inherently describes
an async behaviour).



> On the Window side it could be a new event on Worker which delivers a
> "Promise" type object which the Window script can later fulfill (or break).
> Behavior on multiple event listeners would need to be defined (all get the
> same "Promise", first "fulfill" wins, others throw?).
>
>

Rick

Synchronous postMessage for Workers?

[Joshua Bell]

Jonas and I were having an offline discussing regarding the synchronous
Indexed Database API and noting how clean and straightforward it will allow
Worker scripts to be. One general Worker issue we noted - independent of
IDB - was that there are cases where Worker scripts may need to fetch data
from the Window. This can be done today using bidirectional postMessage,
but of course this requires the Worker to then be coded in now common
asynchronous JavaScript fashion, with either a tangled mess of callbacks or
some sort of Promises/Futures library, which removes some of the benefits
of introducing sync APIs to Workers in the first place.

Wouldn't it be lovely if the Worker script could simply make a synchronous
call to fetch data from the Window?

GTNW.prototype.end = function () {
var result = self.sendMessage({action: "prompt_user", prompt: "How
about a nice game of chess?"});
if (result) { chess_game.begin(); }
}

The requirement would be that the Window side is asynchronous (of course).
Continuing the silly example above, the Window script responds to the
message by fetching some new HTML UI via async XHR, adding it to the DOM,
and only after user input and validation events is a response sent back to
the Worker, which proceeds merrily on its way.

I don't have a specific API suggestion in mind. On the Worker side it
should take the form of a single blocking call taking the data to be passed
and possibly a timeout, and allowing a return value (on
timeout return undefined or throw?). On the Window side it could be a new
event on Worker which delivers a "Promise" type object which the Window
script can later fulfill (or break). Behavior on multiple event listeners
would need to be defined (all get the same "Promise", first "fulfill" wins,
others throw?).