Re: More flexibility in the ECMAScript part?
Le 18/04/2013 05:07, Tab Atkins Jr. a écrit : On Wed, Apr 17, 2013 at 3:57 PM, Allen Wirfs-Brock al...@wirfs-brock.com wrote: A a rule of thumb, if a library does something that can not be expressed in the its base language there is a good chance it is extending the virtual machine of the language and it should at least be reviewed from that perspective and iframe semantics. These are both examples of browser design choices that have deep semantics impact upon the language. Note that Futures are entirely expressible in today's JS semantics. Allen was concerned about the ECMAScript definition of what you're loosely calling JS while it seems you're referring to web platform enhanced JavaScript Specifically, the current semantics uses queue a task which interacts with the event loop which is part of the web platform, but not ECMAScript. I've read intention to move the event loop from HTML Living Standard to ECMAScript 7, but we're not there yet and that work is necessary before adding promises to ECMAScript. David ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part? (was: Re: Futures
On Thu, Apr 18, 2013 at 4:07 AM, Tab Atkins Jr. jackalm...@gmail.com wrote: Note that Futures are entirely expressible in today's JS semantics. (Not to say that it shouldn't be reviewed by the language gurus here, just saying.) JavaScript does not have an event loop (as I mentioned elsewhere) so that is not true. HTML defines the event loop model and processing model for any asynchronous JavaScript execution. Lifting that up to JavaScript seems difficult. -- http://annevankesteren.nl/ ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: DOM EventStreams (take two on Streams): Request for feedback
On Thu, Apr 18, 2013 at 3:11 AM, Tab Atkins Jr. jackalm...@gmail.com wrote: On Wed, Apr 17, 2013 at 5:50 PM, Jason Orendorff jason.orendo...@gmail.com wrote: That seems sensible. OTOH FutureResolver seems to make all those methods no-ops instead. (Step 1 of each method's implementation: If the context object's resolved flag is set, terminate these steps.) I'm not sure what that's about. Maybe worth asking. Hm, we should be consistent. Unsure which is better; I'll ping Anne. They do not throw so the resolver can intentionally race. -- http://annevankesteren.nl/ ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
Le 18/04/2013 09:40, Anne van Kesteren a écrit : On Thu, Apr 18, 2013 at 4:07 AM, Tab Atkins Jr. jackalm...@gmail.com wrote: Note that Futures are entirely expressible in today's JS semantics. (Not to say that it shouldn't be reviewed by the language gurus here, just saying.) JavaScript does not have an event loop (as I mentioned elsewhere) so that is not true. HTML defines the event loop model and processing model for any asynchronous JavaScript execution. Lifting that up to JavaScript seems difficult. What do you expect to be difficult? I foresee that it's going to be *a lot* of work on both sides (W3C/WHATWG TC39) to move this major piece from one place to another without breaking anything. But I would say it's a lot of easy work. It's going to take a lot of eyeballs and probably tests to make sure what the new spec jonction between HTML LS and ES7 conforms to what exists in reality (expecially in the new prose). Is there a particular part of this work that you expect to be difficult. David ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part? (was: Re: Futures
On Thu, Apr 18, 2013 at 3:40 AM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Apr 18, 2013 at 4:07 AM, Tab Atkins Jr. jackalm...@gmail.com wrote: Note that Futures are entirely expressible in today's JS semantics. (Not to say that it shouldn't be reviewed by the language gurus here, just saying.) HTML defines the event loop model and processing model for any asynchronous JavaScript execution. This is true. JavaScript does not have an event loop (as I mentioned elsewhere) so [Futures are entirely expressible] is not true. The only part of futures that isn't expressible today in ES5 (I believe) is the requirement that callbacks be called in the next tick when .then() is called on a resolved promise. Lifting that up to JavaScript seems difficult. Fortunately, this isn't true, because the module loader system requires us to add the event loop to ES6. Sam ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
On Thu, Apr 18, 2013 at 8:48 AM, David Bruant bruan...@gmail.com wrote: Le 18/04/2013 09:40, Anne van Kesteren a écrit : On Thu, Apr 18, 2013 at 4:07 AM, Tab Atkins Jr. jackalm...@gmail.com wrote: Note that Futures are entirely expressible in today's JS semantics. (Not to say that it shouldn't be reviewed by the language gurus here, just saying.) JavaScript does not have an event loop (as I mentioned elsewhere) so that is not true. HTML defines the event loop model and processing model for any asynchronous JavaScript execution. Lifting that up to JavaScript seems difficult. What do you expect to be difficult? I foresee that it's going to be *a lot* of work on both sides (W3C/WHATWG TC39) to move this major piece from one place to another without breaking anything. I don't think that's true at all. We're never going to be able to standardize on ONE event-loop, nor even be able to require that implementations have them...indeed, we've been pretty careful in designs like Object.observe() to avoid dependencies on it. What we can (and should) do is to say if you have an event loop, some primitives behave in this way relative to turns. That doesn't require breaking anything. But I would say it's a lot of easy work. It's going to take a lot of eyeballs and probably tests to make sure what the new spec jonction between HTML LS and ES7 conforms to what exists in reality (expecially in the new prose). Yes, we'll need lots of tests, but the real world gets on fine with event loops and lots of fine-grained, timing-dependent APIs in the web platform today across multiple implementations. We don't have it any harder than any other spec does in that regard. Is there a particular part of this work that you expect to be difficult. David __**_ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/**listinfo/es-discusshttps://mail.mozilla.org/listinfo/es-discuss ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
Le 18/04/2013 13:28, Alex Russell a écrit : On Thu, Apr 18, 2013 at 8:48 AM, David Bruant bruan...@gmail.com mailto:bruan...@gmail.com wrote: Le 18/04/2013 09:40, Anne van Kesteren a écrit : On Thu, Apr 18, 2013 at 4:07 AM, Tab Atkins Jr. jackalm...@gmail.com mailto:jackalm...@gmail.com wrote: Note that Futures are entirely expressible in today's JS semantics. (Not to say that it shouldn't be reviewed by the language gurus here, just saying.) JavaScript does not have an event loop (as I mentioned elsewhere) so that is not true. HTML defines the event loop model and processing model for any asynchronous JavaScript execution. Lifting that up to JavaScript seems difficult. What do you expect to be difficult? I foresee that it's going to be *a lot* of work on both sides (W3C/WHATWG TC39) to move this major piece from one place to another without breaking anything. I don't think that's true at all. We're never going to be able to standardize on ONE event-loop, nor even be able to require that implementations have them...indeed, we've been pretty careful in designs like Object.observe() to avoid dependencies on it. What we can (and should) do is to say if you have an event loop, some primitives behave in this way relative to turns. That doesn't require breaking anything. I believe the ES spec should provide the primitives of what type of interaction with the message queue is allowed and what isn't. This set of primitives would obviously contain all the interactions allowed today by HTML5. For now, I'm aware of only these types of interactions: 1) add a message to the queue LIFO-style (which is the default?) 2) add a message to the queue FIFO-style also known as add for the next turn 3) remove a message (clearTimeout which cancels a message added via a setTimeout message). At least it's not possible to remove an arbitrary message. There are probably reordering policies that need to be described as well. I'm not saying these primitives should necessarily be exposed to the runtime directly, but at least to other specs. Are there other types of interactions with the message queue I'm forgetting? David ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
On 18/04/2013, at 14:40, David Bruant wrote: I believe the ES spec should provide the primitives of what type of interaction with the message queue is allowed and what isn't. This set of primitives would obviously contain all the interactions allowed today by HTML5. For now, I'm aware of only these types of interactions: 1) add a message to the queue LIFO-style (which is the default?) This isn't neccessary, 2) add a message to the queue FIFO-style also known as add for the next turn This is the only one needed, 3) remove a message (clearTimeout which cancels a message added via a setTimeout message). and this has nothing to do with the events queue. -- ( Jorge )(); ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: Futures (was: Request for JSON-LD API review)
On 17/04/2013, at 17:46, Anne van Kesteren wrote: If it was up to me JavaScript would just be part of the W3C and we would not have to deal with that layer of distraction. On 17/04/2013, at 19:48, Tab Atkins Jr. wrote: I strongly support any efforts to move JS standardization into the umbrella of the W3C. The very thought of it sends chills down my spine. The w3c has demonstrated blindness and incompetence. Remember how and why the whatwg came to be? Stop pretending. You guys ought to be deeply embarrassed because HTML5 is *not* your child. Who wants a JS infested of inconvenient APIs w3c-style? -- ( Jorge )(); ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: Futures (was: Request for JSON-LD API review)
On Thu, Apr 18, 2013 at 2:00 PM, Jorge jo...@jorgechamorro.com wrote: You guys ought to be deeply embarrassed because HTML5 is *not* your child. I don't even -- http://annevankesteren.nl/ ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
Le 18/04/2013 14:59, Jorge a écrit : On 18/04/2013, at 14:40, David Bruant wrote: I believe the ES spec should provide the primitives of what type of interaction with the message queue is allowed and what isn't. This set of primitives would obviously contain all the interactions allowed today by HTML5. For now, I'm aware of only these types of interactions: 1) add a message to the queue LIFO-style (which is the default?) This isn't neccessary, 2) add a message to the queue FIFO-style also known as add for the next turn This is the only one needed, If you only have this one, then you can end up in situations where user inputs are never processed. The 1) is necessary. I assume some re-ordering policies/primitives might be necessary too. 3) remove a message (clearTimeout which cancels a message added via a setTimeout message). and this has nothing to do with the events queue. Can you develop your 3 answers, please? Throwing agreement/disagreement without providing reasons doesn't help the debate to move forward. Timeouts do interact with the event loop at least in the WHATWG spec (see step 10 of [1] as an example) David [1] http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#dom-windowtimers-settimeout ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
On Thu, Apr 18, 2013 at 6:11 AM, David Bruant bruan...@gmail.com wrote: Le 18/04/2013 14:59, Jorge a écrit : On 18/04/2013, at 14:40, David Bruant wrote: I believe the ES spec should provide the primitives of what type of interaction with the message queue is allowed and what isn't. This set of primitives would obviously contain all the interactions allowed today by HTML5. Agreed. Further, ES can't avoid specifying event loops, perhaps even in an ES6 timeframe!! It would indeed be unfortunate if we need to specify this in ES6. But I don't see how we can specify module loaders without doing so. The only alternative I see is that we postpone module loaders till ES7, which many of us (including myself) would find unpleasant. Although I do not want to see module loaders postponed till ES7, for completeness I will mention one further advantage of doing so. Currently, module loaders are specified in terms of callbacks. Were they postponed, we could instead specify their API in terms of promises. The similarity to the spec dependency dilemmas motivating this thread are indeed striking. For now, I'm aware of only these types of interactions: 1) add a message to the queue LIFO-style (which is the default?) This isn't neccessary, 2) add a message to the queue FIFO-style also known as add for the next turn This is the only one needed, If you only have this one, then you can end up in situations where user inputs are never processed. The 1) is necessary. I assume some re-ordering policies/primitives might be necessary too. LIFO isn't the way the browser specs deals with this issue. Rather, both the browser specs[*] and, lately, nodejs have adopted a two priority event loop, which can be accurately modeled (and implemented) as two fifo queues; call them inner (or high priority) and outer (or low priority). An outer turn thus consists of a sequence of inner turns. In the browser, rendering only happens as an outer turn. Object.observe and DOMFutures are both careful to queue events only in the inner queue, so all the observation events queued in an outer turn happen within inner turns at the end of that outer turn without the possibility of any intervening rendering. IO events at least usually queue in the outer queue. Is this always the case? Do any IO events queue in the inner queue? The Q API has historically queued only in the outer queue. The Q community has lately been debating this issue and tending towards having local Q events queue in the inner queue. Of course, distributed messaging, being layered on IO, would still queue only in the outer queue, but this doesn't seem to create any conflicts. [*] The HTML5 specs adopt different terminology which obscures the similarity of the two levels of event loop queueing. But it is equivalent to this uniform two-level model. 3) remove a message (clearTimeout which cancels a message added via a setTimeout message). and this has nothing to do with the events queue. Can you develop your 3 answers, please? Throwing agreement/disagreement without providing reasons doesn't help the debate to move forward. Timeouts do interact with the event loop at least in the WHATWG spec (see step 10 of [1] as an example) You don't need to account for cancellation in terms of event queue manipulation. And I agree with (what I believe to be) Jorge's position, that it is less clean to do so. Rather, you can just think of the queued event as having the behavior do this if it isn't cancelled and the cancellation as setting a cancellation state to be so tested by that queued event once it fires. Of course, if implemented this way, it would fail to reclaim as much storage. But that's not an observable difference, so the spec can simply use the simpler model. David [1] http://www.whatwg.org/specs/**web-apps/current-work/** multipage/timers.html#dom-**windowtimers-settimeouthttp://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#dom-windowtimers-settimeout -- Cheers, --MarkM ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part? (was: Re: Futures
Note that Futures are entirely expressible in today's JS semantics.
Even setting aside the event loop, this is arguable. Futures *as
implemented in libraries today* are expressible, sure. That's a tautology.
But there are cross-cutting issues at play, as Allen explained.
In the case of Futures, there's that strange little method named `done`.
The `done` method has always been conceived as a stop-gap solution for
making unhandled rejections visible to the programmer. The usability of
`done` is, well, not so great.
It's quite clear that the garbage collector provides us with an upper bound
on how long we must wait to know that a rejected future is truly unhandled.
When the future is collected, then obviously no additional error handlers
can be assigned to it.
WeakRefs would give us just the information we need, but no consensus has
been reached on them yet. Does it make sense to move forward with `done`
when WeakRefs are sitting on the horizon? I don't have the answer, but
these are the kind of cross-cutting issues that need to be carefully
considered. Preferably on es-discuss. : )
{ Kevin }
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
Agreed. Further, ES can't avoid specifying event loops, perhaps even in an
ES6 timeframe!! It would indeed be unfortunate if we need to specify this
in ES6. But I don't see how we can specify module loaders without doing so.
The only alternative I see is that we postpone module loaders till ES7,
which many of us (including myself) would find unpleasant.
Another upside would be easing the near-term workload on TC39 (and Allen
most of all).
Although I do not want to see module loaders postponed till ES7, for
completeness I will mention one further advantage of doing so. Currently,
module loaders are specified in terms of callbacks. Were they postponed, we
could instead specify their API in terms of promises. The similarity to the
spec dependency dilemmas motivating this thread are indeed striking.
Part of the problem is that hitherto the core standard library and platform
library (DOM) share the same namespace. As such, any activity in this
shared namespace has an oversized influence on other designers. Perhaps
modules can rectify this situation. What if, going forward, DOM APIs were
homed in their own module namespace?
import Future from dom:future;
let future = new Future({ resolve, reject } = resolve(Anne rocks!));
I would probably leave the standard core library in the global lexical
namespace:
let promise = new Promise({ resolve, reject } = resolve(Mark
rocks!));
There is still a clash, to a certain degree, but the effects are much more
localized.
{ Kevin }
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part? (was: Re: Futures
On Thu, Apr 18, 2013 at 7:45 AM, Kevin Smith zenpars...@gmail.com wrote: Note that Futures are entirely expressible in today's JS semantics. Even setting aside the event loop, this is arguable. Futures *as implemented in libraries today* are expressible, sure. That's a tautology. But there are cross-cutting issues at play, as Allen explained. In the case of Futures, there's that strange little method named `done`. The `done` method has always been conceived as a stop-gap solution for making unhandled rejections visible to the programmer. The usability of `done` is, well, not so great. It's quite clear that the garbage collector provides us with an upper bound on how long we must wait to know that a rejected future is truly unhandled. When the future is collected, then obviously no additional error handlers can be assigned to it. WeakRefs would give us just the information we need, but no consensus has been reached on them yet. Does it make sense to move forward with `done` when WeakRefs are sitting on the horizon? I don't have the answer, but these are the kind of cross-cutting issues that need to be carefully considered. Preferably on es-discuss. : ) I think we have informal consensus on the general functionality that WeakRefs will provide and the security constraints they must not violate. We still of course need to argue through many details. But the non-controversial parts of WeakRefs are clearly adequate for the scenario you have in mind -- except for one thing ;). GC is never required to be complete. We must allow the collector to not collect some unreachable objects. This means that, without .done, there's no guarantee that an unseen-rejection bug will ever get diagnosed. Therefore we still need .done. -- Cheers, --MarkM ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part? (was: Re: Futures
GC is never required to be complete. We must allow the collector to not
collect some unreachable objects. This means that, without .done, there's
no guarantee that an unseen-rejection bug will ever get diagnosed.
Therefore we still need .done.
A perfect example of why the discussion should take place on es-discuss. :
)
{ Kevin }
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss
RE: More flexibility in the ECMAScript part? (was: Re: Futures
From: Mark S. Miller [erig...@google.com] GC is never required to be complete. We must allow the collector to not collect some unreachable objects. This means that, without .done, there's no guarantee that an unseen-rejection bug will ever get diagnosed. Therefore we still need .done. I still think the best solution to this is for the developer tools to curate a list of unhandled rejections. Just like `window.onerror` and the developer console work together to display unhandled exceptions, unhandled rejections could be treated very similarly. They would appear in the console while unhandled, then disappear when/if handled. (And there could be programmatic hooks too, just like window.onerror, e.g. `window.onunhandledrejection`/`window.onrejectionhandled`.) In case others weren't aware, the Promises/A+ group has been compiling ideas for the unhandled rejection problem at https://github.com/promises-aplus/unhandled-rejections-spec/issues?state=open ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part? (was: Re: Futures
I still think the best solution to this is for the developer tools to
curate a list of unhandled rejections. Just like `window.onerror` and the
developer console work together to display unhandled exceptions, unhandled
rejections could be treated very similarly. They would appear in the
console while unhandled, then disappear when/if handled. (And there could
be programmatic hooks too, just like window.onerror, e.g.
`window.onunhandledrejection`/`window.onrejectionhandled`.)
That could certainly work for browsers, but what about node and its console?
Also, can someone point me to a real-world example of delayed rejection
handling?
{ Kevin }
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
Le 18/04/2013 17:35, Kevin Smith a écrit : Also, can someone point me to a real-world example of delayed rejection handling? What do you mean by delayed? If that is: a promise has been rejected and someone later add a .fail/.catch handler to it, then I've never done that personally. If you have another definition, I may have real-world examples, based on your definition. David ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
Also, can someone point me to a real-world example of delayed rejection
handling?
What do you mean by delayed? If that is: a promise has been rejected and
someone later add a .fail/.catch handler to it, then I've never done that
personally.
Yes, that's what I mean. Specifically, a real-world example where the
rejection handler is added in some future turn, using a fuzzy definition
of that phrase. A contrived example would be:
// Create a future and get the resolver
let resolver, future = new Future(r = resolver = r);
// Reject it
resolver.reject(error);
// Attach a rejection handler 5 seconds later
setTimeout(() = future.then(null, err = null), 5000);
{ Kevin }
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
On Thu, Apr 18, 2013 at 9:14 AM, Kevin Smith zenpars...@gmail.com wrote: Also, can someone point me to a real-world example of delayed rejection handling? What do you mean by delayed? If that is: a promise has been rejected and someone later add a .fail/.catch handler to it, then I've never done that personally. Yes, that's what I mean. Specifically, a real-world example where the rejection handler is added in some future turn, using a fuzzy definition of that phrase. A contrived example would be: // Create a future and get the resolver let resolver, future = new Future(r = resolver = r); // Reject it resolver.reject(error); // Attach a rejection handler 5 seconds later setTimeout(() = future.then(null, err = null), 5000); There are several DOM APIs planning to use Futures which will return the same future across multiple calls. This future may have been rejected in a previous turn. An example of this is the Font Load API built with Futures that I'm currently proposing: http://www.xanthir.com/b4PV1. The Font#ready function tracks the loading status of a single @font-face rule, so it just returns the same future over and over. These kinds of APIs *could* be done by tracking future state internally, and always returning a fresh future matching the internal state, but that seems silly. ~TJ ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part?
Hi Tab, thanks for a concrete example. From previous experience in E with similar promises, I agree that registering handlers later is quite natural, but I didn't have time to hunt for an example. It would indeed be silly to require such a counter-intuitive freshness workaround in order to avoid a counter-intuitive diagnostic. On Thu, Apr 18, 2013 at 9:43 AM, Tab Atkins Jr. jackalm...@gmail.comwrote: On Thu, Apr 18, 2013 at 9:14 AM, Kevin Smith zenpars...@gmail.com wrote: Also, can someone point me to a real-world example of delayed rejection handling? What do you mean by delayed? If that is: a promise has been rejected and someone later add a .fail/.catch handler to it, then I've never done that personally. Yes, that's what I mean. Specifically, a real-world example where the rejection handler is added in some future turn, using a fuzzy definition of that phrase. A contrived example would be: // Create a future and get the resolver let resolver, future = new Future(r = resolver = r); // Reject it resolver.reject(error); // Attach a rejection handler 5 seconds later setTimeout(() = future.then(null, err = null), 5000); There are several DOM APIs planning to use Futures which will return the same future across multiple calls. This future may have been rejected in a previous turn. An example of this is the Font Load API built with Futures that I'm currently proposing: http://www.xanthir.com/b4PV1. The Font#ready function tracks the loading status of a single @font-face rule, so it just returns the same future over and over. These kinds of APIs *could* be done by tracking future state internally, and always returning a fresh future matching the internal state, but that seems silly. ~TJ ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss -- Text by me above is hereby placed in the public domain Cheers, --MarkM ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: More flexibility in the ECMAScript part? (was: Re: Futures
I'm not sure this is a perfect match, but:
The futures library and task scheduler I've been using in my applications
for around ~5 years does unhandled error detection and delayed error
handling.
For the former, the model is that the future has an internal callback that
is fired when a consumer checks its error status. This is done via a
property accessor on the rough equivalent of Future.Error, along with any
other methods that implicitly check errors (for example, Future.Result will
throw instead of returning a value, if the future contains an error).
Paired with this is logic in the task scheduler such that any future that
goes through the task scheduler (to be waited on, etc) will be tracked to
ensure that the consumer task(s) have handled any errors that were returned
to them in their next step. I don't currently abort if you fail to check a
future for errors unless the future actually contained an error - that is,
this only is used to detect ignored errors, not to detect code quality
issues (though the latter is feasible with this method).
The downside is that this doesn't give you complete coverage for unhandled
errors - futures used outside the task scheduling mechanism don't get
tracked. Given ES promises' potential level of integration, though, you
could probably integrate this kind of diagnostic more fully if you can find
the right place to put it. I do agree that relying on GC to detect unseen
rejections is unacceptable; I likewise found that having the failure be
delayed for any amount of time was a problem. This is why I moved it into
the task scheduler, so that it can synchronously warn you as soon as you
fail to handle it. Let me know if you would like to see more detail on this
and I can link to my implementation and explain more.
By delayed rejection handling, do you mean the ability to delay response to
an error indefinitely? I am not sure what else this would mean. If so, I've
built a few apps atop futures and a task scheduler where the error handling
policy is that any errors that occur are stored immediately, then
propagated up the chain of task dependencies (which means they aren't
handled until the whole chain of tasks has woken up to see them) and then
if they walk up the entire chain of dependencies without being handled, the
task scheduler fires a last chance 'unhandled background error' callback
(which could be occurring seconds after the error). I usually put an error
dialog in that callback, or in cases where the error is recoverable I log
it to a file. If this is a close fit to what you were thinking I can link
you to the source to a couple of the applications.
On Thu, Apr 18, 2013 at 8:35 AM, Kevin Smith zenpars...@gmail.com wrote:
I still think the best solution to this is for the developer tools to
curate a list of unhandled rejections. Just like `window.onerror` and the
developer console work together to display unhandled exceptions, unhandled
rejections could be treated very similarly. They would appear in the
console while unhandled, then disappear when/if handled. (And there could
be programmatic hooks too, just like window.onerror, e.g.
`window.onunhandledrejection`/`window.onrejectionhandled`.)
That could certainly work for browsers, but what about node and its
console?
Also, can someone point me to a real-world example of delayed rejection
handling?
{ Kevin }
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss
--
-kg
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss
Re: DOM EventStreams (take two on Streams): Request for feedback
(narrowing to the part that seems most productive) On Wed, Apr 17, 2013 at 9:11 PM, Tab Atkins Jr. wrote: On Wed, Apr 17, 2013 at 5:50 PM, Jason Orendorff wrote: Bacon offers two equivalent ways of unsubscribing. 1. Bacon's equivalent of the StreamInit callback returns an unsubscribe function. Each subscriber therefore gets its very own unsubscribe callback. Ah, that's an interesting idea. I'm unsure how each subscriber gets its own unsubscribe callback. The init callback is only called once, and so returns only the single value, right? Or is the subscribe callback called every time someone starts listening, so the stream can potentially act different to different listeners? That seems like it would be hard to make compatible with a multi-listener approach. Right. It's not quite per-listener; as in your design, the EventStream class copes with multiple simultaneous listeners. But when the number of listeners on an EventStream goes from 0 to 1, it calls the subscribe hook; when it goes from 1 to 0, it calls the unsubscribe hook. This is because Bacon turns off all the taps when no one's listening. Futures are not like that. Example: var clock = Bacon.interval(100, tick); clock.take(5).log(); setTimeout(() = clock.take(5).log(), 2000); The call to .log() on line 2 causes .take(5) to have a consumer, so clock's subscribe hook is called. We log some events; 500 msec later, .take(5) receives its fifth event, so it's all done. It unsubscribes itself and floats away. clock goes to 0 consumers, so it calls the unsubscribe hook. At 2000 msec, clock once again has a downstream consumer, so its subscribe hook is called a second time. 2. Additionally, Bacon's equivalent of the EventStreamResolver.push() method can return a special value (Bacon.noMore) that means unsubscribe me. That just kicks out all the listeners to the stream? Or does it end the stream? Or do you mean something else, given that you use the pronoun me, which implies it's the *listener* with somehow sends the signal? If the latter, you're confused about the role of a stream resolver. Bacon's equivalent of EventStreamResolver.push() returns Bacon.noMore when it finds that the number of listeners has gone to 0. This is indeed usually caused by the last listener unsubscribing by returning Bacon.noMore; and this is the point where I think it'll be quickest to cut short the discussion and just read some Bacon source code. This is called for every event: https://github.com/raimohanska/bacon.js/blob/6318160839d76ed4ce4eceeefe5d0d78b8e45403/src/Bacon.coffee#L777 -j ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: DOM EventStreams (take two on Streams): Request for feedback
Bacon's equivalent of EventStreamResolver.push() returns Bacon.noMore when it finds that the number of listeners has gone to 0. It should also be noted that streams2 in node return `false` from `push()` when the stream is full. Generally returning a value from `push()` as some kind of message to the source to either pause or abort is a good strategy. On Thu, Apr 18, 2013 at 4:26 PM, Jason Orendorff jason.orendo...@gmail.comwrote: (narrowing to the part that seems most productive) On Wed, Apr 17, 2013 at 9:11 PM, Tab Atkins Jr. wrote: On Wed, Apr 17, 2013 at 5:50 PM, Jason Orendorff wrote: Bacon offers two equivalent ways of unsubscribing. 1. Bacon's equivalent of the StreamInit callback returns an unsubscribe function. Each subscriber therefore gets its very own unsubscribe callback. Ah, that's an interesting idea. I'm unsure how each subscriber gets its own unsubscribe callback. The init callback is only called once, and so returns only the single value, right? Or is the subscribe callback called every time someone starts listening, so the stream can potentially act different to different listeners? That seems like it would be hard to make compatible with a multi-listener approach. Right. It's not quite per-listener; as in your design, the EventStream class copes with multiple simultaneous listeners. But when the number of listeners on an EventStream goes from 0 to 1, it calls the subscribe hook; when it goes from 1 to 0, it calls the unsubscribe hook. This is because Bacon turns off all the taps when no one's listening. Futures are not like that. Example: var clock = Bacon.interval(100, tick); clock.take(5).log(); setTimeout(() = clock.take(5).log(), 2000); The call to .log() on line 2 causes .take(5) to have a consumer, so clock's subscribe hook is called. We log some events; 500 msec later, .take(5) receives its fifth event, so it's all done. It unsubscribes itself and floats away. clock goes to 0 consumers, so it calls the unsubscribe hook. At 2000 msec, clock once again has a downstream consumer, so its subscribe hook is called a second time. 2. Additionally, Bacon's equivalent of the EventStreamResolver.push() method can return a special value (Bacon.noMore) that means unsubscribe me. That just kicks out all the listeners to the stream? Or does it end the stream? Or do you mean something else, given that you use the pronoun me, which implies it's the *listener* with somehow sends the signal? If the latter, you're confused about the role of a stream resolver. Bacon's equivalent of EventStreamResolver.push() returns Bacon.noMore when it finds that the number of listeners has gone to 0. This is indeed usually caused by the last listener unsubscribing by returning Bacon.noMore; and this is the point where I think it'll be quickest to cut short the discussion and just read some Bacon source code. This is called for every event: https://github.com/raimohanska/bacon.js/blob/6318160839d76ed4ce4eceeefe5d0d78b8e45403/src/Bacon.coffee#L777 -j ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Internationalization: Comments on Text Segmentation straw man
In preparation for tomorrow's internationalization ad-hoc meeting, I reviewed the Text Segmentation strawman: http://wiki.ecmascript.org/doku.php?id=globalization:text_segmentation Some issues go beyond internationalization and into general API design question - I'd appreciate input from the es-discuss crowd on them. 1) Text segmentation often works on small portions of potentially large documents. For example, when the user clicks or taps on a word, an application may need to find the word being selected. Only the text under the click/tap location is of interest, but that text may be part of a large DOM tree (not necessarily all within one node!). Using String values as input means that the application has to create a string including enough context so that the complete desired segment and some text outside the segment is guaranteed to be included - typically a paragraph. Some libraries, including ICU, accept alternative input types: iterators that can move forward and (unlike ES6 iterators) backward over strings, or generic interfaces that provide access to text. Should our text segmenters allow such alternative input types, or is it reasonable to expect callers to construct String values? 2) The strawman includes an extension to String.prototype.split, allowing it to accept a TextSegmenter as the separator argument. The only way to detect whether this argument will be understood would be indirect, by checking whether Intl.TextSegmenter exists. Is that acceptable? Other comments are mostly for the ad-hoc team: 3) It would be useful to have more detail on the use cases, including how the text to be segmented might be represented, and whether segments would typically be accessed sequentially or randomly. Some indication of how common each use case is expected to be in JavaScript applications would also be useful. 4) I think we should drop paragraph breaking. Paragraphs are usually defined by the document type - in plain text possibly any CR/LF combination, or two consecutive such combinations, or U+2029; in HTML the text within a p element and various other entities; etc. ES text segmenters shouldn't have to know document types. 5) Do we expect tailored grapheme clusters to be supported and commonly used? It seems to me that default grapheme clusters should be handled in regular expressions, not in this special-purpose API. 6) Line breaks on the other hand should be provided. 7) Is numSegments() really needed? If so, it should be countSegments() or such to indicate that it's a rather expensive operation. 8) We need to define more precisely what is meant by the different segment types. E.g., The notes on segmentType indicate that whitespace and punctuation are returned as separate words - is that generally accepted? Should sequences of punctuation or whitespace be treated as one word or multiple? Will line breaking report U+00AD as a breaking opportunity? Is it safe to assume that in the absence of U+00AD the segments reported are appropriate as input to a separate hyphenation engine, or do such engines need more context? Normative references to UTRs 14 and 29 might help. 9) Why isn't segmentType just a property of the object returned by segmentContaining? 10) Should positions be based on UTF-16 code units or Unicode code points? Code points seem more logical, but make it more difficult to map to underlying strings. 11) Should the end property of the object returned by segmentContaining be the index of the last character/code unit included in the segment, or the index after that character/code unit? The latter would be more compatible with String.prototype.substring. 12) If the second edition of ECMA-402 is based on ES6, this API should provide iterators: http://wiki.ecmascript.org/doku.php?id=harmony:iterators 13) Anything else defaults to “word”. should be Anything else results in a RangeError exception. Norbert ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
RE: DOM EventStreams (take two on Streams): Request for feedback
One bigger question: what is the DOM use case for event streams?
That is, it's very clear what the DOM use cases are for binary data streams.
(Most urgently, streaming XHR, but also perhaps unifying the many interfaces
that use object URLs as a means of connecting separate streams of data; also
exposing the browser's GZIP capabilities; and so on [1].)
But for event streams it's less clear what urgent problem they solve. The
example you've shown so far is basically just a different way of doing
Object.observe, with some nice sugar and of course those combinators. But the
basic capabilities of the platform are not expanded, and sugar seems like a
library-level concern. Nevertheless, there's many allusions to DOM use cases in
your blog posts, so a listing of those would be helpful.
In other words: if there are many use cases for the DOM where event streams
make sense, great! In the spirit of standardizing promises, it's good to
standardize a common idiom so we don't do things in many different ways across
the DOM APIs. But if the only use case is just to notify of property changes,
Object.observe handles that nicely without streams. What else needs event
streams?
[1]: http://imgur.com/a/9vFGa#11
-Original Message-
From: es-discuss-boun...@mozilla.org [mailto:es-discuss-
boun...@mozilla.org] On Behalf Of Tab Atkins Jr.
Sent: Tuesday, April 16, 2013 19:29
To: es-discuss
Subject: DOM EventStreams (take two on Streams): Request for feedback
My first attempt at feedback for my Stream proposal was unfortunately
bogged down with a lot of confusion over terminology and meaning. I'd like
to start fresh and hopefully head off a lot of confusion early-on, so here's
take two.
Now that DOM has added Futures, I've started looking into converting
various event-based APIs into being future-based. This has been a great
success, but there are some things that can't be turned into futures (because
they update multiple times, or don't have a notion of completing), but
have the same lack-of-need for the full DOM Event baggage. I think these
cases will be fairly common, and further, that a good solution to the problem
for DOM will be pretty useful for general programming as well.
This is explicitly *not* an attempt to solve the binary/IO stream
use-case,
as exemplified by Node Streams http://nodejs.org/api/stream.html.
While structurally similar, binary streams have a lot of unique features and
pitfalls that make event streams a poor fit for them: they need to batch up
data by default, they need to be able to apply backfill data, etc. I think we
*also* need to develop such an API, but it'll be separate from this one. (I
suspect it may look very similar, though, so it's good to keep that in mind
when naming.)
So, without further ado, here's the basic API for my proposal for
EventStreams:
callback StreamInit = void (StreamResolver resolver); callback AnyCallback =
any (optional any value); typedef (EventStream or Future or Iterable)
StreamLike; typedef (string or number or boolean or AnyCallback)
updateFilter;
[Constructor(StreamInit init)]
interface EventStream {
EventStream listen(optional AnyCallback? listenCB = null, optional
AnyCallback? completeCB = null, optional AnyCallback? rejectCB = null);
Future complete(optional AnyCallback cb);
Future catch(optional AnyCallback cb);
Future next(optional updateFilter, optional anyCallback cb); }
interface EventStreamResolver {
void push(optional any value);
void complete(optional any value);
void continueWith(optional any value);
void reject(optional any value);
};
This API is intentionally very similar to that of Futures, because it's
intended
to solve similar problems, and I think the shape of the Futures API is pretty
good.
An EventStream represents a stream of events or values. It's roughly
equivalent to the concept of signals or event streams from functional
reactive programming, or the concept of an observable or task from
several functional async programming models.
An EventStream pushes out 0 or more updates, then optionally completes or
rejects. The .listen() function is the basic way to respond to an event
stream,
allowing you to register callbacks for any of those three events. It returns
the same event stream back, for chaining.
For convenience, event streams have several functions that let you listen to
just a single event, returning a Future. You can listen for the stream
completing, rejecting, or for the next update (possibly filtered).
**Important
note**: consuming an event stream using repeated .next() calls rather than
a single .listen() call is lossy - multiple updates can happen between the
tick
that .next() is called and the tick that the future resolves, and the future
will
only contain the value of the first one.
Like Futures, EventStreams separate the power to read/respond to an event
stream and the
Re: DOM EventStreams (take two on Streams): Request for feedback
I've updated my blog post with a refactoring of the API: http://www.xanthir.com/b4PV0. I hadn't fully appreciated the underlying abstraction of futures (that they are the representation of async control flow), and so I was designing the stream API in an inconsistent way. This has been fixed, so now Event Streams are far more consistent. The core function is .then(), and its callbacks can all be called multiple times. (Streams can adopt other streams, and when those streams reject, listener's reject callbacks are called.) If I have engineered things correctly, this makes Streams consistently represent an async loop. I've also reorganized the blog post for easier reading, separating out the basic operations from the constructor functions and the combinators. I've also made a saner API for ValueStream (previously called UpdateStream), and temporarily killed the text about a lossless single-listener stream until I'm sure I've got the semantics right for the lossy multi-listener cases. The big thing I'm concerned with is the semantics of adoption. I'm very unsure I've gotten it right, but I've stared at it too long for now, and can't think about it anymore. ~TJ ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: DOM EventStreams (take two on Streams): Request for feedback
On Thu, Apr 18, 2013 at 4:26 PM, Jason Orendorff jason.orendo...@gmail.com wrote: (narrowing to the part that seems most productive) On Wed, Apr 17, 2013 at 9:11 PM, Tab Atkins Jr. wrote: On Wed, Apr 17, 2013 at 5:50 PM, Jason Orendorff wrote: Bacon offers two equivalent ways of unsubscribing. 1. Bacon's equivalent of the StreamInit callback returns an unsubscribe function. Each subscriber therefore gets its very own unsubscribe callback. Ah, that's an interesting idea. I'm unsure how each subscriber gets its own unsubscribe callback. The init callback is only called once, and so returns only the single value, right? Or is the subscribe callback called every time someone starts listening, so the stream can potentially act different to different listeners? That seems like it would be hard to make compatible with a multi-listener approach. Right. It's not quite per-listener; as in your design, the EventStream class copes with multiple simultaneous listeners. But when the number of listeners on an EventStream goes from 0 to 1, it calls the subscribe hook; when it goes from 1 to 0, it calls the unsubscribe hook. This is because Bacon turns off all the taps when no one's listening. Futures are not like that. Hm, I'm not sure I'm grasping exactly what this does or how it does it. I'm not sure what Bacon's Dispatcher concept maps to, or how its API actually works. This is indeed usually caused by the last listener unsubscribing by returning Bacon.noMore; and this is the point where I think it'll be quickest to cut short the discussion and just read some Bacon source code. This is called for every event: https://github.com/raimohanska/bacon.js/blob/6318160839d76ed4ce4eceeefe5d0d78b8e45403/src/Bacon.coffee#L777 I don't understand how the subscribers reply to the dispatcher with a value. I think there's a decent mismatch in API shapes, which is making it hard for me to follow. ~TJ ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
RE: DOM EventStreams (take two on Streams): Request for feedback
Makes sense, and thanks for clarifying!
I guess my only hesitation is that promises evolved over many years, with the
design we see in Promises/A+ today and its many implementations, including DOM
Futures, being the result of convergent evolution in library-space. While your
sugar is nice, I'd be hesitant to bless it as the one true async-loops sugar
without at least some library-space evolution.
In other words, why not try releasing your event streams as a library, and see
what kind of adoption they get? If libraries as diverse as jQuery, WinJS,
Ember, Angular, Dojo, and YUI end up all having some form of event stream in
roughly that format, you know you're on to a winner :).
It's a different scenario from promises, since they're part of the API contract
that needs to be exposed when designing web APIs. It sounds like the
fundamental API contract for your case is just a changing property, and the
method of consuming that changing property could be done with sugar from any
library. (Indeed, this is somewhat like how the fundamental API contract for
async DOM APIs is now to return a relatively feature-less DOMFuture, which
consumers can use alongside any full-featured promise library in order to get
the sugar they desire.)
From another perspective, it would be somewhat of a shame to quash the nascent
FRP-in-JS industry in its tracks by handing down an API from on high in the
WHATWG. Already we're seeing very diverse implementations, from RxJS to
Bacon.js to the various Node.js experiments (which range from core
object-mode streams to the user-land experiments that seem to have sprung up in
the last few months, like Dominic Tarr's pull-stream). It reminds me of the
.NET space, where Microsoft's entry into an area—whether it be ORMs with Entity
Framework, or package managers with NuGet, or build tools with
MSBuild—immediately grabbed the majority of developer mindshare, despite
arguably-better options being in development and use.
Finally, on the issue of async loops, I'd argue that it's a tentative analogy.
It's at least not nearly as clear how you could do a syntactic transformation
like you can with promises, e.g. by introducing coroutines (cf. `await`). In
fact, async loops to me seem like they'd be expressed as
```js
while (await asyncCondition()) {
await asyncAction();
}
```
or, of course, its ES6 counterpart with `yield` and task.js-style wrappers.
-Original Message-
From: Tab Atkins Jr. [mailto:jackalm...@gmail.com]
Sent: Thursday, April 18, 2013 21:34
To: Domenic Denicola
Cc: es-discuss
Subject: Re: DOM EventStreams (take two on Streams): Request for
feedback
On Thu, Apr 18, 2013 at 5:11 PM, Domenic Denicola
dome...@domenicdenicola.com wrote:
One bigger question: what is the DOM use case for event streams?
That is, it's very clear what the DOM use cases are for binary data
streams. (Most urgently, streaming XHR, but also perhaps unifying the
many interfaces that use object URLs as a means of connecting separate
streams of data; also exposing the browser's GZIP capabilities; and so
on [1].)
But for event streams it's less clear what urgent problem they solve. The
example you've shown so far is basically just a different way of doing
Object.observe, with some nice sugar and of course those combinators. But
the basic capabilities of the platform are not expanded, and sugar seems like
a library-level concern. Nevertheless, there's many allusions to DOM use
cases in your blog posts, so a listing of those would be helpful.
In other words: if there are many use cases for the DOM where event
streams make sense, great! In the spirit of standardizing promises, it's good
to standardize a common idiom so we don't do things in many different ways
across the DOM APIs. But if the only use case is just to notify of property
changes, Object.observe handles that nicely without streams. What else
needs event streams?
Almost every use-case for streams *could* be done by just exposing the
data as a property on some object and using Object.observe. That loses all
the really wonderful control-flow properties of streams, though. One simple
but very nice example is explained in my blog while illustrating the switch()
method. I'll reproduce it here.
Say you want to provide autocomplete suggestions as the user types into
some field, based on data on your server. Using today's technologies, this is
absolutely possible:
1. Register an input listener on the input.
2. Possibly throttle the input events to keep them from coming in too quickly.
3. As the input events come in, construct an XHR to retrieve suggestions from
your server.
4. As each XHR finishes, verify that the results aren't already obsolete by a
later XHR returning faster.
5. If they're not obsolete, update your UI with the returned suggestions.
Actually writing the code to do all this is surprisingly non-trivial.
With EventStream and Future, though,
RE: parallel arrays and sorting
Thanks for the overview of thrust. I had completely missed the issue of stability before. This needs careful consideration and I have not yet made up my mind. I understand the desire to have unstable sort for performance reasons, but it also introduces a form of non-determinism. Histogram can also be expressed using scatter in out API. Assuming some array A of color values, you can do A.map((v) = 1).scatter(A.map((v) = bucket(v)), 0, (a,b) = a+b, numberOfBuckets) So, essentially create a vector of 1's and then scatter those to your buckets based on their bucket value, using addition as the conflict resolution. Sorting the scatter indices indeed makes a lot of sense and an implementation would be free to first sort the vector of scatter indices to reduce synchronization cost. I also thought about an explicit version directly using sort and some other primitives of our API but I could not come up with a formulation that would benefit from sorting. The array of structures (AoS) vs. structures of arrays (SoA) transformation is problematic in JavaScript. In C like languages one has a lot of structural information, in particular the fields and types of a structure are statically known and homogeneous across an array. In JavaScript, on the other hand, objects can be used like structures but in general there may be a lot more dynamism: Arrays may contain objects with varying labels, properties may have different types and, probably worst of all, the programmer can add new properties to objects at any time. So even though a structure may start out as a homogeneous array, this can change during its lifetime. Analyzing these data structures and tracking the 'anomalies' seems not straight forward to me and I am not convinced that this can be implemented efficiently. However, there is also good news. The binary data proposal (see http://wiki.ecmascript.org/doku.php?id=harmony:binary_data) brings C like data types, including arrays of structures, to JavaScript and we have proposed an extension to Parallel JavaScript that makes use of binary data as storage format (see http://wiki.ecmascript.org/doku.php?id=strawman:data_parallelism#support_for_binary_data). In such a setting, doing automatic AoS to SoA transformations is possible and our programming model does not preclude this. Stephan From: Norm Rubin [mailto:nru...@nvidia.com] Sent: Thursday, April 11, 2013 11:36 AM To: Herhut, Stephan A; es-discuss@mozilla.orgmailto:es-discuss@mozilla.org Cc: dev-tech-js-engine-rivertr...@lists.mozilla.orgmailto:dev-tech-js-engine-rivertr...@lists.mozilla.org Subject: RE: parallel arrays and sorting Good set of questions about sort, thanks for looking at this carefully. You guys have been great about responding to my emails. One sorting application that I would expect is physics particle simulation, where the code might sort objects based on distance from the viewer. This could be used so that far away objects get rendered smaller. Another might be computing a histogram over a visual image - sort the data first so nearby values will go into the same bucket, without needing atomic synchronization. Sorting with an optional compare seems ok to me- even if implementations can only optimize for some comparisons One interesting question you brought up as part of sorting by key is pretty tricky. In general parallel data arrays would like to be stored in transposed order from classic cpu styles. We usually call this array of structures or structures of arrays Do we sort an array of objects each with multiple fields stored one object after another or do we sort multiple arrays, with all the values of field1 followed by all the values of field2 etc I'd hope we allow implementations to reorder the data within a parallelArray as they like without needing to expose the layout to developers. Thrust, on the other hand, pushes the layout to the developer and thus has a significant set of transpose routines. Just for a point of reference there are 8 versions of sort in the thrust library. As you can see, Thrust never aimed to be a minimal set it just gained operations as applications appeared. AMD has a similar library called bolt which uses the same interface. Two different parallel libraries with the same routines does suggest that this set is enough to do useful work. Internally thrust notices that the compare ison primitive types and uses radix sort (on the gpu). Thurst includes: Stable and unstable forms Ascending only or with a comparison function Single array or a pair of keys and values 1) Sort(array) - sorts the array into ascending order, not guaranteed to be stable 2) Sort (array)- with a comparison function 3) Sort by key (keys, values) returns both the reordered keys and the reordered values, sorted by the array keys. This one is kind of confusing so here is an example // an example of key sorting #include
RE: parallel arrays and sorting
Thanks, this indeed is an interesting use case. Previously, I thought that using a function to map objects to a sort key is at least as expressive as having two arrays, as a naïve implementation of the mapping function could just be to read the appropriate value from the key array (assuming the mapping function would know the index). However, such a solution does not generate a sorted key array and thus the correlation between the sorted objects and their keys is lost. After reading the blog post you linked, I am still not sure whether that actually is an issue. It seems for a rendering pipeline only the sorted objects are of importance. So, potential runtime efficiency issues aside, do you know a use case where one also needs the array of sorted keys? Stephan From: Kevin Gadd [mailto:kevin.g...@gmail.com] Sent: Thursday, April 11, 2013 11:43 AM To: Norm Rubin Cc: Herhut, Stephan A; es-discuss@mozilla.orgmailto:es-discuss@mozilla.org; dev-tech-js-engine-rivertr...@lists.mozilla.orgmailto:dev-tech-js-engine-rivertr...@lists.mozilla.org Subject: Re: parallel arrays and sorting One simple-ish real world example of a use case that can benefit from a parallel sort: It's common in games and other realtime rendering scenarios to want to sort your queue of rendering operations by various attributes, in order to minimize the number of hardware state changes and allow you to batch up operations. http://realtimecollisiondetection.net/blog/?p=86 is an article that describes an approach used to generate 64-bit 'sort keys' for drawing operations that was used in some current-gen console games. At present, I use a similar technique to sort drawing operations in my buffers, and more importantly, I do all the sorting and preparation operations in parallel. For complex scenes I could imagine it being quite common to want to leverage parallel machinery (or even GPU hardware) to sort the thousands of individual drawing operations in a given scene before clustering them up into larger hardware operations. The proposed compromise of providing a function that maps the objects being sorted to 'sort keys' seems a suitable choice for this kind of scenario - the draw calls can have a sort key implicitly (like described in that blog post), or you can trivially manufacture one (that's what I do). This is definitely a use case where only being able to sort primitives would not be adequate. However, a sort primitive that operates on a pair of arrays would also suffice - Norm calls this 'sort by key', and the .NET BCL exposes this as an overload of Array.Sort. It's quite useful and it seems like it might be feasible to expose this, even if only the key-sorting occurs in parallel (and the value sorting can't because the values are ordinary JS objects). On Thu, Apr 11, 2013 at 11:35 AM, Norm Rubin nru...@nvidia.commailto:nru...@nvidia.com wrote: Good set of questions about sort, thanks for looking at this carefully. You guys have been great about responding to my emails. One sorting application that I would expect is physics particle simulation, where the code might sort objects based on distance from the viewer. This could be used so that far away objects get rendered smaller. Another might be computing a histogram over a visual image – sort the data first so nearby values will go into the same bucket, without needing atomic synchronization. Sorting with an optional compare seems ok to me- even if implementations can only optimize for some comparisons One interesting question you brought up as part of sorting by key is pretty tricky. In general parallel data arrays would like to be stored in transposed order from classic cpu styles. We usually call this array of structures or structures of arrays Do we sort an array of objects each with multiple fields stored one object after another or do we sort multiple arrays, with all the values of field1 followed by all the values of field2 etc I’d hope we allow implementations to reorder the data within a parallelArray as they like without needing to expose the layout to developers. Thrust, on the other hand, pushes the layout to the developer and thus has a significant set of transpose routines. Just for a point of reference there are 8 versions of sort in the thrust library. As you can see, Thrust never aimed to be a minimal set it just gained operations as applications appeared. AMD has a similar library called bolt which uses the same interface. Two different parallel libraries with the same routines does suggest that this set is enough to do useful work. Internally thrust notices that the compare ison primitive types and uses radix sort (on the gpu). Thurst includes: Stable and unstable forms Ascending only or with a comparison function Single array or a pair of keys and values 1) Sort(array) - sorts the array into ascending order, not guaranteed to be stable 2) Sort (array)–
