Can someone tell me exactly how just omitting "await" doesn't broadly achieve the "concurrency" objective?
On Fri, 6 Sep 2019, 20:04 Cyril Auburtin, <cyril.aubur...@gmail.com> wrote: > It could be probably added as a `Promise.all(iterable, concurrency?)` > > Some existing implementations: > - http://bluebirdjs.com/docs/api/promise.map.html > - https://caolan.github.io/async/v3/docs.html#mapLimit > - I tried to write one: > https://github.com/caub/misc/blob/master/utils/promise-concurrent.js > > On Fri, Sep 6, 2019 at 8:33 PM Tom Boutell <t...@apostrophecms.com> wrote: > >> I am more interested in syntax two than syntax one, which I felt should >> probably be included for completeness. But hey, as you say, maybe not since >> unguarded concurrency is indeed usually a mistake. >> >> Taken on its own, do you have an objection to `for (item of items >> concurrency 5) { ... }`? >> >> >> On Fri, Sep 6, 2019 at 1:46 PM C. Scott Ananian <ecmascr...@cscott.net> >> wrote: >> >>> The current way to write what you want is: >>> >>> await Promise.all(itemsCursor.map(item => db.insert(item)); >>> >>> or >>> >>> await Promise.all(itemsCursor.map(Promise.guard(5, item => >>> db.insert(item)))); >>> >>> if you are using a library like `prfun` ( >>> https://github.com/cscott/prfun#promiseguardfunctionnumber-condition-function-fn--function >>> ). >>> >>> I'm not sure adding a new parallel loop construct is an obvious >>> improvement on this, especially since unguarded concurrent execution is >>> usually a mistake (can cause memory requirements to blow up), as you point >>> out yourself. >>> >>> I'd be more in favor of new syntax if it was integrated with a >>> work-stealing mechanism, since (a) that can't as easily be done in a >>> library function (you need access to the entire set of runnable tasks, not >>> just the ones created in this loop), and (b) is more likely to be >>> correct/fast by default and not lead to subtle resource-exhaustion problems. >>> --scott >>> >>> On Fri, Sep 6, 2019 at 12:40 PM Tom Boutell <t...@apostrophecms.com> >>> wrote: >>> >>>> *Specifying concurrency for "for...of" loops potentially containing >>>> "await" statements in the loop body* >>>> >>>> In the async/await era, I see most developers using the async and await >>>> keywords in 90% of situations, shifting to "Promise.all" or the bluebird >>>> library only to cope with concurrency issues. >>>> >>>> The most common case in my experience is the need to admit a manageable >>>> level of parallelism when iterating a large array (or iterator, see the >>>> final section) and performing asynchronous work on each item. Unlimited >>>> concurrency (Promise.all) tends to overwhelm backends involved in a way >>>> that confuses developers as to what is happening. Bluebird's "Promise.map" >>>> permits concurrency to be specified, which is great, but requires pulling >>>> in a library and switching of mental gears ("OK right, these async >>>> functions return promises," etc). >>>> >>>> To build on the friendliness of async/await, I propose these two >>>> syntaxes be accepted: >>>> >>>> SYNTAX ONE >>>> >>>> ```js >>>> for (item of items concurrent) { >>>> // db.insert is an async function >>>> await db.insert(item); >>>> } >>>> ``` >>>> >>>> In Syntax One, all loop bodies commence concurrently (see below for the >>>> definition of "concurrently" with regard to async). If an exception is not >>>> caught inside the loop, it is thrown beyond the loop, and all exceptions >>>> subsequently thrown by concurrently executing loop bodies are discarded >>>> (like Promise.all). >>>> >>>> *While I feel that unlimited concurrency is usually a mistake in a >>>> situation where you have an array of items of unpredictable number, it >>>> seems odd not to have a syntax for this case, and "concurrency 0" seems >>>> clunky.* >>>> >>>> SYNTAX TWO >>>> >>>> ```js >>>> for (item of items concurrency 5) { >>>> // db.insert is an async function >>>> await db.insert(item); >>>> } >>>> ``` >>>> >>>> in Syntax Two, up to 5 loop bodies commence concurrently (see below). >>>> There is no guarantee that item 3 will finish before item 2, or that item 4 >>>> won't start (due to 3 being finished) before item 2 ends, etc. If an >>>> exception is not caught inside the loop, it is thrown beyond the loop, and >>>> all exceptions subsequently thrown by concurrently executing loop bodies >>>> are discarded (like Promise.all in this respect, except for the restriction >>>> of concurrency). >>>> >>>> DEFINING CONCURRENCY FOR ASYNC >>>> >>>> For purposes of this proposal, "concurrent" execution means that >>>> multiple loop bodies may be suspended via "await" at any given time. It >>>> does NOT refer to multithreaded execution, worker threads, etc. >>>> >>>> CONSIDERATIONS FOR ASYNC ITERATORS >>>> >>>> Async iterator syntax for "for...of" loops, as in: >>>> >>>> ```js >>>> for await (item of itemsCursor) { ... } >>>> ``` >>>> >>>> Should also support concurrency for the loop body, with the same syntax: >>>> >>>> ```js >>>> for await (item of itemsCursor concurrency 5) { ... } >>>> ``` >>>> >>>> *It is important to note that this syntax does not add concurrency to >>>> the async iterator itself, *at least not at this time, as I believe >>>> the interface for defining async iterators does not currently accommodate >>>> this. However this syntax is still useful because it *fetches the >>>> items sequentially from the iterator, but may "fill the hopper" with up to >>>> five iterator results* that are currently being actively processed by >>>> loop bodies. In many cases, fetching items via an iterator is much faster >>>> than the processing that will be done to them in the loop bodies, and so >>>> this is still useful. >>>> >>>> Thanks for reading! >>>> >>>> -- >>>> Chief Software Architect >>>> Apostrophe Technologies >>>> Pronouns: he / him / his >>>> _______________________________________________ >>>> es-discuss mailing list >>>> es-discuss@mozilla.org >>>> https://mail.mozilla.org/listinfo/es-discuss >>>> >>> >> >> -- >> Chief Software Architect >> Apostrophe Technologies >> Pronouns: he / him / his >> _______________________________________________ >> es-discuss mailing list >> es-discuss@mozilla.org >> https://mail.mozilla.org/listinfo/es-discuss >> > _______________________________________________ > es-discuss mailing list > es-discuss@mozilla.org > https://mail.mozilla.org/listinfo/es-discuss >
_______________________________________________ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
