It could be probably added as a `Promise.all(iterable, concurrency?)` Some existing implementations: - http://bluebirdjs.com/docs/api/promise.map.html - https://caolan.github.io/async/v3/docs.html#mapLimit - I tried to write one: https://github.com/caub/misc/blob/master/utils/promise-concurrent.js
On Fri, Sep 6, 2019 at 8:33 PM Tom Boutell <[email protected]> wrote: > I am more interested in syntax two than syntax one, which I felt should > probably be included for completeness. But hey, as you say, maybe not since > unguarded concurrency is indeed usually a mistake. > > Taken on its own, do you have an objection to `for (item of items > concurrency 5) { ... }`? > > > On Fri, Sep 6, 2019 at 1:46 PM C. Scott Ananian <[email protected]> > wrote: > >> The current way to write what you want is: >> >> await Promise.all(itemsCursor.map(item => db.insert(item)); >> >> or >> >> await Promise.all(itemsCursor.map(Promise.guard(5, item => >> db.insert(item)))); >> >> if you are using a library like `prfun` ( >> https://github.com/cscott/prfun#promiseguardfunctionnumber-condition-function-fn--function >> ). >> >> I'm not sure adding a new parallel loop construct is an obvious >> improvement on this, especially since unguarded concurrent execution is >> usually a mistake (can cause memory requirements to blow up), as you point >> out yourself. >> >> I'd be more in favor of new syntax if it was integrated with a >> work-stealing mechanism, since (a) that can't as easily be done in a >> library function (you need access to the entire set of runnable tasks, not >> just the ones created in this loop), and (b) is more likely to be >> correct/fast by default and not lead to subtle resource-exhaustion problems. >> --scott >> >> On Fri, Sep 6, 2019 at 12:40 PM Tom Boutell <[email protected]> >> wrote: >> >>> *Specifying concurrency for "for...of" loops potentially containing >>> "await" statements in the loop body* >>> >>> In the async/await era, I see most developers using the async and await >>> keywords in 90% of situations, shifting to "Promise.all" or the bluebird >>> library only to cope with concurrency issues. >>> >>> The most common case in my experience is the need to admit a manageable >>> level of parallelism when iterating a large array (or iterator, see the >>> final section) and performing asynchronous work on each item. Unlimited >>> concurrency (Promise.all) tends to overwhelm backends involved in a way >>> that confuses developers as to what is happening. Bluebird's "Promise.map" >>> permits concurrency to be specified, which is great, but requires pulling >>> in a library and switching of mental gears ("OK right, these async >>> functions return promises," etc). >>> >>> To build on the friendliness of async/await, I propose these two >>> syntaxes be accepted: >>> >>> SYNTAX ONE >>> >>> ```js >>> for (item of items concurrent) { >>> // db.insert is an async function >>> await db.insert(item); >>> } >>> ``` >>> >>> In Syntax One, all loop bodies commence concurrently (see below for the >>> definition of "concurrently" with regard to async). If an exception is not >>> caught inside the loop, it is thrown beyond the loop, and all exceptions >>> subsequently thrown by concurrently executing loop bodies are discarded >>> (like Promise.all). >>> >>> *While I feel that unlimited concurrency is usually a mistake in a >>> situation where you have an array of items of unpredictable number, it >>> seems odd not to have a syntax for this case, and "concurrency 0" seems >>> clunky.* >>> >>> SYNTAX TWO >>> >>> ```js >>> for (item of items concurrency 5) { >>> // db.insert is an async function >>> await db.insert(item); >>> } >>> ``` >>> >>> in Syntax Two, up to 5 loop bodies commence concurrently (see below). >>> There is no guarantee that item 3 will finish before item 2, or that item 4 >>> won't start (due to 3 being finished) before item 2 ends, etc. If an >>> exception is not caught inside the loop, it is thrown beyond the loop, and >>> all exceptions subsequently thrown by concurrently executing loop bodies >>> are discarded (like Promise.all in this respect, except for the restriction >>> of concurrency). >>> >>> DEFINING CONCURRENCY FOR ASYNC >>> >>> For purposes of this proposal, "concurrent" execution means that >>> multiple loop bodies may be suspended via "await" at any given time. It >>> does NOT refer to multithreaded execution, worker threads, etc. >>> >>> CONSIDERATIONS FOR ASYNC ITERATORS >>> >>> Async iterator syntax for "for...of" loops, as in: >>> >>> ```js >>> for await (item of itemsCursor) { ... } >>> ``` >>> >>> Should also support concurrency for the loop body, with the same syntax: >>> >>> ```js >>> for await (item of itemsCursor concurrency 5) { ... } >>> ``` >>> >>> *It is important to note that this syntax does not add concurrency to >>> the async iterator itself, *at least not at this time, as I believe the >>> interface for defining async iterators does not currently accommodate this. >>> However this syntax is still useful because it *fetches the items >>> sequentially from the iterator, but may "fill the hopper" with up to five >>> iterator results* that are currently being actively processed by loop >>> bodies. In many cases, fetching items via an iterator is much faster than >>> the processing that will be done to them in the loop bodies, and so this is >>> still useful. >>> >>> Thanks for reading! >>> >>> -- >>> Chief Software Architect >>> Apostrophe Technologies >>> Pronouns: he / him / his >>> _______________________________________________ >>> es-discuss mailing list >>> [email protected] >>> https://mail.mozilla.org/listinfo/es-discuss >>> >> > > -- > Chief Software Architect > Apostrophe Technologies > Pronouns: he / him / his > _______________________________________________ > es-discuss mailing list > [email protected] > https://mail.mozilla.org/listinfo/es-discuss >
_______________________________________________ es-discuss mailing list [email protected] https://mail.mozilla.org/listinfo/es-discuss
