The current way to write what you want is: await Promise.all(itemsCursor.map(item => db.insert(item));
or await Promise.all(itemsCursor.map(Promise.guard(5, item => db.insert(item)))); if you are using a library like `prfun` ( https://github.com/cscott/prfun#promiseguardfunctionnumber-condition-function-fn--function ). I'm not sure adding a new parallel loop construct is an obvious improvement on this, especially since unguarded concurrent execution is usually a mistake (can cause memory requirements to blow up), as you point out yourself. I'd be more in favor of new syntax if it was integrated with a work-stealing mechanism, since (a) that can't as easily be done in a library function (you need access to the entire set of runnable tasks, not just the ones created in this loop), and (b) is more likely to be correct/fast by default and not lead to subtle resource-exhaustion problems. --scott On Fri, Sep 6, 2019 at 12:40 PM Tom Boutell <[email protected]> wrote: > *Specifying concurrency for "for...of" loops potentially containing > "await" statements in the loop body* > > In the async/await era, I see most developers using the async and await > keywords in 90% of situations, shifting to "Promise.all" or the bluebird > library only to cope with concurrency issues. > > The most common case in my experience is the need to admit a manageable > level of parallelism when iterating a large array (or iterator, see the > final section) and performing asynchronous work on each item. Unlimited > concurrency (Promise.all) tends to overwhelm backends involved in a way > that confuses developers as to what is happening. Bluebird's "Promise.map" > permits concurrency to be specified, which is great, but requires pulling > in a library and switching of mental gears ("OK right, these async > functions return promises," etc). > > To build on the friendliness of async/await, I propose these two syntaxes > be accepted: > > SYNTAX ONE > > ```js > for (item of items concurrent) { > // db.insert is an async function > await db.insert(item); > } > ``` > > In Syntax One, all loop bodies commence concurrently (see below for the > definition of "concurrently" with regard to async). If an exception is not > caught inside the loop, it is thrown beyond the loop, and all exceptions > subsequently thrown by concurrently executing loop bodies are discarded > (like Promise.all). > > *While I feel that unlimited concurrency is usually a mistake in a > situation where you have an array of items of unpredictable number, it > seems odd not to have a syntax for this case, and "concurrency 0" seems > clunky.* > > SYNTAX TWO > > ```js > for (item of items concurrency 5) { > // db.insert is an async function > await db.insert(item); > } > ``` > > in Syntax Two, up to 5 loop bodies commence concurrently (see below). > There is no guarantee that item 3 will finish before item 2, or that item 4 > won't start (due to 3 being finished) before item 2 ends, etc. If an > exception is not caught inside the loop, it is thrown beyond the loop, and > all exceptions subsequently thrown by concurrently executing loop bodies > are discarded (like Promise.all in this respect, except for the restriction > of concurrency). > > DEFINING CONCURRENCY FOR ASYNC > > For purposes of this proposal, "concurrent" execution means that multiple > loop bodies may be suspended via "await" at any given time. It does NOT > refer to multithreaded execution, worker threads, etc. > > CONSIDERATIONS FOR ASYNC ITERATORS > > Async iterator syntax for "for...of" loops, as in: > > ```js > for await (item of itemsCursor) { ... } > ``` > > Should also support concurrency for the loop body, with the same syntax: > > ```js > for await (item of itemsCursor concurrency 5) { ... } > ``` > > *It is important to note that this syntax does not add concurrency to the > async iterator itself, *at least not at this time, as I believe the > interface for defining async iterators does not currently accommodate this. > However this syntax is still useful because it *fetches the items > sequentially from the iterator, but may "fill the hopper" with up to five > iterator results* that are currently being actively processed by loop > bodies. In many cases, fetching items via an iterator is much faster than > the processing that will be done to them in the loop bodies, and so this is > still useful. > > Thanks for reading! > > -- > Chief Software Architect > Apostrophe Technologies > Pronouns: he / him / his > _______________________________________________ > es-discuss mailing list > [email protected] > https://mail.mozilla.org/listinfo/es-discuss >
_______________________________________________ es-discuss mailing list [email protected] https://mail.mozilla.org/listinfo/es-discuss
