> With the promise.all, would each item in the iterable be an async function (or function returning a promise)? I mean, I'm presuming the point is to not have every item execute at the same time.
Ah, correct, like in my [example]( https://github.com/caub/misc/blob/master/utils/promise-concurrent.js) it should be an array of thunks (function with no arguments returning the promise) So it should also have a different name than `Promise.all`, `Promise.allThunks` maybe? On Fri, Sep 6, 2019 at 10:08 PM Michael J. Ryan <[email protected]> wrote: > With the promise.all, would each item in the iterable be an async function > (or function returning a promise)? I mean, I'm presuming the point is to > not have every item execute at the same time. > > -- > > Michael J. Ryan > Website: https://www.tracker1.info/ > Email: [email protected] > Mobile: 480-270-4509 > > > On Fri, Sep 6, 2019 at 12:04 PM Cyril Auburtin <[email protected]> > wrote: > >> It could be probably added as a `Promise.all(iterable, concurrency?)` >> >> Some existing implementations: >> - http://bluebirdjs.com/docs/api/promise.map.html >> - https://caolan.github.io/async/v3/docs.html#mapLimit >> - I tried to write one: >> https://github.com/caub/misc/blob/master/utils/promise-concurrent.js >> >> On Fri, Sep 6, 2019 at 8:33 PM Tom Boutell <[email protected]> wrote: >> >>> I am more interested in syntax two than syntax one, which I felt should >>> probably be included for completeness. But hey, as you say, maybe not since >>> unguarded concurrency is indeed usually a mistake. >>> >>> Taken on its own, do you have an objection to `for (item of items >>> concurrency 5) { ... }`? >>> >>> >>> On Fri, Sep 6, 2019 at 1:46 PM C. Scott Ananian <[email protected]> >>> wrote: >>> >>>> The current way to write what you want is: >>>> >>>> await Promise.all(itemsCursor.map(item => db.insert(item)); >>>> >>>> or >>>> >>>> await Promise.all(itemsCursor.map(Promise.guard(5, item => >>>> db.insert(item)))); >>>> >>>> if you are using a library like `prfun` ( >>>> https://github.com/cscott/prfun#promiseguardfunctionnumber-condition-function-fn--function >>>> ). >>>> >>>> I'm not sure adding a new parallel loop construct is an obvious >>>> improvement on this, especially since unguarded concurrent execution is >>>> usually a mistake (can cause memory requirements to blow up), as you point >>>> out yourself. >>>> >>>> I'd be more in favor of new syntax if it was integrated with a >>>> work-stealing mechanism, since (a) that can't as easily be done in a >>>> library function (you need access to the entire set of runnable tasks, not >>>> just the ones created in this loop), and (b) is more likely to be >>>> correct/fast by default and not lead to subtle resource-exhaustion >>>> problems. >>>> --scott >>>> >>>> On Fri, Sep 6, 2019 at 12:40 PM Tom Boutell <[email protected]> >>>> wrote: >>>> >>>>> *Specifying concurrency for "for...of" loops potentially containing >>>>> "await" statements in the loop body* >>>>> >>>>> In the async/await era, I see most developers using the async and >>>>> await keywords in 90% of situations, shifting to "Promise.all" or the >>>>> bluebird library only to cope with concurrency issues. >>>>> >>>>> The most common case in my experience is the need to admit a >>>>> manageable level of parallelism when iterating a large array (or iterator, >>>>> see the final section) and performing asynchronous work on each item. >>>>> Unlimited concurrency (Promise.all) tends to overwhelm backends involved >>>>> in >>>>> a way that confuses developers as to what is happening. Bluebird's >>>>> "Promise.map" permits concurrency to be specified, which is great, but >>>>> requires pulling in a library and switching of mental gears ("OK right, >>>>> these async functions return promises," etc). >>>>> >>>>> To build on the friendliness of async/await, I propose these two >>>>> syntaxes be accepted: >>>>> >>>>> SYNTAX ONE >>>>> >>>>> ```js >>>>> for (item of items concurrent) { >>>>> // db.insert is an async function >>>>> await db.insert(item); >>>>> } >>>>> ``` >>>>> >>>>> In Syntax One, all loop bodies commence concurrently (see below for >>>>> the definition of "concurrently" with regard to async). If an exception is >>>>> not caught inside the loop, it is thrown beyond the loop, and all >>>>> exceptions subsequently thrown by concurrently executing loop bodies are >>>>> discarded (like Promise.all). >>>>> >>>>> *While I feel that unlimited concurrency is usually a mistake in a >>>>> situation where you have an array of items of unpredictable number, it >>>>> seems odd not to have a syntax for this case, and "concurrency 0" seems >>>>> clunky.* >>>>> >>>>> SYNTAX TWO >>>>> >>>>> ```js >>>>> for (item of items concurrency 5) { >>>>> // db.insert is an async function >>>>> await db.insert(item); >>>>> } >>>>> ``` >>>>> >>>>> in Syntax Two, up to 5 loop bodies commence concurrently (see below). >>>>> There is no guarantee that item 3 will finish before item 2, or that item >>>>> 4 >>>>> won't start (due to 3 being finished) before item 2 ends, etc. If an >>>>> exception is not caught inside the loop, it is thrown beyond the loop, and >>>>> all exceptions subsequently thrown by concurrently executing loop bodies >>>>> are discarded (like Promise.all in this respect, except for the >>>>> restriction >>>>> of concurrency). >>>>> >>>>> DEFINING CONCURRENCY FOR ASYNC >>>>> >>>>> For purposes of this proposal, "concurrent" execution means that >>>>> multiple loop bodies may be suspended via "await" at any given time. It >>>>> does NOT refer to multithreaded execution, worker threads, etc. >>>>> >>>>> CONSIDERATIONS FOR ASYNC ITERATORS >>>>> >>>>> Async iterator syntax for "for...of" loops, as in: >>>>> >>>>> ```js >>>>> for await (item of itemsCursor) { ... } >>>>> ``` >>>>> >>>>> Should also support concurrency for the loop body, with the same >>>>> syntax: >>>>> >>>>> ```js >>>>> for await (item of itemsCursor concurrency 5) { ... } >>>>> ``` >>>>> >>>>> *It is important to note that this syntax does not add concurrency to >>>>> the async iterator itself, *at least not at this time, as I believe >>>>> the interface for defining async iterators does not currently accommodate >>>>> this. However this syntax is still useful because it *fetches the >>>>> items sequentially from the iterator, but may "fill the hopper" with up to >>>>> five iterator results* that are currently being actively processed by >>>>> loop bodies. In many cases, fetching items via an iterator is much faster >>>>> than the processing that will be done to them in the loop bodies, and so >>>>> this is still useful. >>>>> >>>>> Thanks for reading! >>>>> >>>>> -- >>>>> Chief Software Architect >>>>> Apostrophe Technologies >>>>> Pronouns: he / him / his >>>>> _______________________________________________ >>>>> es-discuss mailing list >>>>> [email protected] >>>>> https://mail.mozilla.org/listinfo/es-discuss >>>>> >>>> >>> >>> -- >>> Chief Software Architect >>> Apostrophe Technologies >>> Pronouns: he / him / his >>> _______________________________________________ >>> es-discuss mailing list >>> [email protected] >>> https://mail.mozilla.org/listinfo/es-discuss >>> >> _______________________________________________ >> es-discuss mailing list >> [email protected] >> https://mail.mozilla.org/listinfo/es-discuss >> >
_______________________________________________ es-discuss mailing list [email protected] https://mail.mozilla.org/listinfo/es-discuss
