Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-06 Thread Paul via Digitalmars-d-learn
On Thursday, 6 April 2023 at 01:44:15 UTC, H. S. Teoh wrote: D ranges are conceptually sequential, but the actual underlying memory access patterns depends on the concrete type at runtime. An array's elements are stored sequentially in memory, and arrays are ranges. But a linked-list can

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-06 Thread Salih Dincer via Digitalmars-d-learn
On Tuesday, 4 April 2023 at 16:22:29 UTC, Steven Schveighoffer wrote: On 4/4/23 11:34 AM, Salih Dincer wrote: On Tuesday, 4 April 2023 at 14:20:20 UTC, Steven Schveighoffer wrote: parallel is a shortcut to `TaskPool.parallel`, which is indeed a foreach-only construct, it does not return a

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-05 Thread H. S. Teoh via Digitalmars-d-learn
On Thu, Apr 06, 2023 at 01:20:28AM +, Paul via Digitalmars-d-learn wrote: [...] > Yes I understand, basically, what's going on in hardware. I just > wasn't sure if the access type was linked to the container type. It > seems obvious now, since you've both made it clear, that it also >

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-05 Thread Paul via Digitalmars-d-learn
On Wednesday, 5 April 2023 at 23:06:54 UTC, H. S. Teoh wrote: So your data structures and algorithms should be designed in a way that takes advantage of linear access where possible. T Yes I understand, basically, what's going on in hardware. I just wasn't sure if the access type was

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-05 Thread H. S. Teoh via Digitalmars-d-learn
On Wed, Apr 05, 2023 at 10:34:22PM +, Paul via Digitalmars-d-learn wrote: > On Tuesday, 4 April 2023 at 22:20:52 UTC, H. S. Teoh wrote: > > > Best practices for arrays in hot loops: [...] > > - Where possible, prefer sequential access over random access (take > > advantage of the CPU cache

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-05 Thread Steven Schveighoffer via Digitalmars-d-learn
On 4/5/23 6:34 PM, Paul wrote: On Tuesday, 4 April 2023 at 22:20:52 UTC, H. S. Teoh wrote: Best practices for arrays in hot loops: - Avoid appending if possible; instead, pre-allocate outside the loop. - Where possible, reuse existing arrays instead of discarding old ones   and allocating new

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-05 Thread Paul via Digitalmars-d-learn
On Tuesday, 4 April 2023 at 22:20:52 UTC, H. S. Teoh wrote: Best practices for arrays in hot loops: - Avoid appending if possible; instead, pre-allocate outside the loop. - Where possible, reuse existing arrays instead of discarding old ones and allocating new ones. - Use slices where

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-04 Thread H. S. Teoh via Digitalmars-d-learn
On Tue, Apr 04, 2023 at 09:35:29PM +, Paul via Digitalmars-d-learn wrote: [...] > Well Steven just making the change you said reduced the execution time > from ~6-7 secs to ~3 secs. Then, including the 'parallel' in the > foreach statement took it down to ~1 sec. > > Boy lesson learned in

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-04 Thread Paul via Digitalmars-d-learn
On Monday, 3 April 2023 at 23:50:48 UTC, Steven Schveighoffer wrote: So what you need is inside `createSpansOfNoBeacons`, take as a reference a `ref Span[MAX_SPANS]`, and have it return a `Span[]` that is a slice of that which was "alocated". See if this helps. Well Steven just making the

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-04 Thread Steven Schveighoffer via Digitalmars-d-learn
On 4/4/23 11:34 AM, Salih Dincer wrote: On Tuesday, 4 April 2023 at 14:20:20 UTC, Steven Schveighoffer wrote: parallel is a shortcut to `TaskPool.parallel`, which is indeed a foreach-only construct, it does not return a range. I think what you want is `TaskPool.map`: ```d // untested, just

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-04 Thread Salih Dincer via Digitalmars-d-learn
On Tuesday, 4 April 2023 at 14:20:20 UTC, Steven Schveighoffer wrote: parallel is a shortcut to `TaskPool.parallel`, which is indeed a foreach-only construct, it does not return a range. I think what you want is `TaskPool.map`: ```d // untested, just looking at the taskPool.map!(/* your map

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-04 Thread Steven Schveighoffer via Digitalmars-d-learn
On 4/4/23 5:24 AM, Salih Dincer wrote: Is it necessary to enclose the code in `foreach()`? I invite Ali to tell me! Please explain why parallel isn't running. parallel is a shortcut to `TaskPool.parallel`, which is indeed a foreach-only construct, it does not return a range. I think what

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-04 Thread Ali Çehreli via Digitalmars-d-learn
On 4/4/23 02:24, Salih Dincer wrote: > I don't understand what `foreach()` does :) Hm. I forgot whether 'parallel' works only with 'foreach'. But there are various other algorithms in std.parallelism that may be more useful with range algorithm chains:

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-04 Thread Salih Dincer via Digitalmars-d-learn
On Monday, 3 April 2023 at 22:24:18 UTC, Steven Schveighoffer wrote: So for example, if you have: ```d foreach(i; iota(0, 2_000_000).parallel) { runExpensiveTask(i); } ``` The foreach is run on the main thread, gets a `0`, then hands off to a task thread `runExpensiveTask(0)`. Then it gets

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-03 Thread Steven Schveighoffer via Digitalmars-d-learn
On 4/3/23 7:22 PM, Paul wrote: ```d // Timed main() vvv void main(string[] args) { auto progStartTime = MonoTime.currTime; //- string filename =

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-03 Thread Paul via Digitalmars-d-learn
On Monday, 3 April 2023 at 23:13:58 UTC, Steven Schveighoffer wrote: Yeah, please post. ```d module aoc2215b2; import std.stdio; import std.file: readText; import std.conv: to; import std.math: abs; import std.traits; import std.parallelism; import std.range; import core.time: MonoTime; //

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-03 Thread Steven Schveighoffer via Digitalmars-d-learn
On 4/3/23 6:56 PM, Paul wrote: On Monday, 3 April 2023 at 22:24:18 UTC, Steven Schveighoffer wrote: If your `foreach` body takes a global lock (like `writeln(i);`), then it's not going to run any faster (probably slower actually). **Ok I did have some debug writelns I commented out.** And

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-03 Thread Paul via Digitalmars-d-learn
On Monday, 3 April 2023 at 22:24:18 UTC, Steven Schveighoffer wrote: If your `foreach` body takes a global lock (like `writeln(i);`), then it's not going to run any faster (probably slower actually). **Ok I did have some debug writelns I commented out.** And did it help? **No** My

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-03 Thread Steven Schveighoffer via Digitalmars-d-learn
On 4/3/23 6:02 PM, Paul wrote: On Sunday, 2 April 2023 at 15:32:05 UTC, Steven Schveighoffer wrote: It's important to note that parallel doesn't iterate the range in parallel, it just runs the body in parallel limited by your CPU count. **?!?** So for example, if you have: ```d

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-03 Thread Paul via Digitalmars-d-learn
On Sunday, 2 April 2023 at 15:32:05 UTC, Steven Schveighoffer wrote: It's important to note that parallel doesn't iterate the range in parallel, it just runs the body in parallel limited by your CPU count. **?!?** If your `foreach` body takes a global lock (like `writeln(i);`), then it's

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-02 Thread Steven Schveighoffer via Digitalmars-d-learn
On 4/1/23 6:32 PM, Paul wrote: On Saturday, 1 April 2023 at 18:30:32 UTC, Steven Schveighoffer wrote: On 4/1/23 2:25 PM, Paul wrote: ```d import std.range; foreach(i; iota(0, 2_000_000).parallel) ``` Is there a way to tell if the parallelism actually divided up the work? Both versions of

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-02 Thread Salih Dincer via Digitalmars-d-learn
On Sunday, 2 April 2023 at 04:34:40 UTC, Salih Dincer wrote: I haven't seen rsFirst256 until now... **Edit:** I saw, I saw :) I am struck with consternation! I've never seen these results before. Interesting, there is such a thing as parallel threading :) Here are my skipPoints: ```d

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-01 Thread Salih Dincer via Digitalmars-d-learn
On Saturday, 1 April 2023 at 22:48:46 UTC, Ali Çehreli wrote: On 4/1/23 15:30, Paul wrote: > Is there a way to verify that it split up the work in to tasks/threads > ...? It is hard to see the difference unless there is actual work in the loop that takes time. I always use the Rowland

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-01 Thread Ali Çehreli via Digitalmars-d-learn
On 4/1/23 15:30, Paul wrote: > Is there a way to verify that it split up the work in to tasks/threads > ...? It is hard to see the difference unless there is actual work in the loop that takes time. You can add a Thread.sleep call. (Commented-out in the following program.) Another option is

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-01 Thread Paul via Digitalmars-d-learn
On Saturday, 1 April 2023 at 18:30:32 UTC, Steven Schveighoffer wrote: On 4/1/23 2:25 PM, Paul wrote: ```d import std.range; foreach(; iota(0, 2_000_000).parallel) ``` -Steve Is there a way to tell if the parallelism actually divided up the work? Both versions of my program run in the

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-01 Thread Paul via Digitalmars-d-learn
```d import std.range; foreach(; iota(0, 2_000_000).parallel) ``` -Steve Is there a way to verify that it split up the work in to tasks/threads ...? The example you gave me works...compiles w/o errors but the execution time is the same as the non-parallel version. They both take about 6

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-01 Thread Paul via Digitalmars-d-learn
Thanks Steve.

Re: foreach (i; taskPool.parallel(0..2_000_000)

2023-04-01 Thread Steven Schveighoffer via Digitalmars-d-learn
On 4/1/23 2:25 PM, Paul wrote: Thanks in advance for any assistance. As the subject line suggests can I do something like? : ```d foreach (i; taskPool.parallel(0..2_000_000)) ``` Obviously this exact syntax doesn't work but I think it expresses the gist of my challenge. ```d import

foreach (i; taskPool.parallel(0..2_000_000)

2023-04-01 Thread Paul via Digitalmars-d-learn
Thanks in advance for any assistance. As the subject line suggests can I do something like? : ```d foreach (i; taskPool.parallel(0..2_000_000)) ``` Obviously this exact syntax doesn't work but I think it expresses the gist of my challenge.