Re: Assertion Error
On Wednesday, 13 September 2017 at 15:27:30 UTC, Moritz Maxeiner
wrote:
On Wednesday, 13 September 2017 at 15:12:57 UTC, Vino.B wrote:
On Wednesday, 13 September 2017 at 11:03:38 UTC, Moritz
Maxeiner wrote:
On Wednesday, 13 September 2017 at 07:39:46 UTC, Vino.B wrote:
[...]
[...]
---
foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
{
MSresult.get ~= coSizeDirList(Fs.strip, SizeDir);
}
---
Hi Max,
It's Moritz, not Max. ;)
Below is the explanation of the above code.
[...]
AFAICT that's a reason why you want parallelization of
coSizeDirList, but not why you need to spawn another thread
inside of an *already parallelelized" task. Try my shortened
parallel foreach loop vs your longer one and monitor system
load (threads, memory, etc).
Hi Moritz,
Thank you very much, it was very helpful and time saving and
fast.
From,
Vino.B
Re: Assertion Error
On Wednesday, 13 September 2017 at 15:12:57 UTC, Vino.B wrote:
On Wednesday, 13 September 2017 at 11:03:38 UTC, Moritz
Maxeiner wrote:
On Wednesday, 13 September 2017 at 07:39:46 UTC, Vino.B wrote:
Hi Max,
[...]
Program Code:
[...]
foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
{
auto FFs = Fs.strip;
auto MSizeDirList = task(&coSizeDirList, FFs, SizeDir);
MSizeDirList.executeInNewThread();
auto MSizeDirListData = MSizeDirList.workForce;
MSresult.get ~= MSizeDirListData;
}
[...]
---
foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
{
MSresult.get ~= coSizeDirList(Fs.strip, SizeDir);
}
---
Hi Max,
It's Moritz, not Max. ;)
Below is the explanation of the above code.
[...]
AFAICT that's a reason why you want parallelization of
coSizeDirList, but not why you need to spawn another thread
inside of an *already parallelelized" task. Try my shortened
parallel foreach loop vs your longer one and monitor system load
(threads, memory, etc).
Re: Assertion Error
On Wednesday, 13 September 2017 at 11:03:38 UTC, Moritz Maxeiner
wrote:
On Wednesday, 13 September 2017 at 07:39:46 UTC, Vino.B wrote:
On Tuesday, 12 September 2017 at 21:01:26 UTC, Moritz Maxeiner
wrote:
On Tuesday, 12 September 2017 at 19:44:19 UTC, vino wrote:
Hi All,
I have a small piece of code which executes perfectly 8 out
of 10 times, very rarely it throws an assertion error, so is
there a way to find which line of code is causing this error.
You should be getting the line number as part of the crash,
like here:
--- test.d ---
void main(string[] args)
{
assert(args.length > 1);
}
--
-
$ dmd -run test.d
core.exception.AssertError@test.d(3): Assertion failure
[Stack trace]
-
If you don't what are the steps to reproduce?
Hi Max,
I tried to run the code for at least 80+ time the code ran
without any issue, will let you know in case if I hit the same
issue in feature, Below is the piece of code, plese do let me
know if you find any issue with the below code.
Program Code:
[...]
foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
{
auto FFs = Fs.strip;
auto MSizeDirList = task(&coSizeDirList, FFs, SizeDir);
MSizeDirList.executeInNewThread();
auto MSizeDirListData = MSizeDirList.workForce;
MSresult.get ~= MSizeDirListData;
}
From reading I don't see anything that I would expect to
assert, but I am wondering why you first parallelize your work
with a thread pool (`parallel(...)`) and then inside each
(implicitly created) task (that is already being serviced by a
thread in the thread pool) you create another task, have it
executed in a new thread, and make the thread pool thread wait
for that thread to complete servicing that new task.
This should yield the same result, but without the overhead of
spawning additional threads:
---
foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
{
MSresult.get ~= coSizeDirList(Fs.strip, SizeDir);
}
---
Hi Max,
Below is the explanation of the above code.
The Fs that is passed to the function ptSizeDirList is any array
of 10 -15 file system (NetApp Filers)from 15 different servers
mounted on a single server (Network share).
The function ptSizeDirList pass each of the FS to the function
coSizeDirList in parallel thread to find the folder size under
each of these Fs,
Each of this FS has around 1000+ folders. so the requirement is
that we need the size of each of the folders under each of these
15 Fs along with the folder name in less than an hour or two as
this script is about to be schedule to run once every 3 hours,
and at present we are able to achieve the same in 10 mins / fs
Each Fs is of about 10-15 TB's
Hence this code was written.
Re: Assertion Error
On Wednesday, 13 September 2017 at 07:39:46 UTC, Vino.B wrote:
On Tuesday, 12 September 2017 at 21:01:26 UTC, Moritz Maxeiner
wrote:
On Tuesday, 12 September 2017 at 19:44:19 UTC, vino wrote:
Hi All,
I have a small piece of code which executes perfectly 8 out
of 10 times, very rarely it throws an assertion error, so is
there a way to find which line of code is causing this error.
You should be getting the line number as part of the crash,
like here:
--- test.d ---
void main(string[] args)
{
assert(args.length > 1);
}
--
-
$ dmd -run test.d
core.exception.AssertError@test.d(3): Assertion failure
[Stack trace]
-
If you don't what are the steps to reproduce?
Hi Max,
I tried to run the code for at least 80+ time the code ran
without any issue, will let you know in case if I hit the same
issue in feature, Below is the piece of code, plese do let me
know if you find any issue with the below code.
Program Code:
[...]
foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
{
auto FFs = Fs.strip;
auto MSizeDirList = task(&coSizeDirList, FFs, SizeDir);
MSizeDirList.executeInNewThread();
auto MSizeDirListData = MSizeDirList.workForce;
MSresult.get ~= MSizeDirListData;
}
From reading I don't see anything that I would expect to assert,
but I am wondering why you first parallelize your work with a
thread pool (`parallel(...)`) and then inside each (implicitly
created) task (that is already being serviced by a thread in the
thread pool) you create another task, have it executed in a new
thread, and make the thread pool thread wait for that thread to
complete servicing that new task.
This should yield the same result, but without the overhead of
spawning additional threads:
---
foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
{
MSresult.get ~= coSizeDirList(Fs.strip, SizeDir);
}
---
Re: Assertion Error
On Tuesday, 12 September 2017 at 21:01:26 UTC, Moritz Maxeiner
wrote:
On Tuesday, 12 September 2017 at 19:44:19 UTC, vino wrote:
Hi All,
I have a small piece of code which executes perfectly 8 out of
10 times, very rarely it throws an assertion error, so is
there a way to find which line of code is causing this error.
You should be getting the line number as part of the crash,
like here:
--- test.d ---
void main(string[] args)
{
assert(args.length > 1);
}
--
-
$ dmd -run test.d
core.exception.AssertError@test.d(3): Assertion failure
[Stack trace]
-
If you don't what are the steps to reproduce?
Hi Max,
I tried to run the code for at least 80+ time the code ran
without any issue, will let you know in case if I hit the same
issue in feature, Below is the piece of code, plese do let me
know if you find any issue with the below code.
Program Code:
import core.stdc.stdlib: exit;
import std.algorithm: all, among, filter, map, setDifference,
sort, uniq, each, joiner;
import std.array: appender, join;
import std.container.array;
import std.conv: to;
import std.datetime.systime: Clock, days, SysTime;
import std.file: SpanMode, dirEntries, exists, isFile, mkdir,
remove, rmdirRecurse;
import std.getopt;
import std.parallelism: parallel, task, taskPool;
import std.path: absolutePath, baseName, dirName,
isValidFilename, isValidPath, globMatch;
import std.range: empty, zip, chain, chunks;
import std.stdio: File, writefln, writeln;
import std.string: chomp, chop, isNumeric, split, strip;
import std.typecons: tuple, Tuple;
import std.uni: isAlpha, toLower, isWhite;
import std.conv;
auto coSizeDirList (string FFs, int SizeDir) {
int subdirTotal;
int subdirTotalGB;
Array!string Subdir;
Array!string Subsize;
Array!string Result;
auto dFiles = Array!string ((dirEntries(FFs,
SpanMode.shallow).filter!(a => a.isDir))[].map!(a => a.name));
foreach (d; dFiles[]) {
auto SdFiles = Array!ulong(dirEntries(d,
SpanMode.depth).map!(a => a.size));
foreach(f; SdFiles[]) { subdirTotal += f; }
subdirTotalGB = (subdirTotal/1024/1024);
if (subdirTotalGB > SizeDir) { Result ~= d; Result ~=
to!string(subdirTotalGB); }
subdirTotal = 0;
}
return Result;
}
void ptSizeDirList (Array!string SizeDirlst, int SizeDir) {
alias DirSizeList = typeof(coSizeDirList(string.init, int.init));
auto MSresult = taskPool.workerLocalStorage!DirSizeList();
foreach (string Fs; parallel(SizeDirlst[0 .. $], 1)) {
auto FFs = Fs.strip;
auto MSizeDirList = task(&coSizeDirList, FFs, SizeDir);
MSizeDirList.executeInNewThread();
auto MSizeDirListData = MSizeDirList.workForce;
MSresult.get ~= MSizeDirListData;
}
foreach(i; MSresult.toRange)
chain(i[]).chunks(2).each!(e => writefln!"%-60s %s"(e[0],
e[1]));
}
void main () {
auto SizeDirlst = Array!string( "C:\\Temp\\TEST2\\BACKUP",
"C:\\Temp\\TEST2\\EXPORT", "C:\\Temp\\TEST2\\PROD_TEAM",
"C:\\Temp\\TEST2\\TEAM", "C:\\Temp\\TEST3\\BACKUP",
"C:\\Temp\\TEST3\\EXPORT" );
int SizeDir = 10;
ptSizeDirList(SizeDirlst, SizeDir);
}
From,
Vino.B
Re: Assertion Error
On Tuesday, 12 September 2017 at 19:44:19 UTC, vino wrote:
Hi All,
I have a small piece of code which executes perfectly 8 out of
10 times, very rarely it throws an assertion error, so is there
a way to find which line of code is causing this error.
You should be getting the line number as part of the crash, like
here:
--- test.d ---
void main(string[] args)
{
assert(args.length > 1);
}
--
-
$ dmd -run test.d
core.exception.AssertError@test.d(3): Assertion failure
[Stack trace]
-
If you don't what are the steps to reproduce?
Assertion Error
Hi All, I have a small piece of code which executes perfectly 8 out of 10 times, very rarely it throws an assertion error, so is there a way to find which line of code is causing this error. From, Vino.B
