Re: Coping files and folders
On 01/24/2013 10:23 PM, Joel wrote:
// parallel foreach for regular files
foreach(fn ; taskPool.parallel(files,100)) {
What is the 100 number for?
It is the work unit size:
http://dlang.org/phobos/std_parallelism.html#.TaskPool
Quoting:
The number of elements processed per work unit is controlled by the
workUnitSize parameter. Smaller work units provide better load
balancing, but larger work units avoid the overhead of communicating
with other threads frequently to fetch the next work unit. Large work
units also avoid false sharing in cases where the range is being
modified. The less time a single iteration of the loop takes, the larger
workUnitSize should be. For very expensive loop bodies, workUnitSize
should be 1. An overload that chooses a default work unit size is also
available.
Ali
Re: Coping files and folders
I also wrote a copy version that orders file sequence on disk efficiently, using write through, and posted it. This speeds up any subsequent file system operations done in the directory order as if you have done a defrag. Great for hard drives, but not needed for ssd. https://github.com/jnorwood Any change you could turn this into a pull request and submit to Phobos? Looks like I never promoted the write through code on github. I just posted it on the forum.dlang.org, along with some timings. I was only experimenting on ntfs, and so the extension I posted doesn't force write-through on linux. http://forum.dlang.org/thread/gmkocaqzmlmfbuozh...@forum.dlang.org After re-reading my post, I don't think I experimented with write through in copydir. I show the timings in the post for ntfs operations on a 2GB layout after unzipping with a write through version of unzip. I suspect you'd see similar improvement following a copydir that uses write through. The problem I saw with ntfs was that the target directory was very fragmented when allowing ntfs to do its lazy flushes. I tried several experiments, and using the write through was the only solution that resulted in a reasonably defragged target immediately as the result of the unzip.
Re: Coping files and folders
On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg
wrote:
On 2013-01-24 07:28, Joel wrote:
How does one copy folders (with files in them) including sub
folders,
and creating any needed folders?
Like:
land\house\cat.d
land\house\rat.exe
land\house\bedroom\ants.txt
to
root\island\house\cat.d
root\island\house\rat.exe
root\island\house\bedroom\ants.txt
One work around is to use 'system' (under std.process).
I don't think Phobos currently has any functions for this.
Someone posted code in these newsgroups of a parallel
implementation of copy and remove.
Mustn't be very hard to manually write copyDir.
The problem with copyDir is its transactional behavior: What
should it do in case of a failure mid copy? Bail? Cleanup?
Continue?
Anyways, I just threw this together. The first version bails on
first error.
The second version keeps going as much as it can, and returns
true on success.
The caller is then free to (or not to) call rmdirRecurse in case
of failure.
//Throws exception on first error.
void copyDir(string inDir, string outDir)
{
if (!exists(outDir))
mkdir(outDir);
else
if (!isDir(outDir))
throw new FileException(format(Destination path %s
is not a folder., outDir));
foreach (entry; dirEntries(inDir.idup, SpanMode.shallow))
{
auto fileName = baseName(entry.name);
auto destName = buildPath(outDir, fileName);
if (entry.isDir())
copyDir(entry.name, destName);
else
copy(entry.name, destName);
}
}
//Silently keeps going as much as it can, then returns true on
success,
//or false if an error occured.
bool copyDirSilent(string inDir, string outDir)
{
if (!exists(outDir))
{
auto e = collectException(mkdir(outDir));
if (e !is null)
return false;
}
else
if (!isDir(outDir))
return false;
foreach (entry; dirEntries(inDir, SpanMode.shallow))
{
auto fileName = baseName(entry.name);
auto destName = buildPath(outDir, fileName);
if (entry.isDir())
{
bool b = copyDirSilent(entry.name, destName);
if (b == false)
return false;
}
else
{
auto e = collectException(mkdir(outDir));
if (e !is null)
return false;
}
}
return true;
}
Re: Coping files and folders
Brilliant! Thanks.
I think the silent one has errors. But I've used the first
function in my program now. With mine, strait after doing the
copying is was supposed then update a file, but didn't. Your one
seems to work though.
On Thursday, 24 January 2013 at 08:58:06 UTC, monarch_dodra wrote:
On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg
wrote:
On 2013-01-24 07:28, Joel wrote:
How does one copy folders (with files in them) including sub
folders,
and creating any needed folders?
Like:
land\house\cat.d
land\house\rat.exe
land\house\bedroom\ants.txt
to
root\island\house\cat.d
root\island\house\rat.exe
root\island\house\bedroom\ants.txt
One work around is to use 'system' (under std.process).
I don't think Phobos currently has any functions for this.
Someone posted code in these newsgroups of a parallel
implementation of copy and remove.
Mustn't be very hard to manually write copyDir.
The problem with copyDir is its transactional behavior: What
should it do in case of a failure mid copy? Bail? Cleanup?
Continue?
Anyways, I just threw this together. The first version bails on
first error.
The second version keeps going as much as it can, and returns
true on success.
The caller is then free to (or not to) call rmdirRecurse in
case of failure.
//Throws exception on first error.
void copyDir(string inDir, string outDir)
{
if (!exists(outDir))
mkdir(outDir);
else
if (!isDir(outDir))
throw new FileException(format(Destination path %s
is not a folder., outDir));
foreach (entry; dirEntries(inDir.idup, SpanMode.shallow))
{
auto fileName = baseName(entry.name);
auto destName = buildPath(outDir, fileName);
if (entry.isDir())
copyDir(entry.name, destName);
else
copy(entry.name, destName);
}
}
//Silently keeps going as much as it can, then returns true on
success,
//or false if an error occured.
bool copyDirSilent(string inDir, string outDir)
{
if (!exists(outDir))
{
auto e = collectException(mkdir(outDir));
if (e !is null)
return false;
}
else
if (!isDir(outDir))
return false;
foreach (entry; dirEntries(inDir, SpanMode.shallow))
{
auto fileName = baseName(entry.name);
auto destName = buildPath(outDir, fileName);
if (entry.isDir())
{
bool b = copyDirSilent(entry.name, destName);
if (b == false)
return false;
}
else
{
auto e = collectException(mkdir(outDir));
if (e !is null)
return false;
}
}
return true;
}
Re: Coping files and folders
On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg wrote: Someone posted code in these newsgroups of a parallel implementation of copy and remove. I posted a parallel implementation a while back, and also put it on github. The parallel trick is to create the folder structure first, then populate the folders with the files. Best for ssd drives. I also wrote a copy version that orders file sequence on disk efficiently, using write through, and posted it. This speeds up any subsequent file system operations done in the directory order as if you have done a defrag. Great for hard drives, but not needed for ssd. https://github.com/jnorwood
Re: Coping files and folders
Sounds cool. I've bookmarked your github hub link.
Just wondering with this:
// parallel foreach for regular files
foreach(fn ; taskPool.parallel(files,100)) {
string dfn = destRoot ~ fn[srcLen..$];
copy(fn,dfn);
}
What is the 100 number for?
On Friday, 25 January 2013 at 05:30:11 UTC, Jay Norwood wrote:
On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg
wrote:
Someone posted code in these newsgroups of a parallel
implementation of copy and remove.
I posted a parallel implementation a while back, and also put
it on github.
The parallel trick is to create the folder structure first,
then populate the folders with the files. Best for ssd drives.
I also wrote a copy version that orders file sequence on disk
efficiently, using write through, and posted it. This speeds
up any subsequent file system operations done in the directory
order as if you have done a defrag. Great for hard drives, but
not needed for ssd.
https://github.com/jnorwood
Coping files and folders
How does one copy folders (with files in them) including sub folders, and creating any needed folders? Like: land\house\cat.d land\house\rat.exe land\house\bedroom\ants.txt to root\island\house\cat.d root\island\house\rat.exe root\island\house\bedroom\ants.txt One work around is to use 'system' (under std.process). -joelcnz
Re: Coping files and folders
On 2013-01-24 07:28, Joel wrote: How does one copy folders (with files in them) including sub folders, and creating any needed folders? Like: land\house\cat.d land\house\rat.exe land\house\bedroom\ants.txt to root\island\house\cat.d root\island\house\rat.exe root\island\house\bedroom\ants.txt One work around is to use 'system' (under std.process). I don't think Phobos currently has any functions for this. Someone posted code in these newsgroups of a parallel implementation of copy and remove. -- /Jacob Carlborg
