Re: Directory recursive walking

2021-01-15 Thread dog2002 via Digitalmars-d-learn
On Friday, 15 January 2021 at 11:05:56 UTC, Daniel Kozak wrote: On Fri, Jan 15, 2021 at 10:30 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: ... Okay, the reason is incredibly stupid: using WinMain instead of main causes high memory usage. I don't know why, I

Re: Directory recursive walking

2021-01-15 Thread Daniel Kozak via Digitalmars-d-learn
On Fri, Jan 15, 2021 at 10:30 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > ... > Okay, the reason is incredibly stupid: using WinMain instead of > main causes high memory usage. I don't know why, I use the same > code. If I replace WinMain with main, the memory

Re: Directory recursive walking

2021-01-15 Thread dog2002 via Digitalmars-d-learn
On Friday, 15 January 2021 at 06:15:06 UTC, dog2002 wrote: On Thursday, 14 January 2021 at 22:28:19 UTC, Paul Backus wrote: On Thursday, 14 January 2021 at 20:23:37 UTC, dog2002 wrote: [...] What code are you using to copy the bytes? If you're reading the whole file into memory at once,

Re: Directory recursive walking

2021-01-14 Thread Ferhat Kurtulmuş via Digitalmars-d-learn
On Friday, 15 January 2021 at 07:16:21 UTC, Daniel Kozak wrote: On Fri, Jan 15, 2021 at 8:00 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote: > On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus

Re: Directory recursive walking

2021-01-14 Thread Daniel Kozak via Digitalmars-d-learn
On Fri, Jan 15, 2021 at 8:20 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > On Friday, 15 January 2021 at 06:56:36 UTC, dog2002 wrote: > > On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote: > >> On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus

Re: Directory recursive walking

2021-01-14 Thread dog2002 via Digitalmars-d-learn
On Friday, 15 January 2021 at 06:56:36 UTC, dog2002 wrote: On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote: On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote: You can save a little bit of memory here by allocating tempBuffer on the stack: ubyte[512]

Re: Directory recursive walking

2021-01-14 Thread Daniel Kozak via Digitalmars-d-learn
On Fri, Jan 15, 2021 at 8:00 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote: > > On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote: > >> > >> You can save a little bit of memory here by

Re: Directory recursive walking

2021-01-14 Thread dog2002 via Digitalmars-d-learn
On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote: On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote: You can save a little bit of memory here by allocating tempBuffer on the stack: ubyte[512] tempBuffer; _inputFile.rawRead(tempBuffer[]); // note the explicit

Re: Directory recursive walking

2021-01-14 Thread Paul Backus via Digitalmars-d-learn
On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote: You can save a little bit of memory here by allocating tempBuffer on the stack: ubyte[512] tempBuffer; _inputFile.rawRead(tempBuffer[]); // note the explicit [] I made a mistake; this should be: ubyte[512]

Re: Directory recursive walking

2021-01-14 Thread Paul Backus via Digitalmars-d-learn
On Friday, 15 January 2021 at 06:15:06 UTC, dog2002 wrote: void func(string inputFile, string outFile, uint chunk_size) { try { File _inputFile = File(inputFile, "r"); File _outputFile = File(outFile, "w"); ubyte[]

Re: Directory recursive walking

2021-01-14 Thread dog2002 via Digitalmars-d-learn
On Thursday, 14 January 2021 at 22:28:19 UTC, Paul Backus wrote: On Thursday, 14 January 2021 at 20:23:37 UTC, dog2002 wrote: About 1000 large files. I want to replace several first bytes in all the files, so I just copy the remaining bytes into a new file. Might this be the reason for high

Re: Directory recursive walking

2021-01-14 Thread Paul Backus via Digitalmars-d-learn
On Thursday, 14 January 2021 at 20:23:37 UTC, dog2002 wrote: About 1000 large files. I want to replace several first bytes in all the files, so I just copy the remaining bytes into a new file. Might this be the reason for high memory consumption? If so, is there a way not to copy the entire

Re: Directory recursive walking

2021-01-14 Thread dog2002 via Digitalmars-d-learn
On Thursday, 14 January 2021 at 16:47:45 UTC, drug wrote: On 1/14/21 7:06 PM, dog2002 wrote: On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote: On 1/14/21 6:55 PM, drug wrote: But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to

Re: Directory recursive walking

2021-01-14 Thread drug via Digitalmars-d-learn
On 1/14/21 7:06 PM, dog2002 wrote: On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote: On 1/14/21 6:55 PM, drug wrote: But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of

Re: Directory recursive walking

2021-01-14 Thread drug via Digitalmars-d-learn
On 1/14/21 7:30 PM, dog2002 wrote: On Thursday, 14 January 2021 at 16:18:28 UTC, drug wrote: On 1/14/21 7:06 PM, dog2002 wrote: On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote: [...] Yes. I forgot to add it in the original post. Does using `ref` changed anything? Try following:

Re: Directory recursive walking

2021-01-14 Thread dog2002 via Digitalmars-d-learn
On Thursday, 14 January 2021 at 16:18:28 UTC, drug wrote: On 1/14/21 7:06 PM, dog2002 wrote: On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote: [...] Yes. I forgot to add it in the original post. Does using `ref` changed anything? Try following: ``` import std; void

Re: Directory recursive walking

2021-01-14 Thread drug via Digitalmars-d-learn
On 1/14/21 7:06 PM, dog2002 wrote: On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote: On 1/14/21 6:55 PM, drug wrote: But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of

Re: Directory recursive walking

2021-01-14 Thread dog2002 via Digitalmars-d-learn
On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote: On 1/14/21 6:55 PM, drug wrote: But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory? DirEntry is a struct.

Re: Directory recursive walking

2021-01-14 Thread drug via Digitalmars-d-learn
On 1/14/21 6:55 PM, drug wrote: But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory? DirEntry is a struct. First of all I would try this: ```D foreach(ref entry;

Re: Directory recursive walking

2021-01-14 Thread drug via Digitalmars-d-learn
On 1/14/21 6:46 PM, dog2002 wrote: I need to make some operations with all the files in a directory and subdirectories. Currently, I do it like this: import std; void DirIteration(string path) {     try {     foreach(entry; dirEntries(path, SpanMode.shallow, false)) {