Re: Any easy way to extract files to memory buffer?
On Monday, 18 March 2019 at 23:40:02 UTC, Michelle Long wrote: On Monday, 18 March 2019 at 23:01:27 UTC, H. S. Teoh wrote: On Mon, Mar 18, 2019 at 10:38:17PM +, Michelle Long via Digitalmars-d-learn wrote: On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev wrote: > On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long > wrote: > > Trying to speed up extracting some files that I first > > have to extract using the command line to files then read > > those in... > > > > Not sure what is taking so long. I imagine windows caches > > the extraction so maybe it is pointless? [...] Why not just use std.mmfile to memory-map the file into memory directly? Let the OS take care of actually paging in the file data. T The files are on disk and there is an external program that read them and converts them and then writes the converted files to disk then my program reads. Ideally the conversion program would take memory instead of disk files but it doesn't. the file that was written by the first program will be in the file cache. mmap() (and the Windows equivalent of that) syscalls are at the core only giving access to the OS file cache. This means that std.mmfile is the way to go. There will be no reloading from disk if the file sizes are within reason.
Re: Any easy way to extract files to memory buffer?
On Monday, 18 March 2019 at 23:01:27 UTC, H. S. Teoh wrote: On Mon, Mar 18, 2019 at 10:38:17PM +, Michelle Long via Digitalmars-d-learn wrote: On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev wrote: > On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long > wrote: > > Trying to speed up extracting some files that I first have > > to extract using the command line to files then read those > > in... > > > > Not sure what is taking so long. I imagine windows caches > > the extraction so maybe it is pointless? [...] Why not just use std.mmfile to memory-map the file into memory directly? Let the OS take care of actually paging in the file data. T The files are on disk and there is an external program that read them and converts them and then writes the converted files to disk then my program reads. Ideally the conversion program would take memory instead of disk files but it doesn't.
Re: Any easy way to extract files to memory buffer?
On Mon, Mar 18, 2019 at 10:38:17PM +, Michelle Long via Digitalmars-d-learn wrote: > On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev wrote: > > On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote: > > > Trying to speed up extracting some files that I first have to > > > extract using the command line to files then read those in... > > > > > > Not sure what is taking so long. I imagine windows caches the > > > extraction so maybe it is pointless? [...] Why not just use std.mmfile to memory-map the file into memory directly? Let the OS take care of actually paging in the file data. T -- Give me some fresh salted fish, please.
Re: Any easy way to extract files to memory buffer?
On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev wrote: On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote: Trying to speed up extracting some files that I first have to extract using the command line to files then read those in... Not sure what is taking so long. I imagine windows caches the extraction so maybe it is pointless? You can speed up such things using a tmpfs or RAM disk. On Linux you just use mount -t tmpfs ... On Windows it is a little more complicated, there are programs like ImDisk that can create RAM drives (which can then be formatted to whatever). If it's just one file, sometimes you can pipe it from the unpacking program (tar etc.) into the consuming program. Yeah, but it seems like a lot of work simply to extract the files to memory.
Re: Any easy way to extract files to memory buffer?
On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote: Trying to speed up extracting some files that I first have to extract using the command line to files then read those in... Not sure what is taking so long. I imagine windows caches the extraction so maybe it is pointless? You can speed up such things using a tmpfs or RAM disk. On Linux you just use mount -t tmpfs ... On Windows it is a little more complicated, there are programs like ImDisk that can create RAM drives (which can then be formatted to whatever). If it's just one file, sometimes you can pipe it from the unpacking program (tar etc.) into the consuming program.
Any easy way to extract files to memory buffer?
Trying to speed up extracting some files that I first have to extract using the command line to files then read those in... Not sure what is taking so long. I imagine windows caches the extraction so maybe it is pointless?
