Re: [fpc-pascal] Compiling from and to memory
Am 26.04.2012 14:57, schrieb OBones: Jonas Maebe wrote: OBones wrote on Mon, 23 Apr 2012: And because I have to split my generated files so that FPC does not hit the 2GB memory limit (which it does for a 15M source file) What is the structure of that source code? (a few giant routines, large constant arrays, ... ?) Thousands of small routines. Splitting them across multiple units allows to compile, with the write to disk issue. You can use a compiler compiled with EXTDEBUG to get some statistics where the memory goes. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Compiling from and to memory
OBones wrote on Mon, 23 Apr 2012: And because I have to split my generated files so that FPC does not hit the 2GB memory limit (which it does for a 15M source file) What is the structure of that source code? (a few giant routines, large constant arrays, ... ?) Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Compiling from and to memory
Jonas Maebe wrote: OBones wrote on Mon, 23 Apr 2012: And because I have to split my generated files so that FPC does not hit the 2GB memory limit (which it does for a 15M source file) What is the structure of that source code? (a few giant routines, large constant arrays, ... ?) Thousands of small routines. Splitting them across multiple units allows to compile, with the write to disk issue. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Compiling from and to memory
On Thu, 19 Apr 2012, OBones wrote: Hello all, I'm currently creating a build chain where FPC is the last step, the one producing a DLL for Windows, x86 and x64. In front of it in the chain, I have designed a generator that creates a program and a set of units from its own representation. Right now, the program and units are written to disk and then FPC is called on those files. This works fine but is somewhat slow because of disk writing and reading. I was thus wondering if there was a way to send the program and unit contents directly to FPC without writing to disk, and then get FPC to output the final DLL to a memory location as well. The last part is optional, being able to send the program and units without disk access would be a nice thing. I looked around in the documentation for standard input, pipe, but apart from ways to access them from within a program, I was not able to find any information. Is this possible? If yes, how should I proceed? It is currently not possible. FPC always reads from and writes to files. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Compiling from and to memory
On Thu, 19 Apr 2012 14:11:59 +0200 OBones obo...@free.fr wrote: Hello all, I'm currently creating a build chain where FPC is the last step, the one producing a DLL for Windows, x86 and x64. In front of it in the chain, I have designed a generator that creates a program and a set of units from its own representation. Right now, the program and units are written to disk and then FPC is called on those files. This works fine but is somewhat slow because of disk writing and reading. I was thus wondering if there was a way to send the program and unit contents directly to FPC without writing to disk, and then get FPC to output the final DLL to a memory location as well. The last part is optional, being able to send the program and units without disk access would be a nice thing. I looked around in the documentation for standard input, pipe, but apart from ways to access them from within a program, I was not able to find any information. Is this possible? If yes, how should I proceed? Maybe you can create a ramdisk. Mattias ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Compiling from and to memory
michael.vancann...@wisa.be wrote: It is currently not possible. FPC always reads from and writes to files. Fair enough, I'll have to deal with it then. Thank you for your answer ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Compiling from and to memory
Mattias Gaertner wrote: Maybe you can create a ramdisk. Thank you for your answer. I have thought about that, but this would create deployment problems as there is no default ramdisk driver under Windows. Might be worth considering should performance become a really big issue. Regards ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Compiling from and to memory
On Fri, 20 Apr 2012 18:26:28 +0200 Florian Klaempfl flor...@freepascal.org wrote: Am 19.04.2012 14:11, schrieb OBones: Hello all, I'm currently creating a build chain where FPC is the last step, the one producing a DLL for Windows, x86 and x64. In front of it in the chain, I have designed a generator that creates a program and a set of units from its own representation. Right now, the program and units are written to disk and then FPC is called on those files. This works fine but is somewhat slow because of disk writing and reading. I was thus wondering if there was a way to send the program and unit contents directly to FPC without writing to disk, and then get FPC to output the final DLL to a memory location as well. The last part is optional, being able to send the program and units without disk access would be a nice thing. I looked around in the documentation for standard input, pipe, but apart from ways to access them from within a program, I was not able to find any information. Is this possible? If yes, how should I proceed? Do you have any proof that writing/reading to the files to disk is an issue? On modern harddisks and OSes the plain I/O speed is normally not the issue when compiling. True. Lazarus make clean all under Linux 64 bit: ramdisk: real1m8.377s user1m2.584s sys 0m4.212s disk: real1m11.661s user1m5.488s sys 0m4.352s Mattias ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Compiling from and to memory
On 4/19/2012 08:11, OBones wrote: Hello all, I'm currently creating a build chain where FPC is the last step, the one producing a DLL for Windows, x86 and x64. In front of it in the chain, I have designed a generator that creates a program and a set of units from its own representation. Right now, the program and units are written to disk and then FPC is called on those files. This works fine but is somewhat slow because of disk writing and reading. I was thus wondering if there was a way to send the program and unit contents directly to FPC without writing to disk, and then get FPC to output the final DLL to a memory location as well. i'm old school so please forgive this if is is out of bounds... but... how about a RAM disk? do the compiling on that and then copy off to physical media when completed? ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Compiling from and to memory
On 4/20/2012 14:13, Mattias Gaertner wrote: On Fri, 20 Apr 2012 18:26:28 +0200 Florian Klaempflflor...@freepascal.org wrote: [...] Do you have any proof that writing/reading to the files to disk is an issue? On modern harddisks and OSes the plain I/O speed is normally not the issue when compiling. True. Lazarus make clean all under Linux 64 bit: ramdisk: real1m8.377s user1m2.584s sys 0m4.212s disk: real1m11.661s user1m5.488s sys 0m4.352s AFAIK, linux also has much better caching capabilities, too... at least as compared to winwhatever... in one project i work with, there are magnitudes of difference when sufficiently large cache memory is available... i'm talking about compiling a complete and stripped linux based OS with all associated tools and apps for a network security device aimed at dumpster diving acquired hardware, though... still, when we're talking about 100 minutes of RAM based IO compared to 200+ or more minutes of strictly hardware based IO writes, it makes a huge difference ;) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
