[fpc-pascal] Effective memory allocation
Hi All,
I am programming a Bloom Filter and need a high-performance way to allocate
and wipe large block of memory. I did the following test:
program getmem;
{$mode objfpc}{$H+}
uses epiktimer;
const
SIZE = 1024 * 1024 * 1024;
CNT = 10;
var
a: array of Byte;
p: Pointer;
et: TEpikTimer;
i: Integer;
t1, t2: TimerData;
begin
et := TEpikTimer.Create(nil);
et.Clear(t1); et.Clear(t2);
for i := 1 to CNT do begin
et.Start(t1);
p := GetMemory(SIZE);
//SetLength(a, SIZE);
et.Stop(t1);
et.Start(t2);
FillQWord(p^, SIZE div 8, 0);
//FillQWord(a[0], SIZE div 8, 0);
et.Stop(t2);
FreeMem(p);
//a := nil;
end;
WriteLn('Alloc: ', et.Elapsed(t1) / CNT);
WriteLn('Clear: ', et.Elapsed(t2) / CNT);
end.
The result is:
Using GetMemory:
Alloc: 9.40781697E-0001
Clear: 2.13420202E-0001
Using SetLength:
Alloc: 2.8100E-0005
Clear: 7.74975504E-0001
It is understandable that GetMemory() is faster than SetLength(), but why
the difference in FillQWord()?
Also, I usually use pointer to pass block of memory to functions. How do I
implement a function with the following signature:
procedure MyProc(var Buf; Len: Integer):
I mean, how to handle var Buf inside the procedure body?
Thanks!
Xiangrong
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Effective memory allocation
Sorry, the results in previous mail was mis-labeled. The result is: Using SetLength: Alloc: 9.40781697E-0001 Clear: 2.13420202E-0001 Using GetMemory: Alloc: 2.8100E-0005 Clear: 7.74975504E-0001 ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] cocoa programming without objective-pascal mode
Hi, Is it possible to do Cocoa GUI programming on Mac/OSX using FreePascal but without objective-pascal mode/dialect? Just pure object pascal or delphi mode. How do I access the Cocoa SDK (foundation, appkit, etc) from pascal code? GUI programming without GUI IDE is fine with me as long as I could use Cocoa SDK from pascal. I don't want to use Lazarus since it only supports Carbon (32-bit only) which will be deprecated in the next few years. It is also breaks one of the most important advantage of FreePascal: cross platform language. And, are FreePascal internal libraries (packages) e.g. fcl-web, sql-db, etc. applicable on Mac/OSX/Cocoa 64-bit platform? Any hints for those questions are very much appreciated. Thank you. Regards, -- -Bee- I'm an old pascal lover who feels left alone on Apple platform because FreePascal/Lazarus are no longer Apple friendly like they used to be. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] cocoa programming without objective-pascal mode
On 02/11/14 15:43, Bee wrote: I'm an old pascal lover who feels left alone on Apple platform because FreePascal/Lazarus are no longer Apple friendly like they used to be. How was FPC more Apple friendly in the past than it is now? Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] cocoa programming without objective-pascal mode
Minggu, 02 November 2014, Jonas Maebe jonas.ma...@elis.ugent.be menulis: On 02/11/14 15:43, Bee wrote: I'm an old pascal lover who feels left alone on Apple platform because FreePascal/Lazarus are no longer Apple friendly like they used to be. How was FPC more Apple friendly in the past than it is now? Do I really need to answer this? It's just my email signature. :) -- -Bee _ Sent from GMail Mobile via iPod ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] cocoa programming without objective-pascal mode
On 02/11/14 16:37, Bee wrote: Minggu, 02 November 2014, Jonas Maebe jonas.ma...@elis.ugent.be mailto:jonas.ma...@elis.ugent.be menulis: On 02/11/14 15:43, Bee wrote: I'm an old pascal lover who feels left alone on Apple platform because FreePascal/Lazarus are no longer Apple friendly like they used to be. How was FPC more Apple friendly in the past than it is now? Do I really need to answer this? It's just my email signature. :) I assume you didn't put it there in the hope that nobody would ever notice it or ask questions about it. If you did in fact hope that, then there are probably more effective approaches than posting it to the fpc-pascal mailing list, where the people who are personally responsible for this alleged degradation (or at least for not addressing it) are subscribed. Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] cocoa programming without objective-pascal mode
On Sun, Nov 2, 2014 at 3:43 PM, Bee bee.ogra...@gmail.com wrote: Is it possible to do Cocoa GUI programming on Mac/OSX using FreePascal but without objective-pascal mode/dialect? Just pure object pascal or delphi mode. How do I access the Cocoa SDK (foundation, appkit, etc) from pascal code? It is possible, I was doing it before Objective Pascal was created. Just use the functions in the objective-c runtime, read: http://wiki.lazarus.freepascal.org/PasCocoa https://developer.apple.com/library/mac/documentation/Cocoa/Reference/ObjCRuntimeRef/index.html But I warn you that doing this is a terrible choice. Objective Pascal is the way to go. I've been working in Objective Pascal in the LCL Cocoa interface and it's pretty good IMHO. But if you are masochistic you can use the runtime instead of Objective Pascal. I don't want to use Lazarus since it only supports Carbon (32-bit only) which will be deprecated in the next few years. The Cocoa interface is advancing fast. I am even running out of known bugs, so other people testing would be appreciated. It is also breaks one of the most important advantage of FreePascal: cross platform language. I don't see how Lazarus would break cross platform. That's a wierd statement. I'm an old pascal lover who feels left alone on Apple platform because FreePascal/Lazarus are no longer Apple friendly like they used to be. You got the order inverted here. It is not FPC/Lazarus that are less Apple friendly. It is Apple that is each time less Pascal friendly, and for that matter also each time less C/C++/Java friendly. Apple wants you to use Swift (or whatever it's called). -- Felipe Monteiro de Carvalho ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Effective memory allocation
Am 02.11.2014 14:55 schrieb Xiangrong Fang xrf...@gmail.com:
Hi All,
I am programming a Bloom Filter and need a high-performance way to
allocate and wipe large block of memory. I did the following test:
program getmem;
{$mode objfpc}{$H+}
uses epiktimer;
const
SIZE = 1024 * 1024 * 1024;
CNT = 10;
var
a: array of Byte;
p: Pointer;
et: TEpikTimer;
i: Integer;
t1, t2: TimerData;
begin
et := TEpikTimer.Create(nil);
et.Clear(t1); et.Clear(t2);
for i := 1 to CNT do begin
et.Start(t1);
p := GetMemory(SIZE);
//SetLength(a, SIZE);
et.Stop(t1);
et.Start(t2);
FillQWord(p^, SIZE div 8, 0);
//FillQWord(a[0], SIZE div 8, 0);
et.Stop(t2);
FreeMem(p);
//a := nil;
end;
WriteLn('Alloc: ', et.Elapsed(t1) / CNT);
WriteLn('Clear: ', et.Elapsed(t2) / CNT);
end.
The result is:
Using GetMemory:
Alloc: 9.40781697E-0001
Clear: 2.13420202E-0001
Using SetLength:
Alloc: 2.8100E-0005
Clear: 7.74975504E-0001
It is understandable that GetMemory() is faster than SetLength(), but why
the difference in FillQWord()?
If you use SetLength the dynamic array consists not only of the array data,
but also of an information record in front of it. This will likely lead to
the data not being aligned correctly (FillQWord works best with 8-Byte
alignment). So what about testing FillDWord or FillChar? Just to see
whether they would be faster.
Also, I usually use pointer to pass block of memory to functions. How do
I implement a function with the following signature:
procedure MyProc(var Buf; Len: Integer):
I mean, how to handle var Buf inside the procedure body?
You need to cast Buf to a pointer if I remember correctly. Take a look at
the implementation of a TStream's descendant's Write method (e.g.
TStringStream) for an example.
Regards,
Sven
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] cocoa programming without objective-pascal mode
Am 02.11.2014 15:44 schrieb Bee bee.ogra...@gmail.com: It is also breaks one of the most important advantage of FreePascal: cross platform language. How will you be cross platform if you access Cocoa directly? Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Effective memory allocation
On Sun, Nov 2, 2014 at 3:54 PM, Xiangrong Fang xrf...@gmail.com wrote: Also, I usually use pointer to pass block of memory to functions. How do I implement a function with the following signature: procedure MyProc(var Buf; Len: Integer): I mean, how to handle var Buf inside the procedure body? Use @Buf to get a pointer to the address of the untyped variable. If you want to cast that variable to an internal known type and the compiler complains you can always use a pointer cast: PMyKnownType(@Buf)^. You can also use pointer arithmetic (e.g. (PElementType(@Buf) + n)^) if the variable is known to be an array. In Delphi (and probably FPC) it is allowed to pass a nil pointer to an untyped variable if you need to: begin MyProc(nil^, 0); end; The compiler understands what you are trying to do and does not complain about the blatant nil dereference. So var Buf; is functionally equivalent to Buf: Pointer;. OTOH const Buf; is slightly different in the sense that the compiler will not let you pass the untyped variable to another routine that may change its value. -- Constantine ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Effective memory allocation
2014-11-03 2:50 GMT+08:00 Sven Barth pascaldra...@googlemail.com: If you use SetLength the dynamic array consists not only of the array data, but also of an information record in front of it. This will likely lead to the data not being aligned correctly (FillQWord works best with 8-Byte alignment). So what about testing FillDWord or FillChar? Just to see whether they would be faster. It is quite strange that if I use SetLength+FillByte, it is really faster (for the FillByte), but if I use GetMemory+FillByte, it is not faster than using FillDWord or FillQWord. I found this in FPC doc (for $ALIGN switch): This switch is recognized for Turbo Pascal Compatibility, but is not yet implemented. The alignment of data will be different in any case, since Free Pascalis a 32-bit compiler. It is a pity that this switch is not recognized yet. But I need to understand the last statement, will alignment be different or not? Thanks! Xiangrong ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] cocoa programming without objective-pascal mode
On Mon, Nov 3, 2014 at 12:28 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: I assume you didn't put it there in the hope that nobody would ever notice it or ask questions about it. If you did in fact hope that, then there are probably more effective approaches than posting it to the fpc-pascal mailing list, where the people who are personally responsible for this alleged degradation (or at least for not addressing it) are subscribed. I'm sorry, Jonas. I just say what I've felt about FPC/Lazarus on Apple platform. I didn't mean to offend anybody. That footer text is now removed. On Mon, Nov 3, 2014 at 1:38 AM, Felipe Monteiro de Carvalho felipemonteiro.carva...@gmail.com wrote: It is possible, I was doing it before Objective Pascal was created. Just use the functions in the objective-c runtime, read: http://wiki.lazarus.freepascal.org/PasCocoa I thought this one is deprecated. https://developer.apple.com/library/mac/documentation/Cocoa/Reference/ObjCRuntimeRef/index.html But I warn you that doing this is a terrible choice. Objective Pascal is the way to go. I've been working in Objective Pascal in the LCL Cocoa interface and it's pretty good IMHO. But if you are masochistic you can use the runtime instead of Objective Pascal. Yes, it's terrible choice. I'd like to use the Cocoa runtime in OOP way but without the objective-pascal. The Cocoa interface is advancing fast. I am even running out of known bugs, so other people testing would be appreciated. Can I compile Lazarus using Cocoa on Mac? Last time I tried, it failed. I don't see how Lazarus would break cross platform. That's a wierd statement. I meant the objective-pascal dialect, not the LCL. You got the order inverted here. It is not FPC/Lazarus that are less Apple friendly. It is Apple that is each time less Pascal friendly, and for that matter also each time less C/C++/Java friendly. Apple wants you to use Swift (or whatever it's called). No, it's not. Apple owns the whole things, the platform, the API, the language, the IDE, the tools, the market, the device, everything. Including brings the money to us (app developers) through its customers. Apple has all its right to do whatever it wants. If we want to join them, we have to do it by their rules. As simple as that. Apple owes pascal nothing, so to be pascal friendly. If we want pascal to be available on Apple platform, we should change pascal. Not the other way around. And that's where I think the problem comes. The way FPC adapt itself to Apple platform, by creating objective-pascal dialect, is so bad that it breaks one of its most important advantage: cross platform. On Mon, Nov 3, 2014 at 1:52 AM, Sven Barth pascaldra...@googlemail.com wrote: How will you be cross platform if you access Cocoa directly? Take a look at Delphi. From a single source code base, we can compile it for Windows and Mac. Hardly any changes are required. But if you want to target Mac using FPC, I believe you have to use objective-pascal to access Cocoa API which is not compatible with Windows API. That's where the cross platform breaks. Am I right? CMIIW. Another example, take a look at Oxygene --another pascal language variant by RemObjects-- is able to target OSX, iOS, .Net, Mono, and Android natively without ruining their pascal syntax. Why can't FPC be like that? I don't understand because I'm not a compiler guy. :) Any explanation would be very much appreciated. Thank you. Regards, -- -Bee- ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] cocoa programming without objective-pascal mode
On Sun, Nov 2, 2014 at 10:10 PM, Bee bee.ogra...@gmail.com wrote: Take a look at Delphi. From a single source code base, we can compile it for Windows and Mac. Hardly any changes are required. But if you want to target Mac using FPC, I believe you have to use objective-pascal to access Cocoa API which is not compatible with Windows API. That's where the cross platform breaks. Am I right? CMIIW. Another example, take a look at Oxygene --another pascal language variant by RemObjects-- is able to target OSX, iOS, .Net, Mono, and Android natively without ruining their pascal syntax. Why can't FPC be like that? I don't understand because I'm not a compiler guy. :) If you code using LCL APIs only then you'll achieve the desired effect: * you'll get an application that can compile on different platforms without any code changes (write once, compile anywhere) * you won't need to use objective-pascal code. Just don't bother yourself with cocoa api don't go too specific. thanks, Dmitry ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Effective memory allocation
On 03.11.2014 02:59, Xiangrong Fang wrote: 2014-11-03 2:50 GMT+08:00 Sven Barth pascaldra...@googlemail.com mailto:pascaldra...@googlemail.com: If you use SetLength the dynamic array consists not only of the array data, but also of an information record in front of it. This will likely lead to the data not being aligned correctly (FillQWord works best with 8-Byte alignment). So what about testing FillDWord or FillChar? Just to see whether they would be faster. It is quite strange that if I use SetLength+FillByte, it is really faster (for the FillByte), but if I use GetMemory+FillByte, it is not faster than using FillDWord or FillQWord. Then it's indeed as I suspected. The memory buffer itself that is allocated by SetLength is allocated, but the pointer returned to you (what you get as a dynamic array) has - at least for FillQWord - a bad alignment. Looking at the source though it should still be 8-Byte aligned though (the information record has a size of 8 on 32-bit systems)... *sigh* Please also note that not on every system the Fill* methods are implemented in assembler. FillChar mostly is, FillWord and FillDWord perhaps, but FillQWord mostly seldomly (it's e.g. not even implemented that way on x86_64). So especially on 32-bit system 2 32-bit moves are used each. Thus FillQWord is not always the optimal choice and I'd suggest that you time the different Fill* functions especially on different platforms. Would you mind to show the timings that you got for FillChar? :) I found this in FPC doc (for $ALIGN switch): This switch is recognized for Turbo Pascal Compatibility, but is not yet implemented. The alignment of data will be different in any case, since Free Pascalis a 32-bit compiler. It is a pity that this switch is not recognized yet. But I need to understand the last statement, will alignment be different or not? I think that last statement is ment regarding that Turbo Pascal is a 16-bit compiler which had different alignment values available than FPC needs. I don't know what TP supported back then, but I don't imagine that it supported e.g. 16-Byte alignment. Additionally this switch won't help you. The memory buffer that is allocated by SetLength is also allocated using GetMemory. So the buffer itself *is* aligned. But you don't get the start byte of the buffer in case of SetLength, but the first byte after the information record which might not be aligned correctly and *no* compiler switch will help you with that. Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] cocoa programming without objective-pascal mode
On 03.11.2014 04:10, Bee wrote: On Mon, Nov 3, 2014 at 1:52 AM, Sven Barth pascaldra...@googlemail.com mailto:pascaldra...@googlemail.com wrote: How will you be cross platform if you access Cocoa directly? Take a look at Delphi. From a single source code base, we can compile it for Windows and Mac. Hardly any changes are required. But if you want to target Mac using FPC, I believe you have to use objective-pascal to access Cocoa API which is not compatible with Windows API. That's where the cross platform breaks. Am I right? CMIIW. Another example, take a look at Oxygene --another pascal language variant by RemObjects-- is able to target OSX, iOS, .Net, Mono, and Android natively without ruining their pascal syntax. Why can't FPC be like that? I don't understand because I'm not a compiler guy. :) Any explanation would be very much appreciated. Thank you. That's because FireMonkey abstracts all those APIs for you. The LCL by Lazarus has the same target: no matter whether you are on Win32/64, GTK, Qt, Carbon, Cocoa, whatever you get the same set of APIs (the API provided by the LCL) and you don't (normally; bugs not withstanding) need to care about platform differences. Even if you'd directly access Cocoa using Object Pascal means you would not be compatible with Object Pascal codes on other platforms. The same is true for Delphi by the way. They interface with Cocoa code using COM interfaces which are not used on the Windows platform (for this purpose). Two different API sets! Also using Objective Pascal does in no way mean that you can not use Object Pascal. You can mix them as you want and you can use Objective Pascal classes in Object Pascal code and vice versa (otherwise this feature would be rather useless). So please use Objective Pascal. It will save you from quite some trouble to implement the APIs manually (afterall there is a reason why we implemented Objective Pascal). Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Effective memory allocation
On 11/02/2014 03:54 PM, Xiangrong Fang wrote: Hi All, ... Also, I usually use pointer to pass block of memory to functions. How do I implement a function with the following signature: procedure MyProc(var Buf; Len: Integer): I mean, how to handle var Buf inside the procedure body? You can take the address of the Buf variable by using @Buf. For example: procedure MyProc(var Buf; Len: Integer); var I: Integer; P: PByte; begin P := @Buf; for I := 0 to Len - 1 do (P+I)^ := 0; end; Nikolay ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Effective memory allocation
On 02.11.14 15:05, Xiangrong Fang wrote: Sorry, the results in previous mail was mis-labeled. The result is: Using SetLength: Alloc: 9.40781697E-0001 Clear: 2.13420202E-0001 Using GetMemory: Alloc: 2.8100E-0005 Clear: 7.74975504E-0001 SetLength will already clear the array before returning it, so the time for SetLength is approximate time(GetMem) + time(Clear). As your test allocates a gigabyte of memory the OS will not immediatly assign physical memory to your process. The pagetable after a the allocation will contain entries which create faults when accessing the virtual addresses and then the OS will fill those pages in. So on first access a lot of time will be spend inside the OSes memory subsystem. For Linux the newly allocated memory will most likely point to a read only shared physical page which only contains zeros. On the first write to one of those virtual addresses the real page allocation will trigger and Linux assigns a new empty physical page to the virtual memory area. On Linux you can avoid this by passing MAP_POPULATE to mmap when allocating the memory. For SetLength this first access is already done when clearing the memory inside SetLength. For GetMemory it is done when your explicitly access the memory by calling FillQWord later. HTH Nico ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Effective memory allocation
2014-11-03 14:39 GMT+08:00 Sven Barth pascaldra...@googlemail.com: Would you mind to show the timings that you got for FillChar? :) Using FillChar is always about 5% (or less) faster than FillQWord when used with GetMemory, but will be around 20%-40% faster if the memory is allocated by SetLength. Additionally this switch won't help you. The memory buffer that is allocated by SetLength is also allocated using GetMemory. So the buffer itself *is* aligned. But you don't get the start byte of the buffer in case of SetLength, but the first byte after the information record which might not be aligned correctly and *no* compiler switch will help you with that. Then the problem remains with SetLength vs. GetMemory... Why SetLength is about 1x (!) slower than GetMemory? allocating 1G memory took about 0.1 second by SetLength, but is about 1E-5 via GetMemory. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
