Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 3:31 PM, R0b0t1 wrote: > On Mon, Jul 30, 2018 at 11:42 AM, Marcos Douglas B. Santos > wrote: >> On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 wrote: >> >> [...] > > If a program isn't long running I see programmers tend to not care > about memory management, nor do I make a habit of caring about it > myself. I find it hard to see anything wrong with this. > > As someone has mentioned this can fail on smaller computers, but how > much RAM is too little? SBCs now typically come with 1-4GB of memory. I'm sorry, but we work with a language that expected programmers should manage their objects. The language even provides a clean way (try-finall) to do that. I use try-finally even to write a snippet to show an example of code... IMO, that must be a habit in such languages. regards, Marcos Douglas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On 07/30/2018 11:35 AM, Michael Van Canneyt wrote: On Mon, 30 Jul 2018, R0b0t1 wrote: It might be hard to imagine FPC taking that much longer than it does currently but ~30min for a large program is the standard with other compilers. I very much enjoy the speed of FPC. > 30 *Minutes*, is this real ? yep, for sufficiently large projects... when we compile OSG (OpenSceneGraph) over here (using -j 8) it takes 20+ minutes for the first time or if we reconfigure from debug to release mode or the other way around... now, after the first time we compile it, the next times takes mere 10's of seconds... then we move on to SceneGear and FlightGear when we're building that whole project which uses OSG... 30-45 minutes for a complete, from the ground up builds are not uncommon... it takes me back to the days of 30 years ago when you'd start the (borland/turbo pascal) compile process and go get some c0ffee ;) note: the machines these builds are being done on have 16Gig RAM, 1Tb HD, and a 4Ghz 8-core AMD FX-8350 CPU... -- NOTE: No off-list assistance is given without prior approval. *Please keep mailing list traffic on the list unless* *a signed and pre-paid contract is in effect with us.* ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 11:42 AM, Marcos Douglas B. Santos wrote: > On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 wrote: >> On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote: >>> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: Is performance more important than being correct? :| >>> In this project, the answer is always taken to be yes. >>> >> >> To hopefully offer some explanation for Mr. Santos - if I can compile >> and run the program 5 times over the course of 20 minutes, that is >> certainly better than running it and testing it once over the same >> amount of time, no? >> >> It might be hard to imagine FPC taking that much longer than it does >> currently but ~30min for a large program is the standard with other >> compilers. I very much enjoy the speed of FPC. > > All us like fast compiling... > However, IMHO, a compiler should use the same good practices that > others programs must use. > If it cannot use them because it's very slow, what is the matter about > good practices? > > "Do what I'm telling you to do, but don't do what I do" ? :) > If a program isn't long running I see programmers tend to not care about memory management, nor do I make a habit of caring about it myself. I find it hard to see anything wrong with this. As someone has mentioned this can fail on smaller computers, but how much RAM is too little? SBCs now typically come with 1-4GB of memory. Cheers, R0b0t1 ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 10:24 AM, Michael Van Canneyt wrote: > > > On Mon, 30 Jul 2018, Sven Barth via fpc-devel wrote: > >> Michael Van Canneyt schrieb am Mo., 30. Juli >> 2018, >> 15:30: >> >>> Obviously provided you don't install another mechanism that does this and >>> don't raise exceptions manually, which - AFAIK - is the case in the >>> compiler... >>> >> >> The compiler does use exceptions when the compilation needs to be aborted >> e.g. when too many errors occurred or an internal error is encountered. > > > This is bad news, because it in effect means that the try/finally is > necessary > if people want to fix memleaks :( > > This will considerably slow down the compiler. > I would very much like to see the compiler usable as a dynamic library. Compartmentalizing the bulk of the code in this manner might aid readability and comprehension even if it is not immediately wise to actually use the shared object in a third party application due to memory leaks. I wouldn't mind running some static analysis on FPC in an attempt to get rid of memory leaks. It's fairly mechanical and probably a good introduction to the codebase. This page[1] is promising as it indicates valgrind should work well on FPC binaries (I didn't have much a reason to expect otherwise, but you never know). Is there any reason this hasn't been done? What could raise statements be replaced with if they must be replaced? Cheers, R0b0t1 [1]: http://wiki.lazarus.freepascal.org/Profiling ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 10:35 AM, Michael Van Canneyt wrote: > > > On Mon, 30 Jul 2018, R0b0t1 wrote: > >> On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote: >>> >>> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: Is performance more important than being correct? :| >>> >>> In this project, the answer is always taken to be yes. >>> >> >> To hopefully offer some explanation for Mr. Santos - if I can compile >> and run the program 5 times over the course of 20 minutes, that is >> certainly better than running it and testing it once over the same >> amount of time, no? >> >> It might be hard to imagine FPC taking that much longer than it does >> currently but ~30min for a large program is the standard with other >> compilers. I very much enjoy the speed of FPC. > > > 30 *Minutes*, is this real ? > > Hm. I just complained to a component provider because Delphi takes 25 > seconds to compile my program with the way they set up their paths... > This is real.[1] I've experienced it myself and you can see some good discussions on https://news.ycombinator.com with a search. FPC/Lazarus has made some waves there in the last few years. FPC does have quite a bit of a lead it could shed in an attempt to be "more correct" but I would caution against this. If at every point in a software stack someone says "modern computers are just so fast, we have a little bit of wiggle room" you end up with something slower than molasses. Perfection requires effort at every single step. A good case study on this is keyboard latency in modern systems.[2] Cheers, R0b0t1 [1]: The most extreme comparison was closer to being something like a couple of minutes vs. a couple of hours, but I can't source that right now. Most people think it's a lie; granted, some of it is likely due to the excellent incremental compilation FPC has by default. [2]: https://danluu.com/keyboard-latency/ ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, R0b0t1 wrote: On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote: Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: Is performance more important than being correct? :| In this project, the answer is always taken to be yes. To hopefully offer some explanation for Mr. Santos - if I can compile and run the program 5 times over the course of 20 minutes, that is certainly better than running it and testing it once over the same amount of time, no? It might be hard to imagine FPC taking that much longer than it does currently but ~30min for a large program is the standard with other compilers. I very much enjoy the speed of FPC. 30 *Minutes*, is this real ? Hm. I just complained to a component provider because Delphi takes 25 seconds to compile my program with the way they set up their paths... Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote: > Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: >> Is performance more important than being correct? :| > In this project, the answer is always taken to be yes. > To hopefully offer some explanation for Mr. Santos - if I can compile and run the program 5 times over the course of 20 minutes, that is certainly better than running it and testing it once over the same amount of time, no? It might be hard to imagine FPC taking that much longer than it does currently but ~30min for a large program is the standard with other compilers. I very much enjoy the speed of FPC. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, Sven Barth via fpc-devel wrote: Michael Van Canneyt schrieb am Mo., 30. Juli 2018, 15:30: Obviously provided you don't install another mechanism that does this and don't raise exceptions manually, which - AFAIK - is the case in the compiler... The compiler does use exceptions when the compilation needs to be aborted e.g. when too many errors occurred or an internal error is encountered. This is bad news, because it in effect means that the try/finally is necessary if people want to fix memleaks :( This will considerably slow down the compiler. Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
Michael Van Canneyt schrieb am Mo., 30. Juli 2018, 15:30: > Obviously provided you don't install another mechanism that does this and > don't raise exceptions manually, which - AFAIK - is the case in the > compiler... > The compiler does use exceptions when the compilation needs to be aborted e.g. when too many errors occurred or an internal error is encountered. Regards, Sven > ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, Mattias Gaertner wrote: On Mon, 30 Jul 2018 14:36:10 +0200 (CEST) Michael Van Canneyt wrote: [...] Many people will probably be surprised, but the use of try/finally is not required for an application: Gradually, some parts of the compiler were switched to classes, but even then, without exceptions: As long as you do not use the sysutils unit, (which was the case for many years for the compiler) exceptions will not be raised, and so there is simply no need for try/finally blocks. Many people will probably surprised, that try/finally/except works without sysutils and try/except does not need "Exception". It works with any TObject. Probably it would be better to remove the dependency on sysutils, to keep the compiler free of try/finally blocks. Sysutils has nothing to do with creating exception frames. It does: if you know the program will never raise exceptions, it makes no sense to install try/finally blocks. Runtime errors are converted to exceptions by the sysutils unit. By not including it, you know you will just get runtime errors, and hence you don't need try..finally. Obviously provided you don't install another mechanism that does this and don't raise exceptions manually, which - AFAIK - is the case in the compiler... Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018 14:36:10 +0200 (CEST) Michael Van Canneyt wrote: >[...] > Many people will probably be surprised, but the use of try/finally > is not required for an application: > > Gradually, some parts of the compiler were switched to classes, but even > then, without exceptions: As long as you do not use the sysutils unit, > (which was the case for many years for the compiler) exceptions will > not be raised, and so there is simply no need for try/finally blocks. Many people will probably surprised, that try/finally/except works without sysutils and try/except does not need "Exception". It works with any TObject. > Probably it would be better to remove the dependency on sysutils, to keep > the compiler free of try/finally blocks. Sysutils has nothing to do with creating exception frames. Mattias ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
From: fpc-devel On Behalf Of Marcos
Douglas B. Santos
Sent: Monday, 30 July 2018 22:25
To: FPC developers' list
Subject: Re: [fpc-devel] Why/how does the compiler have a non-trivial number
ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel
mailto:fpc-devel@lists.freepascal.org> > wrote:
J. Gareth Moreton mailto:gar...@moreton-family.com>
> schrieb am Mo., 30. Juli 2018, 13:31:
I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.
Correct. Even implicit try-finally frame generation is disabled for the
compiler source.
Is performance more important than being correct? :|
Freeing memory is irrelevant if your process is going to exit anyway right
after freeing it. The OS doesn’t care about any memory your process has
allocated at that time. All pages go back into the available pool.
Depending on how the allocation/free behaviour of the compiler looks like, but
it might even make sense to use a memory manager that doesn’t even know how to
free memory.
(I’m using windows terminology here, but I’m sure equivalent can be found on
other platforms)
Reserve a large block of address space.
Install a vectored exception handler that reacts to an access violation that
falls inside the reserved block by committing the memory between start and
error address + 1MB (or more or less).
Naïvely and without error checking, Alloc gets reduced to:
var
CurrentPtr: PByte;
function Alloc(Size: Integer):Pointer;
begin
Size := (Size + 3) and not 3; //adjust depending on what allocation alignment
is desired;
If Size > 0 then begin
Result := CurrentPtr;
CurrentPtr := CurrentPtr + Size;
end else
Result := nil;
end;
procedure Free(p: Pointer);
begin
{do nothing}
end;
___
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, Marcos Douglas B. Santos wrote: On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel < fpc-devel@lists.freepascal.org> wrote: J. Gareth Moreton schrieb am Mo., 30. Juli 2018, 13:31: I've noticed that the compiler doesn't use try...finally blocks to help with freeing blocks. I'm not sure why this is the case, but might be speed related. Correct. Even implicit try-finally frame generation is disabled for the compiler source. Is performance more important than being correct? :| You've been around long enough to know that complaints about the speed of the compiler is something that pops up on a regular basis. Also, and this is important: the compiler was developed for TP. It had no concept of exceptions. Everything was done using records and pointers. Many people will probably be surprised, but the use of try/finally is not required for an application: Gradually, some parts of the compiler were switched to classes, but even then, without exceptions: As long as you do not use the sysutils unit, (which was the case for many years for the compiler) exceptions will not be raised, and so there is simply no need for try/finally blocks. Probably it would be better to remove the dependency on sysutils, to keep the compiler free of try/finally blocks. But the compiler devs will be able to provide a more correct picture than this. Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: > Is performance more important than being correct? :| In this project, the answer is always taken to be yes. -- Regards, Martok ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel < fpc-devel@lists.freepascal.org> wrote: > J. Gareth Moreton schrieb am Mo., 30. Juli > 2018, 13:31: > >> I've noticed that the compiler doesn't use >> try...finally blocks to help with freeing >> blocks. I'm not sure why this is the case, >> but might be speed related. >> > > Correct. Even implicit try-finally frame generation is disabled for the > compiler source. > Is performance more important than being correct? :| regards, Marcos Douglas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
J. Gareth Moreton schrieb am Mo., 30. Juli 2018, 13:31: > I've noticed that the compiler doesn't use > try...finally blocks to help with freeing > blocks. I'm not sure why this is the case, > but might be speed related. > Correct. Even implicit try-finally frame generation is disabled for the compiler source. Regards, Sven > ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, J. Gareth Moreton wrote: On Mon 30/07/18 01:58 , Ben Grasset operato...@gmail.com sent: I was even able to successfully actually fix some of the leaks, but others boiled down to a "node" being created as a local variable in the middle of some 1000+ line method, getting assigned to the "left" or "right" property of some other globally visible node, and then never being freed. To add to that, some of then actually *could not* be freed even when I tried to do so without raising an untraceable access violation later in the compilation process, also, which means the compiler is basically knowingly relying on undebuggable undefined behaviour in multiple places that could easily be affected or altered by anything else anywhere else in the codebase at any time. The way-too-short, highly undescriptive naming of variables doesn't exactly help, either! "hp" and "p" are not good variable names. They really aren't! Overall, I'm not trying to put blame on anyone in particular here, but as someone who loves using FPC and would like to see it continue to grow, it would be nice if we could collectively address some of these basic, fundamental issues (which I'd be happy to help with myself, of course.) I too love Free Pascal and seek to improve it. If you've found a fix, by all means submit it as a patch in the bug tracker. I've noticed that the compiler doesn't use try...finally blocks to help with freeing blocks. I'm not sure why this is the case, but might be speed related. Yes. try/finally blocks cause *tremendous* slowdown. Michael.___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon 30/07/18 01:58 , Ben Grasset operato...@gmail.com sent: > I was even able to successfully actually fix some of the leaks, but others > boiled down to a "node" being created as a local variable in the middle of > some 1000+ line method, getting assigned to the "left" or "right" property > of some other globally visible node, and then never being freed. > To add to that, some of then actually *could not* be freed even when I > tried to do so without raising an untraceable access violation later in the > compilation process, also, which means the compiler is basically knowingly > relying on undebuggable undefined behaviour in multiple places that could > easily be affected or altered by anything else anywhere else in the > codebase at any time. > The way-too-short, highly undescriptive naming of variables doesn't > exactly help, either! "hp" and "p" are not good variable names. They really > aren't! > > Overall, I'm not trying to put blame on anyone in particular here, but as > someone who loves using FPC and would like to see it continue to grow, it > would be nice if we could collectively address some of these basic, > fundamental issues (which I'd be happy to help with myself, of course.) > I too love Free Pascal and seek to improve it. If you've found a fix, by all means submit it as a patch in the bug tracker. I've noticed that the compiler doesn't use try...finally blocks to help with freeing blocks. I'm not sure why this is the case, but might be speed related. More complex refactoring might require deeper planning and approval from Florian. For example, the node construction relies a lot on global variables (especially current_procinfo) that could possibly use some improvement. I think a big issue with refactoring is the amount of testing required because of the numerous target platforms. Don't be disheartened though. With enough eyes, all bugs are shallow! Gareth aka. Kit ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
