Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Nikolay Nikolov]

On 05/27/2017 07:59 AM, Paul Robinson wrote:
Graeme Geldenhuys asked in Vol 108, Issue 27, "What makes a compiler 
project special?"


Well, I'm not a member of the FPC but I've worked on several compilers 
and I'll throw in my 0.02 Euro into the discussion.


> Since Florian mentioned that a compiler project is "rocket science" 
[not his
> direct words, but he hinted at that] and totally different to any 
other software
> project... It has really bugged me... Why is it different, and What 
is different?


I'm going to have to disagree here, and it may simply display my own 
ignorance of the subject, but, then again, even a stopped clock is 
right twice a day.


A compiler is a "language processor," an application that converts 
code in one language into something else. If it's a translating 
compiler it converts it to another language. If it's a language 
compiler it converts it to binary code or potentially to assembly 
language. (I'm making a bit of a distinction in that a compiler that 
translates to assembly code isn't a "translator" because it is using 
the assembler to save some of the work in not "reinventing the wheel" 
and not having to create its own object file writer, and because 
compilers generating assembly are usually creating a finished output 
requiring no manual intervention. Most translators that change source 
from one (high level) language to another produce results that often 
require manual correction. Few translators produce "perfect" 
high-level to high-level conversions without some work. They'll do the 
"heavy lifting" but often minor "tweaks" or checking is required by 
the person.)


At its core, a language processor is a text processing application. It 
takes a fixed combination of rules on what the programmer can and must 
"say" in order to specify the particular actions they want a program 
to accomplish. Given these rules, which are called "grammars" the 
programmer describes the program and the compiler takes that 
description and turns it into the target representation of that 
description.


Well, a mathematician is also a text processor, therefore mathematics is 
simple, right? :)


Nikolay
___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Paul Robinson]

Graeme Geldenhuys asked in Vol 108, Issue 27, "What makes a compiler project 
special?"
Well, I'm not a member of the FPC but I've worked on several compilers and I'll 
throw in my 0.02 Euro into the discussion.
> Since Florian mentioned that a compiler project is "rocket science" [not his 
> direct words, but he hinted at that] and totally different to any other 
> software 
> project... It has really bugged me... Why is it different, and What is 
> different?
I'm going to have to disagree here, and it may simply display my own ignorance 
of the subject, but, then again, even a stopped clock is right twice a day.
A compiler is a "language processor," an application that converts code in one 
language into something else. If it's a translating compiler it converts it to 
another language. If it's a language compiler it converts it to binary code or 
potentially to assembly language. (I'm making a bit of a distinction in that a 
compiler that translates to assembly code isn't a "translator" because it is 
using the assembler to save some of the work in not "reinventing the wheel" and 
not having to create its own object file writer, and because compilers 
generating assembly are usually creating a finished output requiring no manual 
intervention. Most translators that change source from one (high level) 
language to another produce results that often require manual correction. Few 
translators produce "perfect" high-level to high-level conversions without some 
work. They'll do the "heavy lifting" but often minor "tweaks" or checking is 
required by the person.) 

At its core, a language processor is a text processing application. It takes a 
fixed combination of rules on what the programmer can and must "say" in order 
to specify the particular actions they want a program to accomplish. Given 
these rules, which are called "grammars" the programmer describes the program 
and the compiler takes that description and turns it into the target 
representation of that description.
In the case of a translator, it produces a new program in a different language. 
Or it may be the same language but converted to a different dialect, such as a 
translation from a different Pascal compiler, or a conversion from HP Cobol to 
IBM Mainframe Cobol, or conversion from C or Fortran to something newer.
Most language processors have gone to using parser generators in order to 
reduce the work involved in scanning a source language. Some may simply do 
language scanning directly. Most older Pascal compilers used "symbol 
substitution" in which as the language was scanned, it would create a symbol 
identifying what had been found. Whether it was an unrecognized word (which 
would indicate a user identifier), a symbol (like :, >, /, comma, etc) or a 
keyword (USES, UNIT, BEGIN, etc). Then the internal "current symbol" was set to 
the value of that symbol. 

Most compilers had about a 1 byte lookahead so that it could determine if it 
was a single byte symbol (comma, ^, or ' ) or a multibyte symbol which may be 
different depending on the second byte (> followed by an identifier vs. >=, : 
vs :=, < vs <> or <=). Okay, all of this was reasonable until object 
orientation came into use.
When one uses a variable, or a constant, which one are you using? Well, it 
depends on the "scope." If you have I defined in the main program (or in the 
definitions of a UNIT), your reference is to that one. If you're inside a 
procedure, function, method or other similar construct and you define I there, 
it uses that one/ But what if your program - or UNIT - calls several others 
UNITs each having a variable I defined, which one does it use? The first one? 
The last one?
Now, the plot thickens if you reference an object. An identifier in that object 
can be fixed or virtual. In which case, it may not be certain until execution 
time which one is being used, a variable or procedure in the base class or an 
overridden one in a descendant object. So a compiler has to read the tables in 
a unit in order to discover what items are visible and where they are in that 
unit, also to know what kind of variable (or procedure, or function) it is, and 
what is legal to do with it (can't add a 64-bit integer to an 8-bit unsigned 
byte because they're not compatible but you can do it the other way around.)

But this is still the translation of symbols and assigning them attributes 
including whether they are a standalone item (like a unit), a dependent item 
(like a variable in a program) or an internal item (like a field in a record or 
a member of an object.) It requires you keep information about these things but 
I don't think this is any worse than the work involved in a video game in 
holding state information about the game map, the player character (PC), 
non-player characters (NPCs), enemies, objects the player can hold (guns, 
Portal Device, radio) or the use or consume (money, ammo, health).
The last time I did a compile of the full compiler, it 

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Sven Barth via fpc-other]

2017-05-25 21:29 GMT+02:00 Graeme Geldenhuys :
> On 2017-05-25 19:47, Nikolay Nikolov wrote:
> So what is Florian going on about regarding workflow and Git not being able
> to cope in a "compiler" based project? He made it out as if FPC will not be
> workable in a Git managed environment. I don't see his analogy. The Linux
> Kernel running on more platforms than FPC does, is just as complex a beast,
> if not more - considering that the Linux Kernel probably has 10's of
> millions of lines of code, 2000+ contributors. The same could be said for
> the KDE and Qt framework. The latter runs on just about every platform out
> there, as multiple rendering engines, font engines, theme engines, layout
> engines etc.

The workflow of the Linux kernel would simply not be acceptable for us
with them having this hierarchy of maintainers with Linus at the top
and doing all the final merging. If at all an approach like Qt uses
with automatic continous integration of pull requests committed by
core devs or forwarded by them from non-core devs would be the way to
go for us - if at all.

Regards,
Sven
___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[noreply]

On 2017-05-25 16:21, Nikolay Nikolov wrote:

On 05/26/2017 12:16 AM, Graeme Geldenhuys wrote:

On 2017-05-25 22:04, Marco van de Voort wrote:

There are no narrow interfaces that are natural seams for
modularization inside the compiler.


Yet the “packages” and “rtl” directories is just that - which by the 
way is part of the FPC project.

"packages" - yes. But "rtl" is a lot more tightly coupled to the
"compiler" than you think.



Indeed, all the string routines and reference counting is tied to the 
compiler.

I learned this when I had an idea for a new string type..

A string(1000) instead of string(255) or a string(arbitrary). Fixed 
length string that can go beyond 255. Oberon has it. When you introduce 
a new type into the RTL that is not a class, but a type that is part of 
the language itself you have to add all the routines in sysutils to deal 
with this type, change the compiler, make overloaded string routines.


And system.pp is tightly intertwined with the compiler and is almost a 
run time for the language. Not as much like Plain C, but probably even 
Plain C has some connections to the include files and the compiler


Then there is writeln too which is tied to the compiler as it is like a 
varargs, but accepts multiple types. So if you introduce a new 
string(1000) into the compiler, writeln also has to be modified to 
accept this new type as a parameter. But not just writeln, other things 
too.


If you don't have this intertwined and tightly integrated system you 
just end up putting things into a Lisp like system where it is all 
defined by not the compiler/rtl, but the libraries/modules which changes 
the language at run time. Powerful but also double edged sword that 
makes the language redefinable - open for abuse and misuse, such as 
lisp's ability to basically rechange the entire language to any thing 
you want as long as you include (some(brackets)) but everything else is 
up for grabs.

___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[noreply]

On 2017-05-25 16:04, mar...@stack.nl wrote:

   But Florian's statements just bugged me, and I see no proof to
   convince me otherwise - a compiler is just a complex project.
   Nothing "special" as he claimed it to be.


I do think Nikolay's point of it being more interconnected describes it
fairly well. There are no narrow interfaces that are natural seams for
modularization inside the compiler.



I've always wanted to modularize the fpc compiler in such a way that you 
could use the fpc compiler for not just compilation, but 
checking/parsing the code for other reasons, i.e. fpdoc.

But this would likely make the compiler slower.

And another modular compiler idea is that you could embed a relational 
database language inside fpc as a plugin, such as SQL or TutorialD, but 
these are just pipe dreams - and a plugin/modular compiler likely will 
be slower to compile code since there are now more things the compiler 
has to choose to do, more wrappers. But, I don't know for certain.


There was also that strange, but rejected, object oriented plugin 
compiler that someone was working on at one time.. Was it Dodi? I forget 
his name. Maybe the delphi decompiler guy?

___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Florian Klämpfl]

Am 25.05.2017 um 16:18 schrieb Graeme Geldenhuys:
> This is directed at Florian primarily, but any other FPC core member is 
> welcome to chip in.
> 
> different to any other software project... It has really bugged me...
> Why is it different, and What is different?

Because they are basically a mess of spaghetti code and modules :) And this is 
not the writers fault
but simply their nature, they are a very complex piece of software were 
everything is inter winded.
In theory, everything sounds simple: one has a front end consisting of a 
scanner and a parser and a
back end consisting of an optimizer, a code generator and an assembler. Front 
end and back end are
accessing the symbol tables. However, in practice things look different. We try 
to keep fpc layered
and everything, nevertheless, the unit dependency graph looks terrible, see 
attachment (resized, I
can send a full fledged pdf if someone is interested). Simple things often mean 
that one has to dig
upside down and one is often bitten by several side effects or interferences of 
changes. I am really
familiar with the code of FPC, but I do not want to know the hours I spent in 
e.g. r36325 (which is
basically 100 lines of new code, rest is copy and moving code around) due 
to such effects.
Code sample for the "spaghetti" nature, just look at the needtemp function in 
fpc/compiler/ncal.pas
at line 4589: it determines if an argument of an inlined subroutine needs to be 
copied to a temp.
The code is commented very well, it is almost literate programming, everything 
is split into
multiple if statements, nevertheless it is very unreadable. Of course,
EveryIfStatmentCouldBeMovedToAnExtraFunctionWithADescribtiveLongName, but I 
doubt that it makes it
more readable in the sense of being understandable.
___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[noreply]

On 2017-05-25 09:18, Graeme Geldenhuys wrote:

This is directed at Florian primarily, but any other FPC core member
is welcome to chip in.

Since Florian mentioned that a compiler project is "rocket science"
[not his direct words, but he hinted at that] and totally different to
any other software project... It has really bugged me... Why is it
different, and What is different?


Sorry I don't speak as a core, but it is in fact rocket science..

If you look at the compiler sources and try to understand what is going 
on, it's either:

 - a mess
 - rocket science
 - both
Because there are so many things to take care of, probably more than a 
space ship... after all, space ships that have their computers on them, 
require a compiler to compile the code that runs on rockets/space ships. 
So imo compilers are more important than space ships and rockets 
themselves, but this, is speaking from an Alien perspective as I never 
was from this Earth planet.

___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Marco van de Voort]

In our previous episode, Graeme Geldenhuys said:
> 
> Yet the ?packages? and ?rtl? directories is just that - which by the way 
> is part of the FPC project.

Yes, except some of the parts directly connected to the compiler and its
features (like exceptions, RTTI etc)

> And that is also where most commits have been going - based on the history
> I queried for the last 4-6 months. 

That overview is skewed by a high amount of work done on pas2js by Michael
and Mattias. It is atypical, and strictly speaking pas2js is ALSO a
compiler.

Nikolay and Karoly (+ rest of Amiga committers) have been persistently high
this cycle though.

___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Nikolay Nikolov]

On 05/26/2017 12:16 AM, Graeme Geldenhuys wrote:

On 2017-05-25 22:04, Marco van de Voort wrote:

There are no narrow interfaces that are natural seams for
modularization inside the compiler.


Yet the “packages” and “rtl” directories is just that - which by the 
way is part of the FPC project.
"packages" - yes. But "rtl" is a lot more tightly coupled to the 
"compiler" than you think.


Nikolay
___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Graeme Geldenhuys]

On 2017-05-25 22:04, Marco van de Voort wrote:

There are no narrow interfaces that are natural seams for
modularization inside the compiler.


Yet the “packages” and “rtl” directories is just that - which by the way 
is part of the FPC project. And that is also where most commits have 
been going - based on the history I queried for the last 4-6 months. And 
then in the “compiler” tree their is the nice natural seams for various 
platforms and cpu types - after all, you guys worked so hard to make FPC 
2.x more modular and maintainable compared to the spaghetti mess of v1.x.


Anyway. I'll keep thinking happy thoughts.

Regards,
  Graeme

___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Marco van de Voort]

In our previous episode, Graeme Geldenhuys said:


>Just to be clear, I'm not pushing Git here - I know you guys will
>not change - Florian made that very clear.

Yes, boundless leaps of faith are out of the question. Git should be a tool,
not a religion.

>But Florian's statements just bugged me, and I see no proof to
>convince me otherwise - a compiler is just a complex project.
>Nothing "special" as he claimed it to be.

I do think Nikolay's point of it being more interconnected describes it
fairly well. There are no narrow interfaces that are natural seams for
modularization inside the compiler.

___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Florian Klämpfl]

Am 25.05.2017 um 21:29 schrieb Graeme Geldenhuys:
> 
> So what is Florian going on about regarding workflow and Git not being able 
> to cope in a "compiler"
> based project? He made it out as if FPC will not be workable in a Git managed 
> environment. I don't
> see his analogy. The Linux Kernel running on more platforms than FPC does, is 
> just as complex a
> beast, 

It is not complex in the sense Nikolay described.

>   I see no proof to
>   convince me otherwise 
> - a compiler is just a complex project.
>   Nothing "special" as he claimed it to be.

See, this is the reason why I do not believe you. You simply cannot understand 
the problems of a
compiler project which requires small linear steps which can be easily reviewed.

And it seems LLVM likes this small steps as well:
http://lists.llvm.org/pipermail/llvm-dev/2011-July/041781.html

___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Graeme Geldenhuys]

On 2017-05-25 19:47, Nikolay Nikolov wrote:

The answer is: much higher complexity and much tighter coupling between
the different components. Everything depends on everything, basically.
And all of that's caused by necessity, not by bad design, because the
task you're solving is very complex.


Ok, so the complexity is in the project code (I never argued or disputed 
that). Many projects have complex code and dependencies - not every 
project is a simple desktop app. eg: You wouldn't believe how complex 
Mortgage Brokerage applications can be. The business logic is insanely 
complex with exceptions to every rule. FPC or any other compiler is not 
that "special" then - it's like any other complex project.


So what is Florian going on about regarding workflow and Git not being 
able to cope in a "compiler" based project? He made it out as if FPC 
will not be workable in a Git managed environment. I don't see his 
analogy. The Linux Kernel running on more platforms than FPC does, is 
just as complex a beast, if not more - considering that the Linux Kernel 
probably has 10's of millions of lines of code, 2000+ contributors. The 
same could be said for the KDE and Qt framework. The latter runs on just 
about every platform out there, as multiple rendering engines, font 
engines, theme engines, layout engines etc.


ps:
  Just to be clear, I'm not pushing Git here - I know you guys will
  not change - Florian made that very clear.
  But Florian's statements just bugged me, and I see no proof to
  convince me otherwise - a compiler is just a complex project.
  Nothing "special" as he claimed it to be.

Regards,
  Graeme

___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

Re: [fpc-other] What makes a Compiler project (like FPC) special?

[Nikolay Nikolov]

On 05/25/2017 05:18 PM, Graeme Geldenhuys wrote:
This is directed at Florian primarily, but any other FPC core member 
is welcome to chip in.


Since Florian mentioned that a compiler project is "rocket science" 
[not his direct words, but he hinted at that] and totally different to 
any other software project... It has really bugged me... Why is it 
different, and What is different?
I'm not Florian, but I also have done extensive work on the compiler 
(including the 16-bit port).


The answer is: much higher complexity and much tighter coupling between 
the different components. Everything depends on everything, basically. 
And all of that's caused by necessity, not by bad design, because the 
task you're solving is very complex.


Of course, this applies to compilers of the scale of FPC. You can find 
much simpler and easier designs in compiler development textbooks, but they:


1) implement a trivial language
2) support only a single platform
3) do very minimal or no optimization

Nikolay
___
fpc-other maillist  -  fpc-other@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other