On Saturday, 19 May 2012 at 20:35:10 UTC, Marco Leise wrote:
Am Fri, 11 May 2012 10:01:28 +0200
schrieb Roman D. Boiko r...@d-coding.com:
There were several discussions about the need for a D compiler
library.
I propose my draft implementation of lexer for community
review:
Am Sun, 20 May 2012 10:09:34 +0200
schrieb Roman D. Boiko r...@d-coding.com:
Could you name a few specific concerns?
Mostly my own gut feeling, that things that sound great in my head turn out to
bite me in the end. Things that one just doesn't think of because of the
limited horizon everyone
On Sunday, 20 May 2012 at 17:42:59 UTC, Marco Leise wrote:
There is one feature I remember caused some head-aches for
Code::Blocks. They used a separate parser instance for every
project in the IDE, which meant that all the standard include
files would be parsed and kept in memory multiple
Am Sun, 20 May 2012 20:37:07 +0200
schrieb Roman D. Boiko r...@d-coding.com:
Since assembly code is usually small I just preallocate an
array of sourceCode.length tokens and realloc it to the
correct size when I'm done parsing. Nothing pretty, but
simple and I am sure it won't get
Am Fri, 11 May 2012 10:01:28 +0200
schrieb Roman D. Boiko r...@d-coding.com:
There were several discussions about the need for a D compiler
library.
I propose my draft implementation of lexer for community review:
https://github.com/roman-d-boiko/dct
Lexer is based on Brian Schott's
Le 15/05/2012 21:59, Roman D. Boiko a écrit :
On Tuesday, 15 May 2012 at 19:27:26 UTC, Timon Gehr wrote:
On 05/14/2012 05:00 PM, Roman D. Boiko wrote:
Currently I think about making token a class instead of struct.
...
Could anybody suggest other pros and cons? Which option would you
choose?
Le 14/05/2012 21:21, Roman D. Boiko a écrit :
On Monday, 14 May 2012 at 19:13:39 UTC, Roman D. Boiko wrote:
On Monday, 14 May 2012 at 19:04:20 UTC, Tove wrote:
What if there were two different lex:er modes... with different
struct:s.
1. For an IDE with on the fly lexing:
Assumption, the error
On Tuesday, 15 May 2012 at 09:33:31 UTC, deadalnix wrote:
Le 14/05/2012 21:21, Roman D. Boiko a écrit :
Just to clarify: different modes in lexer in my view are like
two
different implementations combined in a non-trivial way (unless
the difference is minor). So complexity goes from two
On 05/14/2012 05:00 PM, Roman D. Boiko wrote:
On Saturday, 12 May 2012 at 03:32:20 UTC, Ary Manzana wrote:
I think you are wasting much more memory and performance by storing
all the tokens in the lexer.
Imagine I want to implement a simple syntax highlighter: just
highlight keywords. How can
On Tuesday, 15 May 2012 at 19:27:26 UTC, Timon Gehr wrote:
On 05/14/2012 05:00 PM, Roman D. Boiko wrote:
Currently I think about making token a class instead of struct.
...
Could anybody suggest other pros and cons? Which option would
you choose?
Just use a circular buffer of value-type
On Saturday, 12 May 2012 at 03:32:20 UTC, Ary Manzana wrote:
I think you are wasting much more memory and performance by
storing all the tokens in the lexer.
Imagine I want to implement a simple syntax highlighter: just
highlight keywords. How can I tell DCT to *not* store all
tokens because
On Monday, 14 May 2012 at 15:00:37 UTC, Roman D. Boiko wrote:
Could anybody suggest other pros and cons? Which option would
you choose?
Further discussion on this topic (struct vs class) is at
http://forum.dlang.org/thread/asdrqlaydzcdpqwsb...@forum.dlang.org
On Monday, 14 May 2012 at 16:30:21 UTC, deadalnix wrote:
Le 14/05/2012 17:00, Roman D. Boiko a écrit :
Making it a class would give several benefits:
* allow not to worry about allocating a big array of tokens.
E.g., on
64-bit OS the largest module in Phobos (IIRC, the
std.datetime) consumes
On Monday, 14 May 2012 at 16:58:42 UTC, Roman D. Boiko wrote:
You are over engineering the whole stuff.
I'm trying to solve this and other tradeoffs. I'd like to
simplify but satisfy my design goals.
What if there were two different lex:er modes... with different
struct:s.
1. For an IDE
On Monday, 14 May 2012 at 19:04:20 UTC, Tove wrote:
On Monday, 14 May 2012 at 16:58:42 UTC, Roman D. Boiko wrote:
You are over engineering the whole stuff.
I'm trying to solve this and other tradeoffs. I'd like to
simplify but satisfy my design goals.
What if there were two different lex:er
On Monday, 14 May 2012 at 19:13:39 UTC, Roman D. Boiko wrote:
On Monday, 14 May 2012 at 19:04:20 UTC, Tove wrote:
What if there were two different lex:er modes... with
different struct:s.
1. For an IDE with on the fly lexing:
Assumption, the error rate is high.(need to keep much info)
2.
On Saturday, 12 May 2012 at 05:41:16 UTC, Jonathan M Davis wrote:
It's great that you're working on this. We definitely need more
of this sort of
stuff. However, I would point out that I'm not sure that it
will be acceptable
for Phobos (it _may_ be, but it's not quite what we've been
looking
On 5/12/12 12:17 PM, Roman D. Boiko wrote:
On Saturday, 12 May 2012 at 03:32:20 UTC, Ary Manzana wrote:
As deadalnix says, I think you are over-complicating things.
I mean, to store the column and line information it's just:
if (isNewLine(c)) {
line++;
column = 0;
} else {
column++;
}
(I
Le 11/05/2012 13:50, Ary Manzana a écrit :
On 5/11/12 4:22 PM, Roman D. Boiko wrote:
What about line and column information?
Indices of the first code unit of each line are stored inside lexer and
a function will compute Location (line number, column number, file
specification) for any index.
On Saturday, 12 May 2012 at 13:24:11 UTC, deadalnix wrote:
Le 11/05/2012 13:50, Ary Manzana a écrit :
Usually tokens are used and discarded. I mean, somebody that
uses the
lexer asks tokens, process them (for example to highlight code
or to
build an AST) and then discards them. So you can
On Saturday, 12 May 2012 at 20:28:34 UTC, Tobias Pankrath wrote:
1. If a struct is a field of heap allocated object, it will be
allocated and garbage collected. Only if it only exists on
stack
(i.e., in method body), GC is not used.
As far as I can tell, it won't be allocated on it's own,
On 05/12/2012 11:08 PM, Roman D. Boiko wrote:
On Saturday, 12 May 2012 at 20:28:34 UTC, Tobias Pankrath wrote:
1. If a struct is a field of heap allocated object, it will be
allocated and garbage collected. Only if it only exists on stack
(i.e., in method body), GC is not used.
As far as I
On Saturday, 12 May 2012 at 22:19:55 UTC, Timon Gehr wrote:
On 05/12/2012 11:08 PM, Roman D. Boiko wrote:
On Saturday, 12 May 2012 at 20:28:34 UTC, Tobias Pankrath
wrote:
1. If a struct is a field of heap allocated object, it will
be
allocated and garbage collected. Only if it only exists on
There were several discussions about the need for a D compiler
library.
I propose my draft implementation of lexer for community review:
https://github.com/roman-d-boiko/dct
Lexer is based on Brian Schott's project
https://github.com/Hackerpilot/Dscanner, but it has been
refactored and
On 2012-05-11 10:01, Roman D. Boiko wrote:
There were several discussions about the need for a D compiler library.
I propose my draft implementation of lexer for community review:
https://github.com/roman-d-boiko/dct
Lexer is based on Brian Schott's project
On Friday, 11 May 2012 at 08:38:36 UTC, Jacob Carlborg wrote:
(Re-posting here)
A couple of questions:
* What's the sate of the lexer
I consider it a draft state, because it has got several rewrites
recently and I plan to do more, especially based on community
feedback. However, implementation
On 2012-05-11 10:01, Roman D. Boiko wrote:
There were several discussions about the need for a D compiler library.
I propose my draft implementation of lexer for community review:
https://github.com/roman-d-boiko/dct
Lexer is based on Brian Schott's project
On 2012-05-11 10:58, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 08:38:36 UTC, Jacob Carlborg wrote:
(Re-posting here)
A couple of questions:
* What's the sate of the lexer
I consider it a draft state, because it has got several rewrites
recently and I plan to do more, especially based on
Am 11.05.2012 11:02, schrieb Jacob Carlborg:
For this to happen, for Walter to start using this, I think there would
be a greater change if the frontend was a port of the DMD frontend and
not changed too much.
or a pure D version of it with the features:
-very fast in parsing/lexing - there
On Friday, 11 May 2012 at 09:08:24 UTC, Jacob Carlborg wrote:
On 2012-05-11 10:58, Roman D. Boiko wrote:
Each token contains:
* start index (position in the original encoding, 0 corresponds
to the first code unit after BOM),
* token value encoded as UTF-8 string,
* token kind (e.g., token.kind
Am 11.05.2012 11:23, schrieb Roman D. Boiko:
On Friday, 11 May 2012 at 09:19:07 UTC, dennis luehring wrote:
does the parser/lexer allow half-finished syntax parsing? for
being useable in an IDE for syntax-highlighting while coding?
That's planned, but I would like to see your usage scenarios
On Friday, 11 May 2012 at 09:02:12 UTC, Jacob Carlborg wrote:
If think that the end goal of a project like this, putting a D
frontend in Phobos, should be that the compiler should be built
using this library. This would result in the compiler and
library always being in sync and having the
On Friday, 11 May 2012 at 09:21:29 UTC, dennis luehring wrote:
Am 11.05.2012 11:02, schrieb Jacob Carlborg:
For this to happen, for Walter to start using this, I think
there would
be a greater change if the frontend was a port of the DMD
frontend and
not changed too much.
or a pure D
On 2012-05-11 11:22, Roman D. Boiko wrote:
What about line and column information?
Indices of the first code unit of each line are stored inside lexer and
a function will compute Location (line number, column number, file
specification) for any index. This way size of Token instance is reduced
On 2012-05-11 11:31, Roman D. Boiko wrote:
My plan is to create frontend that would be much better than existing,
both in design and implementation. I decided to work on this full time
for several months.
That's good news.
Front end will not produce the same data as DMD front end does, so
On 2012-05-11 11:23, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 09:19:07 UTC, dennis luehring wrote:
does the parser/lexer allow half-finished syntax parsing? for being
useable in an IDE for syntax-highlighting while coding?
That's planned, but I would like to see your usage scenarios
Am 11.05.2012 11:33, schrieb Roman D. Boiko:
-very fast in parsing/lexing - there need to be a benchmark
enviroment from the very start
Will add that to May roadmap.
are using slices for prevent coping everything around?
the parser/lexer need to be as fast as the original one - maybe even
On Friday, 11 May 2012 at 09:28:36 UTC, dennis luehring wrote:
Am 11.05.2012 11:23, schrieb Roman D. Boiko:
On Friday, 11 May 2012 at 09:19:07 UTC, dennis luehring wrote:
does the parser/lexer allow half-finished syntax parsing? for
being useable in an IDE for syntax-highlighting while coding?
On Friday, 11 May 2012 at 09:36:28 UTC, Jacob Carlborg wrote:
On 2012-05-11 11:22, Roman D. Boiko wrote:
Locations will possible to calculate
both taking into account special token sequences (e.g., #line 3
ab/c.d), or discarding them.
Aha, clever. As long as I can get out the information I'm
On Friday, 11 May 2012 at 10:40:43 UTC, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 10:01:17 UTC, dennis luehring wrote:
Am 11.05.2012 11:33, schrieb Roman D. Boiko:
-very fast in parsing/lexing - there need to be a benchmark
enviroment from the very start
Will add that to May roadmap.
Le 11/05/2012 11:31, Roman D. Boiko a écrit :
On Friday, 11 May 2012 at 09:02:12 UTC, Jacob Carlborg wrote:
If think that the end goal of a project like this, putting a D
frontend in Phobos, should be that the compiler should be built using
this library. This would result in the compiler and
Le 11/05/2012 12:01, dennis luehring a écrit :
Am 11.05.2012 11:33, schrieb Roman D. Boiko:
-very fast in parsing/lexing - there need to be a benchmark
enviroment from the very start
Will add that to May roadmap.
are using slices for prevent coping everything around?
the parser/lexer need
On 5/11/12 4:22 PM, Roman D. Boiko wrote:
What about line and column information?
Indices of the first code unit of each line are stored inside lexer and
a function will compute Location (line number, column number, file
specification) for any index. This way size of Token instance is reduced
On Friday, 11 May 2012 at 11:41:34 UTC, alex wrote:
Ever thought of asking the VisualD developer to integrate your
library into his IDE extension? Might be cool to do so because
of extended completion abilities etc. (lol I'm the Mono-D dev
-- but why not? ;D)
Didn't think about that yet,
On Friday, 11 May 2012 at 11:50:14 UTC, deadalnix wrote:
Le 11/05/2012 11:31, Roman D. Boiko a écrit :
My plan is to create frontend that would be much better than
existing,
both in design and implementation. I decided to work on this
full time
for several months.
I this is your plan, I'm
On Friday, 11 May 2012 at 11:49:23 UTC, Jacob Carlborg wrote:
On 2012-05-11 12:35, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 09:36:28 UTC, Jacob Carlborg wrote:
Aha, clever. As long as I can get out the information I'm
happy :) How
about adding properties for this in the token struct?
Le 11/05/2012 14:04, Roman D. Boiko a écrit :
That makes sense. Is it possible to switch SDC to the Boost license? I'm
trying to keep it for all DCT code.
Let me do a clean package of my code this week end. For now it is mixed
with SDC source code, which was enough as I was working alone,
Am 11.05.2012 13:50, schrieb Ary Manzana:
On 5/11/12 4:22 PM, Roman D. Boiko wrote:
What about line and column information?
Indices of the first code unit of each line are stored inside lexer and
a function will compute Location (line number, column number, file
specification) for any
On Friday, 11 May 2012 at 11:55:47 UTC, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 11:41:34 UTC, alex wrote:
Ever thought of asking the VisualD developer to integrate your
library into his IDE extension? Might be cool to do so because
of extended completion abilities etc. (lol I'm the
On Friday, 11 May 2012 at 11:47:18 UTC, deadalnix wrote:
From the beginning, I'm think AST macro using CTFE.
Could you please elaborate?
I plan to strictly follow published D specification.
Exceptions from this rule are possible provided either of the
following is true:
* new functionality
On Friday, 11 May 2012 at 12:13:53 UTC, alex wrote:
Mono-D is written in C#, VisualD uses D -- so it actually
should be easier to integrate into the second one :)
Sorry, I meant D-IDE. But there might exist the reason to consume
D implementation from C# also. I would happily collaborate to
Le 11/05/2012 14:14, Roman D. Boiko a écrit :
On Friday, 11 May 2012 at 11:47:18 UTC, deadalnix wrote:
From the beginning, I'm think AST macro using CTFE.
Could you please elaborate?
I plan to strictly follow published D specification.
Exceptions from this rule are possible provided either of
On Friday, 11 May 2012 at 11:50:29 UTC, Ary Manzana wrote:
On 5/11/12 4:22 PM, Roman D. Boiko wrote:
What about line and column information?
Indices of the first code unit of each line are stored inside
lexer and
a function will compute Location (line number, column number,
file
On Friday, 11 May 2012 at 12:30:01 UTC, deadalnix wrote:
Your 3 points seem reasonable. Mine were :
* Implement something that can parse D as it is currently
defined/implemented (if dmd's behavior and spec differs, it is
handled on a per case basis).
All differences should be documented.
*
On 2012-05-11 14:07, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 11:49:23 UTC, Jacob Carlborg wrote:
Found it now, calculateFor. It not sure if it's the most intuitive
name though. I get the feeling: calculate what?.
calculateLocation was original name, but I don't like repeating
On Friday, 11 May 2012 at 12:55:58 UTC, Jacob Carlborg wrote:
On 2012-05-11 14:07, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 11:49:23 UTC, Jacob Carlborg wrote:
Found it now, calculateFor. It not sure if it's the most
intuitive
name though. I get the feeling: calculate what?.
On 2012-05-11 14:14, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 11:47:18 UTC, deadalnix wrote:
From the beginning, I'm think AST macro using CTFE.
Could you please elaborate?
I plan to strictly follow published D specification.
That won't be easy, nobody know what the specification is
On 2012-05-11 15:01, Roman D. Boiko wrote:
What about the following signature: Location locate(size_t index)?
Or even better:
alias size_t CodeUnitIndex;
Location locateFor(CodeUnitIndex position);
That is better although I would prefer to pass in a token (assuming that
is where index is
Le 11/05/2012 15:01, Roman D. Boiko a écrit :
On Friday, 11 May 2012 at 12:55:58 UTC, Jacob Carlborg wrote:
On 2012-05-11 14:07, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 11:49:23 UTC, Jacob Carlborg wrote:
Found it now, calculateFor. It not sure if it's the most intuitive
name
? my guess is that the spec is TDPL + TDPL errata. dlang.org should be
updated as people notice inaccuracies.
This project would be an ideal time to update dlang.org as people notice
its not in sync with TDPL.
If TDPL doesn't cover it then the community should review it.
On Fri, May 11, 2012 at
On Friday, 11 May 2012 at 13:28:21 UTC, deadalnix wrote:
Le 11/05/2012 15:01, Roman D. Boiko a écrit :
The problem with placing it in Token is that Token should not
know
anything about source as a whole.
I don't really see the benefit of this. You are trading a O(1)
operation to an
On Friday, 11 May 2012 at 13:25:53 UTC, Jacob Carlborg wrote:
On 2012-05-11 15:01, Roman D. Boiko wrote:
What about the following signature: Location locate(size_t
index)?
Or even better:
alias size_t CodeUnitIndex;
Location locateFor(CodeUnitIndex position);
That is better although I would
On Friday, 11 May 2012 at 13:30:49 UTC, Rory McGuire wrote:
? my guess is that the spec is TDPL + TDPL errata. dlang.org
should be
updated as people notice inaccuracies.
This project would be an ideal time to update dlang.org as
people notice
its not in sync with TDPL.
If TDPL doesn't cover
On Friday, 11 May 2012 at 13:28:21 UTC, deadalnix wrote:
Le 11/05/2012 15:01, Roman D. Boiko a écrit :
The problem with placing it in Token is that Token should not
know
anything about source as a whole.
I don't really see the benefit of this. You are trading a O(1)
operation to an
On Friday, 11 May 2012 at 12:20:27 UTC, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 12:13:53 UTC, alex wrote:
Mono-D is written in C#, VisualD uses D -- so it actually
should be easier to integrate into the second one :)
Sorry, I meant D-IDE. But there might exist the reason to
consume D
On 2012-05-11 15:30, Rory McGuire wrote:
? my guess is that the spec is TDPL + TDPL errata. dlang.org
http://dlang.org should be updated as people notice inaccuracies.
This project would be an ideal time to update dlang.org
http://dlang.org as people notice its not in sync with TDPL.
If TDPL
Le 11/05/2012 16:02, Roman D. Boiko a écrit :
On Friday, 11 May 2012 at 13:28:21 UTC, deadalnix wrote:
Le 11/05/2012 15:01, Roman D. Boiko a écrit :
The problem with placing it in Token is that Token should not know
anything about source as a whole.
I don't really see the benefit of this.
On Friday, 11 May 2012 at 14:45:45 UTC, Jacob Carlborg wrote:
On 2012-05-11 15:30, Rory McGuire wrote:
? my guess is that the spec is TDPL + TDPL errata. dlang.org
http://dlang.org should be updated as people notice
inaccuracies.
This project would be an ideal time to update dlang.org
On Friday, 11 May 2012 at 15:05:19 UTC, deadalnix wrote:
Le 11/05/2012 16:02, Roman D. Boiko a écrit :
Technically, I'm trading N*0(1) operations needed to track
line and
column while consuming each character to M*0(log(n))
operations when
calculating them on demand. N = number of characters,
yip, but TDPL still has to take precedence because that is the one that
walter + andrei + community put the most focused effort into.
On Fri, May 11, 2012 at 4:45 PM, Jacob Carlborg d...@me.com wrote:
On 2012-05-11 15:30, Rory McGuire wrote:
? my guess is that the spec is TDPL + TDPL errata.
On Friday, May 11, 2012 23:00:15 Rory McGuire wrote:
yip, but TDPL still has to take precedence because that is the one that
walter + andrei + community put the most focused effort into.
It doesn't necessarily. Each place that they differ is examined individually
and decided on its own merits.
On 5/11/12 10:14 PM, Roman D. Boiko wrote:
On Friday, 11 May 2012 at 15:05:19 UTC, deadalnix wrote:
Le 11/05/2012 16:02, Roman D. Boiko a écrit :
Technically, I'm trading N*0(1) operations needed to track line and
column while consuming each character to M*0(log(n)) operations when
calculating
On Saturday, 12 May 2012 at 03:32:20 UTC, Ary Manzana wrote:
As deadalnix says, I think you are over-complicating things.
I mean, to store the column and line information it's just:
if (isNewLine(c)) {
line++;
column = 0;
} else {
column++;
}
(I think you need to add that to the
On Saturday, 12 May 2012 at 03:32:20 UTC, Ary Manzana wrote:
As deadalnix says, I think you are over-complicating things.
I mean, to store the column and line information it's just:
if (isNewLine(c)) {
line++;
column = 0;
} else {
column++;
}
(I think you need to add that to the
On Friday, May 11, 2012 10:01:28 Roman D. Boiko wrote:
There were several discussions about the need for a D compiler
library.
I propose my draft implementation of lexer for community review:
https://github.com/roman-d-boiko/dct
Lexer is based on Brian Schott's project
75 matches
Mail list logo
