Re: DScanner is ready for use
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote: * Prints out a complete AST of a source file in XML format. Neat feature! I met two bugs that need some love: https://github.com/Hackerpilot/Dscanner/issues/168 https://github.com/Hackerpilot/Dscanner/issues/169 Aside: the D grammar that I reverse-engineered can be located here: https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html Thanks for making this available! Very useful.
Re: DScanner is ready for use
On 04/22/2014 07:08 PM, bearophile wrote:
This code gives four problems to Dscanner:
void main() {
auto x = float(5);
This is not valid. DMD and the grammar spec both do not allow
this.
auto r = 1. + 2;
Fixed. (https://issues.dlang.org/show_bug.cgi?id=12623)
int items[5];
I don't support C-style declarations. DMD doesn't really support
them either. (https://issues.dlang.org/show_bug.cgi?id=953)
import std.stdio;
int.max.writeln;
Fixed.
}
Andrei Alexandrescu was very wrong to think that implicit
string concatenation is a speck of dust. I am not going to
close down that enhancement request.
It's a very bad feature. Every time I've ever used it has been a
bug.
Re: DScanner is ready for use
Brian Schott: This is not valid. DMD and the grammar spec both do not allow this. This was changed weeks ago. Now D accepts that code. int items[5]; I don't support C-style declarations. DMD doesn't really support them either. (https://issues.dlang.org/show_bug.cgi?id=953) Then I suggest DScanner to support the half-C-declarations, because D compilers digests them just fine and they are very common in D code you will find in the wild. A code analyzer has to accept the real world code people write, otherwise it's much less useful :-) Bye, bearophile
Re: DScanner is ready for use
On Thursday, 24 April 2014 at 21:36:24 UTC, bearophile wrote: This was changed weeks ago. Now D accepts that code. You want tooling to support language features that aren't released? Then I suggest DScanner to support the half-C-declarations, because D compilers digests them just fine and they are very common in D code you will find in the wild. A code analyzer has to accept the real world code people write, otherwise it's much less useful :-) Several people have indicated that they care about C-style declarations more than I do, and yet they still don't care enough to implement support for them and create a pull request.
Re: DScanner is ready for use
On Thursday, 24 April 2014 at 22:19:29 UTC, Brian Schott wrote: Then I suggest DScanner to support the half-C-declarations, because D compilers digests them just fine and they are very common in D code you will find in the wild. A code analyzer has to accept the real world code people write, otherwise it's much less useful :-) Several people have indicated that they care about C-style declarations more than I do, and yet they still don't care enough to implement support for them and create a pull request. I think if such support is ever to be implemented it should result in immediate analysis message with demand to change it to D style declarations and never do it again. This is an unfortunate legacy that should not be encouraged.
Re: DScanner is ready for use
Brian Schott: You want tooling to support language features that aren't released? I think it's nice for a language tool to try to follow closely the improvements in the language. But you are right, the update rhythms of DScanner are left to you. Several people have indicated that they care about C-style declarations more than I do, and yet they still don't care enough to implement support for them and create a pull request. Sorry, I understand the feeling, I was just trying to help a little, underlying a missing part in DScanner. There are several open source projects I report bugs to, but I don't sent patched to. Bye, bearophile
Re: DScanner is ready for use
On 28-7-2013 0:27, Brian Schott wrote: DScanner is a tool for analyzing D source code. It has the following features: * Prints out a complete AST of a source file in XML format. * Syntax checks code and prints warning/error messages * Prints a listing of modules imported by a source file * Syntax highlights code in HTML format * Provides more meaningful line of code count than wc * Counts tokens in a source file The lexer/parser/AST are located in the std/d directory in the repository. These files should prove useful to anyone else working on D tooling. https://github.com/Hackerpilot/Dscanner Aside: the D grammar that I reverse-engineered can be located here: https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html If I do a styleCheck on a source file that contains a BOM, I get the following message: test.d(1:14)[error]: Expected ; instead of .
Re: DScanner is ready for use
Brian Schott:
DScanner is a tool for analyzing D source code. It has the
following features:
...
https://github.com/Hackerpilot/Dscanner
I have just compiled it on Windows32 and tried it.
The compilation using the given bat has failed to link:
OPTLINK (R) for Win32 Release 8.00.15
Copyright (C) Digital Mars 1989-2013 All rights reserved.
http://www.digitalmars.com/ctg/optlink.html
dscanner.obj(dscanner)
Error 42: Symbol Undefined
_D8analysis10ifelsesame15IfElseSameCheck6__ctorMFAyaZC8analysis10ifelsesame15IfElseSameCheck
dscanner.obj(dscanner)
Error 42: Symbol Undefined
_D8analysis12constructors16ConstructorCheck7__ClassZ
dscanner.obj(dscanner)
Error 42: Symbol Undefined
_D8analysis12constructors16ConstructorCheck6__ctorMFAyaZC8analysis12constructors16ConstructorCheck
dscanner.obj(dscanner)
Error 42: Symbol Undefined
_D8analysis10ifelsesame15IfElseSameCheck7__ClassZ
--- errorlevel 4
But I have built the DScanner successfully using the old bud
tool.
I have seen that this basic usage is not supported:
dscanner -s *.d
I have seen it doesn't support source code with unicode
identifiers or chars.
How do you enable/disable specific tests when you use -s?
This code gives four problems to Dscanner:
void main() {
auto x = float(5);
auto r = 1. + 2;
int items[5];
import std.stdio;
int.max.writeln;
}
In about 25_000 CLOC lines of my code DScanner has found several
usages of the old-style alias syntax, that I will fix. Plus it
has found three usages of the implicit string concatenation, and
two of them are (the same) bug! Andrei Alexandrescu was very
wrong to think that implicit string concatenation is a speck of
dust. I am not going to close down that enhancement request.
Bye,
bearophile
Re: DScanner is ready for use
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote: DScanner is a tool for analyzing D source code. It has the following features: * Prints out a complete AST of a source file in XML format. * Syntax checks code and prints warning/error messages * Prints a listing of modules imported by a source file * Syntax highlights code in HTML format * Provides more meaningful line of code count than wc * Counts tokens in a source file The lexer/parser/AST are located in the std/d directory in the repository. These files should prove useful to anyone else working on D tooling. https://github.com/Hackerpilot/Dscanner Aside: the D grammar that I reverse-engineered can be located here: https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html Any idea on when we might see json output(i am not a fan of xml)? Other than that this is a fantastic project! I am planing some projects in the future and this will be of great help. Keep up the good work!
Re: DScanner is ready for use
On Friday, 2 August 2013 at 13:52:14 UTC, Tofu Ninja wrote: Any idea on when we might see json output(i am not a fan of xml)? Roughly the same time somebody submits a pull request. I'm currently focusing my spare time on DCD, so the JSON output will happen after I'm able to get auto-completion working.
Re: DScanner is ready for use
On Friday, 2 August 2013 at 18:01:01 UTC, Brian Schott wrote: On Friday, 2 August 2013 at 13:52:14 UTC, Tofu Ninja wrote: Any idea on when we might see json output(i am not a fan of xml)? Roughly the same time somebody submits a pull request. I'm currently focusing my spare time on DCD, so the JSON output will happen after I'm able to get auto-completion working. I will look into adding it my self if I get some time, but I don't think I will need to use this for a while. For what I want it for, there is a lot of legwork to be done before I get around to needing this. Also roughly how difficult would it be to re-create source code from the xml? And does the xml preserve comments and if so does it do anything with ddoc?
Re: DScanner is ready for use
On Friday, 2 August 2013 at 18:12:15 UTC, Tofu Ninja wrote: I will look into adding it my self if I get some time, but I don't think I will need to use this for a while. For what I want it for, there is a lot of legwork to be done before I get around to needing this. The XML output is handled by this class: https://github.com/Hackerpilot/Dscanner/blob/master/astprinter.d It shouldn't be much more difficult than changing the print statements. Also roughly how difficult would it be to re-create source code from the xml? And does the xml preserve comments and if so does it do anything with ddoc? Aside from the comments, it should be possible to recreate the source from the AST. If it's not, there's a bug in the AST output. Comments are skipped when syntax checking or generating the AST.
Re: DScanner is ready for use
okay, I look forward to its release. I'm really looking forward to code completion in Sublime Text 2 On Wed, Jul 31, 2013 at 10:44 PM, Brian Schott briancsch...@gmail.comwrote: On Wednesday, 31 July 2013 at 18:41:17 UTC, Justin Whear wrote: On Wed, 31 Jul 2013 20:30:17 +0200, Rory McGuire wrote: Any chance of you turning this into a daemon? Something likt margo or gocode? The author has another project here: https://github.com/**Hackerpilot/DCDhttps://github.com/Hackerpilot/DCD I wouldn't bother trying to use that yet. Maybe next week, but not now. When I get it working there will be a thread on D.announce.
Re: DScanner is ready for use
On Sun, 28 Jul 2013 00:27:34 +0200 Brian Schott briancsch...@gmail.com wrote: DScanner is a tool for analyzing D source code. [...] When I try to compile it (DMD 2.063.2 Win32) I get this: C:\Users\Nick\AppData\Roaming\dvm\compilers\dmd-2.063.2\bin\..\src\phobos\std\array.d(321): Error: function core.memory.GC.malloc (uint sz, uint ba = 0u) is not callable using argument types (ulong, BlkAttr) C:\Users\Nick\AppData\Roaming\dvm\compilers\dmd-2.063.2\bin\..\src\phobos\std\array.d(272): Error: template instance std.array.arrayAllocImpl!(false, ubyte[], ulong) error instantiating main.d(77):instantiated from here: uninitializedArray!(ubyte[], ulong) main.d(77): Error: template instance std.array.uninitializedArray!(ubyte[], ulong) error instantiating
Re: DScanner is ready for use
On Thu, 1 Aug 2013 21:52:00 -0400 Nick Sabalausky seewebsitetocontac...@semitwist.com wrote: On Sun, 28 Jul 2013 00:27:34 +0200 Brian Schott briancsch...@gmail.com wrote: DScanner is a tool for analyzing D source code. [...] When I try to compile it (DMD 2.063.2 Win32) I get this: C:\Users\Nick\AppData\Roaming\dvm\compilers\dmd-2.063.2\bin\..\src\phobos\std\array.d(321): Error: function core.memory.GC.malloc (uint sz, uint ba = 0u) is not callable using argument types (ulong, BlkAttr) C:\Users\Nick\AppData\Roaming\dvm\compilers\dmd-2.063.2\bin\..\src\phobos\std\array.d(272): Error: template instance std.array.arrayAllocImpl!(false, ubyte[], ulong) error instantiating main.d(77):instantiated from here: uninitializedArray!(ubyte[], ulong) main.d(77): Error: template instance std.array.uninitializedArray!(ubyte[], ulong) error instantiating Actually, I dug into this more (it was problem on 32-bit) and made a pull request: https://github.com/Hackerpilot/Dscanner/pull/42
Re: DScanner is ready for use
Any chance of you turning this into a daemon? Something likt margo or
gocode?
On 29 Jul 2013 11:05, qznc q...@web.de wrote:
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote:
DScanner is a tool for analyzing D source code. It has the following
features:
* Prints out a complete AST of a source file in XML format.
* Syntax checks code and prints warning/error messages
* Prints a listing of modules imported by a source file
* Syntax highlights code in HTML format
* Provides more meaningful line of code count than wc
* Counts tokens in a source file
The lexer/parser/AST are located in the std/d directory in the
repository. These files should prove useful to anyone else working on D
tooling.
https://github.com/**Hackerpilot/Dscannerhttps://github.com/Hackerpilot/Dscanner
Aside: the D grammar that I reverse-engineered can be located here:
https://rawgithub.com/**Hackerpilot/DGrammar/master/**grammar.htmlhttps://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Dscanner looks like a good starting point for a code formatting tool (like
gofmt). However, there seems to be a tradeoff with performance involved.
For compilation you want a fast lexer and parser. For formatting you need
to preserve comments, though.
For example, convert this from source to AST to source without losing the
comments:
void /*hello*/ /*world*/ main () { }
Re: DScanner is ready for use
On Wed, 31 Jul 2013 20:30:17 +0200, Rory McGuire wrote: Any chance of you turning this into a daemon? Something likt margo or gocode? The author has another project here: https://github.com/Hackerpilot/DCD
Re: DScanner is ready for use
On Wednesday, 31 July 2013 at 18:41:17 UTC, Justin Whear wrote: On Wed, 31 Jul 2013 20:30:17 +0200, Rory McGuire wrote: Any chance of you turning this into a daemon? Something likt margo or gocode? The author has another project here: https://github.com/Hackerpilot/DCD I wouldn't bother trying to use that yet. Maybe next week, but not now. When I get it working there will be a thread on D.announce.
Re: DScanner is ready for use
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote:
DScanner is a tool for analyzing D source code. It has the
following features:
* Prints out a complete AST of a source file in XML format.
* Syntax checks code and prints warning/error messages
* Prints a listing of modules imported by a source file
* Syntax highlights code in HTML format
* Provides more meaningful line of code count than wc
* Counts tokens in a source file
The lexer/parser/AST are located in the std/d directory in
the repository. These files should prove useful to anyone else
working on D tooling.
https://github.com/Hackerpilot/Dscanner
Aside: the D grammar that I reverse-engineered can be located
here:
https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Dscanner looks like a good starting point for a code formatting
tool (like gofmt). However, there seems to be a tradeoff with
performance involved. For compilation you want a fast lexer and
parser. For formatting you need to preserve comments, though.
For example, convert this from source to AST to source without
losing the comments:
void /*hello*/ /*world*/ main () { }
Re: DScanner is ready for use
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote: DScanner is a tool for analyzing D source code. It has the following features: * Prints out a complete AST of a source file in XML format. * Syntax checks code and prints warning/error messages * Prints a listing of modules imported by a source file * Syntax highlights code in HTML format * Provides more meaningful line of code count than wc * Counts tokens in a source file The lexer/parser/AST are located in the std/d directory in the repository. These files should prove useful to anyone else working on D tooling. https://github.com/Hackerpilot/Dscanner Aside: the D grammar that I reverse-engineered can be located here: https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html Great! :) I do not understand the LoC count, though. The description sounds like Number of Statements?
Re: DScanner is ready for use
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote: DScanner is a tool for analyzing D source code. It has the following features: * Prints out a complete AST of a source file in XML format. * Syntax checks code and prints warning/error messages * Prints a listing of modules imported by a source file * Syntax highlights code in HTML format * Provides more meaningful line of code count than wc * Counts tokens in a source file The lexer/parser/AST are located in the std/d directory in the repository. These files should prove useful to anyone else working on D tooling. https://github.com/Hackerpilot/Dscanner Aside: the D grammar that I reverse-engineered can be located here: https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html Awesome! I hope it won't be forgotten by the time I need it :) By the way, how far is that std.d.* stuff from ongoing Phobos inclusion review? I suppose currently it does not do any semantical analysis? How hard it would be to implement dmd warnings on top of dscanner instead?
Re: DScanner is ready for use
On Sunday, 28 July 2013 at 12:49:34 UTC, Dicebot wrote: Awesome! I hope it won't be forgotten by the time I need it :) By the way, how far is that std.d.* stuff from ongoing Phobos inclusion review? I suppose currently it does not do any semantical analysis? How hard it would be to implement dmd warnings on top of dscanner instead? I don't think it's necessary for semantic analysis to be included in Phobos. It's enough to start with a lexer, then later add a parser and semantic analysis. -- /Jacob Carlborg
Re: DScanner is ready for use
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote: Aside: the D grammar that I reverse-engineered can be located here: https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html Does arrayLiteral support stray comma?
Re: DScanner is ready for use
Also looks like enumBody supports multiple commas with nothing
between them, but dmd doesn't support it. Sould be
'{' enumMember (',' enumMember)* ','? '}'
Re: DScanner is ready for use
And arrayLiteral should be
'[' (assignExpression (',' assignExpression)* ','?)? ']'
Re: DScanner is ready for use
The same for arrayInitializer:
'[' (arrayMemberInitialization (',' arrayMemberInitialization)*
','?)? ']'
Re: DScanner is ready for use
Like typedef alias supports multiple declarators.
aliasDeclaration:
'alias' (aliasInitializer (',' aliasInitializer)* | type
declarator (',' declarator)*) ';'
Also declarator supports initializer, but alias doesn't.
declarator:
Identifier ('=' initializer)?
;
alias int a=1;
Error: alias cannot have initializer
Re: DScanner is ready for use
Well, knowing the lean and mean way dmd is written, alias initializer can be a semantical error rather than syntactical.
Re: DScanner is ready for use
On Sunday, 28 July 2013 at 13:44:03 UTC, Jacob Carlborg wrote: I don't think it's necessary for semantic analysis to be included in Phobos. It's enough to start with a lexer, then later add a parser and semantic analysis. Those were 2 separate not related questions ;)
Re: DScanner is ready for use
Am 28.07.2013 00:27, schrieb Brian Schott: DScanner is a tool for analyzing D source code. It has the following features: * Prints out a complete AST of a source file in XML format. * Syntax checks code and prints warning/error messages * Prints a listing of modules imported by a source file * Syntax highlights code in HTML format * Provides more meaningful line of code count than wc * Counts tokens in a source file The lexer/parser/AST are located in the std/d directory in the repository. These files should prove useful to anyone else working on D tooling. https://github.com/Hackerpilot/Dscanner Aside: the D grammar that I reverse-engineered can be located here: https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html would be nice to habe some sort of binary output (beside xml and json) to ease (and maybe speed-up) the usage
Re: DScanner is ready for use
On Sun, 28 Jul 2013 00:27:34 +0200 Brian Schott briancsch...@gmail.com wrote: DScanner is a tool for analyzing D source code. It has the following features: * Prints out a complete AST of a source file in XML format. * Syntax checks code and prints warning/error messages * Prints a listing of modules imported by a source file * Syntax highlights code in HTML format * Provides more meaningful line of code count than wc * Counts tokens in a source file The lexer/parser/AST are located in the std/d directory in the repository. These files should prove useful to anyone else working on D tooling. https://github.com/Hackerpilot/Dscanner Aside: the D grammar that I reverse-engineered can be located here: https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html Sweet, I was *just* thinking about writing a D - HTML syntax highlighter no more than about five minutes ago. Glad to see I won't have to :) I'll definitely be checking this out.
