+1 cnd.nextapt java -Dlexer.print=false -jar target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include .... -- /usr/include/complex.h -- Syntax error on 59:12 token recognition error at: '"clang diagnostic ignored \"'
Will also take a look. Thanks Antonio. Great initiative. P.S. I guess .hpp/.cpp files are next? On Thu, 6 May 2021 at 09:52, antonio <[email protected]> wrote: > Hi all, > > As you may remember from previous threads, the NetBeans 8.2 "cnd.antlr" > module contained a fork of Antlr 2.7.7 (compiler & runtime) that was not > donated to the ASF. As a substitute Oracle donated a > "cnd/libs.cnd.antlr" library wrapper that wrapped the NetBeans 8.2 jar, > that we cannot use in Apache NetBeans due to licensing problems. > > The "cnd.apt" package contains an old "aptlexer.g" (on top of this Antlr > 2.7.7 runtime) lexer grammar that is responsible for lexing C/C++ and > Fortran files. > > I've done a small experiment (see [1]) transforming this old "aptexpr.g" > grammar with 1403 lines [2] to a brand new "APTLexer.g4" lexer grammar > [3] with 480 lines. > > This is still on early stages, but it seems the resulting Antlrv4 lexer > outperforms the old Antlrv2 lexer (we go from ~2600 files/second to > ~4000 files/second). > > I think we can continue the path of removing the old "libs.antlr.cnd" > module from CND and then refactor "cnd.apt" (and "cnd.asm" et al) on top > of the new lexer and Antlrv4. The idea is to upgrade all existing > grammars (including the CXX parser at > > https://github.com/apache/netbeans/blob/cnd/cnd/cnd.modelimpl/src/org/netbeans/modules/cnd/modelimpl/parser/CXXParser.g) > > to Antlrv4 on top of this new lexer. > > Since the "libs.antlr.cnd" API permeates different modules of CND this > may take quite a long time. > > I think we can create a brand new "cnd2" branch in the main repository > (forked from the "cnd" branch and refreshed with latest 12.4 version) > and do this CND refactoring on top of Antlrv4 there, right? Any > objections to this? Or shall I continue this experiment in a repo of > mine? Any other plans/suggestions to have CND back? > > Thanks, > Antonio > > > P.S.: In order to test the lexer I'd appreciate if you run the new lexer > (instructions at [1]) on some of your include/source directories and > report back any syntax errors you may see (the lexer will halt on any > lexer error), so we can fine-tune any extra lexer rules we may need. > > > [1] > https://github.com/vieiro/cnd.nextapt > > To lex a whole directory of (*.h/*.c files) run the command: > java -Dlexer.print=false -jar > target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include/ > > To lex a specific file (dumping all tokens to stdout by removing > -Dlexer.print=false): > java -jar target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar > your-file.[ch] > > [2] > https://github.com/vieiro/cnd.nextapt/blob/master/cnd.apt.nb82/aptlexer.g > > [3] > > https://github.com/vieiro/cnd.nextapt/blob/master/cnd.apt.antlr4/src/main/antlr4/org/netbeans/modules/cnd/apt/impl/support/generated/APTLexer.g4 > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > For further information about the NetBeans mailing lists, visit: > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists > > > >
