Re: [sqlite] Lemon source code?

[Richard Hipp]

On 12/20/19, David Given  wrote:
> Does anyone know if the original source code for the lemon parser still
> exists, and if so, where to find it?
>
> The lemon.c file shipped with sqlite, which the hwaci website points me at,
> does not appear to be it --- instead it's an amalgamation of a bunch of
> other files, including some generated by aagen from data files; but I have
> been able to track down neither the data files nor the aagen tool itself.
>
> Or does maintainence these days happen directly on the amalgamation?

I have long since lost the original source files (which I wrote on a
Sun4 in K C).  All maintenance these days happens on the amalgamated
"lemon.c" source file, which isn't that big really - less than 6K
lines total and less than 5K SLOC.

The complete edit history of the "lemon.c" source file, going back
almost 20 years, can be seen here:
https://www.sqlite.org/src/finfo/tool/lemon.c

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon Parser vs bubble-generator.tcl

[Richard Hipp]

On 8/31/18, Warren Young  wrote:
> They’re separate.  Here’s the Tcl source for the bubble diagrams:
>
> https://www.sqlite.org/docsrc/file/?name=art/syntax/bubble-generator-data.tcl
>
> …and here’s the Lemon grammar for SQLite’s SQL parser:
>
> https://www.sqlite.org/src/file?name=src/parse.y
>

The parse.y file is designed to generate a fast parser automaton.  The
bubble-generator-data.tcl file is designed to generate output that is
easily understandable by humans.  These two files are therefore at
cross-purposes.

One little-noticed aspect of SQLite that the source code to the
LALR(1) parser generator (Lemon) is included in the source tree.  The
fact that SQLite uses its own parser generator, rather than relying
yacc/bison or similar, has proven very helpful over the years.  We've
been able to add features, fix bugs, and make optimizations to the
parser that would not have been possible had we been dependent on an
outside tool.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon Parser vs bubble-generator.tcl

[Scott Robison]

On Fri, Aug 31, 2018 at 2:59 PM Warren Young  wrote:
>
> On Aug 31, 2018, at 1:55 PM, Scott Robison  wrote:
> >
> > Is one generated from the other, or are they maintained separately?
>
> They’re separate.  Here’s the Tcl source for the bubble diagrams:

As I suspected having looked at them in the source tree. Thanks for
the confirmation.

-- 
Scott Robison
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon Parser vs bubble-generator.tcl

[Warren Young]

On Aug 31, 2018, at 1:55 PM, Scott Robison  wrote:
> 
> Is one generated from the other, or are they maintained separately?

They’re separate.  Here’s the Tcl source for the bubble diagrams:


https://www.sqlite.org/docsrc/file/?name=art/syntax/bubble-generator-data.tcl

…and here’s the Lemon grammar for SQLite’s SQL parser:

https://www.sqlite.org/src/file?name=src/parse.y

Here’s a brief article on the script that may explain a few things:

https://wiki.tcl.tk/21708

> I...want to do something related to
> identifying SQL syntax at C++ compile time.

In that case, I’d study how ./src/parse.y gets turned into ./parse.[ch] in the 
SQLite source tree, then try to reuse as much of parse.y and its associated 
bits as you can.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon-generated parser gives an assertion failure

[Richard Hipp]

On 4/25/17, Kelvin Sherlock  wrote:
>
> I believe the issue is line 4164 which should be … = LEMON_TRUE.  Currently,
> when i=0, all rules will erroneously be optimized out.
>
>   4157/* Mark rules that are actually used for reduce actions after all
>   4158** optimizations have been applied
>   4159*/
>   4160for(rp=lemp->rule; rp; rp=rp->next) rp->doesReduce = LEMON_FALSE;
>   4161for(i=0; inxstate; i++){
>   4162  for(ap=lemp->sorted[i]->ap; ap; ap=ap->next){
>   4163if( ap->type==REDUCE || ap->type==SHIFTREDUCE ){
>   4164  ap->x.rp->doesReduce = i;
>   4165}
>   4166  }
>   4167}
>

Well done.  I fear that you have had to learn more about Lemon and
LALR(1) parsing than perhaps you really ever wanted to know.  Your
efforts are greatly appreciated.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon documention (small error)

[Richard Hipp]

On 10/8/16, Conor O'Rourke  wrote:
> I believe the Lemon documentation at:
> http://www.hwaci.com/sw/lemon/lemon.html is out of date with respect to the
> default token type. If I create a parser with no token_type override, the
> default type is:
>
> #define ParseTOKENTYPE void*
>
> not int as stated.

Fix checked in.  Thanks.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] lemon parser bug, %destructors directive not appended into source code

[Richard Hipp]

On 8/16/16, xiaobing  wrote:
> In lemon.c:ReportTable() , when " Combine duplicate destructors into a
> single case ", sp2->destructor is set to 0, but later it is used in
> tranlate_code to generate destructor.
> so if you have grammar like this:
> %destructor expr_a { expr_free($$); }
> %destructor expr_b { expr_free($$); }  // this one will be lost, unless
> you insert some comment so strcmp return nonzero
> expr ::= expr_a expr_b.
> the generated code will include destructor of expr_a, however, destructor
> of expr_b will be lost.

Excellent analysis, Xiaobing.  Thanks.

I have checked in a slightly different fix.  Please confirm that what
I have checked in actually fixes your problem.
https://www.sqlite.org/src/fdiff?dc=20=e3aa9ba3469804d7=e4fb7d73ac88


>
> Here is my patch to lemon.c:
>
> --- old/lemon.c 2016-05-18 18:07:00.0 +0800
> +++ new/lemon.c   2016-08-16 20:57:36.799178091 +0800
> @@ -263,6 +263,7 @@
>int useCnt;  /* Number of times used */
>char *destructor;/* Code which executes whenever this symbol is
> ** popped from the stack during error
> processing */
> +  int destructor_emitted;  /* Is the destructor code emitted */
>int destLineno;  /* Line number for start of destructor */
>char *datatype;  /* The data type of information held by this
> ** object. Only used if type==NONTERMINAL */
> @@ -4351,8 +4352,9 @@
>}
>for(i=0; insymbol; i++){
>  struct symbol *sp = lemp->symbols[i];
> -if( sp==0 || sp->type==TERMINAL || sp->destructor==0 ) continue;
> +if( sp==0 || sp->type==TERMINAL || sp->destructor==0 ||
> sp->destructor_emitted) continue;
>  fprintf(out,"case %d: /* %s */\n", sp->index, sp->name); lineno++;
> +sp->destructor_emitted = 1;
>
>  /* Combine duplicate destructors into a single case */
>  for(j=i+1; jnsymbol; j++){
> @@ -4362,7 +4364,7 @@
>&& strcmp(sp->destructor,sp2->destructor)==0 ){
>   fprintf(out,"case %d: /* %s */\n",
>   sp2->index, sp2->name); lineno++;
> - sp2->destructor = 0;
> + sp2->destructor_emitted = 1;
>}
>  }
>
> @@ -4876,6 +4878,7 @@
>  sp->firstset = 0;
>  sp->lambda = LEMON_FALSE;
>  sp->destructor = 0;
> +sp->destructor_emitted = 0;
>  sp->destLineno = 0;
>  sp->datatype = 0;
>  sp->useCnt = 0;
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Nick Wellnhofer]

On 12/07/2016 22:01, Richard Hipp wrote:

OK.  Another fix.  Please try the latest trunk version.


This version works for me. Thanks.

Nick

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Richard Hipp]

On 7/9/16, Nick Wellnhofer  wrote:
>
> This still doesn't work for me.

OK.  Another fix.  Please try the latest trunk version.

Note to passive readers of this thread: none of this has any impact on
SQLite.  SQLite does not use the feature of the Lemon LALR(1) parser
generator that Nick is fixing.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Nick Wellnhofer]

On 08/07/2016 21:54, Richard Hipp wrote:

Please try again with the latest version of Lemon.  Thanks.


This still doesn't work for me. I created a GitHub repo to demonstrate the 
problem:


https://github.com/nwellnhof/lemon-bug

Nick

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Nick Wellnhofer]

On 08/07/2016 21:54, Richard Hipp wrote:

Please try again with the latest version of Lemon.  Thanks.

On 7/6/16, Nick Wellnhofer  wrote:

On 05/07/2016 18:12, Richard Hipp wrote:

Please try https://www.sqlite.org/src/info/2683b375ad129117 and verify
that the changes on trunk are working.  Thanks.


Still doesn't work for me. The structure of the #ifdefs in `Parse` is:

 #ifdef YYERRORSYMBOL
   ...
 #elif defined(YYNOERRORRECOVERY)
   ...
 #else  /* YYERRORSYMBOL is not defined */
   ...
 #endif

Your first check-in modifies the first branch, your second check-in the
second
branch, resulting in:

 #ifdef YYERRORSYMBOL
   ...
   #ifndef YYNOERRORRECOVERY
 yypParser->yyerrcnt = -1;
   #endif
   ...
 #elif defined(YYNOERRORRECOVERY)
   ...
   #ifndef YYNOERRORRECOVERY
 yypParser->yyerrcnt = -1;
   #endif
   ...
 #else  /* YYERRORSYMBOL is not defined */
   ...
 #endif

The change to the second branch has no effect because YYNOERRORRECOVERY is
always defined. My patch modifies the third branch ("YYERRORSYMBOL is not
defined"). This fixes code that defines neither YYERRORSYMBOL nor
YYNOERRORRECOVERY. I think the code should look like this:

 #ifdef YYERRORSYMBOL
   ...
   #ifndef YYNOERRORRECOVERY
 yypParser->yyerrcnt = -1;
   #endif
   ...
 #elif defined(YYNOERRORRECOVERY)
   ...
 #else  /* YYERRORSYMBOL is not defined */
   ...
   yypParser->yyerrcnt = -1;
   ...
 #endif

(Another check for YYNOERRORRECOVERY isn't really needed in the third
branch.
It will always be undef.)

Nick








--
aevum GmbH
Nadistr. 12
80809 München
Germany

Tel: +49 89 35747589
http://aevum.de/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Richard Hipp]

Please try again with the latest version of Lemon.  Thanks.

On 7/6/16, Nick Wellnhofer  wrote:
> On 05/07/2016 18:12, Richard Hipp wrote:
>> Please try https://www.sqlite.org/src/info/2683b375ad129117 and verify
>> that the changes on trunk are working.  Thanks.
>
> Still doesn't work for me. The structure of the #ifdefs in `Parse` is:
>
>  #ifdef YYERRORSYMBOL
>...
>  #elif defined(YYNOERRORRECOVERY)
>...
>  #else  /* YYERRORSYMBOL is not defined */
>...
>  #endif
>
> Your first check-in modifies the first branch, your second check-in the
> second
> branch, resulting in:
>
>  #ifdef YYERRORSYMBOL
>...
>#ifndef YYNOERRORRECOVERY
>  yypParser->yyerrcnt = -1;
>#endif
>...
>  #elif defined(YYNOERRORRECOVERY)
>...
>#ifndef YYNOERRORRECOVERY
>  yypParser->yyerrcnt = -1;
>#endif
>...
>  #else  /* YYERRORSYMBOL is not defined */
>...
>  #endif
>
> The change to the second branch has no effect because YYNOERRORRECOVERY is
> always defined. My patch modifies the third branch ("YYERRORSYMBOL is not
> defined"). This fixes code that defines neither YYERRORSYMBOL nor
> YYNOERRORRECOVERY. I think the code should look like this:
>
>  #ifdef YYERRORSYMBOL
>...
>#ifndef YYNOERRORRECOVERY
>  yypParser->yyerrcnt = -1;
>#endif
>...
>  #elif defined(YYNOERRORRECOVERY)
>...
>  #else  /* YYERRORSYMBOL is not defined */
>...
>yypParser->yyerrcnt = -1;
>...
>  #endif
>
> (Another check for YYNOERRORRECOVERY isn't really needed in the third
> branch.
> It will always be undef.)
>
> Nick
>
>


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Nick Wellnhofer]

On 05/07/2016 18:12, Richard Hipp wrote:

Please try https://www.sqlite.org/src/info/2683b375ad129117 and verify
that the changes on trunk are working.  Thanks.


Still doesn't work for me. The structure of the #ifdefs in `Parse` is:

#ifdef YYERRORSYMBOL
  ...
#elif defined(YYNOERRORRECOVERY)
  ...
#else  /* YYERRORSYMBOL is not defined */
  ...
#endif

Your first check-in modifies the first branch, your second check-in the second 
branch, resulting in:


#ifdef YYERRORSYMBOL
  ...
  #ifndef YYNOERRORRECOVERY
yypParser->yyerrcnt = -1;
  #endif
  ...
#elif defined(YYNOERRORRECOVERY)
  ...
  #ifndef YYNOERRORRECOVERY
yypParser->yyerrcnt = -1;
  #endif
  ...
#else  /* YYERRORSYMBOL is not defined */
  ...
#endif

The change to the second branch has no effect because YYNOERRORRECOVERY is 
always defined. My patch modifies the third branch ("YYERRORSYMBOL is not 
defined"). This fixes code that defines neither YYERRORSYMBOL nor 
YYNOERRORRECOVERY. I think the code should look like this:


#ifdef YYERRORSYMBOL
  ...
  #ifndef YYNOERRORRECOVERY
yypParser->yyerrcnt = -1;
  #endif
  ...
#elif defined(YYNOERRORRECOVERY)
  ...
#else  /* YYERRORSYMBOL is not defined */
  ...
  yypParser->yyerrcnt = -1;
  ...
#endif

(Another check for YYNOERRORRECOVERY isn't really needed in the third branch. 
It will always be undef.)


Nick

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Richard Hipp]

Please try https://www.sqlite.org/src/info/2683b375ad129117 and verify
that the changes on trunk are working.  Thanks.

On 7/5/16, Nick Wellnhofer  wrote:
> On 05/07/2016 17:15, Richard Hipp wrote:
>> On 7/5/16, Nick Wellnhofer  wrote:
>>> No, this doesn't fix my problem. The check-in only changes the "#ifdef
>>> YYERRORSYMBOL" branch which I don't define. But if I add the change to
>>> the
>>> "YYERRORSYMBOL is not defined" branch as well, everything works as
>>> expected.
>>
>> Can you show me your patch, please?
>
> Here it is.
>
> Nick
>
>


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Nick Wellnhofer]

On 05/07/2016 17:15, Richard Hipp wrote:

On 7/5/16, Nick Wellnhofer  wrote:

No, this doesn't fix my problem. The check-in only changes the "#ifdef
YYERRORSYMBOL" branch which I don't define. But if I add the change to the
"YYERRORSYMBOL is not defined" branch as well, everything works as
expected.


Can you show me your patch, please?


Here it is.

Nick

diff --git a/lemon/lempar.c b/lemon/lempar.c
index 112d0bd..25a9525 100644
--- a/lemon/lempar.c
+++ b/lemon/lempar.c
@@ -919,6 +919,7 @@ void Parse(
   if( yyendofinput ){
 yy_parse_failed(yypParser);
   }
+  yypParser->yyerrcnt = -1;
   yymajor = YYNOCODE;
 #endif
 }
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Richard Hipp]

On 7/5/16, Nick Wellnhofer  wrote:
> No, this doesn't fix my problem. The check-in only changes the "#ifdef
> YYERRORSYMBOL" branch which I don't define. But if I add the change to the
> "YYERRORSYMBOL is not defined" branch as well, everything works as
> expected.

Can you show me your patch, please?

>
> Thanks for the quick response!
>
> Nick
>
> On 05/07/2016 14:48, Richard Hipp wrote:
>> Please try the latest check-in
>> (https://www.sqlite.org/src/info/91889fa30e84760e) and let me know
>> whether or not it clears your problem.
>>
>> On 7/5/16, Nick Wellnhofer  wrote:
>>> Hello,
>>>
>>> I hope this is right place to report Lemon issues. I ran into a problem
>>> after
>>> upgrading to the latest version of the Lemon source code from `trunk`.
>>>
>>> The following commit removed the initialization of `yyerrcnt` from the
>>> `Parse`
>>> function:
>>>
>>>
>>> http://www.sqlite.org/src/fdiff?sbs=1=872383ebf36c13fd=8569dd3e4c22831e
>>>
>>> Then this commit added the initialization to `ParseAlloc`:
>>>
>>>
>>> http://www.sqlite.org/src/fdiff?sbs=1=f06b7e98a6b7efb4=66a16b5e00fefff2
>>>
>>> I'd like to use a parser instance for multiple inputs without
>>> reallocating.
>>> If
>>> I parse input that causes a syntax error, the error is reported for the
>>> first
>>> time. But subsequent errors aren't reported anymore. I presume this is
>>> because
>>> `yyerrcnt` should be reset to -1 somewhere in the code.
>>>
>>> Nick
>>> ___
>>> sqlite-users mailing list
>>> sqlite-users@mailinglists.sqlite.org
>>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>>
>>
>>
>
>
> --
> aevum GmbH
> Nadistr. 12
> 80809 München
> Germany
>
> Tel: +49 89 35747589
> http://aevum.de/
>


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Nick Wellnhofer]

No, this doesn't fix my problem. The check-in only changes the "#ifdef 
YYERRORSYMBOL" branch which I don't define. But if I add the change to the 
"YYERRORSYMBOL is not defined" branch as well, everything works as expected.


Thanks for the quick response!

Nick

On 05/07/2016 14:48, Richard Hipp wrote:

Please try the latest check-in
(https://www.sqlite.org/src/info/91889fa30e84760e) and let me know
whether or not it clears your problem.

On 7/5/16, Nick Wellnhofer  wrote:

Hello,

I hope this is right place to report Lemon issues. I ran into a problem
after
upgrading to the latest version of the Lemon source code from `trunk`.

The following commit removed the initialization of `yyerrcnt` from the
`Parse`
function:


http://www.sqlite.org/src/fdiff?sbs=1=872383ebf36c13fd=8569dd3e4c22831e

Then this commit added the initialization to `ParseAlloc`:


http://www.sqlite.org/src/fdiff?sbs=1=f06b7e98a6b7efb4=66a16b5e00fefff2

I'd like to use a parser instance for multiple inputs without reallocating.
If
I parse input that causes a syntax error, the error is reported for the
first
time. But subsequent errors aren't reported anymore. I presume this is
because
`yyerrcnt` should be reset to -1 somewhere in the code.

Nick
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users







--
aevum GmbH
Nadistr. 12
80809 München
Germany

Tel: +49 89 35747589
http://aevum.de/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Richard Hipp]

Please try the latest check-in
(https://www.sqlite.org/src/info/91889fa30e84760e) and let me know
whether or not it clears your problem.

On 7/5/16, Nick Wellnhofer  wrote:
> Hello,
>
> I hope this is right place to report Lemon issues. I ran into a problem
> after
> upgrading to the latest version of the Lemon source code from `trunk`.
>
> The following commit removed the initialization of `yyerrcnt` from the
> `Parse`
> function:
>
>
> http://www.sqlite.org/src/fdiff?sbs=1=872383ebf36c13fd=8569dd3e4c22831e
>
> Then this commit added the initialization to `ParseAlloc`:
>
>
> http://www.sqlite.org/src/fdiff?sbs=1=f06b7e98a6b7efb4=66a16b5e00fefff2
>
> I'd like to use a parser instance for multiple inputs without reallocating.
> If
> I parse input that causes a syntax error, the error is reported for the
> first
> time. But subsequent errors aren't reported anymore. I presume this is
> because
> `yyerrcnt` should be reset to -1 somewhere in the code.
>
> Nick
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon bug in shiftreduce action for error symbol

[Vincent Zweije]

On Fri, Jun 17, 2016 at 11:50:48AM -0400, Richard Hipp wrote:

> On 6/6/16, Vincent Zweije  wrote:
>
> >
> > When the shiftreduce action is used in an error context,
> > such as:
> >
> >X -> alpha error.
> >
> > the adjustment is not made. This causes error handling to fail.
> >
>
> Is this problem fixed by
> https://www.sqlite.org/src/fdiff?sbs=1=66a16b5e00fefff2=8c4e9d8517e50da3
> from earlier this month?  Or have you identified a new problem?

It is essentially the same change, and made on the day I sent
the bug report for the first time. I conclude it was made
because of my bug report, even though I haven't seen the bug
report back on the mailing list, or seen any response.

Some sort of feedback would have been nice.

I'll let you know what happens when sqlite-3.14 is released.

Vincent.
-- 
WCC - Smart Search & Match
NL  +31 30 7503222
vzwe...@wcc-group.com
www.wcc-group.com
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon bug in shiftreduce action for error symbol

[Richard Hipp]

On 6/6/16, Vincent Zweije  wrote:

>
> When the shiftreduce action is used in an error context,
> such as:
>
>X -> alpha error.
>
> the adjustment is not made. This causes error handling to fail.
>

Is this problem fixed by
https://www.sqlite.org/src/fdiff?sbs=1=66a16b5e00fefff2=8c4e9d8517e50da3
from earlier this month?  Or have you identified a new problem?
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] lemon - %wildcar

[Sylvain Pointeau]

Hello,

Le samedi 26 juillet 2014, Richard Hipp  a écrit :
>
> Historical note:  I wrote Lemon back in the 80s (on a Sun3, IIRC)


You were a visionary, Lemon is probably the only viable option for a parser
generator in C/C++ at this moment.

because
> the $1,$2,$3 notation of Yacc wasn't working for me and also because Yacc
> wanted the parser to call the lexer, not the other way around, and because
> it was difficult to stop memory leaks in Yacc when there was a syntax
> error.  There was a companion LL(1) parser generator called "Lime" which
> has been lost.   Lemon was originally a bunch of separate source files,
> many of which were automatically generated.  But at some point I
> concatenated them all together into the single "lemon.c" file.  All that
> happened prior to the beginning of SQLite.  Anyhow, the current lemon.c
> code is a mess, for which I apologize.

But it does mostly work, so I've
> never gotten around to rewriting it from scratch, which is what it
> desperately needs.


Lemon is a master piece, it deserves it.
I propose my help, I am interested in the domain, but I might not be
capable enough. However I would be highly motivated if I am guided by
someone like you.


>
> The %wildcard token matches any token that cannot be matched by another
> rule in the current context.
>
>
It is exactly what I needed, I tried it and it is wonderful. I try to parse
some plsql code, but I only need to be superficial, then everything between
begin / end will be eaten by the wildcar


> I had to add the %wildcard feature to the parser generator when we added
> virtual tables to SQLite back in 2006 - in order to support the full text
> search extension.  The virtual table syntax allows one put any arbitrary
> sequence of tokens as the argument, as long as parentheses match up.
> Keywords, literals, operators - any tokens are allowed.  And the most
> memory-efficient way to do that was to enhance the parser generate with
> some special features to support it.  Hence:  "%wildcard".
>
>
I am lucky ;-)



> The lack of documentation is probably just because we never got around to
> it


> > if yes, could it be added in the documentation?
> >
>
> Do you have a patch :-)
>
>
Your documentation is among the best, I can help also there I think, how do
you want the patch to be done, I just need a little bit of explanation to
start. (I am on MacOS X)


>
> >
> > ps: I am pretty impressed by lemon, I am really having fun.
> >
> > Best regards,
> > Sylvain
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org 
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
>
>
>
> --
> D. Richard Hipp
> d...@sqlite.org 
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org 
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>

Best regards,
Sylvain
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] lemon - %wildcar

[Richard Hipp]

On Sat, Jul 26, 2014 at 11:10 AM, Sylvain Pointeau <
sylvain.point...@gmail.com> wrote:

> Hello,
>
> I would like to know if %wildcard is an undocumented feature on purpose, is
> this safe to use it?
>

Historical note:  I wrote Lemon back in the 80s (on a Sun3, IIRC) because
the $1,$2,$3 notation of Yacc wasn't working for me and also because Yacc
wanted the parser to call the lexer, not the other way around, and because
it was difficult to stop memory leaks in Yacc when there was a syntax
error.  There was a companion LL(1) parser generator called "Lime" which
has been lost.   Lemon was originally a bunch of separate source files,
many of which were automatically generated.  But at some point I
concatenated them all together into the single "lemon.c" file.  All that
happened prior to the beginning of SQLite.  Anyhow, the current lemon.c
code is a mess, for which I apologize.  But it does mostly work, so I've
never gotten around to rewriting it from scratch, which is what it
desperately needs.

The %wildcard token matches any token that cannot be matched by another
rule in the current context.

I had to add the %wildcard feature to the parser generator when we added
virtual tables to SQLite back in 2006 - in order to support the full text
search extension.  The virtual table syntax allows one put any arbitrary
sequence of tokens as the argument, as long as parentheses match up.
Keywords, literals, operators - any tokens are allowed.  And the most
memory-efficient way to do that was to enhance the parser generate with
some special features to support it.  Hence:  "%wildcard".

The lack of documentation is probably just because we never got around to
it


> if yes, could it be added in the documentation?
>

Do you have a patch :-)


>
> ps: I am pretty impressed by lemon, I am really having fun.
>
> Best regards,
> Sylvain
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] (Lemon) (Patch) adding a -f option to Lemon to emit function prototypes

[Tiago Rodrigues]

I see; well, that's unfortunate.  At any rate, parsing lempar.c will
probably be much more complex than the code I have now, which solves a
simple (but common) case.  Maybe I'll look into parsing lempar.c, but since
this solves my use case, I probably won't be in a hurry to do it.  I
apologise to everyone this doesn't help, then.

Cheers,

  -Tiago


On Mon, Jan 7, 2013 at 9:13 PM, Richard Hipp  wrote:

> On Mon, Jan 7, 2013 at 4:18 PM, Tiago Rodrigues  wrote:
>
> > Oops, Dominique alerted me to the fact that the patch I included was
> > stripped by the listserver...  I'm including it inline, then.
> >
>
>
> This patch does not work in the general case.  I don't doubt that it works
> for your use case, but in general it does not.  The reason:  The function
> prototypes depend on the "lempar.c" template file.  Some applications (ex:
> SQLite) change the lempar.c parser template in ways that could change the
> function signatures.  (The function signatures for SQLite are unchanged,
> but that doesn't exclude the possibility that they might in the future.)
>
> For this to really work in general, I suppose you would have to somehow
> extract the function signatures from the lempar.c template file.  Or maybe
> have a separate lempar.h template that contains the header.  Or something.
>
>
>
> >
> > (Begin patch)
> >
> > --- lemon.c2013-01-04 20:39:20 +
> > +++ lemon-new.c2013-01-04 23:09:59 +
> > @@ -109,7 +109,7 @@
> >  void Reprint(struct lemon *);
> >  void ReportOutput(struct lemon *);
> >  void ReportTable(struct lemon *, int);
> > -void ReportHeader(struct lemon *);
> > +void ReportHeader(struct lemon *, int);
> >  void CompressTables(struct lemon *);
> >  void ResortStates(struct lemon *);
> >
> > @@ -1393,11 +1393,13 @@
> >static int mhflag = 0;
> >static int nolinenosflag = 0;
> >static int noResort = 0;
> > +  static int fpflag = 0;
> >static struct s_options options[] = {
> >  {OPT_FLAG, "b", (char*), "Print only the basis in
> report."},
> >  {OPT_FLAG, "c", (char*), "Don't compress the action
> table."},
> >  {OPT_FSTR, "D", (char*)handle_D_option, "Define an %ifdef macro."},
> >  {OPT_FSTR, "T", (char*)handle_T_option, "Specify a template file."},
> > +{OPT_FLAG, "f", (char*), "Generate function prototypes in
> > header."},
> >  {OPT_FLAG, "g", (char*), "Print grammar without actions."},
> >  {OPT_FLAG, "m", (char*), "Output a makeheaders compatible
> > file."},
> >  {OPT_FLAG, "l", (char*), "Do not print #line
> > statements."},
> > @@ -1502,7 +1504,7 @@
> >  /* Produce a header file for use by the scanner.  (This step is
> >  ** omitted if the "-m" option is used because makeheaders will
> >  ** generate the file for us.) */
> > -if( !mhflag ) ReportHeader();
> > +if( !mhflag ) ReportHeader(, fpflag);
> >}
> >if( statistics ){
> >  printf("Parser statistics: %d terminals, %d nonterminals, %d
> rules\n",
> > @@ -4009,16 +4011,20 @@
> >  }
> >
> >  /* Generate a header file for the parser */
> > -void ReportHeader(struct lemon *lemp)
> > +void ReportHeader(struct lemon *lemp, int fpflag)
> >  {
> >FILE *out, *in;
> > +  const char *name;
> >const char *prefix;
> >char line[LINESIZE];
> >char pattern[LINESIZE];
> >int i;
> > +  int protok = 1;
> >
> >if( lemp->tokenprefix ) prefix = lemp->tokenprefix;
> >elseprefix = "";
> > +  if( lemp->name ) name = lemp->name;
> > +  else name = "Parse";
> >in = file_open(lemp,".h","rb");
> >if( in ){
> >  int nextChar;
> > @@ -4026,9 +4032,24 @@
> >sprintf(pattern,"#define %s%-30s
> > %2d\n",prefix,lemp->symbols[i]->name,i);
> >if( strcmp(line,pattern) ) break;
> >  }
> > +if( fpflag ){
> > +sprintf(pattern,"void *%sAlloc(void *(*)(size_t));\n",name);
> > +if( !fgets(line,LINESIZE,in) || strcmp(line,pattern) ){ protok =
> > 0; goto after; }
> > +
> > +if( lemp->arg ){
> > +sprintf(pattern,"void %s(void
> > *,int,%s,%s);\n",name,lemp->tokentype,lemp->arg);
> > +}else{
> > +sprintf(pattern,"void %s(void
> > *,int,%s);\n",name,lemp->tokentype);
> > +}
> > +if( !fgets(line,LINESIZE,in) || strcmp(line,pattern) ){ protok =
> > 0; goto after; }
> > +
> > +sprintf(pattern,"void *%sFree(void *,void (*)(void
> *));\n",name);
> > +if( !fgets(line,LINESIZE,in) || strcmp(line,pattern) ){ protok =
> > 0; goto after; }
> > +}
> > +after:
> >  nextChar = fgetc(in);
> >  fclose(in);
> > -if( i==lemp->nterminal && nextChar==EOF ){
> > +if( i==lemp->nterminal && protok && nextChar==EOF ){
> >/* No change in the file.  Don't rewrite it. */
> >return;
> >  }
> > @@ -4038,6 +4059,16 @@
> >  for(i=1; interminal; i++){
> >fprintf(out,"#define %s%-30s
> > %2d\n",prefix,lemp->symbols[i]->name,i);
> >  }
> > +

Re: [sqlite] (Lemon) (Patch) adding a -f option to Lemon to emit function prototypes

[Richard Hipp]

On Mon, Jan 7, 2013 at 4:18 PM, Tiago Rodrigues  wrote:

> Oops, Dominique alerted me to the fact that the patch I included was
> stripped by the listserver...  I'm including it inline, then.
>


This patch does not work in the general case.  I don't doubt that it works
for your use case, but in general it does not.  The reason:  The function
prototypes depend on the "lempar.c" template file.  Some applications (ex:
SQLite) change the lempar.c parser template in ways that could change the
function signatures.  (The function signatures for SQLite are unchanged,
but that doesn't exclude the possibility that they might in the future.)

For this to really work in general, I suppose you would have to somehow
extract the function signatures from the lempar.c template file.  Or maybe
have a separate lempar.h template that contains the header.  Or something.



>
> (Begin patch)
>
> --- lemon.c2013-01-04 20:39:20 +
> +++ lemon-new.c2013-01-04 23:09:59 +
> @@ -109,7 +109,7 @@
>  void Reprint(struct lemon *);
>  void ReportOutput(struct lemon *);
>  void ReportTable(struct lemon *, int);
> -void ReportHeader(struct lemon *);
> +void ReportHeader(struct lemon *, int);
>  void CompressTables(struct lemon *);
>  void ResortStates(struct lemon *);
>
> @@ -1393,11 +1393,13 @@
>static int mhflag = 0;
>static int nolinenosflag = 0;
>static int noResort = 0;
> +  static int fpflag = 0;
>static struct s_options options[] = {
>  {OPT_FLAG, "b", (char*), "Print only the basis in report."},
>  {OPT_FLAG, "c", (char*), "Don't compress the action table."},
>  {OPT_FSTR, "D", (char*)handle_D_option, "Define an %ifdef macro."},
>  {OPT_FSTR, "T", (char*)handle_T_option, "Specify a template file."},
> +{OPT_FLAG, "f", (char*), "Generate function prototypes in
> header."},
>  {OPT_FLAG, "g", (char*), "Print grammar without actions."},
>  {OPT_FLAG, "m", (char*), "Output a makeheaders compatible
> file."},
>  {OPT_FLAG, "l", (char*), "Do not print #line
> statements."},
> @@ -1502,7 +1504,7 @@
>  /* Produce a header file for use by the scanner.  (This step is
>  ** omitted if the "-m" option is used because makeheaders will
>  ** generate the file for us.) */
> -if( !mhflag ) ReportHeader();
> +if( !mhflag ) ReportHeader(, fpflag);
>}
>if( statistics ){
>  printf("Parser statistics: %d terminals, %d nonterminals, %d rules\n",
> @@ -4009,16 +4011,20 @@
>  }
>
>  /* Generate a header file for the parser */
> -void ReportHeader(struct lemon *lemp)
> +void ReportHeader(struct lemon *lemp, int fpflag)
>  {
>FILE *out, *in;
> +  const char *name;
>const char *prefix;
>char line[LINESIZE];
>char pattern[LINESIZE];
>int i;
> +  int protok = 1;
>
>if( lemp->tokenprefix ) prefix = lemp->tokenprefix;
>elseprefix = "";
> +  if( lemp->name ) name = lemp->name;
> +  else name = "Parse";
>in = file_open(lemp,".h","rb");
>if( in ){
>  int nextChar;
> @@ -4026,9 +4032,24 @@
>sprintf(pattern,"#define %s%-30s
> %2d\n",prefix,lemp->symbols[i]->name,i);
>if( strcmp(line,pattern) ) break;
>  }
> +if( fpflag ){
> +sprintf(pattern,"void *%sAlloc(void *(*)(size_t));\n",name);
> +if( !fgets(line,LINESIZE,in) || strcmp(line,pattern) ){ protok =
> 0; goto after; }
> +
> +if( lemp->arg ){
> +sprintf(pattern,"void %s(void
> *,int,%s,%s);\n",name,lemp->tokentype,lemp->arg);
> +}else{
> +sprintf(pattern,"void %s(void
> *,int,%s);\n",name,lemp->tokentype);
> +}
> +if( !fgets(line,LINESIZE,in) || strcmp(line,pattern) ){ protok =
> 0; goto after; }
> +
> +sprintf(pattern,"void *%sFree(void *,void (*)(void *));\n",name);
> +if( !fgets(line,LINESIZE,in) || strcmp(line,pattern) ){ protok =
> 0; goto after; }
> +}
> +after:
>  nextChar = fgetc(in);
>  fclose(in);
> -if( i==lemp->nterminal && nextChar==EOF ){
> +if( i==lemp->nterminal && protok && nextChar==EOF ){
>/* No change in the file.  Don't rewrite it. */
>return;
>  }
> @@ -4038,6 +4059,16 @@
>  for(i=1; interminal; i++){
>fprintf(out,"#define %s%-30s
> %2d\n",prefix,lemp->symbols[i]->name,i);
>  }
> +if( fpflag ){
> +/* emit function prototypes */
> +fprintf(out,"void *%sAlloc(void *(*)(size_t));\n",name);
> +if( lemp->arg ){
> +fprintf(out,"void %s(void
> *,int,%s,%s);\n",name,lemp->tokentype,lemp->arg);
> +}else{
> +fprintf(out,"void %s(void *,int,%s);\n",name,lemp->tokentype);
> +}
> +fprintf(out,"void *%sFree(void *,void (*)(void *));\n",name);
> +}
>  fclose(out);
>}
>return;
>
> (End patch)
>
> Thanks,
>
>   -Tiago
>
>
> On Mon, Jan 7, 2013 at 6:15 AM, Dominique Devienne  >wrote:
>
> > On Sat, Jan 5, 2013 at 6:22 PM, Tiago Rodrigues 

Re: [sqlite] (Lemon) (Patch) adding a -f option to Lemon to emit function prototypes

[Tiago Rodrigues]

Oops, Dominique alerted me to the fact that the patch I included was
stripped by the listserver...  I'm including it inline, then.

(Begin patch)

--- lemon.c2013-01-04 20:39:20 +
+++ lemon-new.c2013-01-04 23:09:59 +
@@ -109,7 +109,7 @@
 void Reprint(struct lemon *);
 void ReportOutput(struct lemon *);
 void ReportTable(struct lemon *, int);
-void ReportHeader(struct lemon *);
+void ReportHeader(struct lemon *, int);
 void CompressTables(struct lemon *);
 void ResortStates(struct lemon *);

@@ -1393,11 +1393,13 @@
   static int mhflag = 0;
   static int nolinenosflag = 0;
   static int noResort = 0;
+  static int fpflag = 0;
   static struct s_options options[] = {
 {OPT_FLAG, "b", (char*), "Print only the basis in report."},
 {OPT_FLAG, "c", (char*), "Don't compress the action table."},
 {OPT_FSTR, "D", (char*)handle_D_option, "Define an %ifdef macro."},
 {OPT_FSTR, "T", (char*)handle_T_option, "Specify a template file."},
+{OPT_FLAG, "f", (char*), "Generate function prototypes in
header."},
 {OPT_FLAG, "g", (char*), "Print grammar without actions."},
 {OPT_FLAG, "m", (char*), "Output a makeheaders compatible
file."},
 {OPT_FLAG, "l", (char*), "Do not print #line
statements."},
@@ -1502,7 +1504,7 @@
 /* Produce a header file for use by the scanner.  (This step is
 ** omitted if the "-m" option is used because makeheaders will
 ** generate the file for us.) */
-if( !mhflag ) ReportHeader();
+if( !mhflag ) ReportHeader(, fpflag);
   }
   if( statistics ){
 printf("Parser statistics: %d terminals, %d nonterminals, %d rules\n",
@@ -4009,16 +4011,20 @@
 }

 /* Generate a header file for the parser */
-void ReportHeader(struct lemon *lemp)
+void ReportHeader(struct lemon *lemp, int fpflag)
 {
   FILE *out, *in;
+  const char *name;
   const char *prefix;
   char line[LINESIZE];
   char pattern[LINESIZE];
   int i;
+  int protok = 1;

   if( lemp->tokenprefix ) prefix = lemp->tokenprefix;
   elseprefix = "";
+  if( lemp->name ) name = lemp->name;
+  else name = "Parse";
   in = file_open(lemp,".h","rb");
   if( in ){
 int nextChar;
@@ -4026,9 +4032,24 @@
   sprintf(pattern,"#define %s%-30s
%2d\n",prefix,lemp->symbols[i]->name,i);
   if( strcmp(line,pattern) ) break;
 }
+if( fpflag ){
+sprintf(pattern,"void *%sAlloc(void *(*)(size_t));\n",name);
+if( !fgets(line,LINESIZE,in) || strcmp(line,pattern) ){ protok =
0; goto after; }
+
+if( lemp->arg ){
+sprintf(pattern,"void %s(void
*,int,%s,%s);\n",name,lemp->tokentype,lemp->arg);
+}else{
+sprintf(pattern,"void %s(void
*,int,%s);\n",name,lemp->tokentype);
+}
+if( !fgets(line,LINESIZE,in) || strcmp(line,pattern) ){ protok =
0; goto after; }
+
+sprintf(pattern,"void *%sFree(void *,void (*)(void *));\n",name);
+if( !fgets(line,LINESIZE,in) || strcmp(line,pattern) ){ protok =
0; goto after; }
+}
+after:
 nextChar = fgetc(in);
 fclose(in);
-if( i==lemp->nterminal && nextChar==EOF ){
+if( i==lemp->nterminal && protok && nextChar==EOF ){
   /* No change in the file.  Don't rewrite it. */
   return;
 }
@@ -4038,6 +4059,16 @@
 for(i=1; interminal; i++){
   fprintf(out,"#define %s%-30s %2d\n",prefix,lemp->symbols[i]->name,i);
 }
+if( fpflag ){
+/* emit function prototypes */
+fprintf(out,"void *%sAlloc(void *(*)(size_t));\n",name);
+if( lemp->arg ){
+fprintf(out,"void %s(void
*,int,%s,%s);\n",name,lemp->tokentype,lemp->arg);
+}else{
+fprintf(out,"void %s(void *,int,%s);\n",name,lemp->tokentype);
+}
+fprintf(out,"void *%sFree(void *,void (*)(void *));\n",name);
+}
 fclose(out);
   }
   return;

(End patch)

Thanks,

  -Tiago


On Mon, Jan 7, 2013 at 6:15 AM, Dominique Devienne wrote:

> On Sat, Jan 5, 2013 at 6:22 PM, Tiago Rodrigues  wrote:
> > [...]
> > This patch works against the current version of lemon.c, as linked to by
> > the Lemon page.  This is probably not correct, as the file is amalgamated
> > from other sources, but still might be useful to somebody.
>
> Hi Tiago. Looks like the patch was stripped by the mailing list.
> Perhaps you should include it inline to your message. Thanks, --DD
>



-- 
In those days, in those distant days, in those nights, in those remote
nights, in those years, in those distant years...
  - Gilgamesh, Enkidu and the Underworld
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon: Non-terminal destructors and cleanup

[Ben]

On 18 Oct 2012, at 20:07, Richard Hipp  wrote:

> On Thu, Oct 18, 2012 at 3:03 PM, Ben  wrote:
> 
>> Hi list,
>> 
>> I'm having a little trouble getting my head around memory management
>> within a Lemon-generated parser. Specifically the part of the docs stating
>> when a destructor will or will not be called.
>> 
>> For example, this is a portion of a grammar based on the SQLite parse.y
>> file:
>> 
>> 
>> columnName ::= nm(N). {
>>BSSQLiteColumn *col = [[BSSQLiteColumn alloc] init];
>>col.name = N.textValue;
>>[[parsedTable columns] addObject:col];
>> }
>> 
>> nm(A) ::= id(X). { A = X; }
>> nm(A) ::= STRING(X). { A = X; }
>> id(A) ::= ID(X). { A = X; }
>> 
>> Notes:
>> - The token type here is a struct containing an Objective-C string which
>> needs freeing when done with.
>> - Only a %token_destructor is defined, not any others
>> 
>> 
>> I know that the last three assignments are leaking memory, but I don't
>> know when I should be explicitly freeing my allocated memory within a token
>> and when I should be relying on the destructor defined by
>> %token_destructor{}. Or for that matter whether I should be declaring a
>> more specific symbol destructor.
>> 
>> Can anyone shed some light on how this should be done?
>> 
> 
> If the nonterminal payload is passed into an action (as in your example
> where N is processed because of nm(N)) then Lemon assume that your code
> will free the content, if needed.
> 
> If the rule had been:  columnName ::= nm {...}   (without the (N) argument
> to nm) then the destructor would have been called.

Got it. I've added two release calls for X at the end of the C code blocks for 
these two:
nm(A) ::= STRING(X). { A = X; }
id(A) ::= ID(X). { A = X; }

and now it's working leak-free.

Thank you.

Ben


> The destructor is also called if nm is popped from the stack for any reason
> other than the columnName ::= nm rule, such as when the stack is popped
> during error recovery.
> 
> 
>> 
>> Thanks,
>> 
>> Ben
>> 
>> 
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>> 
> 
> 
> 
> -- 
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon: Non-terminal destructors and cleanup

[Richard Hipp]

On Thu, Oct 18, 2012 at 3:03 PM, Ben  wrote:

> Hi list,
>
> I'm having a little trouble getting my head around memory management
> within a Lemon-generated parser. Specifically the part of the docs stating
> when a destructor will or will not be called.
>
> For example, this is a portion of a grammar based on the SQLite parse.y
> file:
>
>
> columnName ::= nm(N). {
> BSSQLiteColumn *col = [[BSSQLiteColumn alloc] init];
> col.name = N.textValue;
> [[parsedTable columns] addObject:col];
> }
>
> nm(A) ::= id(X). { A = X; }
> nm(A) ::= STRING(X). { A = X; }
> id(A) ::= ID(X). { A = X; }
>
> Notes:
> - The token type here is a struct containing an Objective-C string which
> needs freeing when done with.
> - Only a %token_destructor is defined, not any others
>
>
> I know that the last three assignments are leaking memory, but I don't
> know when I should be explicitly freeing my allocated memory within a token
> and when I should be relying on the destructor defined by
> %token_destructor{}. Or for that matter whether I should be declaring a
> more specific symbol destructor.
>
> Can anyone shed some light on how this should be done?
>

If the nonterminal payload is passed into an action (as in your example
where N is processed because of nm(N)) then Lemon assume that your code
will free the content, if needed.

If the rule had been:  columnName ::= nm {...}   (without the (N) argument
to nm) then the destructor would have been called.

The destructor is also called if nm is popped from the stack for any reason
other than the columnName ::= nm rule, such as when the stack is popped
during error recovery.


>
> Thanks,
>
> Ben
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon: broken lookahead propagation?

[Vincent Zweije]

On Sat, Jan 07, 2012 at 10:18:12AM -0500, Richard Hipp wrote:

||  On Wed, Jan 4, 2012 at 11:00 AM, Vincent Zweije  wrote:
||
||  > I may have hit a bug in the lemon parser generator.
||  >
||
||  Please see if the following fix clears your problem:
||
||  http://www.sqlite.org/src/info/ce32775b23

Looks good: it produces a few more 's and other terminals in
the starters list, but I'm sure you checked that too. Haven't checked
the generated parser yet on actual input yet.

Thanks.  Vincent.
-- 
Vincent Zweije    | "If you're flamed in a group you
  | don't read, does anybody get burnt?"
[Xhost should be taken out and shot] |-- Paul Tomblin on a.s.r.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon: broken lookahead propagation?

[Richard Hipp]

On Wed, Jan 4, 2012 at 11:00 AM, Vincent Zweije  wrote:

> I may have hit a bug in the lemon parser generator.
>

Please see if the following fix clears your problem:

http://www.sqlite.org/src/info/ce32775b23



>
> It looks like lookahead symbols aren't propagated properly, in some cases.
>
> The consequence is that in some states, valid symbols are rejected with
> a syntax error.
>
> The complete grammar can be found at
>
>http://www.zweije.nl/~vzweije/parser.y
>
> I'm processing the grammar with lemon version 3.7.7-2ubuntu2. I've
> compared that version to the current head in sqlite, and it only differs
> in non-essential areas (calls to fclose recently added in trunk, calls
> to snprintf added in ubuntu). I'm pretty sure the current head version
> of lemon will exhibit the problem as well.
>
> For completeness' sake, this was found on an armv7l system.
>
> I have just reproduced the problem with the head lemon.c on an i386
> system. Valgrind reports no stray memory accesses (though quite a few
> memory leaks, but those are probably harmless).
>
> The command to process the parser is:
>
> lemon -c -s parser.y
>
> The generated parser has, among others, the following state:
>
> State 244:
>  constructor_def ::= existental_quant_variables_opt upper_name *
> fix_opt type_clos
>(200) fix_opt ::= *
>  fix_opt ::= * Fix
>
>  Open reduce 200
> CurlyOpen reduce 200
>  Hash reduce 200
>  RectOpen reduce 200
> A reduce 200
>   Dot reduce 200
>   Exclaim reduce 200
>  Star reduce 200
>   Int reduce 200
>  Real reduce 200
>  Char reduce 200
>  Bool reduce 200
>  RectOpenExclaimRectClose reduce 200
>   Dynamic reduce 200
>  File reduce 200
>String reduce 200
> World reduce 200
>   Fix shift  486
>   LowerCaseId reduce 200
>   UpperCaseId reduce 200
>   FunnyId reduce 200
>   fix_opt shift  130
>
> As can be seen, fix_opt is nullable - it may derive the empty string.
> Therefore, any symbol that can start what follows it (type_clos),
> or any lookahead symbol (because type_clos is nullable too), must
> be acceptable in this state. However, some tokens (Bar, Semicolon,
> LowerCaseId, UpperCaseId, FunnyId) are not.
>
> The next state lists the acceptable symbols there:
>
> State 130:
>  type_clos ::= * open_type_clos
>  type_clos ::= * closed_type_clos
>  open_type_clos ::= * closed_type_clos type_prefix_seq
>  closed_type_seq ::= * closed_type_clos type_prefix_clos
> type_expression
>  closed_type_clos ::= * closed_type_seq
>(144) closed_type_clos ::= *
>  constructor_def ::= existental_quant_variables_opt upper_name
> fix_opt * type_clos
>
>  Open reduce 144
> CurlyOpen reduce 144
>   Bar reduce 144
>  Hash reduce 144
>  RectOpen reduce 144
> A reduce 144
>   Dot reduce 144
>   Exclaim reduce 144
>  Star reduce 144
>   Int reduce 144
>  Real reduce 144
>  Char reduce 144
>  Bool reduce 144
>  RectOpenExclaimRectClose reduce 144
>   Dynamic reduce 144
>  File reduce 144
>String reduce 144
> World reduce 144
>   LowerCaseId reduce 144
>   UpperCaseId reduce 144
>   FunnyId reduce 144
> Semicolon reduce 144
> type_clos shift  563
>   closed_type_seq shift  375
>  closed_type_clos shift  179
>open_type_clos shift  491
>
> Some of the missing symbols (LowerCaseId, UpperCaseId, FunnyId) are
> starters of type_clos:
>
>  164: type_clos: Open CurlyOpen Hash RectOpen A Dot Exclaim Star Int Real
> Char Bool RectOpenExclaimRectClose Dynamic File String World LowerCaseId
> UpperCaseId FunnyId
>
> The other two (Bar, Semicolon) are not; these are followers of the
> constructor_def symbol.
>
> Looking at it now,  is not in the starters of type_clos,
> though according to the grammar rules, it is obviously nullable
> (type_clos => closed_type_close => ). Curious.
>
> The problem disappears when fix_opt is deleted from the RHS of the
> rule. (Of course, this means that a present Fix token is no longer
> accepted.)
>
> Ciao.   

Re: [sqlite] Lemon behavior

[Christian Smith]

On Tue, Dec 07, 2010 at 08:09:53PM +0100, Begelman, Jamie wrote:
> I'm using Lemon for a non-sqlite related project and it is exiting with an 
> assertion failure that I would like to understand. I have extracted the 
> following small set of productions from a larger grammar. The "list" 
> production happens to be the start symbol in the larger grammar.
> 
> list::= list DELIMITER command.
> list::= command.
> command ::= TERMINAL1.
> command ::= TERMINAL2.
> 
> When I place these in a .y file by themselves and build the file, Lemon fails 
> with:
> 
> Assertion failed: apx->type==SH_RESOLVED || apx->type==RD_RESOLVED || 
> apx->type==SSCONFLICT || apx->type==SRCONFLICT || apx->type==RRCONFLICT || 
> apy->type==SH_RESOLVED || apy->type==RD_RESOLVED || apy->type==SSCONFLICT || 
> apy->type==SRCONFLICT || apy->type==RRCONFLICT, file lemon.c, line 1065
> 
> The odd thing is I use this pattern to parse lists of things elsewhere in the 
> grammar without issue. Any insight appreciated.
> 


I observed that adding a single rule to the beginning of the grammar:

main::= list.

Fixed things for me. Try that. I've no idea why it makes a difference, mind. 
This was tried
with lemon in SQLite 3.5.9, from Debian lenny.

Christian
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon maintainer

[Richard Hipp]

On Tue, Oct 12, 2010 at 6:54 PM, Vincent Adam Burns wrote:

> Is there an active maintainer for the Lemon Parser? I'm getting some
> parsing conflicts, ex:
>
> statement ::= IF LEFT_PAREN expression RIGHT_PAREN statement ELSE
> statement.
> statement ::= IF LEFT_PAREN expression RIGHT_PAREN statement.
>
> expr ::= expr EQ|NE|GT|GE|LT|LE expr.
> expr ::= expr MOD|DIV|MOD expr.
> expr ::= expr ADD|SUB expr.
> expr ::= primaryExpr.
>

Problems with Lemon should be directed to the SQLite mailing list.

However, parsing conflicts are not problems with Lemon, but rather problems
with your grammar.  Unfortunately there is (to my knowledge) no tutorial on
how to write conflict-free LALR(1) grammers for Lemon.  You have to grok
LALR(1) first, then Lemon just makes sense.

Perhaps you should start with a copy of the Dragon book.  New copies are
wildly over-priced.  But it looks like you can get a used copy from Amazon
for just a few bucks.
http://www.amazon.com/Compilers-Principles-Techniques-Alfred-Aho/dp/0201100886



>
> --
> --- You know you've achieved perfection in design, not when you have
> nothing more to add, but when you have nothing more to take away.
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon parser

[Chris verBurg]

On Tue, Apr 6, 2010 at 10:40 PM, Dan Kennedy  wrote:

>
> On Apr 7, 2010, at 3:23 AM, Chris verBurg wrote:
>
> > Hehe, okay, here I go.  :)
> >
> >
> > I'm trying to replace an existing flex/bison parser with an re2c/lemon
> > parser, but I'm running into a methodological problem.  I have a
> > hypothetical grammar like this:
> >
> >  file ::= FOO str .
> >  file ::= BAR str .
>
> See the use of the "%fallback" directive in the parse.y
> file that is part of SQLite sources. I think you want
> something like this:
>
>   %fallback str BAR.
>
>
That does almost exactly what I want.  Thank you thank you thank you!  :)
Do you know why that isn't documented?  (Or, am I looking at the wrong
documentation?)

-Chris
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon parser

[Dan Kennedy]

On Apr 7, 2010, at 3:23 AM, Chris verBurg wrote:

> Hehe, okay, here I go.  :)
>
>
> I'm trying to replace an existing flex/bison parser with an re2c/lemon
> parser, but I'm running into a methodological problem.  I have a
> hypothetical grammar like this:
>
>  file ::= FOO str .
>  file ::= BAR str .

See the use of the "%fallback" directive in the parse.y
file that is part of SQLite sources. I think you want
something like this:

   %fallback str BAR.


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon parser

[Chris verBurg]

Hehe, okay, here I go.  :)


I'm trying to replace an existing flex/bison parser with an re2c/lemon
parser, but I'm running into a methodological problem.  I have a
hypothetical grammar like this:

  file ::= FOO str .
  file ::= BAR str .

Where my keywords are FOO and BAR, and str is any ol' generic string.

I'm giving it the input "FOO BAR" in the hopes that it will match the first
rule and give str the value "BAR".  To keep Lemon from choking on a syntax
error when it sees BAR, I want to tell the lexer (re2c) to force the next
token after "FOO" to be a string.  Thus I broke the rule up like so:

  file ::= fooDefinition .

  fooDefinition ::= fooStart fooEnd .
  fooStart ::= FOO . { lexer->forceNextThingToBeAString(); }
  fooEnd ::= str .

However, the call to forceNextThingToBeAString() happens after re2c reads
"BAR", and by that time it's already identified it as the keyword BAR
instead of as a string that happens to be set to "BAR".  It seems like Lemon
always reads a lookahead token, even though it doesn't need to.

I considered using the lexer-only hack of having the lexer force a string
after "FOO", but that only works if every FOO in the grammar is followed by
str.  In my real-world problem, that's not the case.  The CFG of the lexer
is not adequate to know when to force strings; it requires control from the
parser.


Would you have any comments/ideas?

Thanks!
-Chris


On Tue, Apr 6, 2010 at 1:12 PM, Wilson, Ronald  wrote:

> > I'm using the Lemon parser and running into a methodological problem
> that
> > I
> > wanted to ask the user base about.  Is there a mailing list or forum
> > specifically for Lemon, or is this it?  :)
> >
> > Thanks,
> > -Chris
>
> There is no mailing list specifically for lemon.  Some of us are
> familiar to various degrees with lemon, so feel free to ask here.
> Though I must say, lemon questions tend languish for a while before
> someone replies.  Don't be afraid to bump your own message if it isn't
> answered.  I think it's safe to say that most sqlite users aren't
> initimately familiar with lemon.
>
> RW
>
> Ron Wilson, Engineering Project Lead
> (o) 434.455.6453, (m) 434.851.1612, www.harris.com
>
> HARRIS CORPORATION   |   RF Communications Division
> assuredcommunications(tm)
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon parser

[Wilson, Ronald]

> I'm using the Lemon parser and running into a methodological problem
that
> I
> wanted to ask the user base about.  Is there a mailing list or forum
> specifically for Lemon, or is this it?  :)
> 
> Thanks,
> -Chris

There is no mailing list specifically for lemon.  Some of us are
familiar to various degrees with lemon, so feel free to ask here.
Though I must say, lemon questions tend languish for a while before
someone replies.  Don't be afraid to bump your own message if it isn't
answered.  I think it's safe to say that most sqlite users aren't
initimately familiar with lemon.

RW

Ron Wilson, Engineering Project Lead
(o) 434.455.6453, (m) 434.851.1612, www.harris.com

HARRIS CORPORATION   |   RF Communications Division
assuredcommunications(tm)
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] lemon: error handling

[D. Richard Hipp]

On Dec 22, 2009, at 6:06 PM, Benjamin Peterson wrote:

> I'm using lemon to write a parser for a little language I'm writing.
> I'm wondering how I indicate to lemon that an error has occurred in
> processing and an exit is needed. For example, if I have:
>
> stmt(A) ::= NAME(B). { A = malloc(sizeof(stmt)); A->name = B; }
>
> If malloc returns NULL, what should I do to escape the parser?

Configure the %extra_argument to be a pointer to a structure that  
holds the state of your parser.

  %extra_argument {Parser *pParser}

The pParser pointer is directly accessible from within your actions.   
So set a flag in that structure that tell the tokenizer to stop  
sending tokens and abort with an error.

 stmt(A) ::= NAME(B).  {
A = malloc(sizeof(stmt));
if( A==0 ){
   pParser->errFlag = 1;
}else{
  A->name = B;
   }
}

>
> -- 
> Regards,
> Benjamin
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

D. Richard Hipp
d...@hwaci.com



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon parser : compile error when using "%token_destructor" directive

[Wilson, Ron P]

Very interesting.  I had no idea that lemon.c and lempar.c were being revised.  
I assumed they were static.

Would it be too much to increment the version that lemon -x prints?  Currently 
it prints out "Lemon version 1.0" which led me to believe it was not being 
actively developed.

RW

P.S.  I like what I see in the revision history.

Ron Wilson, Engineering Project Lead, 434.455.6453

HARRIS CORPORATION   |   RF Communications Division 
assuredcommunications(tm)


-Original Message-
From: Vincent Zweije [mailto:vzwe...@wcc-group.com] 
Sent: Monday, June 15, 2009 12:54 PM
To: Wilson, Ron P
Cc: General Discussion of SQLite Database
Subject: Re: [sqlite] Lemon parser : compile error when using 
"%token_destructor" directive

On Mon, Jun 15, 2009 at 11:42:26AM -0400, Wilson, Ron P wrote:

||  It has been a while since I used lemon (big fan though).  Did you resolve
||  this issue or do you still need help?

[It appears my previous response did not get through.]

Looks suspiciously like this problem, which was fixed in version 3.6.2:

http://www.sqlite.org/cvstrac/tktview?tn=3299

Ciao.

||  -Original Message-
||  From: sqlite-users-boun...@sqlite.org 
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of ferrety ferrety
||  Sent: Monday, June 08, 2009 7:15 PM
||  To: sqlite-users@sqlite.org
||  Subject: [sqlite] Lemon parser : compile error when using 
"%token_destructor" directive
||
||  Hi List,
||  This is the only place I found to ask for "lemon parser" error.
||
||  When trying to use the "%token_destructor" directive:
||
||  %include {
||  #include 
||  void token_dtor (struct Token * t)
||{
||  fprintf(stderr, "In token_destructor: t -> value=%s\n", t -> value);
||}
||  }
||  %token_destructor { token_dtor($$); }
||
||
||  I got the following error:
||  error: too many arguments to function 'yy_destructor'
||
||  I tried to understand how "lemon" generated the calls for that function and
||  found that:
||
||  1) The definition of "yy_destructor" is with only 2 arguments:
||  static void yy_destructor(YYCODETYPE yymajor, YYMINORTYPE *yypminor){
||switch( yymajor ){
||  /* Here is inserted the actions which take place when a
||  ...
||
||  2) "Lemon" generates different calls to "yy_destructor", sometimes with 2 or
||  3 parameters :
||  yy_destructor( yymajor, >minor); <- HERE with 2 arguments,
||  which is fine
||  yy_destructor(yypParser,4,[0].minor); <- HERE with 3 arguments, which
||  is an error
||  yy_destructor(yypParser,5,[-2].minor); <- same
||  yy_destructor(yypParser,6,[0].minor);  <- same
||  ...
||
||  Sometime, the "yy_destructor" si called with 2 arguments as defined and some
||  other time,
||  it's called with 3 arguments.
||
||  Is it a bug or am I missing something?
-- 
WCC - Smart Search & Match
NL  +31 30 7503222
vzwe...@wcc-group.com
www.wcc-group.com
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon parser : compile error when using "%token_destructor" directive

[Vincent Zweije]

On Mon, Jun 15, 2009 at 11:42:26AM -0400, Wilson, Ron P wrote:

||  It has been a while since I used lemon (big fan though).  Did you resolve
||  this issue or do you still need help?

[It appears my previous response did not get through.]

Looks suspiciously like this problem, which was fixed in version 3.6.2:

http://www.sqlite.org/cvstrac/tktview?tn=3299

Ciao.

||  -Original Message-
||  From: sqlite-users-boun...@sqlite.org 
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of ferrety ferrety
||  Sent: Monday, June 08, 2009 7:15 PM
||  To: sqlite-users@sqlite.org
||  Subject: [sqlite] Lemon parser : compile error when using 
"%token_destructor" directive
||
||  Hi List,
||  This is the only place I found to ask for "lemon parser" error.
||
||  When trying to use the "%token_destructor" directive:
||
||  %include {
||  #include 
||  void token_dtor (struct Token * t)
||{
||  fprintf(stderr, "In token_destructor: t -> value=%s\n", t -> value);
||}
||  }
||  %token_destructor { token_dtor($$); }
||
||
||  I got the following error:
||  error: too many arguments to function 'yy_destructor'
||
||  I tried to understand how "lemon" generated the calls for that function and
||  found that:
||
||  1) The definition of "yy_destructor" is with only 2 arguments:
||  static void yy_destructor(YYCODETYPE yymajor, YYMINORTYPE *yypminor){
||switch( yymajor ){
||  /* Here is inserted the actions which take place when a
||  ...
||
||  2) "Lemon" generates different calls to "yy_destructor", sometimes with 2 or
||  3 parameters :
||  yy_destructor( yymajor, >minor); <- HERE with 2 arguments,
||  which is fine
||  yy_destructor(yypParser,4,[0].minor); <- HERE with 3 arguments, which
||  is an error
||  yy_destructor(yypParser,5,[-2].minor); <- same
||  yy_destructor(yypParser,6,[0].minor);  <- same
||  ...
||
||  Sometime, the "yy_destructor" si called with 2 arguments as defined and some
||  other time,
||  it's called with 3 arguments.
||
||  Is it a bug or am I missing something?
-- 
WCC - Smart Search & Match
NL  +31 30 7503222
vzwe...@wcc-group.com
www.wcc-group.com
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon parser : compile error when using "%token_destructor" directive

[Wilson, Ron P]

It has been a while since I used lemon (big fan though).  Did you resolve this 
issue or do you still need help?

RW

Ron Wilson, Engineering Project Lead, 434.455.6453

HARRIS CORPORATION   |   RF Communications Division 
assuredcommunications(tm)


-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of ferrety ferrety
Sent: Monday, June 08, 2009 7:15 PM
To: sqlite-users@sqlite.org
Subject: [sqlite] Lemon parser : compile error when using "%token_destructor" 
directive

Hi List,
This is the only place I found to ask for "lemon parser" error.

When trying to use the "%token_destructor" directive:

%include {
#include 
void token_dtor (struct Token * t)
  {
fprintf(stderr, "In token_destructor: t -> value=%s\n", t -> value);
  }
}
%token_destructor { token_dtor($$); }


I got the following error:
error: too many arguments to function 'yy_destructor'

I tried to understand how "lemon" generated the calls for that function and
found that:

1) The definition of "yy_destructor" is with only 2 arguments:
static void yy_destructor(YYCODETYPE yymajor, YYMINORTYPE *yypminor){
  switch( yymajor ){
/* Here is inserted the actions which take place when a
...

2) "Lemon" generates different calls to "yy_destructor", sometimes with 2 or
3 parameters :
yy_destructor( yymajor, >minor); <- HERE with 2 arguments,
which is fine
yy_destructor(yypParser,4,[0].minor); <- HERE with 3 arguments, which
is an error
yy_destructor(yypParser,5,[-2].minor); <- same
yy_destructor(yypParser,6,[0].minor);  <- same
...

Sometime, the "yy_destructor" si called with 2 arguments as defined and some
other time,
it's called with 3 arguments.

Is it a bug or am I missing something?

Thanks in advance
Frederic
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon: Functionality like Bison's %error-verbose / Access to last follow set from %syntax_error

[Markus Thiele]

Greetings,

> Lemon does not have any feature that will provide the application with  
> access to the follow-set.  You could perhaps tease that informatino  
> out of the "*.out" output file using a script, though.

Capital idea! That does indeed do the trick. It's straightforward to 
extract the terminals that can be shifted in each state from the .out 
file and thus generate a mapping from states to follow sets.

Afterwards, the current state can be queried from %syntax_error as 
yypParser->yystack[yypParser->yyidx].stateno (not part of the public 
interface of course, but it works), and thus a nice error message can be 
generated from the previously created collection of follow sets.

Thank you very much,
-- Markus Thiele
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon: Functionality like Bison's %error-verbose / Access to last follow set from %syntax_error

[D. Richard Hipp]

On Aug 17, 2008, at 1:48 PM, Markus Thiele wrote:

> Greetings,
>
> I've been using Lemon for a small custom compiler project. I've used
> Bison before, and I very much prefer the way Lemon does things,  
> there's
> just one feature I'm missing and haven't been able to find.
>
> Bison generates a human readable error message on syntax errors, and
> when setting %error-verbose this message contains some nice extra
> information, e.g. "Unexpected token A, expected C or D".
>
> As far as I can see, Lemon does not generate human readable messages  
> at
> all (except when ParseTrace is set, but then only to a stream and
> non-selectively).
>
> Now obviously I can generate the "Unexpected token A." part from the
> last token read, but for the rest I'd need access to the last follow  
> set
> before the error. So I've been wondering if there's any halfway nice  
> and
> clean solution for this, or if I'm maybe just missing an existing  
> feature.


Lemon does not have any feature that will provide the application with  
access to the follow-set.  You could perhaps tease that informatino  
out of the "*.out" output file using a script, though.

D. Richard Hipp
[EMAIL PROTECTED]



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon

[arjunkumar keerti]

Hi Arjen,

Thanks for the mail...I found it very helpful and i started to execute
it..I succeeded in doing it...

However i started writing a sample program for converting "Infix to
Prefix"...I am struck at this point ...so can u help me in
writing this program...

Thanks in advance

Regards,
Arjun

On Sat, Jun 28, 2008 at 12:54 AM, Arjen Markus <[EMAIL PROTECTED]>
wrote:

> > Hi,
> > Can u send me any documentation on LEMON that u have worked out.
> > I have some queries also.
> > 1.Does LEMON work on Windows environment?
> > 2.I tried but it is not.I heard that it works on Linux environment.I am
> > trying to know why not on windows...can u give me some info about it
> >
> > I am reading the material u have suggested me but it did not have any
> > information regarding to my queries.
> >
> > Can u mail me if u have made any documentation on LEMON.
> >
>
> Well, basically lemon works as follows:
> 1. It reads the definition of the parser from the input file
>   you give it as the first argument. From that it produces
>   a set of tables that implement the parser. This information
>   gets filled in in a template.
> 2. The result is a C file that you can use to parse (interpret)
>   your input.
> 3. What it does not do is split up the input into tokens. You
>   have to do that yourself.
>
> I attach a small - not very efficient - example that shows a
> few features of lemon.
>
> As for your questions:
> There is no reason it would not work on Windows: it is a
> perfectly straghtforward C program, you get the source code
> and compile it. One thing though: the template file must be
> in the directory where you run the program, where the
> executable lives or in the PATH.
>
> (I am using it myself on Windows :). I am adapting it so that
> it will produce Fortran code instead of C code. But that is
> off topic.)
>
> Here is my sample parser definition:
>
> // expr.y
> // Simple parser:
> // expr = vaue + value + value + ...
>
> %extra_argument {State *state}
> %token_type {Token*}
> %type term {int}
>
> expr ::= firstterm plusterms . {
>printf( "Result: %d\n", state->sum );
> }
>
> firstterm ::= term(T) . {
>state->sum = T;
>printf("First term: %d\n", T );
> }
>
> term(V) ::= NUMBER(N) . {
>sscanf( N->token, "%d",  );
>printf("Term: %d -- %s\n", V, N->token );
> }
>
> plusterms ::= .
> plusterms ::= plusterms plusterm .
>
> plusterm ::= PLUS term(T) . {
>state->sum = state->sum + T;
>printf( "Result so far: %d\n", state->sum );
> }
>
> --
> And here is the main program that includes the
> resulting C code:
>
> #include 
> #include 
> #include 
>
> typedef struct {
>char *token;
> } Token;
> typedef struct {
>int sum;
> } State;
>
> #include "expr.h"
> #include "expr.c"
>
> int main( int argc, char *argv[] ) {
>Token token[10];
>State state;
>
>void *pParser = (void *) ParseAlloc( malloc );
>
>ParseTrace( stdout, ">>" );
>
>token[0].token = "1"; Parse( pParser, NUMBER, [0],  );
>token[1].token = "+"; Parse( pParser, PLUS,   [1],  );
>token[2].token = "2"; Parse( pParser, NUMBER, [2],  );
>token[3].token = "+"; Parse( pParser, PLUS,   [3],  );
>token[4].token = "3"; Parse( pParser, NUMBER, [4],  );
>
>Parse( pParser, 0, [0],  );
>
>ParseFree( pParser, free );
>
>return 0;
> }
>
> -
> One thing to note:
> As the tokens are not always treated the moment they are passed
> to the parser - this depends on the grammar and the moment in the
> parsing process, you need to make sure that the tokens remain
> available. I have done that in this silly program by using an
> array of them. But a smarter program would use lemon's
> abilities to destroy the tokens when ready.
>
> Regards,
>
> Arjen
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon

[Arjen Markus]

> Hi,
> Can u send me any documentation on LEMON that u have worked out.
> I have some queries also.
> 1.Does LEMON work on Windows environment?
> 2.I tried but it is not.I heard that it works on Linux environment.I am
> trying to know why not on windows...can u give me some info about it
>
> I am reading the material u have suggested me but it did not have any
> information regarding to my queries.
>
> Can u mail me if u have made any documentation on LEMON.
>

Well, basically lemon works as follows:
1. It reads the definition of the parser from the input file
   you give it as the first argument. From that it produces
   a set of tables that implement the parser. This information
   gets filled in in a template.
2. The result is a C file that you can use to parse (interpret)
   your input.
3. What it does not do is split up the input into tokens. You
   have to do that yourself.

I attach a small - not very efficient - example that shows a
few features of lemon.

As for your questions:
There is no reason it would not work on Windows: it is a
perfectly straghtforward C program, you get the source code
and compile it. One thing though: the template file must be
in the directory where you run the program, where the
executable lives or in the PATH.

(I am using it myself on Windows :). I am adapting it so that
it will produce Fortran code instead of C code. But that is
off topic.)

Here is my sample parser definition:

// expr.y
// Simple parser:
// expr = vaue + value + value + ...

%extra_argument {State *state}
%token_type {Token*}
%type term {int}

expr ::= firstterm plusterms . {
printf( "Result: %d\n", state->sum );
}

firstterm ::= term(T) . {
state->sum = T;
printf("First term: %d\n", T );
}

term(V) ::= NUMBER(N) . {
sscanf( N->token, "%d",  );
printf("Term: %d -- %s\n", V, N->token );
}

plusterms ::= .
plusterms ::= plusterms plusterm .

plusterm ::= PLUS term(T) . {
state->sum = state->sum + T;
printf( "Result so far: %d\n", state->sum );
}

--
And here is the main program that includes the
resulting C code:

#include 
#include 
#include 

typedef struct {
char *token;
} Token;
typedef struct {
int sum;
} State;

#include "expr.h"
#include "expr.c"

int main( int argc, char *argv[] ) {
Token token[10];
State state;

void *pParser = (void *) ParseAlloc( malloc );

ParseTrace( stdout, ">>" );

token[0].token = "1"; Parse( pParser, NUMBER, [0],  );
token[1].token = "+"; Parse( pParser, PLUS,   [1],  );
token[2].token = "2"; Parse( pParser, NUMBER, [2],  );
token[3].token = "+"; Parse( pParser, PLUS,   [3],  );
token[4].token = "3"; Parse( pParser, NUMBER, [4],  );

Parse( pParser, 0, [0],  );

ParseFree( pParser, free );

return 0;
}

-
One thing to note:
As the tokens are not always treated the moment they are passed
to the parser - this depends on the grammar and the moment in the
parsing process, you need to make sure that the tokens remain
available. I have done that in this silly program by using an
array of them. But a smarter program would use lemon's
abilities to destroy the tokens when ready.

Regards,

Arjen

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon

[Arjen Markus]

arjunkumar keerti wrote:

>Hi,
>I found in wikipedia that Lemon parser is a part of SQLite project but i
>couldn't found any sort of information regarding to LEMON.
>Can u give me any documentation regarding how to install it and how to work
>for some programs on Lemon parser generator or any URL's that might be
>helpful and can download the information regarding LEMON.
>
>  
>
You can find information on lemon at: http://www.hwaci.com/sw/lemon/

(And having gained some limited experience with it myself, this last 
week, I can
say it works quite nicely, with one or two caveats, perhaps :))

Regards,

Arjen
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon: Conflicts with repeated TERMINALS

[Ralf Junker]

Joe Wilson <[EMAIL PROTECTED]> wrote:

>The following grammar may be clearer to you:

Yes, it is many thanks! I believe I am making progress! At least I can see the 
picture much clearer now and was able to come up with the following grammar 
with just one conflict unsolved:

  %left NEWLINE.   /* Do these matter here at all? */
  %nonassoc TEXT LINK.
  %left HEADING_START.
  %left HEADING_END.

  article ::= blocks.

  blocks ::= block. /* EOF */
  blocks ::= blocks NEWLINE./* EOF */
  blocks ::= blocks NEWLINE NEWLINE block.

  block ::= .   /* EOF */
  block ::= paragraph.
  block ::= heading.

  heading ::= HEADING_START text HEADING_END.

  paragraph ::= line.
  paragraph ::= paragraph NEWLINE line.

  line ::= text.

  text ::= textpiece.
  text ::= text textpiece.

  textpiece ::= TEXT.
  textpiece ::= LINK.

I of course appreciate any comments ;-) My idea is that

* A block can be either a paragraph or a heading. Multiple blocks are separated 
by two NEWLINEs.

* A paragraph is made up of n >= 1 lines. Each line within a paragraph ends 
with a single NEWLINE. Two NEWLINEs start a new block (see above).

* A line consists of text, which can be TEXT or LINK.

Not all works well with the grammer, and unfortunately I do not understand why. 
Given this input, for example:

  TEXT, NEWLINE

the parser gets stuck at

  paragraph ::= paragraph NEWLINE line.

instead of falling back to the line above
  
  paragraph ::= line.

to find the conditions of a paragraph fulfilled. Why does it not try the other 
alternatives? Or are there none in the grammar?

>Try reading some papers on parsing or search for the book
>"Compilers: Principles, Techniques, and Tools" (a.k.a. 
>the dragon book).

I certainly will.

>Also try writing on paper random sequences of tokens and 
>manually parse your grammar to see the conflicts firsthand.

As I throw different token sequences to my experimental parser I am slowly 
starting to make sense of the debugger output.

Ralf 


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon: Conflicts with repeated TERMINALS

[Joe Wilson]

--- Ralf Junker <[EMAIL PROTECTED]> wrote:
> >  paragraph ::= PARA text.
> 
> I observed the new PARA terminal token (the clear separator!?). Unfortunately 
> the lexer does not
> generate such a token. Paragraph repeats are also removed.

It was just an HTML-like example. I just wanted to demonstrate one
possible way to remove the conflicts by adding a special tag. 
I'm not suggesting that you alter your grammar in this way.

> >Here's another:
> >
> >  article ::= blocks.
> >
> >  blocks ::= block.
> >  blocks ::= blocks block.
> >
> >  block ::= heading NEWLINE.
> >  block ::= paragraph NEWLINE.
> >
> >  heading ::= HEADING_START text HEADING_END.
> >  heading ::= HEADING_START text.
> >  heading ::= HEADING_START.
> >
> >  paragraph ::= text.
> >
> >  text ::= textpiece.
> >  text ::= text textpiece.
> >
> >  textpiece ::= TEXT.
> >  textpiece ::= LINK.
> 
> This one also removes paragraph repeats, doesn't it? Unfortunately paragraphs 
> need to repeat for
> my grammar. Is there a way to achieve this without conflicts?

In your original grammar, you could have random sequences of TEXT 
and LINK and NEWLINE tokens without any way of differentiating whether 
they were part of a "text" or "paragraph" or "heading", hence the conflict.
So I figured that a paragraph may as well be any combination of
TEXT and LINK tokens ending with NEWLINE. The headings in my 2nd grammar
also have to end with NEWLINE. "paragraph" will not repeat, per se, but you 
can repeat "block" (see the "blocks" rules), where you can have several 
consecutive "block"s that happen to be of type paragraph so you can achieve 
the same effect. 

The following grammar may be clearer to you:

  article ::= blocks.

  blocks ::= block.
  blocks ::= blocks block.

  block ::= heading.
  block ::= paragraph.

  heading ::= HEADING_START text HEADING_END.
  heading ::= HEADING_START text NEWLINE.
  heading ::= HEADING_START NEWLINE.

  paragraph ::= NEWLINE.
  paragraph ::= text NEWLINE.

  text ::= textpiece.
  text ::= text textpiece.

  textpiece ::= TEXT.
  textpiece ::= LINK.

It's much the same as the previous grammar, but puts the NEWLINEs as part 
of the paragraph and heading rules instead of in the block rule.
The difference being that heading no longer needs to end with NEWLINE 
in one case if HEADING_END is encountered, as it is not ambiguous:

  heading ::= HEADING_START text HEADING_END.

and a paragraph in this grammar may also be empty so you can parse 
consecutive NEWLINE tokens with this rule:

  paragraph ::= NEWLINE.

Again, this was just an example of how to disambiguate the grammar. 
You can find other ways.

> >Lemon generates an .out file for the .y file processed.
> >You can examine it for errors.
> 
> I have tried to make sense of the .out file before. It tells me where to look 
> for the problem,
> but not how to fix it ...

Try reading some papers on parsing or search for the book
"Compilers: Principles, Techniques, and Tools" (a.k.a. 
the dragon book).

Also try writing on paper random sequences of tokens and 
manually parse your grammar to see the conflicts firsthand.



  

Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon: Conflicts with repeated TERMINALS

[Ralf Junker]

Many thanks, Joe,

>Your grammar is ambiguous. The text tokens run together for 
>various rules because the grammar lacks clear separators between 
>them. 

OK, I begin to understand. The "clear separators" need to be TERMINALs, right? 
I believed that these were imlicit because there are TEXT and LINK after all 
text tokens are fully expanded. Therefore I thought that the grammar would not 
be ambiguous.

>You can fix it a million ways by altering your grammar.

Thanks for the suggestions - I can see that they do not generate conflicts, but 
they certainly alter the grammar.

>Here is one way:
>
>  article ::= blocks.
>
>  blocks ::= block.
>  blocks ::= blocks block.
>
>  block ::= heading.
>  block ::= paragraph.
>
>  heading ::= HEADING_START text HEADING_END.
>  heading ::= HEADING_START text.
>  heading ::= HEADING_START.
>
>  paragraph ::= PARA text.
>
>  text ::= textpiece.
>  text ::= text textpiece.
>
>  textpiece ::= TEXT.
>  textpiece ::= LINK.

I observed the new PARA terminal token (the clear separator!?). Unfortunately 
the lexer does not generate such a token. Paragraph repeats are also removed.

>Here's another:
>
>  article ::= blocks.
>
>  blocks ::= block.
>  blocks ::= blocks block.
>
>  block ::= heading NEWLINE.
>  block ::= paragraph NEWLINE.
>
>  heading ::= HEADING_START text HEADING_END.
>  heading ::= HEADING_START text.
>  heading ::= HEADING_START.
>
>  paragraph ::= text.
>
>  text ::= textpiece.
>  text ::= text textpiece.
>
>  textpiece ::= TEXT.
>  textpiece ::= LINK.

This one also removes paragraph repeats, doesn't it? Unfortunately paragraphs 
need to repeat for my grammar. Is there a way to achieve this without conflicts?

>Lemon generates an .out file for the .y file processed.
>You can examine it for errors.

I have tried to make sense of the .out file before. It tells me where to look 
for the problem, but not how to fix it ...

I am sorry to appear stupid, but I still can not make sense of it all. Can 
someone still help, please?

Ralf 


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon: Conflicts with repeated TERMINALS

[Joe Wilson]

--- Ralf Junker <[EMAIL PROTECTED]> wrote:
> article ::= blocks.
> 
> blocks ::= block.
> blocks ::= blocks block.
> 
> block ::= heading.
> block ::= paragraph.
> 
> heading ::= HEADING_START text HEADING_END.
> heading ::= HEADING_START text.
> heading ::= HEADING_START.
> 
> paragraph ::= text NEWLINE.
> paragraph ::= paragraph text NEWLINE.
> paragraph ::= text.
> paragraph ::= paragraph text.
> 
> text ::= textpiece.
> text ::= text textpiece.
> 
> textpiece ::= TEXT.
> textpiece ::= LINK.

Your grammar is ambiguous. The text tokens run together for 
various rules because the grammar lacks clear separators between 
them. You can fix it a million ways by altering your grammar.

Here is one way:

  article ::= blocks.

  blocks ::= block.
  blocks ::= blocks block.

  block ::= heading.
  block ::= paragraph.

  heading ::= HEADING_START text HEADING_END.
  heading ::= HEADING_START text.
  heading ::= HEADING_START.

  paragraph ::= PARA text.

  text ::= textpiece.
  text ::= text textpiece.

  textpiece ::= TEXT.
  textpiece ::= LINK.

Here's another:

  article ::= blocks.

  blocks ::= block.
  blocks ::= blocks block.

  block ::= heading NEWLINE.
  block ::= paragraph NEWLINE.

  heading ::= HEADING_START text HEADING_END.
  heading ::= HEADING_START text.
  heading ::= HEADING_START.

  paragraph ::= text.

  text ::= textpiece.
  text ::= text textpiece.

  textpiece ::= TEXT.
  textpiece ::= LINK.

Lemon generates an .out file for the .y file processed.
You can examine it for errors.



  

Be a better sports nut!  Let your teams follow you 
with Yahoo Mobile. Try it now.  
http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon problem

[Téragone]

Hello,

I finally find the bug. In fact that was many bugs.

- Grammar rule order.
- Input string not reseted properly.

Thank you.

David
On Nov 18, 2007 10:30 AM, Joe Wilson <[EMAIL PROTECTED]> wrote:

> It's most likely your bug. Just add some debug prints in your grammar
> and tokenizer to see what's going on.
> See also: ParseTrace() in http://www.hwaci.com/sw/lemon/lemon.html
>
> --- Téragone <[EMAIL PROTECTED]> wrote:
> > I have a problem with a rule of a small calculator which accept
> variables :
> >
> > assignment(A) ::= VARIABLE(C) EQUAL expr(B).
> >
> > I can set variable with simple value or expression like :
> > a=2
> > b=2+3
> >
> > but when I try :
> > c=a+b
> >
> > The result is put in variable b instead of c.
> >
> > Is it my bug or a Lemon bug ?
>
>
>
>
>  
> 
> Be a better pen pal.
> Text or chat with friends inside Yahoo! Mail. See how.
> http://overview.mail.yahoo.com/
>
>
> -
> To unsubscribe, send email to [EMAIL PROTECTED]
>
> -
>
>

Re: [sqlite] Lemon problem

[Joe Wilson]

It's most likely your bug. Just add some debug prints in your grammar 
and tokenizer to see what's going on.
See also: ParseTrace() in http://www.hwaci.com/sw/lemon/lemon.html

--- Téragone <[EMAIL PROTECTED]> wrote:
> I have a problem with a rule of a small calculator which accept variables :
> 
> assignment(A) ::= VARIABLE(C) EQUAL expr(B).
> 
> I can set variable with simple value or expression like :
> a=2
> b=2+3
> 
> but when I try :
> c=a+b
> 
> The result is put in variable b instead of c.
> 
> Is it my bug or a Lemon bug ?



  

Be a better pen pal. 
Text or chat with friends inside Yahoo! Mail. See how.  
http://overview.mail.yahoo.com/

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon not reducing

[Gaspard Bucher]

For those reading this thread, I could solve my problme by using ragel
(http://www.cs.queensu.ca/~thurston/ragel/). You can define leaving
actions but also 'any change' actions. It was also easier to include
in a C++ project then lemon/flex. If my grammar becomes more
complicated, I heard it is possible to make a good collaboration
between ragel and lemon (ragel is then used as a tokenizer).

2007/10/25, Gaspard Bucher <[EMAIL PROTECTED]>:
> >  I do not understand why lemon waits for one more token when it has 
> >  enough information to reduce
> ...
> > 
> > >>> I don't think you can.  Why do you want to?  Why not just go
> > >>> ahead and send it the next token?
> > >>>
> > >> Most people find a way around this problem using white-space. This
> > >> could be a solution but then my grammar will be filled with
> > >> "white-space | nothing" rules and I thought Lemon could reduce when
> > >> there is no other way out of the current stack as it is more elegant.
> > >>
> > >>
> > LA(LR) is the answer - just drop Your's tokens as their arrive and give
> > a chance for the parser to be LALR, not LR or SLR :)
> >
> > mak
> >
> Lookahead improves what we can do with a grammar, but when there is no
> ambiguity, it should resolve without waiting for the next token. If
> your grammar is so simple it's an SLR, why not treat it so and ease
> the creation of grammars without needing to add "separator" tokens
> just so there is a lookahead (most languages use ";" or white space
> for that purpose).
>
> My grammar works with all "whitespace or nothing" rules, but I have to
> be careful to avoid conflicts. When I write it without the whitespace
> stuff (eaten in the tokenizer), I have no conflict. Simple example:
> main ::= commands.
> commands ::= .
> commands ::= commands ws command.
> command  ::= variable EQUAL value.
> variable ::= ws IDENTIFIER ws.
> value::= ws NUMBER ws.
> ws   ::= .
> ws   ::= WHITE.
> > 2 parsing conflicts.
>
> Whithout whitespace stuff:
> main ::= commands.
> commands ::= .
> commands ::= commands command.
> command  ::= variable EQUAL value.
> variable ::= IDENTIFIER.
> value::= NUMBER.
> > no conflicts, easier and cleaner.
>
> I know how to fix the first grammar, but you have to think of the
> actual succession to avoid two "ws" from coming next to the other,
> this thinking is not really related to the grammar from my point of
> view.
>
> Gaspard
>

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon not reducing

[Gaspard Bucher]

>  I do not understand why lemon waits for one more token when it has 
>  enough information to reduce
...
> 
> >>> I don't think you can.  Why do you want to?  Why not just go
> >>> ahead and send it the next token?
> >>>
> >> Most people find a way around this problem using white-space. This
> >> could be a solution but then my grammar will be filled with
> >> "white-space | nothing" rules and I thought Lemon could reduce when
> >> there is no other way out of the current stack as it is more elegant.
> >>
> >>
> LA(LR) is the answer - just drop Your's tokens as their arrive and give
> a chance for the parser to be LALR, not LR or SLR :)
>
> mak
>
Lookahead improves what we can do with a grammar, but when there is no
ambiguity, it should resolve without waiting for the next token. If
your grammar is so simple it's an SLR, why not treat it so and ease
the creation of grammars without needing to add "separator" tokens
just so there is a lookahead (most languages use ";" or white space
for that purpose).

My grammar works with all "whitespace or nothing" rules, but I have to
be careful to avoid conflicts. When I write it without the whitespace
stuff (eaten in the tokenizer), I have no conflict. Simple example:
main ::= commands.
commands ::= .
commands ::= commands ws command.
command  ::= variable EQUAL value.
variable ::= ws IDENTIFIER ws.
value::= ws NUMBER ws.
ws   ::= .
ws   ::= WHITE.
> 2 parsing conflicts.

Whithout whitespace stuff:
main ::= commands.
commands ::= .
commands ::= commands command.
command  ::= variable EQUAL value.
variable ::= IDENTIFIER.
value::= NUMBER.
> no conflicts, easier and cleaner.

I know how to fix the first grammar, but you have to think of the
actual succession to avoid two "ws" from coming next to the other,
this thinking is not really related to the grammar from my point of
view.

Gaspard

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon not reducing

[Grzegorz Makarewicz]

Gaspard Bucher wrote:
> PS: There is another reason, aside from aesthetics and simpler grammar
> to filter white spaces inside the tokenizer: you avoid all the parser
> conflicts you could get with "empty | or space" rules.
>
> 2007/10/24, Gaspard Bucher <[EMAIL PROTECTED]>:
>   
>>> Gaspard Bucher <[EMAIL PROTECTED]> wrote:
>>>   
 I do not understand why lemon waits for one more token when it has
 enough information to reduce.

 I want to recognize :
 foo = Bar()
 when the token CLOSE_PAR is received, not when an extra token is parsed..

 How can I avoid lemon waiting for the extra token before reducing ?

 
>>> I don't think you can.  Why do you want to?  Why not just go
>>> ahead and send it the next token?
>>>   
>> Most people find a way around this problem using white-space. This
>> could be a solution but then my grammar will be filled with
>> "white-space | nothing" rules and I thought Lemon could reduce when
>> there is no other way out of the current stack as it is more elegant.
>> I went into the sources and saw this comment:
>>
>> 
LA(LR) is the answer - just drop Your's tokens as their arrive and give
a chance for the parser to be LALR, not LR or SLR :)

mak


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon not reducing

[Gaspard Bucher]

PS: There is another reason, aside from aesthetics and simpler grammar
to filter white spaces inside the tokenizer: you avoid all the parser
conflicts you could get with "empty | or space" rules.

2007/10/24, Gaspard Bucher <[EMAIL PROTECTED]>:
> > Gaspard Bucher <[EMAIL PROTECTED]> wrote:
> > > I do not understand why lemon waits for one more token when it has
> > > enough information to reduce.
> > >
> > > I want to recognize :
> > > foo = Bar()
> > > when the token CLOSE_PAR is received, not when an extra token is parsed..
> > >
> > > How can I avoid lemon waiting for the extra token before reducing ?
> > >
> >
> > I don't think you can.  Why do you want to?  Why not just go
> > ahead and send it the next token?
> Most people find a way around this problem using white-space. This
> could be a solution but then my grammar will be filled with
> "white-space | nothing" rules and I thought Lemon could reduce when
> there is no other way out of the current stack as it is more elegant.
> I went into the sources and saw this comment:
>
> I think I might find something to do here:
> in yy_find_shift_action :
> when we find the action:
>
> // return yy_action[i];
> result = yy_action[i];
> printf("==> next : %i\n", result);
> printf("==> next_next: %i\n", yy_shift_ofst[result % (YYNSTATE+YYNRULE)]);
> // if the shift results in a default action do this action now.
> if (result < (YYNSTATE+YYNRULE) && yy_shift_ofst[result] ==
> YY_SHIFT_USE_DFLT) {
>   // do the change ourself
>
>   YYMINORTYPE yyminorunion;
>   yy_shift(pParser,result,iLookAhead,);
>   pParser->yyerrcnt--;
>   return yy_default[result];
> } else
>   return result;
>
> This works except when the reduction reaches state '1'. Somehow, it
> should jump directly to state '0', clearing the current lookahead.
>
> > --
> > D. Richard Hipp <[EMAIL PROTECTED]>
> >
> >
> > -
> > To unsubscribe, send email to [EMAIL PROTECTED]
> > -
> >
> >
> > !DSPAM:471f82d0320381804284693!
> >
> >
>

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon not reducing

[Gaspard Bucher]

> Gaspard Bucher <[EMAIL PROTECTED]> wrote:
> > I do not understand why lemon waits for one more token when it has
> > enough information to reduce.
> >
> > I want to recognize :
> > foo = Bar()
> > when the token CLOSE_PAR is received, not when an extra token is parsed..
> >
> > How can I avoid lemon waiting for the extra token before reducing ?
> >
>
> I don't think you can.  Why do you want to?  Why not just go
> ahead and send it the next token?
Most people find a way around this problem using white-space. This
could be a solution but then my grammar will be filled with
"white-space | nothing" rules and I thought Lemon could reduce when
there is no other way out of the current stack as it is more elegant.
I went into the sources and saw this comment:

I think I might find something to do here:
in yy_find_shift_action :
when we find the action:

// return yy_action[i];
result = yy_action[i];
printf("==> next : %i\n", result);
printf("==> next_next: %i\n", yy_shift_ofst[result % (YYNSTATE+YYNRULE)]);
// if the shift results in a default action do this action now.
if (result < (YYNSTATE+YYNRULE) && yy_shift_ofst[result] ==
YY_SHIFT_USE_DFLT) {
  // do the change ourself

  YYMINORTYPE yyminorunion;
  yy_shift(pParser,result,iLookAhead,);
  pParser->yyerrcnt--;
  return yy_default[result];
} else
  return result;

This works except when the reduction reaches state '1'. Somehow, it
should jump directly to state '0', clearing the current lookahead.

> --
> D. Richard Hipp <[EMAIL PROTECTED]>
>
>
> -
> To unsubscribe, send email to [EMAIL PROTECTED]
> -
>
>
> !DSPAM:471f82d0320381804284693!
>
>

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon not reducing

[drh]

Gaspard Bucher <[EMAIL PROTECTED]> wrote:
> I do not understand why lemon waits for one more token when it has
> enough information to reduce.
> 
> I want to recognize :
> foo = Bar()
> when the token CLOSE_PAR is received, not when an extra token is parsed.
> 
> How can I avoid lemon waiting for the extra token before reducing ?
> 

I don't think you can.  Why do you want to?  Why not just go
ahead and send it the next token?
--
D. Richard Hipp <[EMAIL PROTECTED]>


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon: Help on conflic resolution?

[Ralf Junker]

Richard,

this helped me greatly! I also derived from your example that I can use 
multiple characters without conflicts like this:

---

doc ::= inline_list.

// List of allowed characters. Add more as you like.

c ::= CHAR.
c ::= SPACE.

// The c character repeat.

chars ::= c.
chars ::= chars CHAR.

// Any sequence of just c and 'c' (c surrounded by apostrophes).

inline ::= c.
inline ::= APOS chars APOS.
 
// The inline repeat.
 
inline_list ::= inline.
inline_list ::= inline_list inline. 

-

Many thanks!

Ralf


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon: Help on conflic resolution?

[Ralf Junker]

Richard,

this helped me greatly! I also derived from your example that I can use 
multiple characters without conflicts like this:

---

doc ::= inline_list.

// List of allowed characters. Add more as you like.

c ::= CHAR.
c ::= SPACE.

// The c character repeat.

chars ::= c.
chars ::= chars CHAR.

// Any sequence of just c and 'c' (c surrounded by apostrophes).

inline ::= c.
inline ::= APOS chars APOS.
 
// The inline repeat.
 
inline_list ::= inline.
inline_list ::= inline_list inline. 

-

Many thanks!

Ralf


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon: Help on conflic resolution?

[drh]

Ralf Junker <[EMAIL PROTECTED]> wrote:
> I am writing to ask for help about how to solve The Lemon parser conflicts.
> 
> As part of a larger grammar, I am need to implement this regular expression 
> in Lemon:
> 
>   (.+|'.+')+
> 
> I tried lots of grammars, but all of them generated Lemon warnings. 
> 
> Maybe someone could have a look at the grammar below and let me know
> how the conflicts can be solved, and why they are generated in the 
> first place?
> 

doc ::= inline_list. 
 
// One ore more CHARs. 
 
chars ::= CHAR. 
chars ::= chars CHAR. 
 
// Any sequence of just CHARs and 'CHARs' (surrounded by apostrophes). 
 
inline ::= CHAR. 
inline ::= APOS chars APOS. 
 
// The repeat. This causes conflicts. Isn't it allowed? Workarounds? 
 
inline_list ::= inline.
inline_list ::= inline_list inline. 

--
D. Richard Hipp <[EMAIL PROTECTED]>


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon Parser - Modular & Extensible ?

[Christian Smith]

Uma Krishnan uttered:

Hey, There's no need to be offensive. I did not mean to be critical. Far 
from it, it does a great a job (far more than I'm capable of producing). 
What I was trying to find out was, if it is possible for a .y files to 
be broken such that it can be built on top on other .y files.



Sorry if I came across as offensive. That was not the intention. I was 
just a little confused about the question.


I think lemon can only handle single input files. But you can can include 
C source into your output C file using the %include directive. Check out 
the documentation at, if you haven't already done so:

http://www.hwaci.com/sw/lemon/lemon.html

Now, what may draw some critical analysis is top posting and hijacking an 
existing thread for a new topic... [snip]





 Not sure if this is the right group. But could not find a lemon parser 
user group.



This is the best group to ask. While not tied to SQLite, it appears to be 
maintained as part of SQLite (but I may be wrong.)


Christian

--
/"\
\ /ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL
 X   - AGAINST MS ATTACHMENTS
/ \

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon Parser - Modular & Extensible ?

[Uma Krishnan]

Hey, There's no need to be offensive. I did not mean to be critical. Far from 
it, it does a great a job (far more than I'm capable of producing). What I was 
trying to find out was, if it is possible for a .y files to be broken such that 
it can be built on top on other .y files. 
   
  Not sure if this is the right group. But could not find a lemon parser user 
group.
   
  

Christian Smith <[EMAIL PROTECTED]> wrote:
  Uma Krishnan uttered:

> Hello:
>
> Is lemon parser modular and extensible?


Extensible to do what? It generates parsers, and is self contained. It 
does a single job, and does it well. What more could you ask for?


>
> Thanks
>
> Uma
>
> Asif Lodhi wrote:
> Hi Everybody,
>
> I have just joined this mailing list as Sqlite looks like a good
> software solution to my needs. What I need right now is RE-assurance
> of "crash-recovery" that is mentioned on your front page. So, I would
> be thankful if you experts would give me an "accurate" and fair
> picture of the crash-recovery aspects of SQLite - without any hype.
>
> --
> Best regards,
>
> Asif
>
> -
> To unsubscribe, send email to [EMAIL PROTECTED]
> -
>
>
>

--
/"\
\ / ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL
X - AGAINST MS ATTACHMENTS
/ \

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon Parser - Modular & Extensible ?

[Christian Smith]

Uma Krishnan uttered:


Hello:

 Is lemon parser modular and extensible?



Extensible to do what? It generates parsers, and is self contained. It 
does a single job, and does it well. What more could you ask for?





 Thanks

 Uma

Asif Lodhi <[EMAIL PROTECTED]> wrote:
 Hi Everybody,

I have just joined this mailing list as Sqlite looks like a good
software solution to my needs. What I need right now is RE-assurance
of "crash-recovery" that is mentioned on your front page. So, I would
be thankful if you experts would give me an "accurate" and fair
picture of the crash-recovery aspects of SQLite - without any hype.

--
Best regards,

Asif

-
To unsubscribe, send email to [EMAIL PROTECTED]
-





--
/"\
\ /ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL
 X   - AGAINST MS ATTACHMENTS
/ \

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

RE: [sqlite] Lemon parser generator question

[Medi Montaseri]

Thanks again Ulrik...the problem was with the tokenizer and not the
parser...

Medi 

-Original Message-
From: Medi Montaseri [mailto:[EMAIL PROTECTED] 
Sent: Friday, May 04, 2007 5:14 PM
To: sqlite-users@sqlite.org
Subject: RE: [sqlite] Lemon parser generator question

Thanks Ulrik...I am now looking for the lexer...

Medi

-Original Message-
From: Ulrik Petersen [mailto:[EMAIL PROTECTED]
Sent: Friday, May 04, 2007 4:30 PM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] Lemon parser generator question

Medi,

Lemon is a parser, not a lexer.  The terminals are defined outside of
Lemon.  Perhaps you inherited the .y file but did not receive the
lexer/tokenizer?

SQLite, for example, has a hand-coded tokenizer.  Other projects (such
as my own) may use a lexer-generator such as flex.


In Lemon, the tokenizer drives the parser, not the other way around.  
For more information, you can see the sources of SQLite, which includes
a document on Lemon, or you can read this:

http://www.hwaci.com/sw/lemon/lemon.html


HTH


Ulrik Petersen




Medi Montaseri wrote:
> Hi,
>  
> Firstly, if this is not the proper forum for Lemon questions, please
let
> me know where I need to go...
>  
> Second, I am looking at a SQL grammer written for Lemon parser
generator
> and am failing to see where some terminals are defined. For example 
> COMMA, FROM, SELECT are terminals and releatively easy to deduce.
> However in the grammer I am studying, I see references to NAME and 
> STRING (all in uppercase, indicating a terminal production).
>  
> Here is my higher level problem...
>  
> I have inherited a SqlGrammer.y that does not parse quoted-table-name 
> and is failing in cases where the FROM clause have things like "my, 
> table". For example select * from "my, table". Currently it works with

> select * from "table one"
> Chasing the grammer rules, it see a rule that reads
> from_source(P) ::= NAME(C).
> {
> someClass:someAction()
> }
> Based on Lemons' doc, NAME must be a terminal rule (all upper case)
and
> since I don't define it, then lemon must.
>  
> Can someone shed some light on this
>  
> Thanks
> Medi
>
>   



-
To unsubscribe, send email to [EMAIL PROTECTED]

-



-
To unsubscribe, send email to [EMAIL PROTECTED]

-


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

RE: [sqlite] Lemon parser generator question

[Medi Montaseri]

Thanks Ulrik...I am now looking for the lexer...

Medi

-Original Message-
From: Ulrik Petersen [mailto:[EMAIL PROTECTED] 
Sent: Friday, May 04, 2007 4:30 PM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] Lemon parser generator question

Medi,

Lemon is a parser, not a lexer.  The terminals are defined outside of
Lemon.  Perhaps you inherited the .y file but did not receive the
lexer/tokenizer?

SQLite, for example, has a hand-coded tokenizer.  Other projects (such
as my own) may use a lexer-generator such as flex.


In Lemon, the tokenizer drives the parser, not the other way around.  
For more information, you can see the sources of SQLite, which includes 
a document on Lemon, or you can read this:

http://www.hwaci.com/sw/lemon/lemon.html


HTH


Ulrik Petersen




Medi Montaseri wrote:
> Hi,
>  
> Firstly, if this is not the proper forum for Lemon questions, please
let
> me know where I need to go...
>  
> Second, I am looking at a SQL grammer written for Lemon parser
generator
> and am failing to see where some terminals are defined. For example
> COMMA, FROM, SELECT are terminals and releatively easy to deduce.
> However in the grammer I am studying, I see references to NAME and
> STRING (all in uppercase, indicating a terminal production). 
>  
> Here is my higher level problem...
>  
> I have inherited a SqlGrammer.y that does not parse quoted-table-name
> and is failing in cases where the FROM clause have things like "my,
> table". For example select * from "my, table". Currently it works with
> select * from "table one" 
> Chasing the grammer rules, it see a rule that reads
> from_source(P) ::= NAME(C).
> {
> someClass:someAction()
> }
> Based on Lemons' doc, NAME must be a terminal rule (all upper case)
and
> since I don't define it, then lemon must.
>  
> Can someone shed some light on this
>  
> Thanks
> Medi
>
>   



-
To unsubscribe, send email to [EMAIL PROTECTED]

-


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon parser generator question

[Ulrik Petersen]

Medi,

Lemon is a parser, not a lexer.  The terminals are defined outside of 
Lemon.  Perhaps you inherited the .y file but did not receive the 
lexer/tokenizer?


SQLite, for example, has a hand-coded tokenizer.  Other projects (such 
as my own) may use a lexer-generator such as flex.



In Lemon, the tokenizer drives the parser, not the other way around.  
For more information, you can see the sources of SQLite, which includes 
a document on Lemon, or you can read this:


http://www.hwaci.com/sw/lemon/lemon.html


HTH


Ulrik Petersen




Medi Montaseri wrote:

Hi,
 
Firstly, if this is not the proper forum for Lemon questions, please let

me know where I need to go...
 
Second, I am looking at a SQL grammer written for Lemon parser generator

and am failing to see where some terminals are defined. For example
COMMA, FROM, SELECT are terminals and releatively easy to deduce.
However in the grammer I am studying, I see references to NAME and
STRING (all in uppercase, indicating a terminal production). 
 
Here is my higher level problem...
 
I have inherited a SqlGrammer.y that does not parse quoted-table-name

and is failing in cases where the FROM clause have things like "my,
table". For example select * from "my, table". Currently it works with
select * from "table one" 
Chasing the grammer rules, it see a rule that reads

from_source(P) ::= NAME(C).
{
someClass:someAction()
}
Based on Lemons' doc, NAME must be a terminal rule (all upper case) and
since I don't define it, then lemon must.
 
Can someone shed some light on this
 
Thanks

Medi

  



-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] lemon compile parse.y error for windows

[drh]

[EMAIL PROTECTED] wrote:
> 
> then I use the command
> 
> lemon parse.y 
> 
> I successfully get the parse.h file
> 
> but the file is not right. 
> I only get 137 ids
> 
> the follwing 15 ids do not exist in the parse.h
> 
> TK_TO_TEXT
> 
> TK_CONST_FUNC 
> 

You need to run the awk script "addopcodes.awk" in order to
add these additional values.
--
D. Richard Hipp  <[EMAIL PROTECTED]>


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon example

[Martin Jenkins]

Cesar Rodas wrote:

The URL is ok, I opened here...


Works fine in .uk too

Martin

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

RE: [sqlite] Lemon example

[Doug Nebeker]

URL didn't work for me either, but you'll see what you want at the root
(at least today):
http://www.cesarodas.com/



-Original Message-
From: Cesar Rodas [mailto:[EMAIL PROTECTED] 
Sent: Monday, March 05, 2007 10:44 AM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] Lemon example

The URL is ok, I opened here... try again and let me know if you could
not and I will email you the content of the example.

On 05/03/07, Clay Dowling <[EMAIL PROTECTED]> wrote:
>
>
> Cesar Rodas wrote:
> > Here is  Lemon tutorial. Shows how to make a calculator with a
> feature  of
> > use Parents "()" in math expression.
> >
> http://www.cesarodas.com/2007/03/creating-basic-calculator-with-lemon.
> html
> .
> >
> > The author disclaims the copyright of the calculator.
>
> There's a problem with that URL.  Could you check it, or check your 
> server?  I'd love to read the example, but my browser steadfastly 
> refuses to resolve an address for cesarodas.com
>
> Clay Dowling
> --
> Simple Content Management
> http://www.ceamus.com
>
>
>
> --
> --- To unsubscribe, send email to 
> [EMAIL PROTECTED]
>
> --
> ---
>
>


--
Cesar Rodas
http://www.sf.net/projects/pagerank (The PageRank made easy...)
http://www.sf.net/projects/fastfs ( The Fast File System) Mobile Phone:
595 961 974165
Phone: 595 21 645590
[EMAIL PROTECTED]
[EMAIL PROTECTED]


This email was sent to you by Reuters, the global news and information company. 
To find out more about Reuters visit www.about.reuters.com

Any views expressed in this message are those of the individual sender, 
except where the sender specifically states them to be the views of Reuters 
Limited.

Reuters Limited is part of the Reuters Group of companies, of which Reuters 
Group PLC is the ultimate parent company.
Reuters Group PLC - Registered office address: The Reuters Building, South 
Colonnade, Canary Wharf, London E14 5EP, United Kingdom
Registered No: 3296375
Registered in England and Wales



-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] Lemon example

[Cesar Rodas]

The URL is ok, I opened here... try again and let me know if you could not
and I will email you the content of the example.

On 05/03/07, Clay Dowling <[EMAIL PROTECTED]> wrote:



Cesar Rodas wrote:
> Here is  Lemon tutorial. Shows how to make a calculator with a
feature  of
> use Parents "()" in math expression.
>
http://www.cesarodas.com/2007/03/creating-basic-calculator-with-lemon.html
.
>
> The author disclaims the copyright of the calculator.

There's a problem with that URL.  Could you check it, or check your
server?  I'd love to read the example, but my browser steadfastly refuses
to resolve an address for cesarodas.com

Clay Dowling
--
Simple Content Management
http://www.ceamus.com



-
To unsubscribe, send email to [EMAIL PROTECTED]

-





--
Cesar Rodas
http://www.sf.net/projects/pagerank (The PageRank made easy...)
http://www.sf.net/projects/fastfs ( The Fast File System)
Mobile Phone: 595 961 974165
Phone: 595 21 645590
[EMAIL PROTECTED]
[EMAIL PROTECTED]

Re: [sqlite] Lemon example

[Clay Dowling]

Cesar Rodas wrote:
> Here is  Lemon tutorial. Shows how to make a calculator with a feature  of
> use Parents "()" in math expression.
> http://www.cesarodas.com/2007/03/creating-basic-calculator-with-lemon.html.
>
> The author disclaims the copyright of the calculator.

There's a problem with that URL.  Could you check it, or check your
server?  I'd love to read the example, but my browser steadfastly refuses
to resolve an address for cesarodas.com

Clay Dowling
-- 
Simple Content Management
http://www.ceamus.com


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] lemon issue

[Gunnar Roth]

[EMAIL PROTECTED] schrieb:
> dear all:
> i am a programmer from China, i use sqlite in my project. 
> from sqlite, i know lemon.
> i work on windows, using vc6.0,  i am trying to do something using lemon, 
> but i have some issues.
> i write  a my_calculator.y file, and generate my_calculator.c and 
> my_calculator.h file from the my_calculator.y file, i rename file name from 
> my_calculator.c to my_calculator.cpp.
> when i  add the files(.h and .cpp) to my project, i find it could not be 
> compiled.
>
> error message:
> MyCal.obj : error LNK2001: unresolved external symbol _PtCalculatorParser
> MyCal.obj : error LNK2001: unresolved external symbol _PtCalculatorParserFree
> MyCal.obj : error LNK2001: unresolved external symbol _PtCalculatorParserAlloc
> but , in my_calculator.cpp, i have define the the function 
> PtCalculatorParser
> what't wrong? who can help me?
>  thanks.
>   
learn about c++ name mangling and about the extern "C" declaration.

regards,
gunnar


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

RE: [sqlite] LEMON

[Downey, Shawn]

yes 


Shawn M. Downey
MPR Associates
10 Maxwell Drive, Suite 204
Clifton Park, New York 12065
518-371-3983 x113 (work)
860-508-5015 (cell)

-Original Message-
From: Cesar Rodas [mailto:[EMAIL PROTECTED] 
Sent: Friday, December 29, 2006 11:47 AM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] LEMON

Lemon generates C files that could be compiled in Win or Linux Right?

On 29/12/06, Downey, Shawn <[EMAIL PROTECTED]> wrote:
>
> http://www.hwaci.com/sw/lemon/lemon.html
> Sorry if this link has already been pointed out to you.
>
>
> http://www.webdotdev.com/nvd/server-side/c/lemon-parser-generator-tuto
> ri
> al.html
> Looks OK but I have not looked at this site before.
>
> Shawn M. Downey
> MPR Associates
> 10 Maxwell Drive, Suite 204
> Clifton Park, New York 12065
> 518-371-3983 x113 (work)
> 860-508-5015 (cell)
>
> -Original Message-
> From: Cesar Rodas [mailto:[EMAIL PROTECTED]
> Sent: Friday, December 29, 2006 11:17 AM
> To: sqlite-users@sqlite.org; [EMAIL PROTECTED]
> Subject: Re: [sqlite] LEMON
>
> Another URL where i can find a tutorial?
>
> On 29/12/06, Lloyd <[EMAIL PROTECTED]> wrote:
> >
> > Ethereal make use of Lemon
> >
> > On Fri, 2006-12-29 at 11:08 -0400, Cesar Rodas wrote:
> > > Where can i find a tutorial with examples of how to use LEMON
> parser...
> > > because i need to build interpreted language
> > >
> > > Thanks to all
> >
> >
> > __
> > Scanned and protected by Email scanner
> >
> >
> > 
> > --
> > --- To unsubscribe, send email to 
> > [EMAIL PROTECTED]
> >
> > 
> > --
> > ---
> >
> >
>
>
> --
> Cesar Rodas
> http://www.phpclasses.org/grank (A PHP implementation of PageRank)
>
>
> --
> --- To unsubscribe, send email to 
> [EMAIL PROTECTED]
>
> --
> ---
>
>


--
Cesar Rodas
http://www.phpclasses.org/grank (A PHP implementation of PageRank)

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] LEMON

[Cesar Rodas]

Lemon generates C files that could be compiled in Win or Linux Right?

On 29/12/06, Downey, Shawn <[EMAIL PROTECTED]> wrote:


http://www.hwaci.com/sw/lemon/lemon.html
Sorry if this link has already been pointed out to you.


http://www.webdotdev.com/nvd/server-side/c/lemon-parser-generator-tutori
al.html
Looks OK but I have not looked at this site before.

Shawn M. Downey
MPR Associates
10 Maxwell Drive, Suite 204
Clifton Park, New York 12065
518-371-3983 x113 (work)
860-508-5015 (cell)

-Original Message-
From: Cesar Rodas [mailto:[EMAIL PROTECTED]
Sent: Friday, December 29, 2006 11:17 AM
To: sqlite-users@sqlite.org; [EMAIL PROTECTED]
Subject: Re: [sqlite] LEMON

Another URL where i can find a tutorial?

On 29/12/06, Lloyd <[EMAIL PROTECTED]> wrote:
>
> Ethereal make use of Lemon
>
> On Fri, 2006-12-29 at 11:08 -0400, Cesar Rodas wrote:
> > Where can i find a tutorial with examples of how to use LEMON
parser...
> > because i need to build interpreted language
> >
> > Thanks to all
>
>
> __
> Scanned and protected by Email scanner
>
>
> --
> --- To unsubscribe, send email to
> [EMAIL PROTECTED]
>
> --
> ---
>
>


--
Cesar Rodas
http://www.phpclasses.org/grank (A PHP implementation of PageRank)


-
To unsubscribe, send email to [EMAIL PROTECTED]

-





--
Cesar Rodas
http://www.phpclasses.org/grank (A PHP implementation of PageRank)

Re: [sqlite] LEMON

[epankoke]

I'm not familiar with Lemon, so I don't know if this will help: 
http://linuxgazette.net/106/chirico.html.  Also, I'd highly recommend just 
doing a Google search for either "Lemon Parser" or "Lemon Tutorial".

--
Eric Pankoke
Founder / Lead Developer
Point Of Light Software
http://www.polsoftware.com/

 -- Original message --
From: "Cesar Rodas" <[EMAIL PROTECTED]>
> Another URL where i can find a tutorial?
> 
> On 29/12/06, Lloyd <[EMAIL PROTECTED]> wrote:
> >
> > Ethereal make use of Lemon
> >
> > On Fri, 2006-12-29 at 11:08 -0400, Cesar Rodas wrote:
> > > Where can i find a tutorial with examples of how to use LEMON parser...
> > > because i need to build interpreted language
> > >
> > > Thanks to all
> >
> >
> > __
> > Scanned and protected by Email scanner
> >
> >
> > -
> > To unsubscribe, send email to [EMAIL PROTECTED]
> >
> > -
> >
> >
> 
> 
> -- 
> Cesar Rodas
> http://www.phpclasses.org/grank (A PHP implementation of PageRank)


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

RE: [sqlite] LEMON

[Downey, Shawn]

http://www.hwaci.com/sw/lemon/lemon.html
Sorry if this link has already been pointed out to you.


http://www.webdotdev.com/nvd/server-side/c/lemon-parser-generator-tutori
al.html
Looks OK but I have not looked at this site before.

Shawn M. Downey
MPR Associates
10 Maxwell Drive, Suite 204
Clifton Park, New York 12065
518-371-3983 x113 (work)
860-508-5015 (cell)

-Original Message-
From: Cesar Rodas [mailto:[EMAIL PROTECTED] 
Sent: Friday, December 29, 2006 11:17 AM
To: sqlite-users@sqlite.org; [EMAIL PROTECTED]
Subject: Re: [sqlite] LEMON

Another URL where i can find a tutorial?

On 29/12/06, Lloyd <[EMAIL PROTECTED]> wrote:
>
> Ethereal make use of Lemon
>
> On Fri, 2006-12-29 at 11:08 -0400, Cesar Rodas wrote:
> > Where can i find a tutorial with examples of how to use LEMON
parser...
> > because i need to build interpreted language
> >
> > Thanks to all
>
>
> __
> Scanned and protected by Email scanner
>
>
> --
> --- To unsubscribe, send email to 
> [EMAIL PROTECTED]
>
> --
> ---
>
>


--
Cesar Rodas
http://www.phpclasses.org/grank (A PHP implementation of PageRank)

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] LEMON

[Cesar Rodas]

Another URL where i can find a tutorial?

On 29/12/06, Lloyd <[EMAIL PROTECTED]> wrote:


Ethereal make use of Lemon

On Fri, 2006-12-29 at 11:08 -0400, Cesar Rodas wrote:
> Where can i find a tutorial with examples of how to use LEMON parser...
> because i need to build interpreted language
>
> Thanks to all


__
Scanned and protected by Email scanner


-
To unsubscribe, send email to [EMAIL PROTECTED]

-





--
Cesar Rodas
http://www.phpclasses.org/grank (A PHP implementation of PageRank)

Re: [sqlite] LEMON

[Lloyd]

Ethereal make use of Lemon

On Fri, 2006-12-29 at 11:08 -0400, Cesar Rodas wrote:
> Where can i find a tutorial with examples of how to use LEMON parser...
> because i need to build interpreted language
> 
> Thanks to all


__
Scanned and protected by Email scanner

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] lemon segfault

[Dennis Cote]

Michael Somos wrote:

I found the following for "lemon" in the sqlite-3.3.6 distribution :

===
  

lemon -x


Lemon version 1.0
  

lemon /dev/null


Segmentation fault
  

gdb ./lemon


GNU gdb 6.4
Copyright 2005 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...Using host libthread_db library 
"/lib/libthread_db.so.1".

(gdb) run /dev/null
Starting program: /home/tmp/lemon /dev/null

Program received signal SIGSEGV, Segmentation fault.
0x08053b5a in confighash (a=0xb740) at lemon.c:4607
4607  h = h*571 + a->rp->index*37 + a->dot;
(gdb) bt
#0  0x08053b5a in confighash (a=0xb740) at lemon.c:4607
#1  0x08053fab in Configtable_find (key=0xb740) at lemon.c:4720
#2  0x0804a83b in Configlist_addbasis (rp=0x41, dot=0) at lemon.c:1165
#3  0x08049935 in FindStates (lemp=0xb7e0) at lemon.c:723
#4  0x0804b37e in main (argc=2, argv=0xb904) at lemon.c:1482
(gdb) quit
===

  

Michael,

You should report this bug at http://www.sqlite.org/cvstrac/tktnew so it 
will be tracked and fixed.


Dennis Cote

Re: [sqlite] Lemon grammar question

[Ulrik Petersen]

Ludvig Strigeus wrote:


With Bison, you can do something like this (not quite bison syntax):

myrule: TYPE IDENT {DoSomethingRightAfterIdent($1,$2); } LP more_rules
RP; {DoSomethingAfterEverything($1,$2,$5); }

I.e. you have a chunk of C code that's called in the middle of the
processing of the production. (In the above case right after TYPE
IDENT)

Can you do this with lemon?
 



I don't know if you can do it bison-style, but you can do this:

1) have a struct that wraps a TYPE(T) and IDENT(I).

2) myrule : prefix(P) LP more_rules(R) RP . { /* process rule */ }

3) prefix(P) : TYPE(T) IDENT(I) . { /* process ident; wrap T and I in 
the struct from (1); */ }


4) remember to set the default destructor of the prefix rule to destroy 
the struct from (1).



I know you can do something like this for other cases,

myrule ::= TYPE(T) IDENT(I) temp LP more_rules(R) RP. {
DoSomethingAfterEverything(T,I,R); }
temp ::= DoSomethingRightAfterIdent(...how would I access TYPE/INDENT
from here..);

but it doesn't quite work for this case...

Any ideas?
/Ludvig
 


HTH

Ulrik Petersen

--
Ulrik Petersen, PhD student, MA, B.Sc.
Aalborg University, Denmark

Re: [sqlite] Lemon Parser Examples - I'll show you my examples

[sporkey]

On Thu, Aug 05, 2004 at 09:07:36AM -0400, Joseph Stewart wrote:
> Many thanks for your contribution!
> -joe
> 
> On Wed, 4 Aug 2004 22:02:48 -0400, [EMAIL PROTECTED]
> <[EMAIL PROTECTED]> wrote:
> > A user earlier posted a question about terminating
> > the grammer for a simple calculator done with lemon.
> > 
> > I ran into the same problem and had to use
> > the following, which worked fine:
> > 
> >   main ::= in.
> >   in ::= .
> >   in ::= in state NEWLINE.
> > 
> > I cannot reduce it any further. For instance,
> > the following does not work:
> > 
> >   in ::= .
> >   in ::= in stmt ENDLINE.
> > 
> > The version of lemon I'm using is from the
> > cvs 3.0.3.
> > 
> > The full calculator, there are 4 examples, can
> > be downloaded from the following link:
> > 
> > http://prdownloads.sourceforge.net/souptonuts/lemon_examples.tar.gz?download
> > 
> > Look at example4.y, if you want to go directly to
> > the terminating issue above.  Everything should be in
> > this download including lemon.c and lempar.c, so you
> > won't have to download anything else.
> > 
> > Take a look at the README. I tried to add documentation
> > on creating custom tokens, destructors, and a little bit
> > on what the grammer does.
> > 
> > To run everything, just run make
> > 
> >$ make
> > 
> > Then, each example can be run separately
> > 
> >$ ./ex1
> > 
> >$ ./ex2
> > 
> > 
> >$ ./ex4
> > 
> > Anyway, I hope this examples will help. Or
> > inspire you to take a took at lemon.
> > 
> > Regards,
> > 
> > Mike Chirico
> > 
> >

I've added an example with flex, and I'm trying write some engaging
documentation. If enough people find it useful, and if it can help
bring people to SQLite, I'll change the license from GPL to the public
domain. But, it will probably have to go through a few revisions first.

Regards,

Mike Chirico

Re: [sqlite] Lemon Parser Examples - I'll show you my examples

[Joseph Stewart]

Many thanks for your contribution!
-joe

On Wed, 4 Aug 2004 22:02:48 -0400, [EMAIL PROTECTED]
<[EMAIL PROTECTED]> wrote:
> A user earlier posted a question about terminating
> the grammer for a simple calculator done with lemon.
> 
> I ran into the same problem and had to use
> the following, which worked fine:
> 
>   main ::= in.
>   in ::= .
>   in ::= in state NEWLINE.
> 
> I cannot reduce it any further. For instance,
> the following does not work:
> 
>   in ::= .
>   in ::= in stmt ENDLINE.
> 
> The version of lemon I'm using is from the
> cvs 3.0.3.
> 
> The full calculator, there are 4 examples, can
> be downloaded from the following link:
> 
> http://prdownloads.sourceforge.net/souptonuts/lemon_examples.tar.gz?download
> 
> Look at example4.y, if you want to go directly to
> the terminating issue above.  Everything should be in
> this download including lemon.c and lempar.c, so you
> won't have to download anything else.
> 
> Take a look at the README. I tried to add documentation
> on creating custom tokens, destructors, and a little bit
> on what the grammer does.
> 
> To run everything, just run make
> 
>$ make
> 
> Then, each example can be run separately
> 
>$ ./ex1
> 
>$ ./ex2
> 
> 
>$ ./ex4
> 
> Anyway, I hope this examples will help. Or
> inspire you to take a took at lemon.
> 
> Regards,
> 
> Mike Chirico
> 
>

Re: [sqlite] LEMON Examples

[Joseph Stewart]

I'll second sporkey, would you be able to post your calc source for us
all to see?
TIA,
-j

On Tue, 03 Aug 2004 15:38:53 -0400, John Cohen <[EMAIL PROTECTED]> wrote:
> Hi all,
> 
> Thanks for all the help!  I got it all working fine.. The wrapper did the
> trick - I also want to try some speed tests with the others, and see which
> is the fastest (which I doubt there is a measurable difference in them, but
> its always worth the effort).
> 
> Thanks again,
> John
> 
> _
> Don't just search. Find. Check out the new MSN Search!
> http://search.msn.click-url.com/go/onm00200636ave/direct/01/
> 
>

Re: [sqlite] LEMON Examples

[D. Richard Hipp]

D. Richard Hipp wrote:
John Cohen wrote:
The last suggestion worked great (thanks!).  But still, I still have a 
small problem.  It won't accept more than one 'statement'.  I know 
why, but can't fix it.  Take a look:

$ ./a.out
7 + 9 + 7 + 3 / (5 + 7);
23.25
3 + 3 + 3;
Fatal Error: Parser is hopelessly lost...
Now, I know why, because my grammar look like this:
in ::= stmt ENDLINE.
stmt(A) ::= expression. { printf("%u\n", a); }
expression(A) ::= INTEGER(B). { A = B; }
expression(A) ::= expression(B) PPLUS expression(C). { A = B + C; }
expression(A) ::= expression(B) PSUB expression(C). { A = B - C; }
...etc...
Now its obvious why that will only accept one statement, but how to 
fix it?  SQLite doesn't offer any help here because the parser seems 
to be reinitialized each statement, but that really isn't an option in 
my case.  Is there any way?  In bison, you can simply make the start 
token right recursive (like in my last message):

in ::= in stmt ENDLINE.

There's a bug in lemon such that it doesn't like the primary
non-terminal to be recursive.  So do this:
 main ::= in.
Then in addition to what you have it should work.
Correction:  There *used* to be this bug in lemon.  I think
it is now fixed.  The problem above is that there is no
initial case for the in term.  You need this:
in ::= .
in ::= in stmt ENDLINE.

--
D. Richard Hipp -- [EMAIL PROTECTED] -- 704.948.4565

Re: [sqlite] LEMON Examples

[Christian Smith]

On Tue, 2 Aug 2004, John Cohen wrote:

>
>in ::= in stmt ENDLINE.
>
>But that doesn't seem to work in lemon.  I've also tried making the 2nd
>statement right recursive, but that doesn't work either because it doesn't
>seem to ever reduce all the way.
>
>Any ideas?  Much thanks to all.


Just make the start token a simple wrapper for a right recursive token to
allow multiple lines.

in ::= stmt_list .

stmt_list ::= stmt_list stmt ENDLINE .
stmt_list ::= stmt ENDLINE .


That should do it.


>
>John
>

Christian

-- 
/"\
\ /ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL
 X   - AGAINST MS ATTACHMENTS
/ \

Re: [sqlite] LEMON Examples

[John Cohen]

Date: Mon, 02 Aug 2004 23:51:15 -0400
Mime-Version: 1.0
Content-Type: text/plain; format=flowed

Aha! =)

The last suggestion worked great (thanks!).  But still, I still have a small 
problem.  It won't accept more than one 'statement'.  I know why, but can't 
fix it.  Take a look:

$ ./a.out
7 + 9 + 7 + 3 / (5 + 7);
23.25
3 + 3 + 3;
Fatal Error: Parser is hopelessly lost...

Now, I know why, because my grammar look like this:
in ::= stmt ENDLINE.
stmt(A) ::= expression. { printf("%u\n", a); }
expression(A) ::= INTEGER(B). { A = B; }
expression(A) ::= expression(B) PPLUS expression(C). { A = B + C; }
expression(A) ::= expression(B) PSUB expression(C). { A = B - C; }
...etc...

Now its obvious why that will only accept one statement, but how to fix it?  
SQLite doesn't offer any help here because the parser seems to be 
reinitialized each statement, but that really isn't an option in my case.  
Is there any way?  In bison, you can simply make the start token right 
recursive (like in my last message):

in ::= in stmt ENDLINE.

But that doesn't seem to work in lemon.  I've also tried making the 2nd 
statement right recursive, but that doesn't work either because it doesn't 
seem to ever reduce all the way.

Any ideas?  Much thanks to all.

John

_
Planning a family vacation? Check out the MSN Family Travel guide! 
http://dollar.msn.com

Re: [sqlite] LEMON Examples

[Doug Currie]

Monday, August 2, 2004, 12:49:50 AM, John Cohen wrote:

> My problem is the fact I cannot use the start token on the right hand of the
> rule.  [...]

> How can I get this to accept things such as:
> 5 + 5 + 5
> 5 + 6
> ?

[I have never used lemon, but perhaps something like...]

expr := term | ...
term := int | int PLUS term | int MINUS term

e

Re: [sqlite] Lemon vs. Bison speed difference/benchmarking...

[D. Richard Hipp]

Sean Chittenden wrote:
Howdy.  Having been fed up with bison/yacc, I switched to using lemon
and have found lemon to be a vastly more enjoyable tool to work with
compared to bison (thank you!).  That said, before I embarked on my
wholesale conversion to lemon, I went and recreated a calculator using
both lemon and bison and found that bison was about 2x faster than
lemon (much to my surprise). 
I'm interested in fixing this.  The last time I compared the speed of
the two (in 1989) lemon was faster. But I imagine bison has changed
somewhat in the intervening years.

[...]
Of note from the gprof output was the following from the lemon based
calc:
  %   cumulative   self  self total
 time   seconds   secondscalls  ms/call  ms/call  name
 47.3   4.76 4.76 __fix_locale_grouping_str [1]
 15.1   6.28 1.52 24312002 0.00 0.00  Parse [2]
 11.3   7.42 1.14 _mcount [5]
 10.5   8.47 1.06  5335001 0.00 0.00  yylex [4]
  2.6   8.73 0.26  4546001 0.00 0.00  symlook [10]
  2.0   8.94 0.20 yy_pop_parser_stack [6]
  1.7   9.11 0.17 main [8]
  1.3   9.24 0.13 yy_shift [7]
  1.2   9.36 0.12 yy_reduce [3]
  1.0   9.46 0.10 strtod [11]
  1.0   9.56 0.10  5335001 0.00 0.00  ParseAlloc [12]
  0.8   9.64 0.08 addfunc [9]
where as bison's profile was the following:

  %   cumulative   self  self total
 time   seconds   secondscalls  ms/call  ms/call  name
 32.4   1.50 1.501  1498.05  3233.40  yyparse [2]
 29.1   2.84 1.35  5335001 0.00 0.00  yylex [3]
 25.9   4.04 1.20 _mcount [4]
  5.2   4.28 0.24  1867000 0.00 0.00  strtod [6]
  4.2   4.48 0.20 .mcount (62)
  0.7   4.51 0.03  1867000 0.00 0.00  atof [5]
I'm using lemon 1.16 from CVS and would love any advice that people
may have with regards to speeding up lemon's generated parsers.  If
anyone has any questions, please feel free to ask.  -sc
What is __fix_locale_grouping_str?  It isn't used by lemon, as far
as I am aware.  And it does not appear in the bison profile.  Getting
rid of __fix_locale_grouping_str would be a big step towards
improving the speed of lemon.
If you omit _fix_locale_grouping_str, then the time for lemon is
the time for Parse (1.52), yy_pop_parser_stack (0.20), yy_shift (0.13)
and yy_reduce (0.12) for a total of 1.97.  Bison is still faster at
1.50, but we are suddenly a lot closer.
--
D. Richard Hipp -- [EMAIL PROTECTED] -- 704.948.4565
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]