We don't need to package it - we only use it at compile time. There are other 
Apache projects such as Lucine that use JFlex.

Olga

-----Original Message-----
From: Dmitriy Ryaboy [mailto:dvrya...@cloudera.com] 
Sent: Tuesday, August 25, 2009 11:58 AM
To: pig-dev@hadoop.apache.org
Cc: pi.so...@gmail.com
Subject: Re: switching to different parser in Pig

Santosh,
Am I missing something about Jflex licensing? I thought that it being
GPL, we can't package it with apache-licensed software, which prevents
it from being a viable option (regardless of technical merits)

-Dmitriy

On Tue, Aug 25, 2009 at 1:58 PM, Santhosh Srinivasan<s...@yahoo-inc.com> wrote:
> Its been 6 months since this topic was discussed but we don't have
> closure on it.
> For SQL on top of Pig, we are using Jflex and CUP
> (https://issues.apache.org/jira/browse/PIG-824). If we have decided on
> the right parser, can we have a plan to move the other parsers in Pig to
> the same technology?
>
> Thanks,
> Santhosh
>
> PS: I am assuming we are not moving to Antlr.
>
>
> -----Original Message-----
> From: Alan Gates [mailto:ga...@yahoo-inc.com]
> Sent: Tuesday, February 24, 2009 10:17 AM
> To: pig-dev@hadoop.apache.org; pi.so...@gmail.com
> Subject: Re: switching to different parser in Pig
>
> Sorry, after I sent that email yesterday I realized I was not very
> clear.  I did not mean to imply that antlr didn't have good
> documentation or good error handling.  What I wanted to say was we
> want all three of those things, and it didn't appear that antlr
> provided all three, since it doesn't separate out scanner and parser.
> Also, from my viewpoint, I prefer bottom up LALR(1) parsers like yacc
> to top down parsers like javacc.  My understanding is that antlr is
> top down like javacc.  My reasoning for this preference is that parser
> books and classes have used those for decades, so there are a large
> number of engineers out there (including me :) ) who know how to work
> with them.  But maybe antlr is close enough to what we need.  I'll
> take a deeper look at it before I vote officially on which way we
> should go.
>
> As for loops and branches, I'm not saying we need those in Pig Latin.
> We need them somehow.  Whether it's better to put them in Pig Latin or
> imbed pig in a existing script language is an ongoing debate.  I don't
> want to make a decision now that effectively ends that debate without
> buy in from those who feel strongly that Pig Latin should include
> those constructs.
>
> I agree with you that we should modify the logical plan to support
> this rather than add another layer.  As for active development, the
> only thing I'm aware of is we hope to start working on a more robust
> optimizer for pig soon, and that will require some additional
> functionality out of the logical operators, but it shouldn't cause any
> fundamental architectural changes.
>
> Alan.
>
>
> On Feb 24, 2009, at 1:27 AM, pi song wrote:
>
>> (1) Lack of good documentation which makes it hard to and time
>> consuming
>> to learn javacc and make changes to Pig grammar
>> <== ANTLR is very very well documented.
>> http://www.pragprog.com/titles/tpantlr/the-definitive-antlr-reference
>> http://media.pragprog.com/titles/tpantlr/toc.pdf
>> http://www.antlr.org/wiki/display/ANTLR3/ANTLR+3+Wiki+Home
>>
>> (2) No easy way to customize error handling and error messages
>> <== ANTLR has very extensive error handling support
>> http://media.pragprog.com/titles/tpantlr/errors.pdf
>>
>> (3) Single path that performs both tokenizing and parsing
>> <== What is the advantage of decoupling tokenizer and parsing ?
>>
>> In addition, "Composite Grammar" is very useful for keeping the parser
>> modular. Things that can be treated as sub-languages such as bag
>> schema
>> definition can be done and unit tested separately.
>>
>> ANTLRWorks http://www.antlr.org/works/index.html
>> <http://www.antlr.org/works/index.html>also
>> makes grammar development very efficient. Think about IDE that helps
>> you
>> debug your code (which is grammar).
>>
>> One question, is there any use case for branching and loops? The
>> current Pig
>> is more like a query (declarative) language. I don't really see how
>> loop
>> constructs would fit. I think what Ted mentioned is more embedding
>> Pig in
>> other languages and use those languages to do loops.
>>
>> We should think about how the logical plan layer can be made simpler
>> for
>> external use so don't have to introduce a new layer. Is there any
>> major
>> active development on it? Currently I have more spare time and
>> should be
>> able to help out. (BTW, I'm slow because this is just my hobby. I
>> don't want
>> to drag you guys)
>>
>> Pi Song
>>
>> On Tue, Feb 24, 2009 at 6:23 AM, nitesh bhatia
> <niteshbhatia...@gmail.com
>> >wrote:
>>
>>> Hi
>>> I got this info from javacc mailing lists. This may prove helpful:
>>>
>>>
>>>
> ------------------------------------------------------------------------
> ------------------------------------------------------------------------
> ----------------
>>> -----Original Message----- From: Ken Beesley
>>> [mailto:ken....@xrce.xerox.com] Sent: Wednesday, August 18, 2004 2:56
>>> PM To: javacc Subject: [JavaCC] Alternatives to JavaCC (was Hello
>>> All)
>>>
>>> Vicas wrote:
>>>
>>> Hello All
>>>
>>> Kindly let me know other parsers available which does the same job as
>>> javacc.
>>>
>>> It would be very nice of you if you can send me some documentation
>>> related to this.
>>>
>>> Thanks Vikas
>>>
>>> (Correction and clarifications to the following would be _very_
>>> welcome. I'm very likely out of date.)
>>>
>>> Of course, no two software tools are likely to do _exactly_ the same
>>> job. Someone already pointed you to ANTLR, which is probably the
>>> best-known alternative to JavaCC. Another possibility is SableCC.
>>> http://sablecc.org
>>>
>>> The criteria include stability, documentation, language of the parser
>>> generated, and abstract-syntax-tree building.
>>>
>>> When I last looked (a couple of years ago) at ANTLR, SableCC and
>>> JavaCC, I chose JavaCC for the following reasons:
>>>
>>> 1. ANTLR could not handle Unicode input. Things change, of course, so
>>> ANTLR might now be more Unicode-friendly. Unicode was important to
>>> me,
>>> so this was a big factor in my decision.
>>>
>>> On the plus side for ANTLR, it has better abstract-syntax-tree
>>> building capabilities (in my opinion) than JJTree/JavaCC. You can
>>> learn to use JJTree commands, but it's not easy for most people.
>>>
>>> And ANTLR can generate either a Java or a C++ parser. JavaCC
>>> generates
>>> only Java parsers.
>>>
>>> Another concern about ANTLR was that it was reputed to change a lot
>>> as
>>> the guru, Terence Parr, experimented with new syntax and
>>> functionality. JavaCC, at least at the time, was reputed to be more
>>> stable, perhaps stable to a fault. I wanted stability and
>>> reliability.
>>>
>>> 2. SableCC is much like JavaCC; it generates a Java parser from a
>>> grammar description; but it had, in my opinion, less flexible
>>> abstract-syntax-tree building than JJTree/JavaCC. In SableCC (when I
>>> looked at it), the AST it built was always a direct reflection of
>>> your
>>> grammar, generating one tree node for each grammar expansion involved
>>> in a parse, much like using JavaCC with Java Tree Builder (JTB
>>> http://www.cs.purdue.edu/jtb/). When using JavaCC, JTB is the
>>> alternative to using JJTree.
>>>
>>> Using SableCC, or the combination JavaCC/JTB, should be _very_
>>> similar
>>> indeed.
>>>
>>> In my opinion, SableCC and JavaCC/JTB have made a conscious choice to
>>> simplify AST building--you get trees that reflect the expansions in
>>> your grammar. Period. But often these default trees will be big, full
>>> of extraneous nodes that reflect precedence hierarchies in the
>>> recursive-descent parsing. If you want to have more control over AST
>>> building, to get more compact and tailored ASTs, you need to pay the
>>> price of learning JJTree.
>>>
>>> Assuming that you need to build ASTs, with JavaCC you have the choice
>>> between JJTree and JTB. With SableCC, when I last looked at it, you
>>> only get the JTB-like option.
>>>
>>> *******
>>>
>>> (Again, corrections and expansions would be much appreciated.)
>>>
>>> Ken Beesley
>>>
>>>
>>>
>>>
>>>
>>>
> ------------------------------------------------------------------------
> ------------------------------------------------------------------------
> ---
>>>
>>>
>>> Of course, no two software tools are likely to do _exactly_ the same
>>> job. Someone already pointed you to ANTLR, which is probably the
>>> best-known alternative to JavaCC. Another possibility is SableCC.
>>> http://sablecc.org
>>>
>>> The criteria include stability, documentation, language of the parser
>>> generated, and abstract-syntax-tree building.
>>>
>>> When I last looked (a couple of years ago) at ANTLR, SableCC and
>>> JavaCC, I chose JavaCC for the following reasons:
>>>
>>> 1. ANTLR could not handle Unicode input. Things change, of course, so
>>> ANTLR might now be more Unicode-friendly. Unicode was important to
>>> me,
>>> so this was a big factor in my decision.
>>>
>>> On the plus side for ANTLR, it has better abstract-syntax-tree
>>> building capabilities (in my opinion) than JJTree/JavaCC. You can
>>> learn to use JJTree commands, but it's not easy for most people.
>>>
>>> And ANTLR can generate either a Java or a C++ parser. JavaCC
>>> generates
>>> only Java parsers.
>>>
>>> Another concern about ANTLR was that it was reputed to change a lot
>>> as
>>> the guru, Terence Parr, experimented with new syntax and
>>> functionality. JavaCC, at least at the time, was reputed to be more
>>> stable, perhaps stable to a fault. I wanted stability and
>>> reliability.
>>>
>>> 2. SableCC is much like JavaCC; it generates a Java parser from a
>>> grammar description; but it had, in my opinion, less flexible
>>> abstract-syntax-tree building than JJTree/JavaCC. In SableCC (when I
>>> looked at it), the AST it built was always a direct reflection of
>>> your
>>> grammar, generating one tree node for each grammar expansion involved
>>> in a parse, much like using JavaCC with Java Tree Builder (JTB
>>> http://www.cs.purdue.edu/jtb/). When using JavaCC, JTB is the
>>> alternative to using JJTree.
>>>
>>> Using SableCC, or the combination JavaCC/JTB, should be _very_
>>> similar
>>> indeed.
>>>
>>> In my opinion, SableCC and JavaCC/JTB have made a conscious choice to
>>> simplify AST building--you get trees that reflect the expansions in
>>> your grammar. Period. But often these default trees will be big, full
>>> of extraneous nodes that reflect precedence hierarchies in the
>>> recursive-descent parsing. If you want to have more control over AST
>>> building, to get more compact and tailored ASTs, you need to pay the
>>> price of learning JJTree.
>>>
>>> Assuming that you need to build ASTs, with JavaCC you have the choice
>>> between JJTree and JTB. With SableCC, when I last looked at it, you
>>> only get the JTB-like option.
>>>
>>> ----------
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Feb 23, 2009 at 10:06 PM, Alan Gates <ga...@yahoo-inc.com>
>>> wrote:
>>>> We looked into antlr.  It appears to be very similar to javacc,
>>>> with the
>>>> added feature that the java code it generates is humanly
>>>> readable.  That
>>>> isn't why we want to switch off of javacc.  Olga listed the 3
>>>> things we
>>> want
>>>> out of a parser that javacc isn't giving us (lack of docs, no easy
>>>> customization of error handle, decoupling of scanning and
>>>> parsing).  So
>>>> antlr doesn't look viable.
>>>>
>>>> In response to Pi's suggestion that we could use the logical plan,
>>>> I hope
>>> we
>>>> could use something close to it.  Whatever we choose we want it to
>>>> be
>>>> flexible enough to represent richer language constructs (like
>>>> branch and
>>>> loop).  I'm not sure our current logical plan can do that.  At the
>>>> same
>>>> time, we don't need another layer of translation (we already have
>>>> logical
>>> ->
>>>> physical -> mapreduce).  I would like to find a representation
>>>> that could
>>>> handle expressing the syntax and what is currently the logical plan.
>>>>
>>>> Alan.
>>>>
>>>> On Feb 20, 2009, at 5:15 PM, pi song wrote:
>>>>
>>>>> Should be pretty close but we may need to cleanup the interface a
>>>>> bit.
>>>>> Then
>>>>> the new parser  module can be switched in easily.
>>>>> BTW, have we already got the solution for the new parser generator?
>>>>>
>>>>> Pi
>>>>>
>>>>>
>>>>> On Fri, Feb 20, 2009 at 9:03 PM, Ted Dunning
>>>>> <ted.dunn...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Probably nearly the same effect as you suggest.  Are the
>>>>>> concepts at
>>> the
>>>>>> logical plan layer similar to those expressed in pig latin?  Or
>>>>>> has a
>>>>>> significant transformation occurred by then?
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 20, 2009 at 1:59 AM, pi song <pi.so...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Sounds good but how about exposing the logical plan layer
>>>>>>> instead?
>>>>>>> Wouldn't
>>>>>>> that yield the same effect?  From python for example you still
>>>>>>> can
>>>>>>> construct
>>>>>>> a logical plan and give to Pig to execute.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ted Dunning, CTO
>>>>>> DeepDyve
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Nitesh Bhatia
>>> Dhirubhai Ambani Institute of Information & Communication Technology
>>> Gandhinagar
>>> Gujarat
>>>
>>> "Life is never perfect. It just depends where you draw the line."
>>>
>>> visit:
>>> http://www.awaaaz.com - connecting through music
>>> http://www.volstreet.com - lets volunteer for better tomorrow
>>> http://www.instibuzz.com - Voice opinions, Transact easily, Have fun
>>>
>
>

Reply via email to