GitHub user hvanhovell opened a pull request:
https://github.com/apache/spark/pull/11557
[SPARK-13713][SQL] Migrate parser from ANTLR3 to ANTLR4 [WIP]
### What changes were proposed in this pull request?
The current ANTLR3 parser is quite complex to maintain and suffers from
code blow-ups. This PR introduces a new parser that is based on ANTLR4.
This parser is based on the [Presto's SQL
parser](https://github.com/facebook/presto/blob/master/presto-parser/src/main/antlr4/com/facebook/presto/sql/parser/SqlBase.g4).
The current implementation can parse and create Catalyst and SQL plans. Large
parts of the HiveQl DDL and some of the DML functionality is currently missing,
the plan is to add this in follow-up PRs.
This PR is a work in progress, and work needs to be done in the following
area's:
- [ ] Error handling should be improved.
- [ ] Documentation should be improved.
- [ ] Multi-Insert needs to be tested.
### How was this patch tested?
Catalyst and SQL unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hvanhovell/spark ngParser
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11557.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11557
----
commit 567032345603f445bbdea14a9f17c638119d93d6
Author: Herman van Hovell <[email protected]>
Date: 2016-02-23T16:30:10Z
Move Presto parser into spark
commit acac8f5233dcea16558c97853315a8f0ff677c74
Author: Herman van Hovell <[email protected]>
Date: 2016-02-23T22:21:13Z
Build parsers side-by-side
commit 1a98c0ccf6c9200a837d3800118128c94590de38
Author: Herman van Hovell <[email protected]>
Date: 2016-02-25T23:26:23Z
Add Expression Parsing
commit 3ebdeecc23a9eb63b7d1f4272b6bf43e234f9722
Author: Herman van Hovell <[email protected]>
Date: 2016-02-27T17:10:56Z
Add Expression Parsing
commit 555a959b7ac333a6f485b0097e532f122cb179e9
Author: Herman van Hovell <[email protected]>
Date: 2016-02-29T23:49:40Z
Add Basic Plan Parsing
commit a130429953713fbec3408e4b0e8db64ae4fa5c1c
Author: Herman van Hovell <[email protected]>
Date: 2016-03-06T00:25:26Z
Catalyst feature parity. Start with SQL/Core.
commit 384f10a4c106200ffc3dae9e0e60e8f3c15a2ee5
Author: Herman van Hovell <[email protected]>
Date: 2016-03-06T21:28:21Z
SQL/core feature parity.
commit d4461ae82316f16a5408eda5773be3502ea8d25d
Author: Herman van Hovell <[email protected]>
Date: 2016-03-06T21:34:21Z
Merge remote-tracking branch 'apache-github/master' into ngParser
# Conflicts:
# sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
commit e991d20e2949ad25c7a26f15567105a52e5b0196
Author: Herman van Hovell <[email protected]>
Date: 2016-03-06T21:34:34Z
Merge remote-tracking branch 'apache-github/master' into ngParser
# Conflicts:
# sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
commit 8ff792988d4b91fdfa0978c65368e560d79266ab
Author: Herman van Hovell <[email protected]>
Date: 2016-03-06T23:24:16Z
Make tests pass.
commit 06303bcc2801b3f7520c3174a982e21056e86386
Author: Herman van Hovell <[email protected]>
Date: 2016-03-07T06:52:36Z
Fix expand star
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
