Hi Kay! On Thu, Apr 09, 2009 at 01:44:42PM +0200, Kay Röpke wrote: > I wouldn't do it in a pre-parser stage, preprocessors are evil. In > reality you almost always want to stay inside the language and do > rewriting on trees rather than strings. > Just look at how hard it can be to understand C/C++ syntax errors when > they result from complex macro expansions (the problem only is that cpp > is not operating on C syntax trees but on strings).
Agreed it is not an ideal solution, but was suggested as a hack for the people that really need the MySQL conversion. The module would be called something like "show_mysql" and would check to see if the first four bytes are "SHOW". If so, it would either do a hash lookup rewrite or a more complex variable rewrite. Nasty? Yes, but it would be a couple hour hack that is non-invasive and would help ease some people into the Drizzle water. :) > How about a recognition pipeline? > > I can see two different approaches: > The first idea would be to chain the recognizers (= lexer + parser), > where the main SQL recognizer pair is the first one to have a shot at an > incoming query. > Should recognition fail, the entire thing get's passed to the next one > (multiple parsers could share lexers, thus avoiding expensive re- > tokenization, but they don't have to). > In case none of the recognizers can make sense of the input, the first > one would supply the error message, because it's the "main one". > In this approach both efficiency and quality of error messages are > likely to be problems, though. However, it's easier to implement than > the next one. If an approach like this were used, I think each recognizer could add an error string if it was appropriate (ie, could parse some of the query but hit an error). If a recognizer can't get anything from it, it would just lease the message empty. If no recognizer sets an error message by the end, you could just have some default "No idea what you're talking about" response. > The second idea would be to have multiple recognizers that recognize > _disjoint_ sets of query syntax. They would form a tree, with the > easiest layout having a tree depth of 1. > Here the main problem is to efficiently pick a parser (not lexer > +parser!). This can be done very efficiently by retrieving the possible > start symbols from all the parsers and then delegating to the one that > knows about the symbol (e.g. a token "SHOW" would cause the "SHOW" parser > to run, the real SQL parser would have "SELECT", "INSERT", ... as start > symbols). > The implementation would simply be a big switch based on the first token > (or even deeper into the token stream, if that's what you want). It can > be implemented by hand as a recursive descent parser with one token > lookahead (LL(1)) pretty easily. Interesting. I think you would want to be able to define your start sets as prefix trees and not just a single symbol. This makes it a bit more complex, but it would not limit how deep the prefixes can go. > The downside is that you need to have exactly one lexer to yield the > token used in picking the right parser. This lexer doesn't not need to > be the one used by the invoked recognizer, but then it would have to be > generated or manually kept in sync as the superset of all individual > lexers. Doing this at runtime, esp looking at supporting dynamically > loaded recognizers, is not impossible, but might not be the most > efficient nor most trivial code ever written ;) > > What's the point of all this? > Well, given this architecture, you could support SHOW commands by having > the SHOW-parser generate a tree that looks exactly like the > corresponding information_schema query. This is completely transparent > for the client and eases maintenance should the I_S schema ever change: > Doing the preprocessing on the client side would mean that an old client > would constantly generate invalid I_S queries. Having it server side you > can keep both the SHOW commands and I_S schema compatible. This is certainly a cleaner way to handle the SHOW rewrite problem, but such a chain still seems a bit limited since you only get to match on the first non-terminal (or prefix). I'm still thinking of ways to have a fully pluggable lexer and parser (possibly through token namespaces and/or functional non-terminals). > P.S.: The concept of recognition delegation could easily extend to other > plugins, too, as long as the token that triggers this delegation is part > of the "main" lexer. Token stream implementation permitting you could > also stack lexers that then feed off of the same character stream. > Ambiguities are the biggest problem, especially when allowing plugins to > "inject" token rules into the main lexer. However, given it's upsides for > modularity and extensibility I think it's worth the risk of backtracking > a lot. Yeah, even things like schedulers could work this way. Depending on the type of query there may be a better scheduler to handle the type of load. Of course this brings up the question of if it makes sense to allow the query to switch schedulers after parsing and optimizing since you have more context to make those decisions. -Eric _______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
