On Wednesday, 2 October 2013 at 14:41:56 UTC, Dicebot wrote:
After brief discussion with Brian and gathering data from the
review thread, I have decided to start voting for `std.d.lexer`
inclusion into Phobos.
No.
I really want to see `std.d.lexer` in Phobos, but have too many
conditions.
Documentation issues:
- please specify the parser algorithm that you used for
`std.d.lexer`. As I understand from review thread, you implement
`GLR parser` - please document it (correct me if I wrong). Also,
add link to the algorithm description, for example to the
wikipedia:
http://en.wikipedia.org/wiki/GLR_parser
It helps to understand how `std.d.lexer` works.
Also, please add best-case and worst-case time complexity (for
example, from O(n) to O(n^3)), and best-case and worst-case
memory complexity.
- please add more usage examples. Currently you have only one big
example how generate HTML markup of D code. Try to add a simple
example for every function.
- explicitly specify functions that can throw: add `Throws:`
block for it and specify conditions when they can throw.
UTF-16/UTF-32 support:
- why standart `std.d.lexer` supports only UTF-8, but not a
UTF-16/UTF-32? The official lexing specification allows all of
them. The conversion from UTF-16/UTF-32 to UTF-8 is not a option
due performance issues.
If Phobos string functions too slow, please add a bug. If Phobos
haven't got necessary functions, please add enhancement request.
I think it's serious issue that affects all string utilities
(like std.xml or std.json), not only `std.d.lexer`.
Exception handling
- please use `ParseException` as a default exception, not the
`Exception`.
Codestyle:
- I don't like `TokenType` enum. You can use Tok!">>=" and
`static if` to compare the token string to the `TokenType` enum.
So, you will not lose performance, because string parsing will be
done at compile time.
Not a condition, but wishlist:
- implement low-level API, not only high-level range-based API. I
hope it can help increase performance for applications that
really need it.
- add ability to use `std.d.lexer` at the compile time.
