Reviewers: dcarney,
Message:
Committed patchset #1 manually as r17939.
Description:
Experimental lexer generator / utf16 support: Fix + add test files.
The whitespace thing is a hack (again).
[email protected]
BUG=
Committed: https://code.google.com/p/v8/source/detail?r=17939
Please review this at https://codereview.chromium.org/78933002/
SVN Base: https://v8.googlecode.com/svn/branches/experimental/parser
Affected files (+0, --1 lines):
A test/lexer/utf-16-1.js
A test/lexer/utf-16-2.js
M tools/lexer_generator/code_generator.jinja
Index: test/lexer/utf-16-1.js
diff --git a/test/lexer/utf-16-1.js b/test/lexer/utf-16-1.js
new file mode 100644
index
0000000000000000000000000000000000000000..1f09d0c9149e618df8496fc23f3bb0cf4732f428
Binary files /dev/null and b/test/lexer/utf-16-1.js differ
Index: test/lexer/utf-16-2.js
diff --git a/test/lexer/utf-16-2.js b/test/lexer/utf-16-2.js
new file mode 100644
index
0000000000000000000000000000000000000000..02764308b38239d3c20fb9bdcf5d6a27260163d6
Binary files /dev/null and b/test/lexer/utf-16-2.js differ
Index: tools/lexer_generator/code_generator.jinja
diff --git a/tools/lexer_generator/code_generator.jinja
b/tools/lexer_generator/code_generator.jinja
index
4ed43af92bed51c5143b80432a0271a028032c87..a6a381d83ca5c96abe4affb17dc9ea8ea262dea5
100644
--- a/tools/lexer_generator/code_generator.jinja
+++ b/tools/lexer_generator/code_generator.jinja
@@ -19,7 +19,8 @@
{%- elif r[1] == 'zero' -%}
(yych == 0 && cursor_ < buffer_end_)
{%- elif r[1] == 'whitespace' and encoding == 'utf16'-%}
- unicode_cache_->IsWhiteSpace(yych)
+ {# FIXME: Add and use unicode_cache_->InNonAsciiWhitespace #}
+ (yych != ' ' && yych != '\t' && yych != '\v' && yych != '\f' &&
yych != '\r' && yych != '\n' && unicode_cache_->IsWhiteSpace(yych))
{%- elif r[1] == 'letter' and encoding == 'utf16'-%}
{# FIXME: Add and use unicode_cache_->InNonAsciiLetter #}
(!(yych >= 'a' && yych <= 'z') && !(yych >= 'A' && yych <= 'Z')
&& unicode_cache_->IsLetter(yych))
--
--
v8-dev mailing list
[email protected]
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
