Revision: 18098
Author: [email protected]
Date: Wed Nov 27 10:26:29 2013 UTC
Log: Experimental scanner: fix two-byte handling in lexer-shell.
In many cases (e.g., when the utf8 file contained an รค), it's a legit utf8
file (not ascii), but it'll still be 1-byte representation (latin1) inside
v8.
[email protected], [email protected]
BUG=
Review URL: https://codereview.chromium.org/91233002
http://code.google.com/p/v8/source/detail?r=18098
Modified:
/branches/experimental/parser/src/lexer/lexer-shell.cc
=======================================
--- /branches/experimental/parser/src/lexer/lexer-shell.cc Tue Nov 26
14:07:28 2013 UTC
+++ /branches/experimental/parser/src/lexer/lexer-shell.cc Wed Nov 27
10:26:29 2013 UTC
@@ -295,10 +295,15 @@
Vector<const uc16>(buffer_16, buffer_end_16 - buffer_16));
// If the string was just an expaneded one byte string, V8 detects
it
// and doesn't store it as two byte.
- CHECK(source->IsTwoByteRepresentation());
- experimental_time = RunExperimentalScanner<uint16_t>(
- source, isolate, encoding, print_tokens || check_tokens,
- &experimental_tokens, repeat, harmony_settings);
+ if (!source->IsTwoByteRepresentation()) {
+ experimental_time = RunExperimentalScanner<uint8_t>(
+ source, isolate, encoding, print_tokens || check_tokens,
+ &experimental_tokens, repeat, harmony_settings);
+ } else {
+ experimental_time = RunExperimentalScanner<uint16_t>(
+ source, isolate, encoding, print_tokens || check_tokens,
+ &experimental_tokens, repeat, harmony_settings);
+ }
break;
}
default:
--
--
v8-dev mailing list
[email protected]
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
