Revision: 17939
Author: [email protected]
Date: Wed Nov 20 17:15:07 2013 UTC
Log: Experimental lexer generator / utf16 support: Fix + add test
files.
The whitespace thing is a hack (again).
[email protected]
BUG=
Review URL: https://codereview.chromium.org/78933002
http://code.google.com/p/v8/source/detail?r=17939
Added:
/branches/experimental/parser/test/lexer/utf-16-1.js
/branches/experimental/parser/test/lexer/utf-16-2.js
Modified:
/branches/experimental/parser/tools/lexer_generator/code_generator.jinja
=======================================
--- /dev/null
+++ /branches/experimental/parser/test/lexer/utf-16-1.js Wed Nov 20
17:15:07 2013 UTC
@@ -0,0 +1,30 @@
+ÿþ/ / C o p y r i g h t 2 0 1 3 t h e V 8 p r o j e c t
a u t h o r s . A l l r i g h t s r e s e r v e d .
+ / / R e d i s t r i b u t i o n a n d u s e i n s o u r c e
a n d b i n a r y f o r m s , w i t h o r w i t h o u t
+ / / m o d i f i c a t i o n , a r e p e r m i t t e d
p r o v i d e d t h a t t h e f o l l o w i n g
c o n d i t i o n s a r e
+ / / m e t :
+ / /
+ / / * R e d i s t r i b u t i o n s o f s o u r c e
c o d e m u s t r e t a i n t h e a b o v e c o p y r i g h t
+ / / n o t i c e , t h i s l i s t o f
c o n d i t i o n s a n d t h e f o l l o w i n g
d i s c l a i m e r .
+ / / * R e d i s t r i b u t i o n s i n b i n a r y
f o r m m u s t r e p r o d u c e t h e a b o v e
+ / / c o p y r i g h t n o t i c e , t h i s l i s t
o f c o n d i t i o n s a n d t h e f o l l o w i n g
+ / / d i s c l a i m e r i n t h e
d o c u m e n t a t i o n a n d / o r o t h e r m a t e r i a l s
p r o v i d e d
+ / / w i t h t h e d i s t r i b u t i o n .
+ / / * N e i t h e r t h e n a m e o f G o o g l e
I n c . n o r t h e n a m e s o f i t s
+ / / c o n t r i b u t o r s m a y b e u s e d t o
e n d o r s e o r p r o m o t e p r o d u c t s d e r i v e d
+ / / f r o m t h i s s o f t w a r e w i t h o u t
s p e c i f i c p r i o r w r i t t e n p e r m i s s i o n .
+ / /
+ / / T H I S S O F T W A R E I S P R O V I D E D B Y T H E
C O P Y R I G H T H O L D E R S A N D C O N T R I B U T O R S
+ / / " A S I S " A N D A N Y E X P R E S S O R
I M P L I E D W A R R A N T I E S , I N C L U D I N G , B U T
N O T
+ / / L I M I T E D T O , T H E I M P L I E D
W A R R A N T I E S O F M E R C H A N T A B I L I T Y A N D
F I T N E S S F O R
+ / / A P A R T I C U L A R P U R P O S E A R E
D I S C L A I M E D . I N N O E V E N T S H A L L T H E
C O P Y R I G H T
+ / / O W N E R O R C O N T R I B U T O R S B E L I A B L E
F O R A N Y D I R E C T , I N D I R E C T , I N C I D E N T A L ,
+ / / S P E C I A L , E X E M P L A R Y , O R
C O N S E Q U E N T I A L D A M A G E S ( I N C L U D I N G , B U T
N O T
+ / / L I M I T E D T O , P R O C U R E M E N T O F
S U B S T I T U T E G O O D S O R S E R V I C E S ; L O S S O F
U S E ,
+ / / D A T A , O R P R O F I T S ; O R B U S I N E S S
I N T E R R U P T I O N ) H O W E V E R C A U S E D A N D O N
A N Y
+ / / T H E O R Y O F L I A B I L I T Y , W H E T H E R I N
C O N T R A C T , S T R I C T L I A B I L I T Y , O R T O R T
+ / / ( I N C L U D I N G N E G L I G E N C E O R
O T H E R W I S E ) A R I S I N G I N A N Y W A Y O U T O F
T H E U S E
+ / / O F T H I S S O F T W A R E , E V E N I F A D V I S E D
O F T H E P O S S I B I L I T Y O F S U C H D A M A G E .
+
+ / / T h i s f i l e i s i n u t f - 1 6 , b u t
d o e s n ' t c o n t a i n a n y w e i r d c h a r a c t e r s .
+
+ v a r f o o b a r = 1 6 ;
=======================================
--- /dev/null
+++ /branches/experimental/parser/test/lexer/utf-16-2.js Wed Nov 20
17:15:07 2013 UTC
@@ -0,0 +1,31 @@
+ÿþ/ / C o p y r i g h t 2 0 1 3 t h e V 8 p r o j e c t
a u t h o r s . A l l r i g h t s r e s e r v e d .
+ / / R e d i s t r i b u t i o n a n d u s e i n s o u r c e
a n d b i n a r y f o r m s , w i t h o r w i t h o u t
+ / / m o d i f i c a t i o n , a r e p e r m i t t e d
p r o v i d e d t h a t t h e f o l l o w i n g
c o n d i t i o n s a r e
+ / / m e t :
+ / /
+ / / * R e d i s t r i b u t i o n s o f s o u r c e
c o d e m u s t r e t a i n t h e a b o v e c o p y r i g h t
+ / / n o t i c e , t h i s l i s t o f
c o n d i t i o n s a n d t h e f o l l o w i n g
d i s c l a i m e r .
+ / / * R e d i s t r i b u t i o n s i n b i n a r y
f o r m m u s t r e p r o d u c e t h e a b o v e
+ / / c o p y r i g h t n o t i c e , t h i s l i s t
o f c o n d i t i o n s a n d t h e f o l l o w i n g
+ / / d i s c l a i m e r i n t h e
d o c u m e n t a t i o n a n d / o r o t h e r m a t e r i a l s
p r o v i d e d
+ / / w i t h t h e d i s t r i b u t i o n .
+ / / * N e i t h e r t h e n a m e o f G o o g l e
I n c . n o r t h e n a m e s o f i t s
+ / / c o n t r i b u t o r s m a y b e u s e d t o
e n d o r s e o r p r o m o t e p r o d u c t s d e r i v e d
+ / / f r o m t h i s s o f t w a r e w i t h o u t
s p e c i f i c p r i o r w r i t t e n p e r m i s s i o n .
+ / /
+ / / T H I S S O F T W A R E I S P R O V I D E D B Y T H E
C O P Y R I G H T H O L D E R S A N D C O N T R I B U T O R S
+ / / " A S I S " A N D A N Y E X P R E S S O R
I M P L I E D W A R R A N T I E S , I N C L U D I N G , B U T
N O T
+ / / L I M I T E D T O , T H E I M P L I E D
W A R R A N T I E S O F M E R C H A N T A B I L I T Y A N D
F I T N E S S F O R
+ / / A P A R T I C U L A R P U R P O S E A R E
D I S C L A I M E D . I N N O E V E N T S H A L L T H E
C O P Y R I G H T
+ / / O W N E R O R C O N T R I B U T O R S B E L I A B L E
F O R A N Y D I R E C T , I N D I R E C T , I N C I D E N T A L ,
+ / / S P E C I A L , E X E M P L A R Y , O R
C O N S E Q U E N T I A L D A M A G E S ( I N C L U D I N G , B U T
N O T
+ / / L I M I T E D T O , P R O C U R E M E N T O F
S U B S T I T U T E G O O D S O R S E R V I C E S ; L O S S O F
U S E ,
+ / / D A T A , O R P R O F I T S ; O R B U S I N E S S
I N T E R R U P T I O N ) H O W E V E R C A U S E D A N D O N
A N Y
+ / / T H E O R Y O F L I A B I L I T Y , W H E T H E R I N
C O N T R A C T , S T R I C T L I A B I L I T Y , O R T O R T
+ / / ( I N C L U D I N G N E G L I G E N C E O R
O T H E R W I S E ) A R I S I N G I N A N Y W A Y O U T O F
T H E U S E
+ / / O F T H I S S O F T W A R E , E V E N I F A D V I S E D
O F T H E P O S S I B I L I T Y O F S U C H D A M A G E .
+
+ / / T h i s f i l e i s i n u t f - 1 6 , a n d
c o n t a i n s w e i r d l e t t e r s .
+
+ v a r f ö ö b a r = 1 6 ; / / ö i n l a t i n 1 , b u t
n o t i n a s c i i .
+ v a r f o o 7 a r = 1 7 ; / / 7 n o t i n l a t i n 1 .
=======================================
---
/branches/experimental/parser/tools/lexer_generator/code_generator.jinja
Wed Nov 20 16:10:09 2013 UTC
+++
/branches/experimental/parser/tools/lexer_generator/code_generator.jinja
Wed Nov 20 17:15:07 2013 UTC
@@ -19,7 +19,8 @@
{%- elif r[1] == 'zero' -%}
(yych == 0 && cursor_ < buffer_end_)
{%- elif r[1] == 'whitespace' and encoding == 'utf16'-%}
- unicode_cache_->IsWhiteSpace(yych)
+ {# FIXME: Add and use unicode_cache_->InNonAsciiWhitespace #}
+ (yych != ' ' && yych != '\t' && yych != '\v' && yych != '\f' &&
yych != '\r' && yych != '\n' && unicode_cache_->IsWhiteSpace(yych))
{%- elif r[1] == 'letter' and encoding == 'utf16'-%}
{# FIXME: Add and use unicode_cache_->InNonAsciiLetter #}
(!(yych >= 'a' && yych <= 'z') && !(yych >= 'A' && yych <= 'Z')
&& unicode_cache_->IsLetter(yych))
--
--
v8-dev mailing list
[email protected]
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
