Package: libopentoken6.1-dev
Version: 6.0b-7
Severity: normal
Dear Maintainer,
When the size of a comment is close to Max_Buffer_Size (default = 1024),
Find_Best_Match may enter an infinite loop or have an incoherent behavior.
The provided test case contains a comment ending with "123".
- ending with "123", the comment length is 1026 : bug, infinite loop.
This is the behavior for Max_Buffer_Size + 2 or more longer comment.
- removing the "3", length = 1025 : bug, it fails with the message :
> 24: 7:60 Unrecognized character '''
(This character is in the first comment line)
- removing the "2", length = 1024 :
> raised OPENTOKEN.PROGRAMMER_ERROR : token larger than buffer size
of 1024
Normal behavior (I suppose that this is the expected behavior for
Max_Buffer_Size or longer comments).
- removing the "1", length = 1023 :
Normal behavior
I didn't further investigate the bug (the code is quite complex).
Workarround :
This size is a parameter of the Analyzer initialization (functions
OpenToken.Token.Enumerated.Analyzer.Initialize).
For the Java lexer, as the syntax is exposed in the spec, it's possible to
declare another Analyzer in user code with different defaults.
But, to avoid users a long investigation, I suggest to change the
default size
of the buffer to 4K, as lots of Java comments may be longer thant 1K.
Last line in java_lexer.ads :
Analyzer : constant Tokenizer.Handle := Tokenizer.Initialize (Syntax,
Buffer_Size => 4096);
Lionel
I joined : test main, gpr, Makefile and Java source
NodeEventTarget.java :
-----------------------------------------------------------------------
public interface NodeEventTarget extends EventTarget {
EventSupport getEventSupport();
// Comment size= 1023
/**
* This method allows the dispatch of events into the implementation's
* event model. The event target of the event is the
* <code>EventTarget</code> object on which <code>dispatchEvent</code>
* is called.
* @param evt The event to be dispatched.
* @return Indicates whether any of the listeners which handled the
* event called <code>Event.preventDefault()</code>. If
* <code>Event.preventDefault()</code> was called the returned value
* is <code>false</code>, else it is <code>true</code>.
* @exception EventException
* UNSPECIFIED_EVENT_TYPE_ERR: Raised if the <code>Event.type</code>
* was not specified by initializing the event before
* <code>dispatchEvent</code> was called. Specification of the
* <code>Event.type</code> as <code>null</code> or an empty string
* will also trigger this exception.
* <br> DISPATCH_REQUEST_ERR: Raised if the <code>Event</code>
object is
* already being dispatched
*/
EventSupport getEventSupport();
// Comment size= 1024
/**
* This method allows the dispatch of events into the implementation's
* event model. The event target of the event is the
* <code>EventTarget</code> object on which <code>dispatchEvent</code>
* is called.
* @param evt The event to be dispatched.
* @return Indicates whether any of the listeners which handled the
* event called <code>Event.preventDefault()</code>. If
* <code>Event.preventDefault()</code> was called the returned value
* is <code>false</code>, else it is <code>true</code>.
* @exception EventException
* UNSPECIFIED_EVENT_TYPE_ERR: Raised if the <code>Event.type</code>
* was not specified by initializing the event before
* <code>dispatchEvent</code> was called. Specification of the
* <code>Event.type</code> as <code>null</code> or an empty string
* will also trigger this exception.
* <br> DISPATCH_REQUEST_ERR: Raised if the <code>Event</code>
object is
* already being dispatched1
*/
EventSupport getEventSupport();
// Comment size= 1025
/**
* This method allows the dispatch of events into the implementation's
* event model. The event target of the event is the
* <code>EventTarget</code> object on which <code>dispatchEvent</code>
* is called.
* @param evt The event to be dispatched.
* @return Indicates whether any of the listeners which handled the
* event called <code>Event.preventDefault()</code>. If
* <code>Event.preventDefault()</code> was called the returned value
* is <code>false</code>, else it is <code>true</code>.
* @exception EventException
* UNSPECIFIED_EVENT_TYPE_ERR: Raised if the <code>Event.type</code>
* was not specified by initializing the event before
* <code>dispatchEvent</code> was called. Specification of the
* <code>Event.type</code> as <code>null</code> or an empty string
* will also trigger this exception.
* <br> DISPATCH_REQUEST_ERR: Raised if the <code>Event</code>
object is
* already being dispatched12
*/
EventSupport getEventSupport();
// Comment size= 1026
/**
* This method allows the dispatch of events into the implementation's
* event model. The event target of the event is the
* <code>EventTarget</code> object on which <code>dispatchEvent</code>
* is called.
* @param evt The event to be dispatched.
* @return Indicates whether any of the listeners which handled the
* event called <code>Event.preventDefault()</code>. If
* <code>Event.preventDefault()</code> was called the returned value
* is <code>false</code>, else it is <code>true</code>.
* @exception EventException
* UNSPECIFIED_EVENT_TYPE_ERR: Raised if the <code>Event.type</code>
* was not specified by initializing the event before
* <code>dispatchEvent</code> was called. Specification of the
* <code>Event.type</code> as <code>null</code> or an empty string
* will also trigger this exception.
* <br> DISPATCH_REQUEST_ERR: Raised if the <code>Event</code>
object is
* already being dispatched123
*/
}
Makefile :
-----------------------------------------------------------------------
all: build check
build:
gprbuild -P run_java_lexer.gpr
check:
obj/run_java_lexer NodeEventTarget.java
run_java_lexer.adb :
-----------------------------------------------------------------------
with Java_Lexer; use Java_Lexer;
with Ada.Text_IO; use Ada.Text_IO;
with Ada.Command_Line; use Ada.Command_Line;
with Ada.Exceptions;
with OpenToken;
with OpenToken.Text_Feeder.Text_IO;
procedure Run_Java_Lexer is
File : File_Type;
begin
Ada.Text_IO.Open (File, In_File, Argument (1));
-- Set_Input_Feeder (File);
Ada.Text_IO.Set_Input (File);
Analyzer.Reset;
Analyzer.Set_Text_Feeder (OpenToken.Text_Feeder.Text_IO.Create
(Ada.Text_IO.Current_Input));
loop
exit when End_Of_File (File);
Analyzer.Find_Next;
Put_Line (Natural'Image (Analyzer.Line)
& ": " & Java_Token'Image (Analyzer.ID)
& ", " & Analyzer.Lexeme);
end loop;
exception
when E : OpenToken.Syntax_Error =>
Put_Line (Count'Image (Line (File)) & ": " &
Ada.Exceptions.Exception_Message (E));
end Run_Java_Lexer;
run_java_lexer.gpr :
-----------------------------------------------------------------------
-- with "../../patched_ot";
project run_java_lexer is
for Source_Dirs use ("./src");
for Object_Dir use "./obj";
for Main use ("run_java_lexer.adb");
package Ide is
for Vcs_Kind use "Git";
-- for Documentation_Dir use "./html";
end Ide;
package Builder is
for Executable ("run_java_lexer.adb") use "run_java_lexer";
for Default_Switches ("ada") use ("-j2");
end Builder;
package Compiler is
Common_Switches := ("-gnat2012");
Common_Debug_Switches := Common_Switches &
("-g", "-O0", "-gnatQ", "-gnatf", "-gnato", "-fstack-check",
"-gnata",
"-gnateE", "-gnatVa", "-gnatwa", "-gnatyabefhiklnprtx");
-- -gnatQ : Don't quit
-- -gnatf : Full errors
-- -gnato : Intermediate overflows to be handled in strict mode.
-- -fstack-check : stack checking
-- -gnata : assertions
-- -gnateE : extra infos in exception messages
-- -gnatVa : all validity checks
-- -gnatwa : activate most optional warning messages
--
-- Style checking :
-- a : Check attribute casing
-- b : Blanks not allowed at statement end
-- e : Check end/exit labels
-- f : No form feeds or vertical tabs
-- h : No horizontal tabs
-- i : Check if-then layout
-- k : Check keyword casing
-- l : Check layout
-- n : Check casing of entities in Standard
-- p : Check pragma casing
-- r : Check references
-- t : Check token spacing
--removed M100 : Set maximum line length
-- x : Check extra parentheses.
for Switches ("ada") use Common_Debug_Switches;
end Compiler;
package Linker is
for Switches ("ada") use ("-g", "-lgcov");
end Linker;
package Binder is
for Switches ("ada") use ("-Es");
end Binder;
package Pretty_Printer is
for Switches ("ada") use ("-M100", "-l2", "-A1", "-A2", "-A3",
"-A4");
end Pretty_Printer;
end run_java_lexer;
-----------------------------------------------------------------------
-- System Information:
Debian Release: buster/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 4.14.0-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=fr_FR.utf8, LC_CTYPE=fr_FR.utf8 (charmap=UTF-8),
LANGUAGE=fr_FR.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages libopentoken6.1-dev depends on:
ii gnat 7
ii gnat-7 7.2.0-18
ii libopentoken9.1 6.0b-7
libopentoken6.1-dev recommends no packages.
Versions of packages libopentoken6.1-dev suggests:
ii libopentoken-doc 6.0b-7
-- no debconf information
--
-- Lionel
