Will,
i try to test more JavaCC compiler and do some optimalizations with
code generation.
In Parser.jjt we can remove USER_CHAR_STREAM=true becouse auto
generated SimpleCharStream is almost identical to our
VelocityCharStream. If we decide to do this it's possible in Parser
class to:
- remove VelocityCharStream velcharstream variable - this will reduce
memory use, every VCS has 24kb arrays in memory
- change Parser CTR to simple
public Parser( RuntimeServices rs)
{
/*
* need to call the CTOR first thing.
*/
this( new ByteArrayInputStream("\n".getBytes()));
/*
* and save the RuntimeServices
*/
rsvc = rs;
}
-- and parse method to simple one:
public SimpleNode parse( Reader reader, String templateName )
throws ParseException
{
SimpleNode sn = null;
currentTemplateName = templateName;
try
{
token_source.clearStateVars();
/*
* now reinit the Parser with this CharStream
*/
ReInit(reader);
[..]
catch (Exception e)
{
rsvc.getLog().error("Parser Error: " + templateName, e);
}
currentTemplateName = "";
// finish with input strem
token_source.clearStateVars();
jj_input_stream.inputStream = null; // gc - to unrefernce
passed Reader from Parser class
return sn;
}
- (optional) TokenMgrError can be regenerated to fix some typos in comment
- regenerate javacc code and recompile.
I run test with latest Jira (3.7.4-#189 Standalone/Java6) Velocity
1.6-dev and JavaCC 4.0 and this still works ;) Of course, this need
more test - someone can test this with larger app with more templates?
Sylwester
2007/3/13, Will Glass-Husain <[EMAIL PROTECTED]>:
Sylvester,
Perhaps RuntimeInstance.parser could call a cleanup method? You can
insert new methods into Parser.java by putting them in Parser.jjt.
Re: VelocityCharStream. I'm a little leery of lazy inits unless they
demonstrate significant performance improvements. I've found them to
be an easy way to convolute the code without always providing
performance benefits.
WILL
On 3/12/07, Will Glass-Husain <[EMAIL PROTECTED]> wrote:
> We need to figure out a way to put this in Parser.jj. Parser.java is
> automatically generated-- it's too complex to generate Parser.java and
> then modify it.
>
> WILL
>
> On 3/12/07, Sylwester Lachiewicz <[EMAIL PROTECTED]> wrote:
> > I do something like this. This works for me, but need some more tests.
> > Also in method parse() - resource cleanup should be in finally block
> > (currentTemplateName, token_source, velcharstream).
> >
> > In VelocityCharStream class we can also do some rework to lazy init int,char
> > buffers
> >
> > I'll try to contact JavaCC developers to find where Done() method should be
> > called during parsing.
> >
> > Sylwester
> >
> > Index: Parser.java
> > ===================================================================
> > --- Parser.java (revision 517275)
> > +++ Parser.java (working copy)
> > @@ -53,8 +53,7 @@
> > * need to call the CTOR first thing.
> > */
> >
> > - this( new VelocityCharStream(
> > - new ByteArrayInputStream("\n".getBytes()), 1, 1 ));
> > + this( (CharStream)null );
> >
> > /*
> > * now setup a VCS for later use
> > @@ -129,7 +128,8 @@
> > }
> >
> > currentTemplateName = "";
> > -
> > + token_source.clearStateVars();
> > + if (velcharstream!=null) velcharstream.Done();
> > return sn;
> > }
> >
> >
> >
> > 2007/3/12, Ryan Smith <[EMAIL PROTECTED]>:
> > >
> > >
> > > Will,
> > >
> > > Where would you put the code to call the Done() method and free up the
> > > memory?
> > >
> > > We currently doing profiling of Velocity as well and are trying to find
> > > places exactly like this.
> > >
> > > Ryan
> > >
> > > -----Original Message-----
> > > From: Will Glass-Husain [mailto:[EMAIL PROTECTED]
> > > Sent: Monday, March 12, 2007 2:47 PM
> > > To: Velocity Developers List
> > > Subject: Re: JavaCC memory problems [was: Re: Problems with build
> > > Velocity]
> > >
> > > Thanks for the tips! Definitely something to investigate.
> > >
> > > WILL
> > >
> > > On 3/12/07, Sylwester Lachiewicz <[EMAIL PROTECTED]> wrote:
> > > > Hi,
> > > >
> > > > today JavaCC works fine - maybe there was bad day ;)
> > > >
> > > > I run profiler with simple velocity app (2 .vm file, simple merge) and
> > > found
> > > > that internal by design CharStream has method :
> > > > ---
> > > > /**
> > > > * The lexer calls this function to indicate that it is done with
> > > the
> > > > stream
> > > > * and hence implementations can free any resources held by this
> > > class.
> > > > * Again, the body of this function can be just empty and it will
> > > not
> > > > * affect the lexer's operation.
> > > > */
> > > > void Done();
> > > > --
> > > > but this method is never called by any JavaCC code (lexer/parser).
> > > > In class VelocityCharStream.Done() implementation buffers are
> > > cleaned.
> > > >
> > > > More, in VelocityCharStream constructor we have 3 times allocation of
> > > 4kb
> > > > buffer
> > > > buffer = new char[buffersize];
> > > > bufline = new int[buffersize];
> > > > bufcolumn = new int[buffersize];
> > > >
> > > > And in class Parser constructor we create 2 times VelocityCharStream.
> > > > So every Parser allocates about 24kb memory, even when is not needed.
> > > >
> > > > When I run trace I can find that memory is deallocated from old
> > > tempate only
> > > > on new parse request, but only to this 24kb.
> > > > In Done method we can also null readerStreem.
> > > >
> > > > Mayby this will help improve memory management.
> > > >
> > > > Sylwester
> > > >
> > > >
> > > >
> > > >
> > > > 2007/3/12, Will Glass-Husain <[EMAIL PROTECTED]>:
> > > > >
> > > > > Hi Sylvester,
> > > > >
> > > > > Thanks for the note on the build script. It really needs to check
> > > for
> > > > > a min of 1.6.
> > > > >
> > > > > I'm not sure why you have problems compiling. I use javacc 3.2 as
> > > > > well. Did you type "ant clean" first?
> > > > >
> > > > > JavaCC generates node files and Parser.java. The node files are not
> > > > > needed and can be deleted. (I think this happens in the build
> > > > > script). If you are just changing existing syntax, no manual
> > > copying
> > > > > of files is required. If you are adding a new Node you'll have to
> > > > > move that into the Node directory.
> > > > >
> > > > > WILL
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
