Some time ago I started this topic... I now doubt there's a memory leak in the
velocity code I use. I've re-written some parts of my program so it uses a
much better data structure and there's no problem now :) this month's data is
even more, I'm running the program now with 510000+ data objects and velocity
seems to do fine! I think the memory problem was in my own code somewhere...
Regards,
Eric
On Fri, 28 Oct 2005 17:31:07 +0200
"E.R. van Es" <[EMAIL PROTECTED]> wrote:
> A) thanks
>
> B) I don't put all 450000+ data objects in the context. the application
> creates selections, wich are put in a new ArrayList of data objects and this
> one is put in the context. Maybe I can try to set those "selections" to null
> after the template is processed, however because I initialize the ArrayList
> for the selection in the same variable for each loop iteration, shouldn't the
> previous one be null automaticly (because re-initialization of the same
> variable)? I'll test next monday, have a nice weekend :)
>
> Regards,
> Eric
>
> On Fri, 28 Oct 2005 14:08:24 +0100
> Andrew Mason <[EMAIL PROTECTED]> wrote:
>
> > A) Your english is very good for a second language
> >
> > B) When he says clings, i think he means are the references in memory
> > needed
> > used by velocity at a later stage (i.e they must stay in memory ) or can
> > you
> > just use them and set it to null.....
> >
> > On Fri October 28 2005 1:42 pm, E.R. van Es wrote:
> > > Unfortunatly I think need the data objects all the time, because all
> > > reports generated contain links to references wich are other data
> > > objects.
> > > I'll check my code again to see if I can drop anything, thanks for the
> > > hint. I don't know what you mean with "velocity cling into your
> > > objects"??
> > > (english is not my natural language...)
> > >
> > > Regards,
> > > Eric
> > >
> > > On Fri, 28 Oct 2005 12:34:10 +0000 (UTC)
> > >
> > > "Henning P. Schmiedehausen" <[EMAIL PROTECTED]> wrote:
> > > > "E.R. van Es" <[EMAIL PROTECTED]> writes:
> > > > > Hmm didn't test that, but it ran out of memory using 512Mb, it runs
> > > > > fine with 800Mb... I admit there might be a memory problem, as
> > > > > there are over 450000 (!) data objects in memory but I don't want
> > > > > the application to do disc reads for every data fetch (as it runs
> > > > > over data fetching methods over 10000 times).
> > > >
> > > > Do you actually need that number of data objects all the time? I
> > > > remember a similar case where adding a single "foo = null" line to the
> > > > code brought the memory usage from ~ 1.2 GB down to < 5 MB. If you
> > > > drop the references to the data objects as soon as you don't need them
> > > > any longer, you can bring your memory usage down dramatically.
> > > >
> > > > Or does Velocity cling onto your objects? That would be an interesting
> > > > case to check.
> > > >
> > > > Best regards
> > > > Henning
> > > >
> > > > >On Thu, 27 Oct 2005 14:14:23 +0100
> > > > >
> > > > >Andrew Mason <[EMAIL PROTECTED]> wrote:
> > > > >> It could just be that it actually DOES need more memory. Out of
> > > > >> curiosity at what amount of memory does it stop being a problem ?
> > > > >>
> > > > >> On Thu October 27 2005 2:09 pm, E.R. van Es wrote:
> > > > >> > Hi,
> > > > >> >
> > > > >> > I gave the program 800Mb of memory, enabled caching and using a
> > > > >> > BufferedWriter, and the program runs with no noticable slow-down at
> > > > >> > all. Maybe I'll stress-test with twice the amount of information
> > > > >> > for
> > > > >> > fun :) Is it possible to do something like "memory leak detection"
> > > > >> > using some kind of debugging on java programs?
> > > > >> >
> > > > >> > Greets,
> > > > >> > Eric
> > > > >> >
> > > > >> > On Thu, 27 Oct 2005 09:26:00 +0200
> > > > >> >
> > > > >> > "E.R. van Es" <[EMAIL PROTECTED]> wrote:
> > > > >> > > The data is read from a plain text file and put in simple 'data'
> > > > >> > > objects first. I need all data from the file to be available
> > > > >> > > before processing the templates, so all data objects are saved in
> > > > >> > > an ArrayList, the getData() method search through the data
> > > > >> > > objects
> > > > >> > > ArrayList for the key matching and returns only those data
> > > > >> > > objects
> > > > >> > > needed in a new ArrayList.
> > > > >> > >
> > > > >> > > Greetings,
> > > > >> > > Eric
> > > > >> > >
> > > > >> > > On Thu, 27 Oct 2005 08:09:57 +0100
> > > > >> > >
> > > > >> > > Andrew Mason <[EMAIL PROTECTED]> wrote:
> > > > >> > > > Just out of curiosity where is it gettng the data from ?
> > > > >> > > > database ? If so are you using a connection pool?
> > > > >> > > >
> > > > >> > > > private
> > > > >> > > >
> > > > >> > > > On Thu October 27 2005 8:03 am, E.R. van Es wrote:
> > > > >> > > > > Hi,
> > > > >> > > > >
> > > > >> > > > > I'm running a test now with the BufferedWriter, we'll see
> > > > >> > > > > what
> > > > >> > > > > happens. About the getData() method, maybe this actually is
> > > > >> > > > > the leak, it does this:
> > > > >> > > > >
> > > > >> > > > > *** CODE ***
> > > > >> > > > > private ArrayList getData(String key) {
> > > > >> > > > > ArrayList temp = new ArrayList();
> > > > >> > > > > ... fetch data here and fill temp ...
> > > > >> > > > > return temp;
> > > > >> > > > > }
> > > > >> > > > > *** CODE END ***
> > > > >> > > > >
> > > > >> > > > > I'm not a complete java guru, but is there a new temp
> > > > >> > > > > ArrayList allocated in memory every time I call this method?
> > > > >> > > > > won't previous created temp ArrayLists be collected by the
> > > > >> > > > > garbage collector?
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > Greetings,
> > > > >> > > > > Eric
> > > > >> > > > >
> > > > >> > > > > On Wed, 26 Oct 2005 11:57:08 -0400
> > > > >> > > > >
> > > > >> > > > > James Kebinger <[EMAIL PROTECTED]> wrote:
> > > > >> > > > > > I would just turn up the amount of memory java can use.
> > > > >> > > > > > Does point to a possible memory leak somewhere though. What
> > > > >> > > > > > does getData() do? Note you're also shadowing the data
> > > > >> > > > > > variable in processData().
> > > > >> > > > > >
> > > > >> > > > > > On 10/26/05, E.R. van Es <[EMAIL PROTECTED]> wrote:
> > > > >> > > > > > > Hi Andrew,
> > > > >> > > > > > >
> > > > >> > > > > > > does this really make a memory difference? The generated
> > > > >> > > > > > > files themselves are not that big (up to 50kb each) and
> > > > >> > > > > > > the writer is closed after each file... I didn't thought
> > > > >> > > > > > > about it so I'll try of course :)
> > > > >> > > > > > >
> > > > >> > > > > > > Greetings,
> > > > >> > > > > > > Eric
> > > > >> > > > > > >
> > > > >> > > > > > > On Wed, 26 Oct 2005 16:12:28 +0100
> > > > >> > > > > > >
> > > > >> > > > > > > Andrew Mason <[EMAIL PROTECTED]> wrote:
> > > > >> > > > > > > > Have you thought about using a buffered writer ?
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Wed October 26 2005 4:08 pm, E.R. van Es wrote:
> > > > >> > > > > > > > > Hello Velocity List,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > i have to create some big monthly reports on CD. I
> > > > >> > > > > > > > > use
> > > > >> > > > > > > > > a simple java program to generate all reports in HTML
> > > > >> > > > > > > > > files using velocity
> > > > >> > > > > > >
> > > > >> > > > > > > templates.
> > > > >> > > > > > >
> > > > >> > > > > > > > > All data is pre-loaded and then the reports are
> > > > >> > > > > > > > > created by iterating
> > > > >> > > > > > >
> > > > >> > > > > > > over
> > > > >> > > > > > >
> > > > >> > > > > > > > > the data. However, there are too many files to create
> > > > >> > > > > > > > > (over 9600) and after file 7850 I get a java memory
> > > > >> > > > > > > > > exception. I use a scheme like
> > > > >> > > > > > >
> > > > >> > > > > > > this:
> > > > >> > > > > > > > > *** JAVA CODE ***
> > > > >> > > > > > > > > private Context context;
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > private void processData() {
> > > > >> > > > > > > > > ArrayList data = getDataKeys(); // this returns an
> > > > >> > > > > > > > > array with all
> > > > >> > > > > > >
> > > > >> > > > > > > "data
> > > > >> > > > > > >
> > > > >> > > > > > > > > keys" Iterator i = data.iterator();
> > > > >> > > > > > > > > while (i.hasNext()) {
> > > > >> > > > > > > > > String key = (String) i.next();
> > > > >> > > > > > > > > ArrayList data = getData(key); // get all data
> > > > >> > > > > > > > > objects
> > > > >> > > > > > > > > for this key. context.put("key", key);
> > > > >> > > > > > > > > context.put("data", data);
> > > > >> > > > > > > > > processTemplate("template.vm", key + ".html");
> > > > >> > > > > > > > > }
> > > > >> > > > > > > > > }
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > private void processTemplate(String template, String
> > > > >> > > > > > > > > outputFile) { FileWriter outputFileWriter = new
> > > > >> > > > > > > > > FileWriter(new File(outputFile));
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Template x = Velocity.getTemplate(template);
> > > > >> > > > > > > > > x.merge(context, outputFileWriter);
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > outputFileWriter.close();
> > > > >> > > > > > > > > }
> > > > >> > > > > > > > > *** JAVA CODE END ***
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *** TEMPLATE CODE ***
> > > > >> > > > > > > > > <html>
> > > > >> > > > > > > > > ... bla bla ...
> > > > >> > > > > > > > > #foreach ( $el in $data )
> > > > >> > > > > > > > > $data.getX()<br>
> > > > >> > > > > > > > > $data.getY()<br>
> > > > >> > > > > > > > > and so on and so forth...
> > > > >> > > > > > > > > #end
> > > > >> > > > > > > > > ... bla bla ...
> > > > >> > > > > > > > > </html>
> > > > >> > > > > > > > > *** TEMPLATE CODE END ***
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > As you can see I put every data class in the context
> > > > >> > > > > > > > > with identifier "data", my thought was to overwrite
> > > > >> > > > > > > > > the previous one and NOT get a
> > > > >> > > > > > >
> > > > >> > > > > > > memory
> > > > >> > > > > > >
> > > > >> > > > > > > > > problem, however I do get the memory problem. I can
> > > > >> > > > > > > > > adjust the java
> > > > >> > > > > > >
> > > > >> > > > > > > memory
> > > > >> > > > > > >
> > > > >> > > > > > > > > allocation to 1Gb but I was wondering if my approach
> > > > >> > > > > > > > > just isn't
> > > > >> > > > > > >
> > > > >> > > > > > > correct?
> > > > >> > > > > > >
> > > > >> > > > > > > > > I'm not sure about the processTemplate method, though
> > > > >> > > > > > > > > it workes
> > > > >> > > > > > >
> > > > >> > > > > > > perfectly
> > > > >> > > > > > >
> > > > >> > > > > > > > > for the first 7800 files :) By the way: I'm using
> > > > >> > > > > > > > > velocity-1.4.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks in advance,
> > > > >> > > > > > > > > Eric
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > ------------------------------------------------------
> > > > >> > > > > > > > >------- ------ -- To unsubscribe, e-mail:
> > > > >> > > > > > > > > [EMAIL PROTECTED] For
> > > > >> > > > > > > > > additional commands, e-mail:
> > > > >> > > > > > > > > [EMAIL PROTECTED]
> > > > >> > > > > > > >
> > > > >> > > > > > > > --------------------------------------------------------
> > > > >> > > > > > > >------- ------ To unsubscribe, e-mail:
> > > > >> > > > > > > > [EMAIL PROTECTED] For
> > > > >> > > > > > > > additional commands, e-mail:
> > > > >> > > > > > > > [EMAIL PROTECTED]
> > > > >> > > > > > >
> > > > >> > > > > > > ----------------------------------------------------------
> > > > >> > > > > > >------- ---- To unsubscribe, e-mail:
> > > > >> > > > > > > [EMAIL PROTECTED] For
> > > > >> > > > > > > additional commands, e-mail:
> > > > >> > > > > > > [EMAIL PROTECTED]
> > > > >> > > > >
> > > > >> > > > > --------------------------------------------------------------
> > > > >> > > > >------- To unsubscribe, e-mail:
> > > > >> > > > > [EMAIL PROTECTED] For additional
> > > > >> > > > > commands, e-mail:
> > > > >> > > > > [EMAIL PROTECTED]
> > > > >> > > >
> > > > >> > > > ----------------------------------------------------------------
> > > > >> > > >----- To unsubscribe, e-mail:
> > > > >> > > > [EMAIL PROTECTED] For additional
> > > > >> > > > commands, e-mail: [EMAIL PROTECTED]
> > > > >> > >
> > > > >> > > ------------------------------------------------------------------
> > > > >> > >--- To unsubscribe, e-mail:
> > > > >> > > [EMAIL PROTECTED] For additional
> > > > >> > > commands, e-mail: [EMAIL PROTECTED]
> > > > >> >
> > > > >> > --------------------------------------------------------------------
> > > > >> >- To unsubscribe, e-mail:
> > > > >> > [EMAIL PROTECTED] For additional
> > > > >> > commands, e-mail: [EMAIL PROTECTED]
> > > > >>
> > > > >> ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: [EMAIL PROTECTED]
> > > > >> For additional commands, e-mail: [EMAIL PROTECTED]
> > > > >
> > > > >---------------------------------------------------------------------
> > > > >To unsubscribe, e-mail: [EMAIL PROTECTED]
> > > > >For additional commands, e-mail: [EMAIL PROTECTED]
> > > >
> > > > --
> > > > Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH
> > > > [EMAIL PROTECTED] +49 9131 50 654 0 http://www.intermeta.de/
> > > >
> > > > RedHat Certified Engineer -- Jakarta Turbine Development -- hero for
> > > > hire Linux, Java, perl, Solaris -- Consulting, Training, Development
> > > >
> > > > 4 - 8 - 15 - 16 - 23 - 42
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > > > For additional commands, e-mail: [EMAIL PROTECTED]
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > > For additional commands, e-mail: [EMAIL PROTECTED]
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail: [EMAIL PROTECTED]
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
