[EMAIL PROTECTED] wrote: >Thank you Andy, > >The -mx option does work rather well in my particular scenario. However I >have to read up on how to set that when the JVM is started through JNI. I >guess it's not too big of an issue initially b/c I'm toying with the best >options for optimization of this app I'm building and will probably just >use the simplest approach on my initial deployment. I get out-of-memory >when I read/write around 3-4k records from this particular DB table that >has roughly 25 columns. It is a pretty average size table and I expect to >be able to process these types of tables with relative ease. I'm not sure >exactly why streaming directly to disk would be a performance hit when >that's what is done ultimately anyhow. I think I have an idea of how it >could be an issue. Right now I'm thinking of putting some constraints or >documentation of possible constraints in place for using my app. > > The very reason why we suck down so much memory is the very reason why "streaming" directly to disk would be a problem.
To clarify. We cannot "stream" due to the structure of the file format. The issue is that records in the Header for a worksheet point to records in the body, meaning we must know WHERE those records will be written before we can even write out the header. Secondly, even if that were not the case, the structure of the OLE 2 Compound Document that encapsulates the Workbook is structured like the old dos fat filesystem. It has the same type of pointers to the body. Doing random file access in java, without a fixed record size is NOT going to be the fastest thing in the world (or even "real"). We cannot memory map in quite the same way we would if we were using C. So its a multifacited problem. Java's poor IO support (there may be a solution in 1.4 but I need to read up on it), lack of lightweight types and the structure of the file format. We have only have real control over the use of objects. The other two we can only work around so much... We CAN reduce our memory consumption in the coming release by flattening our object model a bit, relying on primtives, etc. We're not going to do it for 2.0 because it will make debugging more difficult and our feature schedule is more ambitious in HSSF for 2.0 than I imagine it will be for 3.0. (build on 2.0 graph support add image support) -Andy >Thanx again, >Cliff > > > >"Andrew C. Oliver" <[EMAIL PROTECTED]> >08/19/2002 12:43 PM > > > > > >Please respond to "POI Users List" <[EMAIL PROTECTED]> > >To: POI Users List <[EMAIL PROTECTED]> >cc: > >Subject: Re: Out of memory error... > > >The only options that drastically affect HSSF's running out of memory is >the -mx option. The "heap space". >I don't think increasing stack space is very important (objects are >stored on the heap and we don't make significant use of recursion). As >for building on disk...we're planning that in a later release, but next >release will mostly focus (as far as this area goes) on reducing memory >consumption in the first place. I want to do it with as little effect on >performance, swapping to disk will have a drastic effect on performance, >but should be an option (later....4.0 probably). > >-Andy > >[EMAIL PROTECTED] wrote: > > > >>Ok, I've done some extensive research on the command line options outlined >>below. These appear to be non-standard options one of which (-oss?) I >>couldn't even find any documentation on. Further more some appear to be >>Solaris/Linux specific. The there doesn't appear to be a -server option >> >> >for > > >>the Windows version of the Java launcher. Would the Windows equivalent be >>-classic? (I don't think so b/c -classic is a lower optimization option >> >> >for > > >>bypassing the Hotspot VM.) I ran command line help on the AS/400 via QSH >>and Java -? and there doesn't appear to be a -classic or a -server option >>available there. The AS/400 help also did not mention either the -oss or >>the -ss options either. The one thing that does seem consistent across the >>board is the -ms and -mx optins. I will try them along with some minor >> >> >code > > >>optimizations but I'm not sure how or if I can rely on them because I plan >>ultimately to run this utlity on the AS/400 via JNI. I'm not sure how to >>set these options via JNI. I'm wondering if there is a way to stream the >>output to a file as I'm reading data from the database as opposed to >>building the document entirely in memory then writing to disk. >> >>Thank you for all of your help, >>Cliff >> >> >> >>"Jackson, Stephen" <[EMAIL PROTECTED]> >>08/09/2002 05:00 PM >> >> >> >> >> >>Please respond to "POI Users List" <[EMAIL PROTECTED]> >> >>To: "'POI Users List'" <[EMAIL PROTECTED]> >>cc: >> >>Subject: RE: Out of memory error... >> >> >>Remember the default setting of the JVM >>The JVM runs as a client unless you specify -server (which will process >> >> >you > > >>code about 50% faster >>by default your JVM is only 64MB. >> >>The follow gives sets the initial heap to 975MB and the max size the same. >>java -server -ms975m -mx975m -oss6400k -ss2048k >> >>You need to do some research and figure out the parameters that best fit >>what you are doing. >>You need to use Runtime to monitor you memory. >>-----Original Message----- >>From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] >>Sent: Friday, August 09, 2002 5:01 PM >>To: POI Users List >>Subject: Out of memory error... >> >> >> >>Hello all, >> >>I recently created a conversion utility that reads processes records from >> >> >a > > >>result set into an HSSFWorksheet object. I just ran across a scenario >> >> >where > > >>I am getting an out of memory error. I was testing it against a moderately >>sized file on an AS/400 system with the code running on my Dell 900Mhz 20G >>PC Win2K PC. I have 128Meg RAM and it was running in the Forte Community >>Edition IDE from Sun. I did have about 3-4 other windows open but I hardly >>think that would have caused a difference. I'm wondering if there's >>anything I can do to alleviate this. I'm thinking the out-of-memory is due >>to too many object being created with the result set containing about 4600 >>records each w/ about 25 or 30 fields. This combined with all of the HSSF >>sheet, row, and font objects for formatting and writing the data was >>probably too much. Is there any low memory alternative to what I'm trying? >> >>Regards, >>Cliff >> >> >> >>-- >>To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> >>For additional commands, e-mail: <mailto:[EMAIL PROTECTED]> >> >>-- >>To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> >>For additional commands, e-mail: <mailto:[EMAIL PROTECTED]> >> >> >> >> >> >> >> >> >> >>-- >>To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> >>For additional commands, e-mail: <mailto:[EMAIL PROTECTED]> >> >> >> >> >> >> > > > > >-- >To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> >For additional commands, e-mail: <mailto:[EMAIL PROTECTED]> > > > > > > > > > >-- >To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> >For additional commands, e-mail: <mailto:[EMAIL PROTECTED]> > > > > -- To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
