I am working with Brian on this. I'm the developer that did the coding. Here is what happened. I wrote the dao logic in Hibernate and tested it and when loading the data file, it got a little over half way through and the server ran out of memory. This file is 10,000 payments, but to load the 10,000 payments, it needs 182,000 sql inserts.
I found out that Hibernate doesn't run SQL when you tell it to. It keeps it all in RAM, then when you commit, it will run the SQL. I went to great lengths to parse this payment file using a SAX parser just so it wouldn't load the whole file in memory. Hibernate is working against me in this case. We are using the Spring framework, so I told Spring not to start a transaction for this file load. That's when it took 17 hours and didn't finish. Later, I found out that it would open the database connection, run one SQL, then close the connection. Obviously, not the way to go. After opening a JBoss support call with the Hibernate support people, they said that we would have to add this code after every SQL operation: session.flush(); session.evict(object); So, just to make sure we had a valid comparison, I added that to all the SQL operations in the Hibernate version of our application. On my development system, it still ran out of memory before it completed the 10,000 payments. On the testing environment, Hibernate was able to load the file in 18 minutes, but it used a ton more RAM than OJB to do so. Hibernate is supposed to be a "transparent" ORM tool, but as you can see, it can be quite invasive. We're very happy we switched to OJB. :) Also, the support on this list is much better than the Hibernate support. "Professional Open Source" means you have to pay to get any help it seems. Jay On Mon, 09 Aug 2004 15:49:24 +0200, Armin Waibel <[EMAIL PROTECTED]> wrote: > Hi Brian, > > > We started out using Hibernate for this, and we found that we had some > > real problems. It just would not scale whether or not we were using > > transactions. We found that it would take greater than 17 hours to load > > only 7500 of the records. Obviously this is unacceptable performance, > > and so we thought to try the same thing using OJB. > > I don't want to defend Hibernate ;-), but I assume there must be > something "wrong" with your setting or the way you handle it. I can't > believe that OJB is 85 time faster than HB. > > > I am happy to report that using OJB we were able to load the whole file > > of 10,000 in under 12 minutes. > > 12 minutes is better, but sounds slow too. Must be a really complex > operation. Storing of 10000 objects should be done in a "normal" > environment < 20 sec. > Can you describe why it take so long in your case or describe a little > bit more detailed what you are doing? > > regards, > Armin > > > > > Mcgough, Brian Joseph wrote: > > All, > > > > I just wanted to share some data points that were recently collected > > that compare OJB and Hibernate and the ability to scale with both. > > > > We had a data file with only 10,000 records in it that we needed to load > > into our database. Typically we use our batch environment, but given > > that we are a java shop now, we wanted to see if we could use java and > > our ORM tool to get the job done. > > > > We started out using Hibernate for this, and we found that we had some > > real problems. It just would not scale whether or not we were using > > transactions. We found that it would take greater than 17 hours to load > > only 7500 of the records. Obviously this is unacceptable performance, > > and so we thought to try the same thing using OJB. > > > > I am happy to report that using OJB we were able to load the whole file > > of 10,000 in under 12 minutes. > > > > In addition to this, we just recently upgraded a project from OJB 1.0 > > rc2 to OJB 1.0 and I am happy to report that for that particular project > > db performance was improve by a factor greater than 10. This is mostly > > due to the new implementation for FieldAccess. > > > > I just wanted to thank the developers for their attention to detail in > > regards to ensuring that the overhead above jdbc was minimal, and for > > all of the tests that they have written to ensure that is the case. We > > are very happy that we are still able to use ORM for this instead of > > straight jdbc, because the rest of the application is written using the > > ORM. > > > > Anyway I just wanted to share these points with the group, for those of > > you that are out there and are on the sidelines as far as which > > framework will scale better. > > > > Brian McGough > > IU - UITS - UIS - SIT > > (812) 856-4871 > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
