Hej Jay, JS> Here is what happened. I wrote the dao logic in Hibernate and tested JS> it and when loading the data file, it got a little over half way JS> through and the server ran out of memory. This file is 10,000 JS> payments, but to load the 10,000 payments, it needs 182,000 sql JS> inserts.
One thing that people tend to forget is that you have to choose the right tool for the job - doing batch updates/operations is not particual well suited for ORM-s (neither OJB nor Hibernate) - but I do understand why you would want to do it as "hey, all the other stuff we have is written for this java domain model, why not also do the batch jobs in that ?" JS> I found out that Hibernate doesn't run SQL when you tell it to. Not true ;) It does exactly what you tell it to - and per default that is to not hit the database before necessary...please again think it through why it in the majority of cases is better to not hit the db...and remember that you do have full control if it if you want! JS> It JS> keeps it all in RAM, then when you commit, it will run the SQL. or when you do a .flush() JS> I JS> went to great lengths to parse this payment file using a SAX parser JS> just so it wouldn't load the whole file in memory. Hibernate is JS> working against me in this case. No - please think about this for a second! If hibernate did not cache the elements associated with a session per default you would not be able to do: x = s.get(X.class, 42); y = s.get(X.class, 42); assertSame(x,y); // without caching this would break - and that you don't want in 99% of the cases! JS> We are using the Spring framework, so I told Spring not to start a JS> transaction for this file load. That's when it took 17 hours and JS> didn't finish. Later, I found out that it would open the database JS> connection, run one SQL, then close the connection. Obviously, not JS> the way to go. hmm...why would you ever run a batchprocessing job across multiple sessions/transactions ? It sounds like you havent the proper division of "labor" in your layers ? JS> After opening a JBoss support call with the Hibernate support people, JS> they said that we would have to add this code after every SQL JS> operation: JS> session.flush(); JS> session.evict(object); JS> So, just to make sure we had a valid comparison, I added that to all JS> the SQL operations in the Hibernate version of our application. If you just added this stuff blindly then you probably did not understand why it was needed in the first place.... JS> On my JS> development system, it still ran out of memory before it completed the JS> 10,000 payments. On the testing environment, Hibernate was able to JS> load the file in 18 minutes, but it used a ton more RAM than OJB to do JS> so. I can make Hibernate batch process indefinitly amount of data as long as I don't fill up the cache! So, something is probably filling up the cache - like, have you adjusted how much of the object graph that is fetched when getting a object ? If not then you again would be filling up the cache.... (or maybe opening a connection again and again or something else ?) JS> Hibernate is supposed to be a "transparent" ORM tool, but as you can JS> see, it can be quite invasive. It's a very transparent ORM tool, not a batch processing tool - the same goes for OJB. Please again remember that I'm quite sure you are using the PB API part of OJB which is not something you should compare Hibernate with - Hibernate is best compared with OJB ODMG og JDO layers...here we are doing much the same things. Please check out http://blog.hibernate.org/cgi-bin/blosxom.cgi/2004/06/24#comparingpersistence to see part of what i'm talking about here ;) JS> We're very happy we switched to OJB. If that's the right tool for you then congratz! ;) (but IMHO you have just been lucky that OJB's defaults is better than Hibernate's for your particular job/algorithm - because the other way around have also been observerd...OJB being much slower than HB) JS> :) Also, the support on this list is much better than the Hibernate JS> support. This is really funny as the above question about batch processing have been asked a zillion times and answer close to just as many times...it should be possible to dig out an answer - and otherwise we try to answer as many questions humanly possible...(but answering the same question over and over again with the same level of detail is hard) JS> "Professional Open Source" means you have to pay to get any JS> help it seems. No - it mean that you have the OPTION of paying to be ensured getting near 24 hour instant support. Oh well - enough from me, just wanted to enlighten you with info on the world is not all black and white ;) Best regards, [EMAIL PROTECTED] JS> Jay JS> On Mon, 09 Aug 2004 15:49:24 +0200, Armin Waibel <[EMAIL PROTECTED]> wrote: >> Hi Brian, >> >> > We started out using Hibernate for this, and we found that we had some >> > real problems. It just would not scale whether or not we were using >> > transactions. We found that it would take greater than 17 hours to load >> > only 7500 of the records. Obviously this is unacceptable performance, >> > and so we thought to try the same thing using OJB. >> >> I don't want to defend Hibernate ;-), but I assume there must be >> something "wrong" with your setting or the way you handle it. I can't >> believe that OJB is 85 time faster than HB. >> >> > I am happy to report that using OJB we were able to load the whole file >> > of 10,000 in under 12 minutes. >> >> 12 minutes is better, but sounds slow too. Must be a really complex >> operation. Storing of 10000 objects should be done in a "normal" >> environment < 20 sec. >> Can you describe why it take so long in your case or describe a little >> bit more detailed what you are doing? >> >> regards, >> Armin >> >> >> >> >> Mcgough, Brian Joseph wrote: >> > All, >> > >> > I just wanted to share some data points that were recently collected >> > that compare OJB and Hibernate and the ability to scale with both. >> > >> > We had a data file with only 10,000 records in it that we needed to load >> > into our database. Typically we use our batch environment, but given >> > that we are a java shop now, we wanted to see if we could use java and >> > our ORM tool to get the job done. >> > >> > We started out using Hibernate for this, and we found that we had some >> > real problems. It just would not scale whether or not we were using >> > transactions. We found that it would take greater than 17 hours to load >> > only 7500 of the records. Obviously this is unacceptable performance, >> > and so we thought to try the same thing using OJB. >> > >> > I am happy to report that using OJB we were able to load the whole file >> > of 10,000 in under 12 minutes. >> > >> > In addition to this, we just recently upgraded a project from OJB 1.0 >> > rc2 to OJB 1.0 and I am happy to report that for that particular project >> > db performance was improve by a factor greater than 10. This is mostly >> > due to the new implementation for FieldAccess. >> > >> > I just wanted to thank the developers for their attention to detail in >> > regards to ensuring that the overhead above jdbc was minimal, and for >> > all of the tests that they have written to ensure that is the case. We >> > are very happy that we are still able to use ORM for this instead of >> > straight jdbc, because the rest of the application is written using the >> > ORM. >> > >> > Anyway I just wanted to share these points with the group, for those of >> > you that are out there and are on the sidelines as far as which >> > framework will scale better. >> > >> > Brian McGough >> > IU - UITS - UIS - SIT >> > (812) 856-4871 >> > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> JS> --------------------------------------------------------------------- JS> To unsubscribe, e-mail: [EMAIL PROTECTED] JS> For additional commands, e-mail: [EMAIL PROTECTED] -- Med venlig hilsen, Max Rydahl Andersen mailto:[EMAIL PROTECTED] ACURE A/S Celluar: 51 56 10 14 Office: 39 11 80 27 --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
