I did some more testing today and found that processing 42 days with individual messages was pegging the development servers CPU for the duration of the saga. the saga was taking minutes to complete. probably with all the read/writes from PHT and database. I abandoned the saga altogether and preformed the 42 day loop within a single consume(message) method. This had a very positive effect across the entire system.
1. the load was lessened on the server as a whole 2. the duration of the load was drastically shorter 3. all 42 days worth of calculations complete faster 4. No write locks with saga persister because I'm not using sagas I haven't run this through NHProf, but I'm sure it will yell about 42 days worth of queries. with a HiLo id scheme, multi-queries and batched writes I can't think of any more optimizations. All in all this was a good learning experience about messaging and an introduction into multithreading. I'm still novice with deadlocking, but I sort of know why/how it's happening. I really like how messaging can decouple background work from the UI with "minimal" effort. The actual code is minimal, understanding the architecture is more complex. RSB is great and I can see this opening huge possibilities for my projects moving forward! On Sep 8, 5:37 pm, Ayende Rahien <[email protected]> wrote: > Jason,That is pretty much for the same reason as before. > Although I would like to see if you can get a repro, I'll see if I can > reduce contention in the PHT or RQ. > > On Wed, Sep 9, 2009 at 12:33 AM, Jason Meckley <[email protected]>wrote: > > > > > spent the day researching dead locks and experimenting with isolation > > levels. One thing I did come to realize is there are no indexes on my > > tables. I applied some indexing to the tables in question. the 2 > > biggest impacts I see are: the responsiveness of the webpages is much > > better and the dead lock messages have decreased significantly. > > > I am receiving an EsentErrorException: Error WriteConflict > > (JET_errWriteConflict, Write lock failed due to outstanding write > > lock). This may be caused by the nature of my saga. I was reading > > another post (can't find the thread now). the users was getting write > > conflicts because they where using PHT in a tight loop with messaging. > > I'm doing something similar with my saga. when MessageX is consummed > > the process performs selects and updates and then sends another > > MessageX. after MessageX is consumed 42 times (6 weeks) the saga is > > complete. > > > I figured this would be a better approach because it removes multiple > > database calls in a loop and keeps the UI responsive. there is still a > > loop (42 days) but the calculations are autonomous. > > > The Esent Exception didn't stop the processing, so that's good. I'll > > keep digging into this more and stress the system again tomorrow. > > > On Sep 5, 5:56 am, Ayende Rahien <[email protected]> wrote: > > > inline > > > > On Fri, Sep 4, 2009 at 11:29 PM, Jason Meckley <[email protected] > > >wrote: > > > > > I have a messaging system that runs calculations in the background and > > > > stores the results in the database. If I run on more than 1 thread and > > > > insert records through the web interface rapidly I will end up with > > > > the follow 2 exceptions in my service log: > > > > > NHibernate.Event.Default.AbstractFlushingEventListener [(null)] - > > > > Could not synchronize database state with session > > > > NHibernate.HibernateException: An exception occurred when executing > > > > batch queries ---> System.Data.SqlClient.SqlException: Transaction > > > > (Process ID 51) was deadlocked on lock resources with another process > > > > and has been chosen as the deadlock victim. Rerun the transaction. > > > > > NHibernate.Impl.AbstractSessionImpl [(null)] - DTC transaction prepre > > > > phase failed > > > > NHibernate.HibernateException: An exception occurred when executing > > > > batch queries ---> System.Data.SqlClient.SqlException: Transaction > > > > (Process ID 51) was deadlocked on lock resources with another process > > > > and has been chosen as the deadlock victim. Rerun the transaction. > > > > Yep, those are expected when you execute multiple threads that conflict > > with > > > one another. > > > With RSB, this will automatically retry the transaction, so the action is > > > actually being performed. > > > You might need to look at transaction contention rate reduction > > techniques > > > or how to reduce deadlocks, but the basic problem is that you have two > > > concurrent transactions that each wants resources that the other has > > > locked. > > > Pay attention to the fact that by default we are running in RSB with > > > serializable isolation level. > > > > > If I switch over to a single thread the problem appears to stop. I > > > > think this makes sense to me. with >1 threads 2 threads attempt to > > > > access the same record at the same time = dead lock. > > > > > the message tells me to re-run the transaction. how do I do that in > > > > RSB? is there a Transaction.TryAgain(); method? I thought part of the > > > > advantage of messaging was parallel processing. If I only run on 1 > > > > thread is there any real benefit to messaging? > > > > Nope, your message will be retried automatically. If you will look at the > > > results you'll see that the action *is* being performed. > > > That is one of the really nice things about messaging approach. > > > > > some other back ground on this procedure. > > > > 1. in a NH post insert/update/delete event I send a message to start > > > > calculating. the service picks up this message and works its magic. > > > > 2. the isolation level for the windows service the RSB default, > > > > > serializable i believe. the isolation level for the website is read > > > > committed. > > > > 3. it appears that after the deadlock the current saga stops > > > > altogether. > > > > That isn't right. Can you post the log from RSB for that? You should see > > the > > > message retrying. > > > > > 4. right now I'm testing on an older box I'm pretty sure it's a single > > > > proc box with a gig of RAM. > > > > 5. both the website and service are hitting the database. the website > > > > does CRUD for the immediate input. the service runs the calculations > > > > and CRUD against the database. > > > > 6. the tables updated by the website are read by the service and vice > > > > versa. wouldn't think this would be an issue though. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en -~----------~----~----~----~------~----~------~--~---
