Jason,I am happy that you found a solution. But I would like to know what you mean, 42 days of messaging.
On Wed, Sep 9, 2009 at 6:08 PM, Jason Meckley <[email protected]>wrote: > > I did some more testing today and found that processing 42 days with > individual messages was pegging the development servers CPU for the > duration of the saga. the saga was taking minutes to complete. > probably with all the read/writes from PHT and database. I abandoned > the saga altogether and preformed the 42 day loop within a single > consume(message) method. This had a very positive effect across the > entire system. > > 1. the load was lessened on the server as a whole > 2. the duration of the load was drastically shorter > 3. all 42 days worth of calculations complete faster > 4. No write locks with saga persister because I'm not using sagas > > I haven't run this through NHProf, but I'm sure it will yell about 42 > days worth of queries. with a HiLo id scheme, multi-queries and > batched writes I can't think of any more optimizations. > > All in all this was a good learning experience about messaging and an > introduction into multithreading. I'm still novice with deadlocking, > but I sort of know why/how it's happening. I really like how messaging > can decouple background work from the UI with "minimal" effort. The > actual code is minimal, understanding the architecture is more > complex. > > RSB is great and I can see this opening huge possibilities for my > projects moving forward! > > > On Sep 8, 5:37 pm, Ayende Rahien <[email protected]> wrote: > > Jason,That is pretty much for the same reason as before. > > Although I would like to see if you can get a repro, I'll see if I can > > reduce contention in the PHT or RQ. > > > > On Wed, Sep 9, 2009 at 12:33 AM, Jason Meckley <[email protected] > >wrote: > > > > > > > > > spent the day researching dead locks and experimenting with isolation > > > levels. One thing I did come to realize is there are no indexes on my > > > tables. I applied some indexing to the tables in question. the 2 > > > biggest impacts I see are: the responsiveness of the webpages is much > > > better and the dead lock messages have decreased significantly. > > > > > I am receiving an EsentErrorException: Error WriteConflict > > > (JET_errWriteConflict, Write lock failed due to outstanding write > > > lock). This may be caused by the nature of my saga. I was reading > > > another post (can't find the thread now). the users was getting write > > > conflicts because they where using PHT in a tight loop with messaging. > > > I'm doing something similar with my saga. when MessageX is consummed > > > the process performs selects and updates and then sends another > > > MessageX. after MessageX is consumed 42 times (6 weeks) the saga is > > > complete. > > > > > I figured this would be a better approach because it removes multiple > > > database calls in a loop and keeps the UI responsive. there is still a > > > loop (42 days) but the calculations are autonomous. > > > > > The Esent Exception didn't stop the processing, so that's good. I'll > > > keep digging into this more and stress the system again tomorrow. > > > > > On Sep 5, 5:56 am, Ayende Rahien <[email protected]> wrote: > > > > inline > > > > > > On Fri, Sep 4, 2009 at 11:29 PM, Jason Meckley < > [email protected] > > > >wrote: > > > > > > > I have a messaging system that runs calculations in the background > and > > > > > stores the results in the database. If I run on more than 1 thread > and > > > > > insert records through the web interface rapidly I will end up with > > > > > the follow 2 exceptions in my service log: > > > > > > > NHibernate.Event.Default.AbstractFlushingEventListener [(null)] - > > > > > Could not synchronize database state with session > > > > > NHibernate.HibernateException: An exception occurred when executing > > > > > batch queries ---> System.Data.SqlClient.SqlException: Transaction > > > > > (Process ID 51) was deadlocked on lock resources with another > process > > > > > and has been chosen as the deadlock victim. Rerun the transaction. > > > > > > > NHibernate.Impl.AbstractSessionImpl [(null)] - DTC transaction > prepre > > > > > phase failed > > > > > NHibernate.HibernateException: An exception occurred when executing > > > > > batch queries ---> System.Data.SqlClient.SqlException: Transaction > > > > > (Process ID 51) was deadlocked on lock resources with another > process > > > > > and has been chosen as the deadlock victim. Rerun the transaction. > > > > > > Yep, those are expected when you execute multiple threads that > conflict > > > with > > > > one another. > > > > With RSB, this will automatically retry the transaction, so the > action is > > > > actually being performed. > > > > You might need to look at transaction contention rate reduction > > > techniques > > > > or how to reduce deadlocks, but the basic problem is that you have > two > > > > concurrent transactions that each wants resources that the other has > > > > locked. > > > > Pay attention to the fact that by default we are running in RSB with > > > > serializable isolation level. > > > > > > > If I switch over to a single thread the problem appears to stop. I > > > > > think this makes sense to me. with >1 threads 2 threads attempt to > > > > > access the same record at the same time = dead lock. > > > > > > > the message tells me to re-run the transaction. how do I do that in > > > > > RSB? is there a Transaction.TryAgain(); method? I thought part of > the > > > > > advantage of messaging was parallel processing. If I only run on 1 > > > > > thread is there any real benefit to messaging? > > > > > > Nope, your message will be retried automatically. If you will look at > the > > > > results you'll see that the action *is* being performed. > > > > That is one of the really nice things about messaging approach. > > > > > > > some other back ground on this procedure. > > > > > 1. in a NH post insert/update/delete event I send a message to > start > > > > > calculating. the service picks up this message and works its magic. > > > > > > 2. the isolation level for the windows service the RSB default, > > > > > > > serializable i believe. the isolation level for the website is read > > > > > committed. > > > > > 3. it appears that after the deadlock the current saga stops > > > > > altogether. > > > > > > That isn't right. Can you post the log from RSB for that? You should > see > > > the > > > > message retrying. > > > > > > > 4. right now I'm testing on an older box I'm pretty sure it's a > single > > > > > proc box with a gig of RAM. > > > > > 5. both the website and service are hitting the database. the > website > > > > > does CRUD for the immediate input. the service runs the > calculations > > > > > and CRUD against the database. > > > > > 6. the tables updated by the website are read by the service and > vice > > > > > versa. wouldn't think this would be an issue though. > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en -~----------~----~----~----~------~----~------~--~---
