On Nov 10, 11:06 pm, Ayende Rahien <[email protected]> wrote:
> They have the exact same issue, as far as I can see. I spoke about this with
> Udi, and he confirmed that he had much the same experience.
> As far as I understand, he is simply making tradeoffs, and using non
> transactional queues (which we can now do in RSB as well) when perf matters.
Today I've been running simple test - two database inserts in separate
db session each (NHibernate), repeated 10000 times.
If these inserts were inside TransactionScope, the test took 2'54'',
without TransactionScope - 2'17'', so the overhead of distributed
transactions was about 20% (and I have run 10000 distributed
transactions, not a huge single transaction). Average 57 distributed
transactions/sec vs 72 normal transaction pairs / sec (that is, 144
inserts/sec).
This doesn't look bad at all (however, at production system where I
was troubleshooting DTC transaction problems, the overhead seems to be
much bigger - maybe because of more complex transactions and reduced
concurrency whenever a distributed transaction occurs). Corey, maybe
you could check the Service Broker performance with and without
distributed transactions?
> And yes, there is a small chance of a problem when you are storing them in
> memory, but I think that the chance of a process crash between the
> transaction commit and sending the messages is small enough that we can
> safely ignore it.
I was rather thinkin about errors occurring when trying to actually
insert messages to some queue outside the transaction.
In such situation the code executing in transaction will never know
about these errors, so your library must take additional steps to
ensure that these messages don't get lost. You might retry sending
several times, persist the messages for manual handling later, notify
the user somehow or report the errors to some error handler in
application.... looks complicated to me
> On Tue, Nov 10, 2009 at 11:06 PM, nightwatch77
> <[email protected]>wrote:
>
>
>
>
>
> > Ayende, I think building non-transactional RSB could be a bit
> > difficult. Reliability is the key, if you call 'Send' you expect that
> > the infrastructure will take responsibility for the message and do its
> > best to deliver it. In any case it cannot lose the message. If
> > messages are simply stored in memory and sent to some queue after
> > transaction they can be lost if the process dies, but also if the
> > queue doesn't accept them for some reason (and the sender will never
> > know that). So you would need some persistent store for the outgoing
> > messages.
> > What about NServicebus, they have been 'in business' for several
> > years, didn't they have problems with distributed transactions? I
> > can't find anything at NServicebus discussion list...
> > R
>
> > On Nov 9, 9:02 pm, Ayende Rahien <[email protected]> wrote:
> > > Rafal,
> > > You are correct in all your assumptions.
> > > I am considering adding a non transactional mode for RSB, which won't be
> > as
> > > safe as using the DTC but will be MUCH more performant.
> > > Without using the DTC, we will simply gather all the messages in memory
> > and
> > > only send them at the end of the message processing.
> > > Thus, we would get similar behavior to now, where if a message has
> > failed,
> > > nothing get sent.
>
> > > On Mon, Nov 9, 2009 at 9:57 PM, nightwatch77 <[email protected]
> > >wrote:
>
> > > > Corey & all others interested in RSB,
>
> > > > Today I've been hunting for mysterious distributed transactions
> > > > appearing out of nowhere in production systems and hanging there
> > > > forever. And I have made quite unpleasant discoveries along the way.
> > > > First of all, I learnt that TransactionScope is not as smart as
> > > > Microsoft says. They say that TransactionScope will not promote the
> > > > transaction to distributed if there's only one resource manager
> > > > involved. I thought that if I have one SQL server and only one
> > > > database I will also be using exactly 1 resource manager, so I will
> > > > not have distributed transactions when making connections to that
> > > > database. Wrong! If you open two database connections inside
> > > > TransactionScope you will be promoted to a distributed transaction!
> > > > This will happen even if these two connections are identical, the same
> > > > database, the same connection string (!)
> > > > Up to today I thought distributed transactions are used when the
> > > > transaction scope crosses the process boundary (be it on the client or
> > > > on the server) - but obviously i've been wrong.
> > > > Now, why I am posting this here - mainly because Corey's work on
> > > > Service Broker transport will probably not improve performance by not
> > > > using distributed transactions. Each transaction will be promoted to
> > > > distributed as soon as you open a second connection to your database.
> > > > And you almost certainly will open new connection when executing some
> > > > application logic in a message hander - for example a new ORM session
> > > > or ADO.Net database update.
> > > > Also, this is no wonder Ayende had performance problems with MSMQ when
> > > > executing operations on two queues in a single transaction. Microsoft
> > > > simply chose to promote the transaction to distributed, no matter if
> > > > there's only one transactional resource or more.
>
> > > > Next issue: default isolation level with MS distributed transactions
> > > > is Serializable. This kills last bits of performance remaining after
> > > > promoting to distributed transaction. Fortunately it can be changed.
>
> > > > And more: sometimes distributed transactions will get 'zombified'.
> > > > They will hang in DTC forever and will not be aborted even after they
> > > > time out. The problem is that they will hold locks in SQL server,
> > > > deadlocking with other transactions in the database. I don't know why
> > > > this happens, googled a bit and found that it can happen when
> > > > application server dies during a distributed transaction. In our
> > > > production system the ASP.Net host process is recycled several times a
> > > > day, so I suspect sometimes a transaction is zombified during such
> > > > recycle. And this is a 'normal' recycle, not a process crash. If the
> > > > DTC transaction hangs we have a catastrophic failure reported by our
> > > > customer (the system throws SQL transaction timeout when executing any
> > > > action) and the admin has to manually kill dtc transactions.
>
> > > > Summing up: Microsoft is certainly abusing distributed transactions.
> > > > They are slow, execute longer, put more locks on database and reduce
> > > > system performance. MSDTC is a nasty pig without any useful monitoring/
> > > > diagnostic tools. I don't know what's the purpose of building system
> > > > on top of a message bus when you have to pay such performance penalty.
>
> > > > Ah, and please note I didn't perform any tests with RSB - I'm talking
> > > > about my own message queuing library built on top of SQL server table.
> > > > But I suspect the problem is also affecting Corey's work. With MSMQ
> > > > transport it's natural that you will have a distributed transaction so
> > > > this is not a big surprise.
>
> > > > Can you verify if I'm correct?
> > > > Best regards
> > > > Rafal
>
> > > > On Nov 3, 8:53 pm, Corey Kaylor <[email protected]> wrote:
> > > > > If you're referring to this
> > > > > article<
> > > >http://javiercrespoalvez.com/2009/03/using-sql-service-broker-in-net...
> > .>.
> > > > > Yes I've read it although I don't agree. It really depends on what is
> > > > > important to you. The comment about poison messages in the article is
> > > > valid.
> > > > > However, I have prevented that by never rolling back the dequeue
> > call. If
> > > > it
> > > > > fails I resend the same message back to the transport endpoint. The
> > > > consume
> > > > > method is called within its own isolated transactionscope preventing
> > a
> > > > > failure rollback for the dequeue as well. The comments in the article
> > > > about
> > > > > distributed transactions aren't valid either because my
> > implementation
> > > > will
> > > > > support it if it requires a promotion to DTC, but in most cases won't
> > > > need
> > > > > it. The comments in the article about not all resources are sql
> > server
> > > > > resources might be valid, but for myself it holds true for 99% of our
> > > > > resources and I don't mind catering to that 99%.
>
> > > > > Regardless of the frustrations you might have with RSB, it has been a
> > > > great
> > > > > thing for us and if it wasn't so nicely designed and extensible I
> > > > wouldn't
> > > > > have even attempted to do this in the first place. The nice thing
> > about
> > > > open
> > > > > source software is where you feel things are lacking or not perfect
> > for
> > > > your
> > > > > scenario you can contribute to make it better, or tweak it and make
> > it
> > > > work
> > > > > for your needs.
>
> > > > > It's not hiding it's located
> > > > > here<http://github.com/CoreyKaylor/rhino-esb/tree/servicebroker>.
> > > > > I may blog about it and our successes with RSB in the next few weeks.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Rhino Tools Dev" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/rhino-tools-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
