Re: ActiveMQ NMS and Failure Recovery

2016-10-13 Thread magmasystems
That's a great idea!



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717873.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Re: ActiveMQ NMS and Failure Recovery

2016-10-13 Thread Jim Gomes
I think you may be able to remove a lot of your exception handling code if
you turn on failover:.  It's so universal, I'm almost tempted to propose
making it the default, and you would have to say nofailover: if you didn't
want it.

On Thu, Oct 13, 2016 at 8:53 AM magmasystems 
wrote:

> Thanks, Jim. If failover works with a single instance, then I surely did
> reinvent the wheel :-)
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717869.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>


Re: ActiveMQ NMS and Failure Recovery

2016-10-13 Thread magmasystems
Thanks, Jim. If failover works with a single instance, then I surely did
reinvent the wheel :-)



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717869.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Re: ActiveMQ NMS and Failure Recovery

2016-10-13 Thread Jim Gomes
Yes, failover works on a single instance. I think I mentioned that in my
first reply. You don't need to list two brokers. You just need to have the
failover: keyword in the connection URI.

On Thu, Oct 13, 2016, 7:49 AM Tim Bain  wrote:

> Yes, the failover transport works in all three dialects (Java, .NET, and
> C++).
>
> On Oct 13, 2016 7:08 AM, "magmasystems" 
> wrote:
>
> > Thanks guys. As I mentioned, the devs tend to do stuff on their laptops
> > without running multiple instances of ActiveMQ. They basically go to the
> > ActiveMQ directory, run the activemq.bat file, do some debugging, and
> then
> > CTRL+C out of ActiveMQ. If there are any patterns or tricks we could use
> to
> > get automatic failover out of a single instance, I would love to know
> about
> > it. Could you do something like:
> >
> > failover:(tcp://localhost:61616,tcp://remotehost:61616)
> >
> > H ... according to
> > http://activemq.apache.org/failover-transport-reference.html, you should
> > be
> > able to. But, will that work with ActiveMQ NMS?
> >
> > -marc
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717859.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>


Re: ActiveMQ NMS and Failure Recovery

2016-10-13 Thread Tim Bain
Yes, the failover transport works in all three dialects (Java, .NET, and
C++).

On Oct 13, 2016 7:08 AM, "magmasystems" 
wrote:

> Thanks guys. As I mentioned, the devs tend to do stuff on their laptops
> without running multiple instances of ActiveMQ. They basically go to the
> ActiveMQ directory, run the activemq.bat file, do some debugging, and then
> CTRL+C out of ActiveMQ. If there are any patterns or tricks we could use to
> get automatic failover out of a single instance, I would love to know about
> it. Could you do something like:
>
> failover:(tcp://localhost:61616,tcp://remotehost:61616)
>
> H ... according to
> http://activemq.apache.org/failover-transport-reference.html, you should
> be
> able to. But, will that work with ActiveMQ NMS?
>
> -marc
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717859.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>


Re: ActiveMQ NMS and Failure Recovery

2016-10-13 Thread magmasystems
Thanks guys. As I mentioned, the devs tend to do stuff on their laptops
without running multiple instances of ActiveMQ. They basically go to the
ActiveMQ directory, run the activemq.bat file, do some debugging, and then
CTRL+C out of ActiveMQ. If there are any patterns or tricks we could use to
get automatic failover out of a single instance, I would love to know about
it. Could you do something like:

failover:(tcp://localhost:61616,tcp://remotehost:61616)

H ... according to
http://activemq.apache.org/failover-transport-reference.html, you should be
able to. But, will that work with ActiveMQ NMS?

-marc



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717859.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Re: ActiveMQ NMS and Failure Recovery

2016-10-13 Thread Tim Bain
Like Jim, I think that the failover transport should be capable of doing
the reconnect for you when debugging (and it's definitely capable of doing
it when there isn't a debugger in the mix), but it's possible that this is
one of the situations that illustrate the saying, "In theory, theory and
practice are the same thing..."

Also, I'd be cautious about that proxy object layer.  If that layer is
performing reconnections outside the context of breakpoints that mess
things up, it probably means something else is wrong with your
configuration (since the failover transport should be taking care of it
seamlessly for you), and you risk having your proxy object layer mask those
configuration problems.  So I'd recommend including a way to tell when the
proxy layer has to do a reconnect (it should never happen), and if you see
it happening, dig in to why the failover transport isn't doing that for
you.  I'm not saying you shouldn't use it, but monitor it and dig in as
necessary.

Tim

On Wed, Oct 12, 2016 at 2:16 PM, Jim Gomes  wrote:

> It seems like you reinvented the wheel a little bit there, but I'm glad you
> got it to work for you.
>
> Cheers!
>
>
> On Wed, Oct 12, 2016, 8:36 AM magmasystems 
> wrote:
>
> > Thanks Jim and Tim. I managed to hook up a poor-man's recover mechanism.
> > Our
> > application uses a framework that sits over the various low-level
> messaging
> > APIs, such ActiveMQ NMS, Tibco EMS .NET client, and the .NET RabbitMQ
> > client. So we actually have thin "proxy objects" sitting on top of the
> > native MessageConsumers and MessageProducers. When we detect a broker
> > failure, we instantiate a new connection and new sessions, and we
> recreate
> > the MessageConsumers and MessageProducers that the application was using.
> > Not the most elegant solution, but it was accomplished in a few lines of
> > code and seems to work.
> >
> > The most difficult thing to do (which is not too difficult, and only
> > involved finding the correct combination of BindingFlags to use) is to
> copy
> > the listener from the old MessageConsumer to the new MessageConsumer.
> >
> > var handler = typeof(MessageConsumer).GetField("listener",
> > BindingFlags.DeclaredOnly | BindingFlags.Instance |
> BindingFlags.NonPublic
> > |
> > BindingFlags.GetField).GetValue(nativeMessageConsumer);
> > newMessageConsumer.Listener += handler;
> >
> > So
> >
> >
> >
> > --
> > View this message in context:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-NMS-and-Failure-Recovery-
> tp4717704p4717848.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>


Re: ActiveMQ NMS and Failure Recovery

2016-10-12 Thread Jim Gomes
It seems like you reinvented the wheel a little bit there, but I'm glad you
got it to work for you.

Cheers!


On Wed, Oct 12, 2016, 8:36 AM magmasystems 
wrote:

> Thanks Jim and Tim. I managed to hook up a poor-man's recover mechanism.
> Our
> application uses a framework that sits over the various low-level messaging
> APIs, such ActiveMQ NMS, Tibco EMS .NET client, and the .NET RabbitMQ
> client. So we actually have thin "proxy objects" sitting on top of the
> native MessageConsumers and MessageProducers. When we detect a broker
> failure, we instantiate a new connection and new sessions, and we recreate
> the MessageConsumers and MessageProducers that the application was using.
> Not the most elegant solution, but it was accomplished in a few lines of
> code and seems to work.
>
> The most difficult thing to do (which is not too difficult, and only
> involved finding the correct combination of BindingFlags to use) is to copy
> the listener from the old MessageConsumer to the new MessageConsumer.
>
> var handler = typeof(MessageConsumer).GetField("listener",
> BindingFlags.DeclaredOnly | BindingFlags.Instance | BindingFlags.NonPublic
> |
> BindingFlags.GetField).GetValue(nativeMessageConsumer);
> newMessageConsumer.Listener += handler;
>
> So
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717848.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>


Re: ActiveMQ NMS and Failure Recovery

2016-10-12 Thread magmasystems
Thanks Jim and Tim. I managed to hook up a poor-man's recover mechanism. Our
application uses a framework that sits over the various low-level messaging
APIs, such ActiveMQ NMS, Tibco EMS .NET client, and the .NET RabbitMQ
client. So we actually have thin "proxy objects" sitting on top of the
native MessageConsumers and MessageProducers. When we detect a broker
failure, we instantiate a new connection and new sessions, and we recreate
the MessageConsumers and MessageProducers that the application was using.
Not the most elegant solution, but it was accomplished in a few lines of
code and seems to work.

The most difficult thing to do (which is not too difficult, and only
involved finding the correct combination of BindingFlags to use) is to copy
the listener from the old MessageConsumer to the new MessageConsumer. 

var handler = typeof(MessageConsumer).GetField("listener",
BindingFlags.DeclaredOnly | BindingFlags.Instance | BindingFlags.NonPublic |
BindingFlags.GetField).GetValue(nativeMessageConsumer);
newMessageConsumer.Listener += handler;

So



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717848.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Re: ActiveMQ NMS and Failure Recovery

2016-10-12 Thread Jim Gomes
Another technique to try when hitting a breakpoint is to not freeze all of
your threads. Let the NMS worker threads continue running in the background
so they are able to respond to the KeepAlive requests from the broker.
Otherwise, your client will be kicked off from the broker. Visual Studio
supports the ability to let background threads continue to run and only
break certain threads for debugging.

However, even though you client gets kicked off, it should go into failover
mode and attempt to reconnect once you start running your application
again. It will need a certain amount of time in order to accomplish this,
so hitting Step Over line-by-line debugging won't give it enough time to
reconnect. You don't have to try to reestablish all of the connections
inside of your exception handler. That would be redundant to what the
failover code already does. It just needs processor time in order to do it.

This seems like a debugging issue, and not really a bug in the client.

On Wed, Oct 12, 2016, 6:16 AM Tim Bain  wrote:

> As Jim said, the failover transport should ensure that your client
> reconnects even if the broker disconnects the client due to a
> breakpoint-induced timeout.  If you're able to isolate specific steps or
> sequences of events that result in that not happening, or to gather more
> detailed information about the symptoms of the failed reconnect, it might
> be possible for someone to figure out what's causing the behavior and fix
> it.  As always when debugging something with no obvious cause, the more
> detail you can provide, the better the odds of someone figuring out what
> the problem is.
>
> Also: tell your developers to stop getting coffee while they're debugging!
> The clock is ticking when you hit your first breakpoint, because as you've
> seen, there's a pretty narrow timeout window before the broker declares
> your client dead and disconnects you.  Do your thing and then detach, as
> quickly as you can.
>
> One technique I use (which might or might not be available in Visual
> Studio; I use Eclipse) is to use custom code in the condition of
> conditional breakpoints that will add a custom log line (to see the value
> of a variable, for example) but then always return false (i.e. don't stop
> at this breakpoint), which lets me add custom logging anywhere I want it
> without causing timeouts.  Obviously there are some technology differences
> between our setups, so YMMV.
>
> Tim
>
> On Oct 10, 2016 8:32 AM, "magmasystems" 
> wrote:
>
> > Thanks, Jim. I neglected to add that we are using the failover URI in
> > production. The main problem seems to be around surviving debugging. For
> > instance, I someone is debugging a service, and goes away for a cup of
> > coffee for a few minutes, they will invariably find that the broker has
> > send
> > an NMSException.
> >
> > The pseudo-code for what we are trying to do is below. Let's assume that
> > the
> > developer is working on his own laptop, with a locally-installed single
> > instance of ActiveMQ.
> >
> > // When initially creating a connection, we set the NMS
> > ExceptionListener handler
> > this.TheConnection.ExceptionListener += this.OnConnectionException;
> >
> > // This will get called when we get a break in the connection
> > public virtual void OnConnectionException(Exception exception)
> > {
> >   if (exception is NMSException)
> >   this.TryExceptionRepair(exception as NMSException);
> > }
> >
> > private void TryExceptionRepair(NMSException nmsException)
> > {
> >// This is where we want to try to re-establish connections,
> > sessions, etc
> > }
> >
> >
> > Let's say that we have a single Connection, a single Session that is
> > associated with the connection, and a number of MessageConsumers and
> > MessageProducers that were created by that Session.
> >
> > Is there a well-known pattern for re-establishing the environment when
> the
> > connection is broker?
> >
> > Also, any advice regarding the URI is welcome.
> >
> > Thanks,
> >
> > Marc
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717707.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>


Re: ActiveMQ NMS and Failure Recovery

2016-10-12 Thread Tim Bain
As Jim said, the failover transport should ensure that your client
reconnects even if the broker disconnects the client due to a
breakpoint-induced timeout.  If you're able to isolate specific steps or
sequences of events that result in that not happening, or to gather more
detailed information about the symptoms of the failed reconnect, it might
be possible for someone to figure out what's causing the behavior and fix
it.  As always when debugging something with no obvious cause, the more
detail you can provide, the better the odds of someone figuring out what
the problem is.

Also: tell your developers to stop getting coffee while they're debugging!
The clock is ticking when you hit your first breakpoint, because as you've
seen, there's a pretty narrow timeout window before the broker declares
your client dead and disconnects you.  Do your thing and then detach, as
quickly as you can.

One technique I use (which might or might not be available in Visual
Studio; I use Eclipse) is to use custom code in the condition of
conditional breakpoints that will add a custom log line (to see the value
of a variable, for example) but then always return false (i.e. don't stop
at this breakpoint), which lets me add custom logging anywhere I want it
without causing timeouts.  Obviously there are some technology differences
between our setups, so YMMV.

Tim

On Oct 10, 2016 8:32 AM, "magmasystems" 
wrote:

> Thanks, Jim. I neglected to add that we are using the failover URI in
> production. The main problem seems to be around surviving debugging. For
> instance, I someone is debugging a service, and goes away for a cup of
> coffee for a few minutes, they will invariably find that the broker has
> send
> an NMSException.
>
> The pseudo-code for what we are trying to do is below. Let's assume that
> the
> developer is working on his own laptop, with a locally-installed single
> instance of ActiveMQ.
>
> // When initially creating a connection, we set the NMS
> ExceptionListener handler
> this.TheConnection.ExceptionListener += this.OnConnectionException;
>
> // This will get called when we get a break in the connection
> public virtual void OnConnectionException(Exception exception)
> {
>   if (exception is NMSException)
>   this.TryExceptionRepair(exception as NMSException);
> }
>
> private void TryExceptionRepair(NMSException nmsException)
> {
>// This is where we want to try to re-establish connections,
> sessions, etc
> }
>
>
> Let's say that we have a single Connection, a single Session that is
> associated with the connection, and a number of MessageConsumers and
> MessageProducers that were created by that Session.
>
> Is there a well-known pattern for re-establishing the environment when the
> connection is broker?
>
> Also, any advice regarding the URI is welcome.
>
> Thanks,
>
> Marc
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717707.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>


Re: ActiveMQ NMS and Failure Recovery

2016-10-10 Thread magmasystems
Thanks, Jim. I neglected to add that we are using the failover URI in
production. The main problem seems to be around surviving debugging. For
instance, I someone is debugging a service, and goes away for a cup of
coffee for a few minutes, they will invariably find that the broker has send
an NMSException. 

The pseudo-code for what we are trying to do is below. Let's assume that the
developer is working on his own laptop, with a locally-installed single
instance of ActiveMQ. 

// When initially creating a connection, we set the NMS
ExceptionListener handler
this.TheConnection.ExceptionListener += this.OnConnectionException;

// This will get called when we get a break in the connection
public virtual void OnConnectionException(Exception exception)
{
  if (exception is NMSException)
  this.TryExceptionRepair(exception as NMSException);
}

private void TryExceptionRepair(NMSException nmsException)
{
   // This is where we want to try to re-establish connections,
sessions, etc
}


Let's say that we have a single Connection, a single Session that is
associated with the connection, and a number of MessageConsumers and
MessageProducers that were created by that Session.

Is there a well-known pattern for re-establishing the environment when the
connection is broker?

Also, any advice regarding the URI is welcome.

Thanks,

Marc



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704p4717707.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Re: ActiveMQ NMS and Failure Recovery

2016-10-09 Thread Jim Gomes
The first thing I notice is that you haven't actually enabled failover. The
connection URI should be based on this:

activemq:failover:tcp://hostname:61616

The inclusion of failover: is important.

Beyond that, if you break inside of a debugger, you most likely will be
disconnected by the broker. So I'm not sure what you are asking. Do you
want to survive breakpoints in a debugger, or do you want normal failover
in production?


On Sun, Oct 9, 2016, 9:00 AM magmasystems 
wrote:

> Hello all,
>
> As we prepare to ship our new product that uses ActiveMQ NMS heavily, I am
> trying to harden the code against possible failures. One of the things that
> our developers have noticed (and others in this group may have noticed) is
> that it is common to get NMSExceptions when you are stopped at a breakpoint
> for a while in Visual Studio. The connection is lost, and often times, we
> need to restart our services.
>
> Our typical connection string is this:
>
>
>
> activemq:tcp://myserver:61616?transport.useInactivityMonitor=falseamp;keepAlive=true
>
> I have a few questions:
>
> 1) Let's say that we have a few cached consumers. If a connection faulted,
> and we need to open a new connection, I assume that we also need to create
> new sessions and therefore, create new consumers and producers? Or is there
> any way to take an existing session and associate it with a new connection?
>
> 2) What are the best patterns for failure recovery, especially when a
> connection goes down because of a "keep-alive" violation? Do we need to
> only
> create a new connection? Do we need to recreate all sessions? Do we need to
> recreate all consumers and producers?
>
> 3) Are there any other parameters that you feel that I should add to my
> URI?
> I see that there are parameters like:
>
> wireFormat.maxInactivityDuration (set to either a large number or to -1 ...
> people have tried both)
> maxInactivityDurationInitalDelay
>
> Thanks very much for any advice.
>
> -marc
>
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-NMS-and-Failure-Recovery-tp4717704.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>