Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

[Tim Bain]

I had forgotten you'd already submitted one. Thank you for doing that.

Tim

On Sat, Mar 28, 2020, 12:13 AM nexus  wrote:

> I have already opened an issue AMQ-7453 with detailed description, logs and
> examples, please, take a look:
> https://issues.apache.org/jira/browse/AMQ-7453
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

[nexus]

I have already opened an issue AMQ-7453 with detailed description, logs and
examples, please, take a look:
https://issues.apache.org/jira/browse/AMQ-7453



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

[Tim Bain]

Thanks for testing that; it will help make sure that whoever works on a fix
focuses quickly on the right part of the code. Would you be able to submit
a bug in JIRA to capture the problem as you observed it? If you're able to
attach any config files that would make it easier to quickly reproduce the
problem, that would be helpful as well.

Thanks for working through this preliminary testing and helping better
characterize the problem.

Tim

On Fri, Mar 27, 2020, 12:55 AM nexus  wrote:

> > if you cycle the remote broker while the local one is down, do you get
> the
> same behavior?
>
> I've performed the test, there's no error after clean restart of the remote
> broker, but the problem repeats whenever inactive destinations are purged
> once again. So, I suppose, your assumption is right "it is purely something
> in-memory on the remote broker".
>
> bad-activemq-restart.log
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/bad-activemq-restart.log>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

[nexus]

> if you cycle the remote broker while the local one is down, do you get the
same behavior?

I've performed the test, there's no error after clean restart of the remote
broker, but the problem repeats whenever inactive destinations are purged
once again. So, I suppose, your assumption is right "it is purely something
in-memory on the remote broker".

bad-activemq-restart.log
  



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

[Tim Bain]

What I meant was, if you cycle the remote broker while the local one is
down, do you get the same behavior? I'm trying to characterize whether the
problem is purely something in-memory on the remote broker, or whether the
problem is also in the content written to disk (and therefore the problem
persists even if you cycle the remote broker).

Tim

On Thu, Mar 26, 2020, 1:38 AM nexus  wrote:

> > can you please submit a bug report for it in JIRA?
>
> Done:
> https://issues.apache.org/jira/projects/AMQ/issues/AMQ-7453
>
> > whether the problem occurs if you cycle the remote broker between steps 5
> > and 6
>
> Steps 5. - 6. can be repeated multiple times -- problem persists.
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

[nexus]

> can you please submit a bug report for it in JIRA?

Done:
https://issues.apache.org/jira/projects/AMQ/issues/AMQ-7453

> whether the problem occurs if you cycle the remote broker between steps 5
> and 6

Steps 5. - 6. can be repeated multiple times -- problem persists.




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

[Tim Bain]

This definitely sounds like a bug in the mKahaDB code, so can you please
submit a bug report for it in JIRA?

I'd be curious to know whether the problem occurs if you cycle the remote
broker between steps 5 and 6. If not, it's likely something in-memory that
represents an inconsistency between the filesystem and what's in memory for
the mKahaDB database within the remote broker.

Tim

On Sun, Mar 22, 2020, 4:03 AM nexus  wrote:

> > Can you please provide a listing (ls -l) of the KahaDB data directory?
> See details below aligned with use case.
> > Also, can you please configure mKahaDB to point to a different (empty)
> > directory rather than the same one that the normal KahaDB points to, and
> > try again? I've never used mKahaDB so I don't have direct experience, but
> > I wouldn't expect the broker to be happy if you switch between KahaDB and
> > mKahaDB using the same data directory (and the same files within it, more
> > importantly). Unless your process for switching the KahaDB type already
> > involved deleting the files in the data directory?
> There was no switch from single to multi kahadb. Problem is reproduced with
> clean installation of  apache-activemq-5.15.12-bin.zip
> <
> http://www.apache.org/dyn/closer.cgi?filename=/activemq/5.15.12/apache-activemq-5.15.12-bin.zip&action=download>
>
> on 2 PCs and using configuration for local and remote brokers I a have
> provided in the first post.
> Some observations
> The main difference I see in logs, that in bad situtation (mKahaDB +
> perDestination="true") following messages appear after inactive
> destinations
> deleted:
> Since this moment local broker cannot establish duplex connection any more,
> and it doesn't matter which destinations has been purged -- with the same
> name (App.Data) or any other. Also it doesn't matter whether local broker
> already had any successful communication with the remote. As soon as these
> messages appear, broker cannot "create responder end of duplex network
> bridge" because of "PageFile is not loaded".
> When I try to do the same with single kahadb instance, these messages do
> not
> appear and no such problem.
> Use case with logs and kahadb listing
> Attached logs
> activemq.log
> 
> wrapper.log <
> http://activemq.2283324.n4.nabble.com/file/t379838/wrapper.log>
> 1. Remote broker started
> kahadb
> 2. Local broker started
> Duplex network connection established.
> Statically included queue "App.Data" created on remote.
> Active consumer is shown in web-console.
> wrapper.log
> kahadb
> 3. Local broker stopped and remote deletes inactive destinations after a
> while
> wrapper.log
> kahadb
> 4. Local broker started again
> wrapper.log
> kahadb
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

[nexus]

> Can you please provide a listing (ls -l) of the KahaDB data directory?
See details below aligned with use case.
> Also, can you please configure mKahaDB to point to a different (empty)
> directory rather than the same one that the normal KahaDB points to, and
> try again? I've never used mKahaDB so I don't have direct experience, but
> I wouldn't expect the broker to be happy if you switch between KahaDB and
> mKahaDB using the same data directory (and the same files within it, more
> importantly). Unless your process for switching the KahaDB type already
> involved deleting the files in the data directory?
There was no switch from single to multi kahadb. Problem is reproduced with
clean installation of  apache-activemq-5.15.12-bin.zip

  
on 2 PCs and using configuration for local and remote brokers I a have
provided in the first post.
Some observations
The main difference I see in logs, that in bad situtation (mKahaDB +
perDestination="true") following messages appear after inactive destinations
deleted:
Since this moment local broker cannot establish duplex connection any more,
and it doesn't matter which destinations has been purged -- with the same
name (App.Data) or any other. Also it doesn't matter whether local broker
already had any successful communication with the remote. As soon as these
messages appear, broker cannot "create responder end of duplex network
bridge" because of "PageFile is not loaded".
When I try to do the same with single kahadb instance, these messages do not
appear and no such problem.
Use case with logs and kahadb listing
Attached logs
activemq.log
  
wrapper.log   
1. Remote broker started
kahadb
2. Local broker started
Duplex network connection established.
Statically included queue "App.Data" created on remote.
Active consumer is shown in web-console.
wrapper.log
kahadb
3. Local broker stopped and remote deletes inactive destinations after a
while
wrapper.log
kahadb
4. Local broker started again
wrapper.log
kahadb




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

[Tim Bain]

Can you please provide a listing (ls -l) of the KahaDB data directory?

Also, can you please configure mKahaDB to point to a different (empty)
directory rather than the same one that the normal KahaDB points to, and
try again? I've never used mKahaDB so I don't have direct experience, but I
wouldn't expect the broker to be happy if you switch between KahaDB and
mKahaDB using the same data directory (and the same files within it, more
importantly). Unless your process for switching the KahaDB type already
involved deleting the files in the data directory?

Tim

Tim

On Sat, Mar 21, 2020, 8:03 AM nexus  wrote:

> Hello,
>
> Encountering following issue with network of brokers on Windows 10 / 64-bit
> / ActiveMQ v5.15.12
>
> Problem
>
>
> Cannot re-establish duplex network connection on remote broker with
> following error:
>
>
>
> Remote broker is configured to  delete inactive destinations
>    and uses
> mKahaDB persistence adapters for different destinations ( Automatic Per
> Destination Persistence Adapter   ).
>
> Same setup, but single kahaDB persistence adapter on remote broker is not
> causing the issue.
>
> See attached files for detailed configuration and logs:
>
> *  local-activemq.xml
> 
> *  good-remote-activemq.xml
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/good-remote-activemq.xml>
>
> *  good-remote-wrapper.log
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/good-remote-wrapper.log>
>
> *  bad-remote-activemq.xml
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/bad-remote-activemq.xml>
>
> *  bad-remote-wrapper.log
> 
>
>
> Use case
>
>
> Simulate network connection loss and then re-astablish duplex communication
> after remote broker destinations were purged due to inactivity:
>
> 1. start remote broker
> 2. start local broker
> 3. destination queue created automatically on remote (active consumer from
> local broker is also shown correctly in web-console)
> 4. stop local broker
> 5. wait for a while until destination is deleted on remote due to
> inactivity
> 6. start local broker again
>
> Results
>
>
> GOOD: single kahaDB persistence adapter on remote broker
>
>
>
>
> BAD: multiple kahaDb persistence adapters on remote broker with
> perDestination="true"
>
>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>