[jira] [Commented] (ARTEMIS-1310) Provide GSSAPI (kerberos) SASL mechanism for AMQP
[ https://issues.apache.org/jira/browse/ARTEMIS-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110808#comment-16110808 ] Gary Tully commented on ARTEMIS-1310: - work in progress branch pending release of required dependencies: 0.24.0-SNAPSHOT 0.20.0-SNAPSHOT https://github.com/gtully/activemq-artemis/tree/sasl-gssapi > Provide GSSAPI (kerberos) SASL mechanism for AMQP > - > > Key: ARTEMIS-1310 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1310 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.2.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 2.3.0 > > > implement GSSAPI sasl mechanism for AMQP transport allowing kerberos single > sign on (authentication) and subsequent authorisation -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ARTEMIS-1310) Provide GSSAPI (kerberos) SASL mechanism for AMQP
[ https://issues.apache.org/jira/browse/ARTEMIS-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107114#comment-16107114 ] Gary Tully commented on ARTEMIS-1310: - The round trip integration test requires qpid-jms gssapi sasl support via QPIDJMS-303 > Provide GSSAPI (kerberos) SASL mechanism for AMQP > - > > Key: ARTEMIS-1310 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1310 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.2.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 2.3.0 > > > implement GSSAPI sasl mechanism for AMQP transport allowing kerberos single > sign on (authentication) and subsequent authorisation -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ARTEMIS-1310) Provide GSSAPI (kerberos) SASL mechanism for AMQP
[ https://issues.apache.org/jira/browse/ARTEMIS-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107108#comment-16107108 ] Gary Tully commented on ARTEMIS-1310: - this feature requires PROTON-1525 b/c SASL GSSAPI requires sending frames that exceed the AMQP limit in place before the open frame can negotiate an appropriate frame size. > Provide GSSAPI (kerberos) SASL mechanism for AMQP > - > > Key: ARTEMIS-1310 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1310 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.2.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 2.3.0 > > > implement GSSAPI sasl mechanism for AMQP transport allowing kerberos single > sign on (authentication) and subsequent authorisation -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ARTEMIS-1310) Provide GSSAPI (kerberos) SASL mechanism for AMQP
[ https://issues.apache.org/jira/browse/ARTEMIS-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107100#comment-16107100 ] Gary Tully commented on ARTEMIS-1310: - ARTEMIS-1264 provides for kerberos single sign on over an insecure channel (b/c of weak/old cyphers). This feature leverages SASL to do single sign on over a potentially TLS secured channel. > Provide GSSAPI (kerberos) SASL mechanism for AMQP > - > > Key: ARTEMIS-1310 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1310 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.2.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 2.3.0 > > > implement GSSAPI sasl mechanism for AMQP transport allowing kerberos single > sign on (authentication) and subsequent authorisation -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (ARTEMIS-1310) Provide GSSAPI (kerberos) SASL mechanism for AMQP
Gary Tully created ARTEMIS-1310: --- Summary: Provide GSSAPI (kerberos) SASL mechanism for AMQP Key: ARTEMIS-1310 URL: https://issues.apache.org/jira/browse/ARTEMIS-1310 Project: ActiveMQ Artemis Issue Type: Bug Components: AMQP Affects Versions: 2.2.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 2.3.0 implement GSSAPI sasl mechanism for AMQP transport allowing kerberos single sign on (authentication) and subsequent authorisation -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AMQ-6778) Advisories are dropped till broker is fully started in error
[ https://issues.apache.org/jira/browse/AMQ-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6778. - Resolution: Fixed > Advisories are dropped till broker is fully started in error > > > Key: AMQ-6778 > URL: https://issues.apache.org/jira/browse/AMQ-6778 > Project: ActiveMQ > Issue Type: Bug > Components: Broker, networkbridge >Affects Versions: 5.15.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.16.0 > > > The Advisory broker fire is conditional on the broker being fully started. > This has been present for the longest time however the started flag has been > modified to include a countDownLatch that is triggered on start completion. > This means that consumers that connect to the active transportConnectors do > not result in advisories which can mean incoming networkConnectors can mis > this advise in error which results in the demand being ignored. > From what I can see, the check for started does not add value and in the > event of a failure to fire an advisory the error will be caught and logged > rather than being ignored as it is now. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AMQ-6778) Advisories are dropped till broker is fully started in error
Gary Tully created AMQ-6778: --- Summary: Advisories are dropped till broker is fully started in error Key: AMQ-6778 URL: https://issues.apache.org/jira/browse/AMQ-6778 Project: ActiveMQ Issue Type: Bug Components: Broker, networkbridge Affects Versions: 5.15.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.16.0 The Advisory broker fire is conditional on the broker being fully started. This has been present for the longest time however the started flag has been modified to include a countDownLatch that is triggered on start completion. This means that consumers that connect to the active transportConnectors do not result in advisories which can mean incoming networkConnectors can mis this advise in error which results in the demand being ignored. >From what I can see, the check for started does not add value and in the event >of a failure to fire an advisory the error will be caught and logged rather >than being ignored as it is now. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AMQ-6771) Improve performance of KahaDB recovery check checkForCorruptJournalFiles=true
[ https://issues.apache.org/jira/browse/AMQ-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6771. - Resolution: Fixed > Improve performance of KahaDB recovery check checkForCorruptJournalFiles=true > - > > Key: AMQ-6771 > URL: https://issues.apache.org/jira/browse/AMQ-6771 > Project: ActiveMQ > Issue Type: Improvement > Components: KahaDB >Affects Versions: 5.15.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.16.0 > > > The KahaDB checkForCorruptJournalFiles option validates the checksum of every > journal batch record on startup. If a single producer writes many small > messages, the batch sizes in the journal will be small. The current check > implementation reads each batch at a time with a fseek/read sequence that can > be very slow over shared disks. > The check can be a quick buffered sequential read using the maxBatchSize > which should already be tuned to match the disk transfer rate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMQ-6772) runtimeConfigurationPlugin - prioritizedMessages property update at Runtime
[ https://issues.apache.org/jira/browse/AMQ-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089937#comment-16089937 ] Gary Tully commented on AMQ-6772: - it won't work at all via the runtimeConfigurationPlugin, you will need a wildcard in the default config to enable it for every queue, this then is applied for queues created at runtime in the normal way. if you want to selectively disable, you will need to restart. It is a performance tweak to disable it, that may not be relevant in your environment. > runtimeConfigurationPlugin - prioritizedMessages property update at Runtime > --- > > Key: AMQ-6772 > URL: https://issues.apache.org/jira/browse/AMQ-6772 > Project: ActiveMQ > Issue Type: Bug > Components: runtime configuration plugin >Affects Versions: 5.14.5 > Environment: Unix >Reporter: Anil P. > > I have configured the 'runtimeConfigurationPlugin' and started the Active MQ. > Then added the below in policyEntries: > prioritizedMessages="true" useCache="false" expireMessagesPeriod="0" /> > After saving the config file, I am able to see that the console log got > updated saying that config changes are detected and loaded. > When tested, however, Message Priority is not being enforced as expected on > the queue. > I checked in JConsole, I am not able to update this particular property there > either. > If I restart with the policy configured, the property is getting applied and > it is working as expected. It is only not happening if I try to have it > updated at run-time. > Tim dug into the code, and discovered that the > PolicyEntry.update() and PolicyEntry.baseUpdate() methods ( > https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=blob;f=activemq-broker/src/main/java/org/apache/activemq/broker/region/policy/PolicyEntry.java;h=5b7ff0e184b9e6b4be6cf61ebcacc0a845bb1a6c;hb=HEAD) > don't attempt to set any of the three properties being passed in. > Please fix this so that those attributes will be set when the policy entry is > reloaded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMQ-6771) Improve performance of KahaDB recovery check checkForCorruptJournalFiles=true
[ https://issues.apache.org/jira/browse/AMQ-6771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089842#comment-16089842 ] Gary Tully commented on AMQ-6771: - [~cshannon] I don't have great numbers from my local testing b/c I am on a ssd and seek is very fast. I saw some strace from a user where the repeated fseek/read was very slow. The current sequential read in batches eliminates the fseek. Would be great to get some early validation. Thanks. > Improve performance of KahaDB recovery check checkForCorruptJournalFiles=true > - > > Key: AMQ-6771 > URL: https://issues.apache.org/jira/browse/AMQ-6771 > Project: ActiveMQ > Issue Type: Improvement > Components: KahaDB >Affects Versions: 5.15.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.16.0 > > > The KahaDB checkForCorruptJournalFiles option validates the checksum of every > journal batch record on startup. If a single producer writes many small > messages, the batch sizes in the journal will be small. The current check > implementation reads each batch at a time with a fseek/read sequence that can > be very slow over shared disks. > The check can be a quick buffered sequential read using the maxBatchSize > which should already be tuned to match the disk transfer rate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMQ-4693) Add kerberos [SASL] authentication for TCP connectors
[ https://issues.apache.org/jira/browse/AMQ-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089692#comment-16089692 ] Gary Tully commented on AMQ-4693: - [~nannou9] thanks for the reply :-) > Add kerberos [SASL] authentication for TCP connectors > - > > Key: AMQ-4693 > URL: https://issues.apache.org/jira/browse/AMQ-4693 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 5.8.0 > Environment: linux, solaris >Reporter: Bhanu >Priority: Minor > Fix For: Unscheduled > > > Hi, > Can kerberos based authentication be added to ActiveMQ's TCP connectors. > Thanks, > Bhanu -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMQ-6772) runtimeConfigurationPlugin - prioritizedMessages property update at Runtime
[ https://issues.apache.org/jira/browse/AMQ-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089683#comment-16089683 ] Gary Tully commented on AMQ-6772: - use a wildcard policy to enable prioritySupport for all queues and disable it selectively if that is required. In that way any new queue will match the wildcard and get priority support. > runtimeConfigurationPlugin - prioritizedMessages property update at Runtime > --- > > Key: AMQ-6772 > URL: https://issues.apache.org/jira/browse/AMQ-6772 > Project: ActiveMQ > Issue Type: Bug > Components: runtime configuration plugin >Affects Versions: 5.14.5 > Environment: Unix >Reporter: Anil P. > > I have configured the 'runtimeConfigurationPlugin' and started the Active MQ. > Then added the below in policyEntries: > prioritizedMessages="true" useCache="false" expireMessagesPeriod="0" /> > After saving the config file, I am able to see that the console log got > updated saying that config changes are detected and loaded. > When tested, however, Message Priority is not being enforced as expected on > the queue. > I checked in JConsole, I am not able to update this particular property there > either. > If I restart with the policy configured, the property is getting applied and > it is working as expected. It is only not happening if I try to have it > updated at run-time. > Tim dug into the code, and discovered that the > PolicyEntry.update() and PolicyEntry.baseUpdate() methods ( > https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=blob;f=activemq-broker/src/main/java/org/apache/activemq/broker/region/policy/PolicyEntry.java;h=5b7ff0e184b9e6b4be6cf61ebcacc0a845bb1a6c;hb=HEAD) > don't attempt to set any of the three properties being passed in. > Please fix this so that those attributes will be set when the policy entry is > reloaded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMQ-6772) runtimeConfigurationPlugin - prioritizedMessages property update at Runtime
[ https://issues.apache.org/jira/browse/AMQ-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089592#comment-16089592 ] Gary Tully commented on AMQ-6772: - the priority support is not a good candidate for runtime modification because not only do the in-memory messages need to retained in priority memory, the store needs to retrieve and store messages in priority order. For this, the store needs a restart which is not practical. > runtimeConfigurationPlugin - prioritizedMessages property update at Runtime > --- > > Key: AMQ-6772 > URL: https://issues.apache.org/jira/browse/AMQ-6772 > Project: ActiveMQ > Issue Type: Bug > Components: runtime configuration plugin >Affects Versions: 5.14.5 > Environment: Unix >Reporter: Anil P. > > I have configured the 'runtimeConfigurationPlugin' and started the Active MQ. > Then added the below in policyEntries: > prioritizedMessages="true" useCache="false" expireMessagesPeriod="0" /> > After saving the config file, I am able to see that the console log got > updated saying that config changes are detected and loaded. > When tested, however, Message Priority is not being enforced as expected on > the queue. > I checked in JConsole, I am not able to update this particular property there > either. > If I restart with the policy configured, the property is getting applied and > it is working as expected. It is only not happening if I try to have it > updated at run-time. > Tim dug into the code, and discovered that the > PolicyEntry.update() and PolicyEntry.baseUpdate() methods ( > https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=blob;f=activemq-broker/src/main/java/org/apache/activemq/broker/region/policy/PolicyEntry.java;h=5b7ff0e184b9e6b4be6cf61ebcacc0a845bb1a6c;hb=HEAD) > don't attempt to set any of the three properties being passed in. > Please fix this so that those attributes will be set when the policy entry is > reloaded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AMQ-6771) Improve performance of KahaDB recovery check checkForCorruptJournalFiles=true
Gary Tully created AMQ-6771: --- Summary: Improve performance of KahaDB recovery check checkForCorruptJournalFiles=true Key: AMQ-6771 URL: https://issues.apache.org/jira/browse/AMQ-6771 Project: ActiveMQ Issue Type: Improvement Components: KahaDB Affects Versions: 5.15.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.16.0 The KahaDB checkForCorruptJournalFiles option validates the checksum of every journal batch record on startup. If a single producer writes many small messages, the batch sizes in the journal will be small. The current check implementation reads each batch at a time with a fseek/read sequence that can be very slow over shared disks. The check can be a quick buffered sequential read using the maxBatchSize which should already be tuned to match the disk transfer rate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (ARTEMIS-1264) Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
[
https://issues.apache.org/jira/browse/ARTEMIS-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully updated ARTEMIS-1264:
Description:
Allow a client authenticated with a kerberos credential to authenticate to the
broker using SSL via the Kerberos cipher suites.
next steps:
- -ensure mapping from kerberos principal to broker identity is locked down-
-- https://github.com/apache/activemq-artemis/pull/1388
- -ensure jms client config is trivial-
-- the connector properties can be configured in the same way as for core.
- -validate broker side ticket expiry and renewal-
- work with qpid-jms to validate amqp client (on hold)
- validate with non java - proton-c client ({color:red}problem{color})
Interop with non java clients is a problem. OpenSSL [has removed
support|http://openssl.6102.n7.nabble.com/openssl-users-Kerberos-tp57906p58095.html]
for [rfc2712|https://www.ietf.org/rfc/rfc2712.txt].
While reusing the TLS handshake was a good idea at the time; it has issues (non
compatible impl between openssl and sun) and the world has moved on to layering
authentication over TLS rather than with.
This makes sense b/c kerberos does two things, authentication over an insecure
connection and session encryption over that connection. With rfc2712 the
available session encryption options are known to be insecure, best to leave
encryption entirely to TLS.
In a java only scenario (sun jdk on both ends), using this feature for kerberos
*authentication only* is viable.
For example, if clients use username/password for authentication and TLS to
encrypt the connection to secure the password, but don't care about encrypting
the rest of the data, there is some value here.
They can swap the username/password for a kerberos token and achieve
authentication. They will essentially drop encryption because the cypher in use
is insecure. Note a kerberos ticket is designed to be validated across an
insecure channel.
The modern approach is to layer kerberos authentication over TLS using
something like the GSSAPI and SASL.
was:
Allow a client authenticated with a kerberos credential to authenticate to the
broker using SSL via the Kerberos cipher suites.
next steps:
- -ensure mapping from kerberos principal to broker identity is locked down-
-- https://github.com/apache/activemq-artemis/pull/1388
- -ensure jms client config is trivial-
-- the connector properties can be configured in the same way as for core.
- -validate broker side ticket expiry and renewal-
- work with qpid-jms to validate amqp client (on hold)
- validate with non java - proton-c client ({color:red}problem{color})
Interop with non java clients is a problem. OpenSSL [has removed
support|http://openssl.6102.n7.nabble.com/openssl-users-Kerberos-tp57906p58095.html]
for [rfc2712|https://www.ietf.org/rfc/rfc2712.txt].
While reusing the TLS handshake was a good idea at the time; it has issues (non
compatible impl between openssl and sun) and the world has moved on to layering
authentication over TLS rather than with.
This makes sense b/c kerberos does two things, authentication over an insecure
connection and session encryption over that connection. With rfc2712 the
available session encryption options are known to be insecure.
In a java only scenario (sun jdk on both ends), using this feature for kerberos
*authentication only* is viable.
For example, if clients use username/password for authentication and TLS to
encrypt the connection to secure the password, but don't care about encrypting
the rest of the data, there is some value here.
They can swap the username/password for a kerberos token and achieve
authentication. They will essentially drop encryption because the cypher in use
is insecure. Note a kerberos ticket is designed to be validated across an
insecure channel.
The modern approach is to layer kerberos authentication over TLS using
something like the GSSAPI and SASL.
> Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
> ---
>
> Key: ARTEMIS-1264
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1264
> Project: ActiveMQ Artemis
> Issue Type: New Feature
>Affects Versions: 2.1.0
>Reporter: Gary Tully
>Assignee: Gary Tully
>
> Allow a client authenticated with a kerberos credential to authenticate to
> the broker using SSL via the Kerberos cipher suites.
> next steps:
> - -ensure mapping from kerberos principal to broker identity is locked down-
> -- https://github.com/apache/activemq-artemis/pull/1388
> - -ensure jms client config is trivial-
> -- the connector properties can be configured in the same way as for core.
> - -validate broker side ticket expiry and renewal-
> - work with qpid-jms to validate amqp client (on hold)
> - validate with non java - proton-c
[jira] [Commented] (AMQ-4693) Add kerberos [SASL] authentication for TCP connectors
[ https://issues.apache.org/jira/browse/AMQ-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085525#comment-16085525 ] Gary Tully commented on AMQ-4693: - [~nannou9] I was inspired by ur PR to implement same in artemis - https://issues.apache.org/jira/browse/ARTEMIS-1264 however it appears the solution is in a bit of a java bubble and maybe a time warp. One I peeked out of the java realm I found the OpenSSL discussions and noted the evolution from RFC2712. It *is* too good to be true that any TLS client can get kerberos authentication for free. It comes at the cost of having an insecure channel. The use case is Kerberos SSO over an insecure channel, have you found that useful in practice? > Add kerberos [SASL] authentication for TCP connectors > - > > Key: AMQ-4693 > URL: https://issues.apache.org/jira/browse/AMQ-4693 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker >Affects Versions: 5.8.0 > Environment: linux, solaris >Reporter: Bhanu >Priority: Minor > Fix For: Unscheduled > > > Hi, > Can kerberos based authentication be added to ActiveMQ's TCP connectors. > Thanks, > Bhanu -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Issue Comment Deleted] (ARTEMIS-1264) Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
[
https://issues.apache.org/jira/browse/ARTEMIS-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully updated ARTEMIS-1264:
Comment: was deleted
(was: next steps:
- -ensure mapping from kerberos principal to broker identity is locked down-
-- https://github.com/apache/activemq-artemis/pull/1388
- -ensure jms client config is trivial-
-- the connector properties can be configured in the same way as for core.
- -validate broker side ticket expiry and renewal-
- work with qpid-jms to validate amqp client (on hold)
- validate with non java - proton-c client ({color:red}problem{color})
Interop with non java clients is a problem. OpenSSL [has removed
support|http://openssl.6102.n7.nabble.com/openssl-users-Kerberos-tp57906p58095.html]
for [rfc2712|https://www.ietf.org/rfc/rfc2712.txt].
While reusing the TLS handshake was a good idea at the time; it has issues (non
compatible impl between openssl and sun) and the world has moved on to layering
authentication over TLS rather than with.
This makes sense b/c kerberos does two things, authentication over an insecure
connection and session encryption over that connection. With rfc2712 the
available session encryption options are known to be insecure.
In a java only scenario (sun jdk on both ends), using this feature for kerberos
*authentication only* is viable.
For example, if clients use username/password for authentication and TLS to
encrypt the connection to secure the password, but don't care about encrypting
the rest of the data, there is some value here.
They can swap the username/password for a kerberos token and achieve
authentication. They will essentially drop encryption because the cypher in use
is insecure. Note a kerberos ticket is designed to be validated across an
insecure channel.
The modern approach is to layer kerberos authentication over TLS using
something like the GSSAPI and SASL.)
> Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
> ---
>
> Key: ARTEMIS-1264
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1264
> Project: ActiveMQ Artemis
> Issue Type: New Feature
>Affects Versions: 2.1.0
>Reporter: Gary Tully
>Assignee: Gary Tully
>
> Allow a client authenticated with a kerberos credential to authenticate to
> the broker using SSL via the Kerberos cipher suites.
> next steps:
> - -ensure mapping from kerberos principal to broker identity is locked down-
> -- https://github.com/apache/activemq-artemis/pull/1388
> - -ensure jms client config is trivial-
> -- the connector properties can be configured in the same way as for core.
> - -validate broker side ticket expiry and renewal-
> - work with qpid-jms to validate amqp client (on hold)
> - validate with non java - proton-c client ({color:red}problem{color})
> Interop with non java clients is a problem. OpenSSL [has removed
> support|http://openssl.6102.n7.nabble.com/openssl-users-Kerberos-tp57906p58095.html]
> for [rfc2712|https://www.ietf.org/rfc/rfc2712.txt].
> While reusing the TLS handshake was a good idea at the time; it has issues
> (non compatible impl between openssl and sun) and the world has moved on to
> layering authentication over TLS rather than with.
> This makes sense b/c kerberos does two things, authentication over an
> insecure connection and session encryption over that connection. With rfc2712
> the available session encryption options are known to be insecure.
> In a java only scenario (sun jdk on both ends), using this feature for
> kerberos *authentication only* is viable.
> For example, if clients use username/password for authentication and TLS to
> encrypt the connection to secure the password, but don't care about
> encrypting the rest of the data, there is some value here.
> They can swap the username/password for a kerberos token and achieve
> authentication. They will essentially drop encryption because the cypher in
> use is insecure. Note a kerberos ticket is designed to be validated across an
> insecure channel.
> The modern approach is to layer kerberos authentication over TLS using
> something like the GSSAPI and SASL.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (ARTEMIS-1264) Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
[
https://issues.apache.org/jira/browse/ARTEMIS-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully updated ARTEMIS-1264:
Description:
Allow a client authenticated with a kerberos credential to authenticate to the
broker using SSL via the Kerberos cipher suites.
next steps:
- -ensure mapping from kerberos principal to broker identity is locked down-
-- https://github.com/apache/activemq-artemis/pull/1388
- -ensure jms client config is trivial-
-- the connector properties can be configured in the same way as for core.
- -validate broker side ticket expiry and renewal-
- work with qpid-jms to validate amqp client (on hold)
- validate with non java - proton-c client ({color:red}problem{color})
Interop with non java clients is a problem. OpenSSL [has removed
support|http://openssl.6102.n7.nabble.com/openssl-users-Kerberos-tp57906p58095.html]
for [rfc2712|https://www.ietf.org/rfc/rfc2712.txt].
While reusing the TLS handshake was a good idea at the time; it has issues (non
compatible impl between openssl and sun) and the world has moved on to layering
authentication over TLS rather than with.
This makes sense b/c kerberos does two things, authentication over an insecure
connection and session encryption over that connection. With rfc2712 the
available session encryption options are known to be insecure.
In a java only scenario (sun jdk on both ends), using this feature for kerberos
*authentication only* is viable.
For example, if clients use username/password for authentication and TLS to
encrypt the connection to secure the password, but don't care about encrypting
the rest of the data, there is some value here.
They can swap the username/password for a kerberos token and achieve
authentication. They will essentially drop encryption because the cypher in use
is insecure. Note a kerberos ticket is designed to be validated across an
insecure channel.
The modern approach is to layer kerberos authentication over TLS using
something like the GSSAPI and SASL.
was:Allow a client authenticated with a kerberos credential to authenticate
to the broker using SSL via the Kerberos cipher suites.
> Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
> ---
>
> Key: ARTEMIS-1264
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1264
> Project: ActiveMQ Artemis
> Issue Type: New Feature
>Affects Versions: 2.1.0
>Reporter: Gary Tully
>Assignee: Gary Tully
>
> Allow a client authenticated with a kerberos credential to authenticate to
> the broker using SSL via the Kerberos cipher suites.
> next steps:
> - -ensure mapping from kerberos principal to broker identity is locked down-
> -- https://github.com/apache/activemq-artemis/pull/1388
> - -ensure jms client config is trivial-
> -- the connector properties can be configured in the same way as for core.
> - -validate broker side ticket expiry and renewal-
> - work with qpid-jms to validate amqp client (on hold)
> - validate with non java - proton-c client ({color:red}problem{color})
> Interop with non java clients is a problem. OpenSSL [has removed
> support|http://openssl.6102.n7.nabble.com/openssl-users-Kerberos-tp57906p58095.html]
> for [rfc2712|https://www.ietf.org/rfc/rfc2712.txt].
> While reusing the TLS handshake was a good idea at the time; it has issues
> (non compatible impl between openssl and sun) and the world has moved on to
> layering authentication over TLS rather than with.
> This makes sense b/c kerberos does two things, authentication over an
> insecure connection and session encryption over that connection. With rfc2712
> the available session encryption options are known to be insecure.
> In a java only scenario (sun jdk on both ends), using this feature for
> kerberos *authentication only* is viable.
> For example, if clients use username/password for authentication and TLS to
> encrypt the connection to secure the password, but don't care about
> encrypting the rest of the data, there is some value here.
> They can swap the username/password for a kerberos token and achieve
> authentication. They will essentially drop encryption because the cypher in
> use is insecure. Note a kerberos ticket is designed to be validated across an
> insecure channel.
> The modern approach is to layer kerberos authentication over TLS using
> something like the GSSAPI and SASL.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (ARTEMIS-1264) Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
[
https://issues.apache.org/jira/browse/ARTEMIS-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074589#comment-16074589
]
Gary Tully edited comment on ARTEMIS-1264 at 7/13/17 10:22 AM:
---
next steps:
- -ensure mapping from kerberos principal to broker identity is locked down-
-- https://github.com/apache/activemq-artemis/pull/1388
- -ensure jms client config is trivial-
-- the connector properties can be configured in the same way as for core.
- -validate broker side ticket expiry and renewal-
- work with qpid-jms to validate amqp client (on hold)
- validate with non java - proton-c client ({color:red}problem{color})
Interop with non java clients is a problem. OpenSSL [has removed
support|http://openssl.6102.n7.nabble.com/openssl-users-Kerberos-tp57906p58095.html]
for [rfc2712|https://www.ietf.org/rfc/rfc2712.txt].
While reusing the TLS handshake was a good idea at the time; it has issues (non
compatible impl between openssl and sun) and the world has moved on to layering
authentication over TLS rather than with.
This makes sense b/c kerberos does two things, authentication over an insecure
connection and session encryption over that connection. With rfc2712 the
available session encryption options are known to be insecure.
In a java only scenario (sun jdk on both ends), using this feature for kerberos
*authentication only* is viable.
For example, if clients use username/password for authentication and TLS to
encrypt the connection to secure the password, but don't care about encrypting
the rest of the data, there is some value here.
They can swap the username/password for a kerberos token and achieve
authentication. They will essentially drop encryption because the cypher in use
is insecure. Note a kerberos ticket is designed to be validated across an
insecure channel.
The modern approach is to layer kerberos authentication over TLS using
something like the GSSAPI and SASL.
was (Author: gtully):
next steps:
- -ensure mapping from kerberos principal to broker identity is locked down-
-- https://github.com/apache/activemq-artemis/pull/1388
- ensure jms client config is trivial
- validate broker side ticket expiry and renewal
- work with qpid-jms to validate amqp client
- validate with non java - proton-c client
> Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
> ---
>
> Key: ARTEMIS-1264
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1264
> Project: ActiveMQ Artemis
> Issue Type: New Feature
>Affects Versions: 2.1.0
>Reporter: Gary Tully
>Assignee: Gary Tully
>
> Allow a client authenticated with a kerberos credential to authenticate to
> the broker using SSL via the Kerberos cipher suites.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (AMQ-6764) Add ended statement to Audit log for JMX ops
[ https://issues.apache.org/jira/browse/AMQ-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6764. - Resolution: Fixed > Add ended statement to Audit log for JMX ops > > > Key: AMQ-6764 > URL: https://issues.apache.org/jira/browse/AMQ-6764 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.16.0 > > > AMQ-3100 introduced an audit log. It is boolean enabled or disabled. > This enhancement allows four values for the system property: > "org.apache.activemq.audit" > true|entry,exit,all > - true, entry: log each called op > - exit: log the ending of each called op > - all: - log both the entry and exit (called and ended) log statements -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AMQ-6764) Add ended statement to Audit log for JMX ops
[ https://issues.apache.org/jira/browse/AMQ-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully updated AMQ-6764: Description: AMQ-3100 introduced an audit log. It is boolean enabled or disabled. This enhancement allows four values for the system property: "org.apache.activemq.audit" true|entry,exit,all - true|entry: log each called op (existing behaviour) - exit: log the ending of each called op - all: - log both the entry and exit (called and ended) log statements was: AMQ-3100 introduced an audit log. It is boolean enabled or disabled. This enhancement allows four values for the system property: "org.apache.activemq.audit" true|entry,exit,all - true, entry: log each called op - exit: log the ending of each called op - all: - log both the entry and exit (called and ended) log statements > Add ended statement to Audit log for JMX ops > > > Key: AMQ-6764 > URL: https://issues.apache.org/jira/browse/AMQ-6764 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.16.0 > > > AMQ-3100 introduced an audit log. It is boolean enabled or disabled. > This enhancement allows four values for the system property: > "org.apache.activemq.audit" > true|entry,exit,all > - true|entry: log each called op (existing behaviour) > - exit: log the ending of each called op > - all: - log both the entry and exit (called and ended) log statements -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AMQ-6764) Add ended statement to Audit log for JMX ops
[ https://issues.apache.org/jira/browse/AMQ-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully updated AMQ-6764: Description: AMQ-3100 introduced an audit log. It is boolean enabled or disabled. This enhancement allows four values for the system property: "org.apache.activemq.audit" true|entry,exit,all - true, entry: log each called op - exit: log the ending of each called op - all: - log both the entry and exit (called and ended) log statements was: AMQ-3100 introduced an audit log. It is boolean enabled or disabled. This enhancement allows four values for the system property org.apache.activemq.audit true|entry,exit,all - true, entry: log each called op - exit: log the ending of each called op - all: - log both the entry and exit (called and ended) log statements > Add ended statement to Audit log for JMX ops > > > Key: AMQ-6764 > URL: https://issues.apache.org/jira/browse/AMQ-6764 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.16.0 > > > AMQ-3100 introduced an audit log. It is boolean enabled or disabled. > This enhancement allows four values for the system property: > "org.apache.activemq.audit" > true|entry,exit,all > - true, entry: log each called op > - exit: log the ending of each called op > - all: - log both the entry and exit (called and ended) log statements -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (ARTEMIS-1264) Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
[ https://issues.apache.org/jira/browse/ARTEMIS-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074589#comment-16074589 ] Gary Tully edited comment on ARTEMIS-1264 at 7/6/17 4:05 PM: - next steps: - -ensure mapping from kerberos principal to broker identity is locked down- -- https://github.com/apache/activemq-artemis/pull/1388 - ensure jms client config is trivial - validate broker side ticket expiry and renewal - work with qpid-jms to validate amqp client - validate with non java - proton-c client was (Author: gtully): next steps: - ensure mapping from kerberos principal to broker identity is locked down - ensure jms client config is trivial - validate broker side ticket expiry and renewal - work with qpid-jms to validate amqp client - validate with non java - proton-c client > Client authentication via Kerberos TLS Cipher Suites (RFC 2712) > --- > > Key: ARTEMIS-1264 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1264 > Project: ActiveMQ Artemis > Issue Type: New Feature >Affects Versions: 2.1.0 >Reporter: Gary Tully >Assignee: Gary Tully > > Allow a client authenticated with a kerberos credential to authenticate to > the broker using SSL via the Kerberos cipher suites. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (ARTEMIS-1264) Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
[ https://issues.apache.org/jira/browse/ARTEMIS-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074589#comment-16074589 ] Gary Tully edited comment on ARTEMIS-1264 at 7/5/17 12:38 PM: -- next steps: - ensure mapping from kerberos principal to broker identity is locked down - ensure jms client config is trivial - validate broker side ticket expiry and renewal - work with qpid-jms to validate amqp client - validate with non java - proton-c client was (Author: gtully): next steps: - ensure jms client config is trivial - validate broker side ticket expiry and renewal - work with qpid-jms to validate amqp client - validate with non java - proton-c client > Client authentication via Kerberos TLS Cipher Suites (RFC 2712) > --- > > Key: ARTEMIS-1264 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1264 > Project: ActiveMQ Artemis > Issue Type: New Feature >Affects Versions: 2.1.0 >Reporter: Gary Tully >Assignee: Gary Tully > > Allow a client authenticated with a kerberos credential to authenticate to > the broker using SSL via the Kerberos cipher suites. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ARTEMIS-1264) Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
[ https://issues.apache.org/jira/browse/ARTEMIS-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074589#comment-16074589 ] Gary Tully commented on ARTEMIS-1264: - next steps: - ensure jms client config is trivial - validate broker side ticket expiry and renewal - work with qpid-jms to validate amqp client - validate with non java - proton-c client > Client authentication via Kerberos TLS Cipher Suites (RFC 2712) > --- > > Key: ARTEMIS-1264 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1264 > Project: ActiveMQ Artemis > Issue Type: New Feature >Affects Versions: 2.1.0 >Reporter: Gary Tully >Assignee: Gary Tully > > Allow a client authenticated with a kerberos credential to authenticate to > the broker using SSL via the Kerberos cipher suites. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (ARTEMIS-1264) Client authentication via Kerberos TLS Cipher Suites (RFC 2712)
Gary Tully created ARTEMIS-1264: --- Summary: Client authentication via Kerberos TLS Cipher Suites (RFC 2712) Key: ARTEMIS-1264 URL: https://issues.apache.org/jira/browse/ARTEMIS-1264 Project: ActiveMQ Artemis Issue Type: New Feature Affects Versions: 2.1.0 Reporter: Gary Tully Assignee: Gary Tully Allow a client authenticated with a kerberos credential to authenticate to the broker using SSL via the Kerberos cipher suites. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMQ-6625) IOException in kahaDB needs to pause pending IOExceptionHandler intervention
[ https://issues.apache.org/jira/browse/AMQ-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050747#comment-16050747 ] Gary Tully commented on AMQ-6625: - The DefaultIOExceptionHandler implements a resume call on kahadb that allows the stopStartConnectors behaviour or ignore all errors behaviour to be chosen if necessary. > IOException in kahaDB needs to pause pending IOExceptionHandler intervention > > > Key: AMQ-6625 > URL: https://issues.apache.org/jira/browse/AMQ-6625 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > KahaDB can recover from kill -9 by replaying the journal from the last > checkpoint or by detecting and reapplying partial writes to the index. > However this activity is compromised if the journal or index accepts > subsequent writes. It an lead to skipped write batches or skipped partial > updates to the index. > The desirable behaviour of treating an IOException as fatal and stopping the > broker in the knowledge that it will restart and fully recover needs to treat > the first IO error as fatal and by default not accept any further writes. > A more advanced IOException handler can facilitate staying alive in more > specific scenarios and reactivate kahadb. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (AMQ-6625) IOException in kahaDB needs to pause pending IOExceptionHandler intervention
[ https://issues.apache.org/jira/browse/AMQ-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907515#comment-15907515 ] Gary Tully edited comment on AMQ-6625 at 6/15/17 4:38 PM: -- Prior to the handIOException up-call the journal and index pause. The appender write thread won't get recreated and the index is marked unloaded. This ensures that subsequent writes will fail allowing auto recovery from a partial write to kick in on restart. The DefaultIOExceptionHandler implements a resume call on kahadb that allows the stopStartConnectors behaviour or ignore all errors behaviour to be chosen if necessary. was (Author: gtully): Prior to the handIOException up-call the journal and index pause. The appender write thread won't get recreated and the index is marked unloaded. This ensures that subsequent writes will fail allowing auto recovery from a partial write to kick in on restart. The new kahaDBIOExceptionHandler implements a resume call on kahadb that allows the stopStartConnectors behaviour or ignore all errors behaviour to be chosen if necessary. > IOException in kahaDB needs to pause pending IOExceptionHandler intervention > > > Key: AMQ-6625 > URL: https://issues.apache.org/jira/browse/AMQ-6625 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > KahaDB can recover from kill -9 by replaying the journal from the last > checkpoint or by detecting and reapplying partial writes to the index. > However this activity is compromised if the journal or index accepts > subsequent writes. It an lead to skipped write batches or skipped partial > updates to the index. > The desirable behaviour of treating an IOException as fatal and stopping the > broker in the knowledge that it will restart and fully recover needs to treat > the first IO error as fatal and by default not accept any further writes. > A more advanced IOException handler can facilitate staying alive in more > specific scenarios and reactivate kahadb. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AMQ-6625) IOException in kahaDB needs to pause pending IOExceptionHandler intervention
[ https://issues.apache.org/jira/browse/AMQ-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6625. - Resolution: Fixed > IOException in kahaDB needs to pause pending IOExceptionHandler intervention > > > Key: AMQ-6625 > URL: https://issues.apache.org/jira/browse/AMQ-6625 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > KahaDB can recover from kill -9 by replaying the journal from the last > checkpoint or by detecting and reapplying partial writes to the index. > However this activity is compromised if the journal or index accepts > subsequent writes. It an lead to skipped write batches or skipped partial > updates to the index. > The desirable behaviour of treating an IOException as fatal and stopping the > broker in the knowledge that it will restart and fully recover needs to treat > the first IO error as fatal and by default not accept any further writes. > A more advanced IOException handler can facilitate staying alive in more > specific scenarios and reactivate kahadb. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (AMQ-6625) IOException in kahaDB needs to pause pending IOExceptionHandler intervention
[ https://issues.apache.org/jira/browse/AMQ-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully reopened AMQ-6625: - The KahaDbIOExceptionHandler clashes with the use of the LeaseLockerIOExceptionHandler - need to push the allowIOResumption support into the persistence adapter. > IOException in kahaDB needs to pause pending IOExceptionHandler intervention > > > Key: AMQ-6625 > URL: https://issues.apache.org/jira/browse/AMQ-6625 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > KahaDB can recover from kill -9 by replaying the journal from the last > checkpoint or by detecting and reapplying partial writes to the index. > However this activity is compromised if the journal or index accepts > subsequent writes. It an lead to skipped write batches or skipped partial > updates to the index. > The desirable behaviour of treating an IOException as fatal and stopping the > broker in the knowledge that it will restart and fully recover needs to treat > the first IO error as fatal and by default not accept any further writes. > A more advanced IOException handler can facilitate staying alive in more > specific scenarios and reactivate kahadb. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AMQ-6703) JMX purge needs to clear the message audit
[ https://issues.apache.org/jira/browse/AMQ-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6703. - Resolution: Fixed > JMX purge needs to clear the message audit > -- > > Key: AMQ-6703 > URL: https://issues.apache.org/jira/browse/AMQ-6703 > Project: ActiveMQ > Issue Type: Bug > Components: Broker, JMX >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > JMX purge clears a queue of messages but does not reset the message audit. > Doing a X.copyTo(Y), X.purge() Y.copyYo(X) results in duplicate detection and > messages going to the DLQ in error. > JMX purge needs to clear the message audit such that the queue is free of old > state. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AMQ-6703) JMX purge needs to clear the message audit
Gary Tully created AMQ-6703: --- Summary: JMX purge needs to clear the message audit Key: AMQ-6703 URL: https://issues.apache.org/jira/browse/AMQ-6703 Project: ActiveMQ Issue Type: Bug Components: Broker, JMX Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.15.0 JMX purge clears a queue of messages but does not reset the message audit. Doing a X.copyTo(Y), X.purge() Y.copyYo(X) results in duplicate detection and messages going to the DLQ in error. JMX purge needs to clear the message audit such that the queue is free of old state. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMQ-6701) Intermittent CPU spikes in one broker in a network bridge
[
https://issues.apache.org/jira/browse/AMQ-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049094#comment-16049094
]
Gary Tully commented on AMQ-6701:
-
there is noting odd in there. it could be gc: see
https://blog.codecentric.de/en/2014/01/useful-jvm-flags-part-8-gc-logging/
> Intermittent CPU spikes in one broker in a network bridge
> -
>
> Key: AMQ-6701
> URL: https://issues.apache.org/jira/browse/AMQ-6701
> Project: ActiveMQ
> Issue Type: Task
> Components: Broker
>Affects Versions: 5.12.0
> Environment: RHEL 7.2 in VMWARE ESXi virtual machines, JDK 1.8
>Reporter: Sridip Bhattacharya
> Attachments: jstack.zip
>
>
> Hi,
> We have a setup with network bridge between two activemq, say *http-broker12
> & http-broker21* in two different VM, where communication occurs in both ways.
> {noformat}
>
> uri="static:(tcp://10.137.80.199:61616)" networkTTL="2"/>
> {noformat}
> {noformat}
>
> uri="static:(tcp://10.137.80.179:61616)" networkTTL="2"/>
> {noformat}
> Now we are facing intermittent CPU spikes (400%) in http-broker12 broker.
> Whereas http-broker21 works normal.
> Could you please confirm if anything abnormal/identified bug or config issue
> here?
> //SB
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (AMQ-6702) Allow dead letter strategy audit configuration
[ https://issues.apache.org/jira/browse/AMQ-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6702. - Resolution: Fixed The dead letter strategy has an message audit that is enabled by default. This prevents duplicate messages from being added to the configured DLQ. From 5.15.0, the limits of the audit can configured via the maxProducersToAudit and maxAuditDepth attributes. The audit can be disabled using enableAudit="false" > Allow dead letter strategy audit configuration > -- > > Key: AMQ-6702 > URL: https://issues.apache.org/jira/browse/AMQ-6702 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > When the default shared dead letter strategy is used with composite queues or > durable subs or virtual topics and messages expire - the dlq gets lots of > requests to store duplicates. The audit on the strategy traps most of these > duplicate add attempts but that audit is not configurable so it is possible > the some get through which may be trapped by the message store or cursor on > subsequent dispatch. > Allowing the audit to be configured past the default parameters allows > duplicates to be stopped at source. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AMQ-6702) Allow dead letter strategy audit configuration
Gary Tully created AMQ-6702: --- Summary: Allow dead letter strategy audit configuration Key: AMQ-6702 URL: https://issues.apache.org/jira/browse/AMQ-6702 Project: ActiveMQ Issue Type: Improvement Components: Broker Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.15.0 When the default shared dead letter strategy is used with composite queues or durable subs or virtual topics and messages expire - the dlq gets lots of requests to store duplicates. The audit on the strategy traps most of these duplicate add attempts but that audit is not configurable so it is possible the some get through which may be trapped by the message store or cursor on subsequent dispatch. Allowing the audit to be configured past the default parameters allows duplicates to be stopped at source. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMQ-6701) Intermittent CPU spikes in one broker in a network bridge
[
https://issues.apache.org/jira/browse/AMQ-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047758#comment-16047758
]
Gary Tully commented on AMQ-6701:
-
post a series of thread dumps during the high cpu period, that will tell you
what is going on.
> Intermittent CPU spikes in one broker in a network bridge
> -
>
> Key: AMQ-6701
> URL: https://issues.apache.org/jira/browse/AMQ-6701
> Project: ActiveMQ
> Issue Type: Task
> Components: Broker
>Affects Versions: 5.12.0
> Environment: RHEL 7.2 in VMWARE ESXi virtual machines, JDK 1.8
>Reporter: Sridip Bhattacharya
>
> Hi,
> We have a setup with network bridge between two activemq, say *http-broker12
> & http-broker21* in two different VM, where communication occurs in both ways.
> {noformat}
>
> uri="static:(tcp://10.137.80.199:61616)" networkTTL="2"/>
> {noformat}
> {noformat}
>
> uri="static:(tcp://10.137.80.179:61616)" networkTTL="2"/>
> {noformat}
> Now we are facing intermittent CPU spikes (400%) in http-broker12 broker.
> Whereas http-broker21 works normal.
> Could you please confirm if anything abnormal/identified bug or config issue
> here?
> //SB
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMQ-6683) NullPointerException on transaction prepare in JdbcMememoryTransactionStore
[
https://issues.apache.org/jira/browse/AMQ-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040724#comment-16040724
]
Gary Tully commented on AMQ-6683:
-
Any chance you can replicate the usage pattern in some test code or maybe with
trace=true on the transport connector broker side such that every command will
be logged.
This looks maybe like some contention between add and prepare. Are there
multiple connections involved in a single XA transaction. The TM(aries) will
treat them as a single resource based on the broker id but maybe there is some
ordering issue. There is an option to have each connection represent a separate
resource - maybe that would workaround - use
"tcp://host:port?jms.rmIdFromConnectionId=true" on the broker url for your
connection factory
> NullPointerException on transaction prepare in JdbcMememoryTransactionStore
> ---
>
> Key: AMQ-6683
> URL: https://issues.apache.org/jira/browse/AMQ-6683
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker, JDBC
>Affects Versions: 5.14.5
>Reporter: Jakub
>Priority: Blocker
>
> From time to time an exception occurs:
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.activemq.store.jdbc.JdbcMemoryTransactionStore.prepare(JdbcMemoryTransactionStore.java:75)
> at
> org.apache.activemq.transaction.XATransaction.prepare(XATransaction.java:188)
> at
> org.apache.activemq.broker.TransactionBroker.prepareTransaction(TransactionBroker.java:247)
> at
> org.apache.activemq.broker.MutableBrokerFilter.prepareTransaction(MutableBrokerFilter.java:133)
> at
> org.apache.activemq.broker.MutableBrokerFilter.prepareTransaction(MutableBrokerFilter.java:133)
> at
> org.apache.activemq.broker.TransportConnection.processPrepareTransaction(TransportConnection.java:519)
> at
> org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:98)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[47:org.apache.activemq.activemq-osgi:5.14.5]
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[47:org.apache.activemq.activemq-osgi:5.14.5]
> at java.lang.Thread.run(Thread.java:745)[:1.8.0_131]
> {code}
> On client side there is an exception:
> {code}
> XID:[1197822575,globalId=2c162f175c1006f72672e6170616368652e61726965732e7472616e73616374696f6e,branchId=2000ffe5ff8319175c1006170616368652e61726965732e7472616e73616374696f6e]
> failed with: javax.jms.JMSException: java.lang.NullPointerException
> javax.jms.JMSException: java.lang.NullPointerException
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1399)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1428)
> at
> org.apache.activemq.TransactionContext.prepare(TransactionContext.java:469)
> at
> org.apache.geronimo.transaction.manager.WrapperNamedXAResource.prepare(WrapperNamedXAResource.java:86)
> at
> org.apache.geronimo.transaction.manager.TransactionImpl.internalPrepare(TransactionImpl.java:429)
> at
> org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:312)
> at
> org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:252)
> at
> org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1020)
> at
> org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
> at
>
[jira] [Commented] (AMQ-6695) MirrorQueue - send a copy to the mirror topic instead of original message
[ https://issues.apache.org/jira/browse/AMQ-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034856#comment-16034856 ] Gary Tully commented on AMQ-6695: - I agree - the mirror should get a copy - and I think it should differ by having an originalDestination set: very like the doForward in the composite case: https://github.com/apache/activemq/blob/7413ee00e1d19563cb0273df954ad71ef1433705/activemq-broker/src/main/java/org/apache/activemq/broker/region/virtual/CompositeDestinationFilter.java#L112 > MirrorQueue - send a copy to the mirror topic instead of original message > - > > Key: AMQ-6695 > URL: https://issues.apache.org/jira/browse/AMQ-6695 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.x >Reporter: Joachim Glink >Priority: Minor > Attachments: > MirrorQueue___send_the_original_message_to_the_original_destination_and_a_copy_to_the_mirr.patch > > > The MirrorQueue implementation currently sends the original message to the > mirror topic and a copy (if copy flag is set to true) of the message to the > requested destination. > Also, the MemoryUsage on the message which is sent to the requested > destination is set to null but the comment in the line notes that it´s done > so the memory size is used from the queue instead of the topic which isn´t > the case; its vise versa. (At least in my opinion) > Another possible bug in my opinion is, that the MemoryUsage is set to null > even if the copy flag is false; so the same message is sent to the mirror and > destination but the MemoryUsage is set to null. This could lead to a false > calculation of the broker storage. > If there are further plugins configured, than these plugins see the original > message in e.g. the send method. But this message isn´t the message which > will be sent to the defined destination but the message which is sent to the > mirror. > I´d suggest to reverse the logic of the MirrorQueue so that a copy of the > message is sent to the mirror topic and the original message to the requested > destination (and also the MemoryUsage of this copy is set to null - and only > if it´s a copy!) > A patch is attached to this issue which would cover the changes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-6695) MirrorQueue - send a copy to the mirror topic instead of original message
[ https://issues.apache.org/jira/browse/AMQ-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034562#comment-16034562 ] Gary Tully commented on AMQ-6695: - guess the copy option could be deprecated in the same way as in CompositeDestination > MirrorQueue - send a copy to the mirror topic instead of original message > - > > Key: AMQ-6695 > URL: https://issues.apache.org/jira/browse/AMQ-6695 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.x >Reporter: Joachim Glink >Priority: Minor > Attachments: > MirrorQueue___send_the_original_message_to_the_original_destination_and_a_copy_to_the_mirr.patch > > > The MirrorQueue implementation currently sends the original message to the > mirror topic and a copy (if copy flag is set to true) of the message to the > requested destination. > Also, the MemoryUsage on the message which is sent to the requested > destination is set to null but the comment in the line notes that it´s done > so the memory size is used from the queue instead of the topic which isn´t > the case; its vise versa. (At least in my opinion) > Another possible bug in my opinion is, that the MemoryUsage is set to null > even if the copy flag is false; so the same message is sent to the mirror and > destination but the MemoryUsage is set to null. This could lead to a false > calculation of the broker storage. > If there are further plugins configured, than these plugins see the original > message in e.g. the send method. But this message isn´t the message which > will be sent to the defined destination but the message which is sent to the > mirror. > I´d suggest to reverse the logic of the MirrorQueue so that a copy of the > message is sent to the mirror topic and the original message to the requested > destination (and also the MemoryUsage of this copy is set to null - and only > if it´s a copy!) > A patch is attached to this issue which would cover the changes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-6695) MirrorQueue - send a copy to the mirror topic instead of original message
[ https://issues.apache.org/jira/browse/AMQ-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034558#comment-16034558 ] Gary Tully commented on AMQ-6695: - btw: this copy not needed has been addressed in https://issues.apache.org/jira/browse/AMQ-6281 for composite destinations > MirrorQueue - send a copy to the mirror topic instead of original message > - > > Key: AMQ-6695 > URL: https://issues.apache.org/jira/browse/AMQ-6695 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.x >Reporter: Joachim Glink >Priority: Minor > Attachments: > MirrorQueue___send_the_original_message_to_the_original_destination_and_a_copy_to_the_mirr.patch > > > The MirrorQueue implementation currently sends the original message to the > mirror topic and a copy (if copy flag is set to true) of the message to the > requested destination. > Also, the MemoryUsage on the message which is sent to the requested > destination is set to null but the comment in the line notes that it´s done > so the memory size is used from the queue instead of the topic which isn´t > the case; its vise versa. (At least in my opinion) > Another possible bug in my opinion is, that the MemoryUsage is set to null > even if the copy flag is false; so the same message is sent to the mirror and > destination but the MemoryUsage is set to null. This could lead to a false > calculation of the broker storage. > If there are further plugins configured, than these plugins see the original > message in e.g. the send method. But this message isn´t the message which > will be sent to the defined destination but the message which is sent to the > mirror. > I´d suggest to reverse the logic of the MirrorQueue so that a copy of the > message is sent to the mirror topic and the original message to the requested > destination (and also the MemoryUsage of this copy is set to null - and only > if it´s a copy!) > A patch is attached to this issue which would cover the changes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AMQ-6696) Memory Usage for composite destination increasing when concurrentSend is used
[
https://issues.apache.org/jira/browse/AMQ-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully resolved AMQ-6696.
-
Resolution: Fixed
> Memory Usage for composite destination increasing when concurrentSend is used
> -
>
> Key: AMQ-6696
> URL: https://issues.apache.org/jira/browse/AMQ-6696
> Project: ActiveMQ
> Issue Type: Bug
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> After message has been forwarded to the underlying queues
> MemoryUsageByteCount is increasing for the topic, would have expected it to
> return to zero.{code}
>
>
>
>forwardOnly="false" concurrentSend="true" >
>
>
>
>
>
>
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (AMQ-6696) Memory Usage for composite destination increasing when concurrentSend is used
Gary Tully created AMQ-6696: --- Summary: Memory Usage for composite destination increasing when concurrentSend is used Key: AMQ-6696 URL: https://issues.apache.org/jira/browse/AMQ-6696 Project: ActiveMQ Issue Type: Bug Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.15.0 After message has been forwarded to the underlying queues MemoryUsageByteCount is increasing for the topic, would have expected it to return to zero. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AMQ-6696) Memory Usage for composite destination increasing when concurrentSend is used
[
https://issues.apache.org/jira/browse/AMQ-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully updated AMQ-6696:
Description:
After message has been forwarded to the underlying queues MemoryUsageByteCount
is increasing for the topic, would have expected it to return to zero.{code}
{code}
was:
After message has been forwarded to the underlying queues MemoryUsageByteCount
is increasing for the topic, would have expected it to return to zero.
> Memory Usage for composite destination increasing when concurrentSend is used
> -
>
> Key: AMQ-6696
> URL: https://issues.apache.org/jira/browse/AMQ-6696
> Project: ActiveMQ
> Issue Type: Bug
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> After message has been forwarded to the underlying queues
> MemoryUsageByteCount is increasing for the topic, would have expected it to
> return to zero.{code}
>
>
>
>forwardOnly="false" concurrentSend="true" >
>
>
>
>
>
>
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AMQ-6695) MirrorQueue - send a copy to the mirror topic instead of original message
[ https://issues.apache.org/jira/browse/AMQ-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034386#comment-16034386 ] Gary Tully commented on AMQ-6695: - thanks for the patch. Changes like this can be tricky (as in it will change behaviour for folks) - that code has not changed since 2007. I think there should always be a copy of the message because sending does change the metadata and of course the usage. >From the comments on setCopyMessage it seems the intent is to set the forward >destination on the copy, maybe that flags should just do that and there is >always a copy. Can you add some unit tests to show the problem and that can protect this changes into the future. Also verify the few unit tests that exercise the mirror queue There is a related issue https://issues.apache.org/jira/browse/AMQ-3669 that seems to be in the same area and has a relevant test that can give inspiration. > MirrorQueue - send a copy to the mirror topic instead of original message > - > > Key: AMQ-6695 > URL: https://issues.apache.org/jira/browse/AMQ-6695 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.x >Reporter: Joachim Glink >Priority: Minor > Attachments: > MirrorQueue___send_the_original_message_to_the_original_destination_and_a_copy_to_the_mirr.patch > > > The MirrorQueue implementation currently sends the original message to the > mirror topic and a copy (if copy flag is set to true) of the message to the > requested destination. > Also, the MemoryUsage on the message which is sent to the requested > destination is set to null but the comment in the line notes that it´s done > so the memory size is used from the queue instead of the topic which isn´t > the case; its vise versa. (At least in my opinion) > Another possible bug in my opinion is, that the MemoryUsage is set to null > even if the copy flag is false; so the same message is sent to the mirror and > destination but the MemoryUsage is set to null. This could lead to a false > calculation of the broker storage. > If there are further plugins configured, than these plugins see the original > message in e.g. the send method. But this message isn´t the message which > will be sent to the defined destination but the message which is sent to the > mirror. > I´d suggest to reverse the logic of the MirrorQueue so that a copy of the > message is sent to the mirror topic and the original message to the requested > destination (and also the MemoryUsage of this copy is set to null - and only > if it´s a copy!) > A patch is attached to this issue which would cover the changes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AMQ-6693) SQLException when concurrently moving messages to DLQ
[ https://issues.apache.org/jira/browse/AMQ-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6693. - Resolution: Fixed > SQLException when concurrently moving messages to DLQ > - > > Key: AMQ-6693 > URL: https://issues.apache.org/jira/browse/AMQ-6693 > Project: ActiveMQ > Issue Type: Bug > Components: JDBC, Message Store >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > with the jdbc message store and the absence of a security context moving > messages to the dlq on transaction rollback and exhausted redelivery is > completed using the broker context. > However the long term connection associated with a jdbc context is not thread > safe. It is typically associated with an incoming connection which is. > If multiple consumers poison ack concurrently the broker can step over itself > using this shared context. > The long term connection context should be ignored for the broker. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6693) SQLException when concurrently moving messages to DLQ
Gary Tully created AMQ-6693: --- Summary: SQLException when concurrently moving messages to DLQ Key: AMQ-6693 URL: https://issues.apache.org/jira/browse/AMQ-6693 Project: ActiveMQ Issue Type: Bug Components: JDBC, Message Store Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.15.0 with the jdbc message store and the absence of a security context moving messages to the dlq on transaction rollback and exhausted redelivery is completed using the broker context. However the long term connection associated with a jdbc context is not thread safe. It is typically associated with an incoming connection which is. If multiple consumers poison ack concurrently the broker can step over itself using this shared context. The long term connection context should be ignored for the broker. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AMQ-6691) JMX - allow DLQ flag to be modified
[ https://issues.apache.org/jira/browse/AMQ-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6691. - Resolution: Fixed Assignee: Gary Tully Test that validates persistence of the dlq flag via the destinations element of the broker xml is at: https://git1-us-west.apache.org/repos/asf?p=activemq.git;a=commit;h=18d05ba5e0a96d231a357eb0a20b1f6388cb6c49 > JMX - allow DLQ flag to be modified > --- > > Key: AMQ-6691 > URL: https://issues.apache.org/jira/browse/AMQ-6691 > Project: ActiveMQ > Issue Type: Improvement > Components: JMX >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > AMQ-4483 introduces the destination flag but it is not persisted in the > message store and does not survive a restart. > Adding the flag to the destinations element in the broker xml works around > this however it assumes prior knowledge. > Allowing the flag to be set allows a retrospective change pending the next > restart when a modification to the destinations element can be picked up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6691) JMX - allow DLQ flag to be modified
Gary Tully created AMQ-6691: --- Summary: JMX - allow DLQ flag to be modified Key: AMQ-6691 URL: https://issues.apache.org/jira/browse/AMQ-6691 Project: ActiveMQ Issue Type: Improvement Components: JMX Affects Versions: 5.14.0 Reporter: Gary Tully Fix For: 5.15.0 AMQ-4483 introduces the destination flag but it is not persisted in the message store and does not survive a restart. Adding the flag to the destinations element in the broker xml works around this however it assumes prior knowledge. Allowing the flag to be set allows a retrospective change pending the next restart when a modification to the destinations element can be picked up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AMQ-6690) Protect against JMX move/copy operations onto self
[ https://issues.apache.org/jira/browse/AMQ-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6690. - Resolution: Fixed I considered throwing an exception but I think it is safer to retain the existing behaviour and report via the return code > Protect against JMX move/copy operations onto self > -- > > Key: AMQ-6690 > URL: https://issues.apache.org/jira/browse/AMQ-6690 > Project: ActiveMQ > Issue Type: Improvement > Components: JMX >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > The move and copy jmx operations are intended to move/copy messages to > another destination. However this is not enforce and if the move/copy > destination is the same queue the logs fill with duplicate warnings etc and > the stats can get out of sync if the operation is repeated or concurrent. > Best to cut this off at the pass with a return return code indicating nothing > was moved/copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6690) Protect against JMX move/copy operations onto self
Gary Tully created AMQ-6690: --- Summary: Protect against JMX move/copy operations onto self Key: AMQ-6690 URL: https://issues.apache.org/jira/browse/AMQ-6690 Project: ActiveMQ Issue Type: Improvement Components: JMX Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.15.0 The move and copy jmx operations are intended to move/copy messages to another destination. However this is not enforce and if the move/copy destination is the same queue the logs fill with duplicate warnings etc and the stats can get out of sync if the operation is repeated or concurrent. Best to cut this off at the pass with a return return code indicating nothing was moved/copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AMQ-6688) org.apache.activemq.broker.region.Queue.doMessageSend() future not completing
[
https://issues.apache.org/jira/browse/AMQ-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully resolved AMQ-6688.
-
Resolution: Fixed
> org.apache.activemq.broker.region.Queue.doMessageSend() future not completing
> -
>
> Key: AMQ-6688
> URL: https://issues.apache.org/jira/browse/AMQ-6688
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker, KahaDB, Message Store
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> A case of a send thread blocked on the send future.get()...
> {code}
> ActiveMQ VMTransport: vm://XX" #375 daemon prio=5 os_prio=0
> tid=0x7f9bdc2f0800 nid=0x4e2 waiting on condition [0x7f9bc7e9e000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xd8e65ef8> (a
> org.apache.activemq.store.kahadb.KahaDBStore$StoreQueueTask$InnerFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:853)
> at org.apache.activemq.broker.region.Queue.send(Queue.java:727)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:419)
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:468)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:296)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)
> at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:293)
> at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)
> at
> org.apache.activemq.broker.util.LoggingBrokerPlugin.send(LoggingBrokerPlugin.java:275)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)
> at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)
> at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:571)
> at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:326)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:190)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:271)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It seems that the
> org.apache.activemq.store.kahadb.KahaDBStore.StoreQueueTask#run() which
> should invoke complete on the future has been invoked but complete is not
> triggered. This can happen in the event that the add results in a Throwable!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (AMQ-6687) destination memory usage incorrect after flow control (PFC) sendTimeout and rollback
[ https://issues.apache.org/jira/browse/AMQ-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6687. - Resolution: Fixed > destination memory usage incorrect after flow control (PFC) sendTimeout and > rollback > > > Key: AMQ-6687 > URL: https://issues.apache.org/jira/browse/AMQ-6687 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > When using PFC, to block a producer on a memory limit and using a sendTimeout > to error out after some timeout. If there is a send transaction the send > completes in error even though the transaction has completed once usage > allows. > Leaving a dangling send that shows up in the memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-6687) destination memory usage incorrect after flow control (PFC) sendTimeout and rollback
[ https://issues.apache.org/jira/browse/AMQ-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024737#comment-16024737 ] Gary Tully commented on AMQ-6687: - With transactions or concurrentStoreAndDispatch - memory is accounted for before it reaches the cache which typically means that the cursor cache stops accepting messages early. If we want the cache and PFC we have to let the cache consume >100% of memory. To achieve this we must set the cursorHighWaterMark>100% something like 120% such that it will still allow a message to be cached while is is <100 that will put it over the 100% (full) limit. Ie: we check is full and we are at 99%, we accept another message that pushes us to 101%, without a cursorHighWaterMark > 101 the cursor won't cache and will stick at 99% and producer flow control (PFC) won't kick in. > destination memory usage incorrect after flow control (PFC) sendTimeout and > rollback > > > Key: AMQ-6687 > URL: https://issues.apache.org/jira/browse/AMQ-6687 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > When using PFC, to block a producer on a memory limit and using a sendTimeout > to error out after some timeout. If there is a send transaction the send > completes in error even though the transaction has completed once usage > allows. > Leaving a dangling send that shows up in the memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-4467) Memory usage percent can be exceeded much if PFC is disabled
[ https://issues.apache.org/jira/browse/AMQ-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024609#comment-16024609 ] Gary Tully commented on AMQ-4467: - With transactions or concurrentStoreAndDispatch - memory is accounted for before it reaches the cache which typically means that the cursor cache stops accepting messages early. If we want the cache and PFC we have to let the cache consume >100% of memory. To achieve this we must set the cursorHighWaterMark>100% something like 120% such that it will still allow a message to be cached while is is <100 that will put it over the 100% (full) limit. Ie: we check is full and we are at 99%, we accept another message that pushes us to 101%, without a cursorHighWaterMark > 101 the cursor won't cache and will stick at 99% and producer flow control (PFC) won't kick in. > Memory usage percent can be exceeded much if PFC is disabled > > > Key: AMQ-4467 > URL: https://issues.apache.org/jira/browse/AMQ-4467 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.x >Reporter: SuoNayi >Assignee: Gary Tully > Fix For: 5.15.0 > > Attachments: AMQ-4467.patch > > > If PFC is disabled, when the store cursor checks limits, it checks only the > memory percentage of its own MemoryUsage and compares it to the high water > mark. Otherwise if PFC is enabled, it checks whether the MemoryUsage is > "full" but the "isFull" method also checks its parents. > This issue arrises when you have memory limits set on queues higher than the > overall system limit, as well as if you have multiple queues who's memory > limits combined are higher than the overall system limit. These settings must > be taken into account. > The original form can be found at > http://activemq.2283324.n4.nabble.com/What-can-be-reason-of-460-memory-usage-limit-td4665651.html -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-6687) destination memory usage incorrect after flow control (PFC) sendTimeout and rollback
[ https://issues.apache.org/jira/browse/AMQ-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024568#comment-16024568 ] Gary Tully commented on AMQ-6687: - AMQ-4467 needed a revisit b/c that change prevented pfc from kicking in in this scenario in error. > destination memory usage incorrect after flow control (PFC) sendTimeout and > rollback > > > Key: AMQ-6687 > URL: https://issues.apache.org/jira/browse/AMQ-6687 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > When using PFC, to block a producer on a memory limit and using a sendTimeout > to error out after some timeout. If there is a send transaction the send > completes in error even though the transaction has completed once usage > allows. > Leaving a dangling send that shows up in the memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6687) destination memory usage incorrect after flow control (PFC) sendTimeout and rollback
Gary Tully created AMQ-6687: --- Summary: destination memory usage incorrect after flow control (PFC) sendTimeout and rollback Key: AMQ-6687 URL: https://issues.apache.org/jira/browse/AMQ-6687 Project: ActiveMQ Issue Type: Bug Components: Broker Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.15.0 When using PFC, to block a producer on a memory limit and using a sendTimeout to error out after some timeout. If there is a send transaction the send completes in error even though the transaction has completed once usage allows. Leaving a dangling send that shows up in the memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AMQ-6684) org.json is cat-x - needs to be removed
[ https://issues.apache.org/jira/browse/AMQ-6684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6684. - Resolution: Fixed Assignee: Gary Tully resolved via exclusions in the pom. > org.json is cat-x - needs to be removed > --- > > Key: AMQ-6684 > URL: https://issues.apache.org/jira/browse/AMQ-6684 > Project: ActiveMQ > Issue Type: Bug > Components: activemq-leveldb-store >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Blocker > Fix For: 5.15.0 > > > org.json is cat-x and cannot be included as a compile dependency going > forward. > org.linkedin:org.linkedin.zookeeper-impl:jar:1.4.0:compile > [INFO] | \- org.linkedin:org.linkedin.util-groovy:jar:1.7.1:compile > [INFO] | +- org.slf4j:jul-to-slf4j:jar:1.5.8:compile > [INFO] | +- org.apache.ant:ant:jar:1.8.4:compile > [INFO] | | \- org.apache.ant:ant-launcher:jar:1.8.4:compile > [INFO] | \- org.json:json:jar:20090211:compile > is used in activemq-leveldb-store and activemq-partition > using curator as a replacement may be a viable option for the zk client > dependency. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AMQ-6684) org.json is cat-x - needs to be removed
[ https://issues.apache.org/jira/browse/AMQ-6684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully updated AMQ-6684: Description: org.json is cat-x and cannot be included as a compile dependency going forward. org.linkedin:org.linkedin.zookeeper-impl:jar:1.4.0:compile [INFO] | \- org.linkedin:org.linkedin.util-groovy:jar:1.7.1:compile [INFO] | +- org.slf4j:jul-to-slf4j:jar:1.5.8:compile [INFO] | +- org.apache.ant:ant:jar:1.8.4:compile [INFO] | | \- org.apache.ant:ant-launcher:jar:1.8.4:compile [INFO] | \- org.json:json:jar:20090211:compile is used in activemq-leveldb-store and activemq-partition using curator as a replacement may be a viable option for the zk client dependency. was: org.json is cat-x and cannot be included as a compile dependency going forward. org.linkedin:org.linkedin.zookeeper-impl:jar:1.4.0:provided [INFO] | \- org.linkedin:org.linkedin.util-groovy:jar:1.7.1:provided [INFO] | +- org.slf4j:jul-to-slf4j:jar:1.5.8:provided [INFO] | +- org.apache.ant:ant:jar:1.8.4:provided [INFO] | | \- org.apache.ant:ant-launcher:jar:1.8.4:provided [INFO] | \- org.json:json:jar:20090211:provided is used in activemq-leveldb-store and activemq-partition using curator as a replacement may be a viable option for the zk client dependency. > org.json is cat-x - needs to be removed > --- > > Key: AMQ-6684 > URL: https://issues.apache.org/jira/browse/AMQ-6684 > Project: ActiveMQ > Issue Type: Bug > Components: activemq-leveldb-store >Affects Versions: 5.14.0 >Reporter: Gary Tully >Priority: Blocker > Fix For: 5.15.0 > > > org.json is cat-x and cannot be included as a compile dependency going > forward. > org.linkedin:org.linkedin.zookeeper-impl:jar:1.4.0:compile > [INFO] | \- org.linkedin:org.linkedin.util-groovy:jar:1.7.1:compile > [INFO] | +- org.slf4j:jul-to-slf4j:jar:1.5.8:compile > [INFO] | +- org.apache.ant:ant:jar:1.8.4:compile > [INFO] | | \- org.apache.ant:ant-launcher:jar:1.8.4:compile > [INFO] | \- org.json:json:jar:20090211:compile > is used in activemq-leveldb-store and activemq-partition > using curator as a replacement may be a viable option for the zk client > dependency. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6684) org.json is cat-x - needs to be removed
Gary Tully created AMQ-6684: --- Summary: org.json is cat-x - needs to be removed Key: AMQ-6684 URL: https://issues.apache.org/jira/browse/AMQ-6684 Project: ActiveMQ Issue Type: Bug Components: activemq-leveldb-store Affects Versions: 5.14.0 Reporter: Gary Tully Priority: Blocker Fix For: 5.15.0 org.json is cat-x and cannot be included as a compile dependency going forward. org.linkedin:org.linkedin.zookeeper-impl:jar:1.4.0:provided [INFO] | \- org.linkedin:org.linkedin.util-groovy:jar:1.7.1:provided [INFO] | +- org.slf4j:jul-to-slf4j:jar:1.5.8:provided [INFO] | +- org.apache.ant:ant:jar:1.8.4:provided [INFO] | | \- org.apache.ant:ant-launcher:jar:1.8.4:provided [INFO] | \- org.json:json:jar:20090211:provided is used in activemq-leveldb-store and activemq-partition using curator as a replacement may be a viable option for the zk client dependency. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AMQ-6678) Move/Retry (destructive) Queue JMX operations need synchronization
[ https://issues.apache.org/jira/browse/AMQ-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6678. - Resolution: Fixed fix and test added. > Move/Retry (destructive) Queue JMX operations need synchronization > -- > > Key: AMQ-6678 > URL: https://issues.apache.org/jira/browse/AMQ-6678 > Project: ActiveMQ > Issue Type: Bug > Components: Broker, JMX >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > concurrent calls to move selector matching messages from one queue to another > or any wildcard type destructive JMX operation can break on concurrent > invocation. > The move results in duplicates in the journal (waste of space) and incorrect > return counts due to inflight messages being visible in error. > Two jmx consoles is enough to cause problems. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6678) Move/Retry (destructive) Queue JMX operations need synchronization
Gary Tully created AMQ-6678: --- Summary: Move/Retry (destructive) Queue JMX operations need synchronization Key: AMQ-6678 URL: https://issues.apache.org/jira/browse/AMQ-6678 Project: ActiveMQ Issue Type: Bug Components: Broker, JMX Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.15.0 concurrent calls to move selector matching messages from one queue to another or any wildcard type destructive JMX operation can break on concurrent invocation. The move results in duplicates in the journal (waste of space) and incorrect return counts due to inflight messages being visible in error. Two jmx consoles is enough to cause problems. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (AMQ-6671) ActiveMQ with PostgreSQL: Not deleting persistent messages sent to a topic
[ https://issues.apache.org/jira/browse/AMQ-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002772#comment-16002772 ] Gary Tully edited comment on AMQ-6671 at 5/9/17 2:28 PM: - [~anujkhandelwal90] it may be that the cleanup task needs to run a few times. messages for durable subs that are unreferenced are deleted periodically with the cleanupPeriod which defaults to 5mins. Then there is an iteration per each priority 1-10, messages with default priority 4 may take 20mins to clear by default see: https://github.com/apache/activemq/blob/57795bafcea290c6879bb288822435c480a9212d/activemq-jdbc-store/src/main/java/org/apache/activemq/store/jdbc/adapter/DefaultJDBCAdapter.java#L829 was (Author: gtully): [~anujkhandelwal90] it may be that the cleanup task needs to run a few times. messages for durable subs that are unreferenced are deleted periodically with the cleanupPeriod which defaults to 5mins. Then there is an iteration per each priority 1-10, messages with default priority 4 may take 20mins to clear by default > ActiveMQ with PostgreSQL: Not deleting persistent messages sent to a topic > --- > > Key: AMQ-6671 > URL: https://issues.apache.org/jira/browse/AMQ-6671 > Project: ActiveMQ > Issue Type: Bug > Components: Broker, JDBC >Affects Versions: 5.11.1 >Reporter: Anuj Khandelwal > > Using ActiveMQv5.11.1 > I ran a test case where a durable subscriber was listening from a topic and a > publisher is publishing persistent messages. I observed that the messages > were stored in the DB (ACTIVEMQ_MSGS table) but even after successful > consumption by the consumer, these messages were not deleted from the backend > DB. However I have seen that this doesn't happen in case of queue. Persistent > messages sent to a queue are automatically deleted from the DB once > successfully consumed by the consumer. > More details: > 1. 1 producer 1 durable subscriber. Subscriber is in auto_ack mode. > 2. Producer sent 100 persistent messages to consumer > 3. Consumer successfully consumed them. > 4. DB still has those messages. > 5. Stop producer and consumer both, but messages are still there. > 6. Even after restarting broker, message does not get deleted. > This is a very generic use case which should never happen. It is also > reproducible. > Thanks, > Anuj -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-6671) ActiveMQ with PostgreSQL: Not deleting persistent messages sent to a topic
[ https://issues.apache.org/jira/browse/AMQ-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002772#comment-16002772 ] Gary Tully commented on AMQ-6671: - [~anujkhandelwal90] it may be that the cleanup task needs to run a few times. messages for durable subs that are unreferenced are deleted periodically with the cleanupPeriod which defaults to 5mins. Then there is an iteration per each priority 1-10, messages with default priority 4 may take 20mins to clear by default > ActiveMQ with PostgreSQL: Not deleting persistent messages sent to a topic > --- > > Key: AMQ-6671 > URL: https://issues.apache.org/jira/browse/AMQ-6671 > Project: ActiveMQ > Issue Type: Bug > Components: Broker, JDBC >Affects Versions: 5.11.1 >Reporter: Anuj Khandelwal > > Using ActiveMQv5.11.1 > I ran a test case where a durable subscriber was listening from a topic and a > publisher is publishing persistent messages. I observed that the messages > were stored in the DB (ACTIVEMQ_MSGS table) but even after successful > consumption by the consumer, these messages were not deleted from the backend > DB. However I have seen that this doesn't happen in case of queue. Persistent > messages sent to a queue are automatically deleted from the DB once > successfully consumed by the consumer. > More details: > 1. 1 producer 1 durable subscriber. Subscriber is in auto_ack mode. > 2. Producer sent 100 persistent messages to consumer > 3. Consumer successfully consumed them. > 4. DB still has those messages. > 5. Stop producer and consumer both, but messages are still there. > 6. Even after restarting broker, message does not get deleted. > This is a very generic use case which should never happen. It is also > reproducible. > Thanks, > Anuj -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AMQ-6670) KahaDB - Inconsistent error handling on corrupt journal reads
[
https://issues.apache.org/jira/browse/AMQ-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully resolved AMQ-6670.
-
Resolution: Fixed
> KahaDB - Inconsistent error handling on corrupt journal reads
> -
>
> Key: AMQ-6670
> URL: https://issues.apache.org/jira/browse/AMQ-6670
> Project: ActiveMQ
> Issue Type: Bug
> Components: KahaDB, Message Store
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> When the journal is corrupt, for example if it is overwritten with null or
> zero. The result of reading from corrupt locations is inconsistent.
> Errors occur but not all result in a trip to the IOExceptionHander where the
> option to stop the broker is available.
> ClassCastExceptions and RuntimeExceptions can bubble up to the cursors in
> error.
> {code}ERROR | Failed to page in more queue messages |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[X]
> Task-300
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassCastException:
> org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to
> org.apache.activemq.store.kahadb.data.KahaAddMessageCommand
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:145){code}
> or {code}ERROR | Failed to load message at: 1084:12816246 |
> org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ BrokerService[]
> Task-702
> org.apache.activemq.protobuf.InvalidProtocolBufferException: Protocol message
> contained an invalid tag (zero).
> at
> org.apache.activemq.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:48)
> at
> org.apache.activemq.protobuf.CodedInputStream.readTag(CodedInputStream.java:75)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (AMQ-6670) KahaDB - Inconsistent error handling on corrupt journal reads
Gary Tully created AMQ-6670:
---
Summary: KahaDB - Inconsistent error handling on corrupt journal
reads
Key: AMQ-6670
URL: https://issues.apache.org/jira/browse/AMQ-6670
Project: ActiveMQ
Issue Type: Bug
Components: KahaDB, Message Store
Affects Versions: 5.14.0
Reporter: Gary Tully
Assignee: Gary Tully
Fix For: 5.15.0
When the journal is corrupt, for example if it is overwritten with null or
zero. The result of reading from corrupt locations is inconsistent.
Errors occur but not all result in a trip to the IOExceptionHander where the
option to stop the broker is available.
ClassCastExceptions and RuntimeExceptions can bubble up to the cursors in error.
{code}ERROR | Failed to page in more queue messages |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[X] Task-300
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassCastException:
org.apache.activemq.store.kahadb.data.KahaTraceCommand cannot be cast to
org.apache.activemq.store.kahadb.data.KahaAddMessageCommand
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:145){code}
or {code}ERROR | Failed to load message at: 1084:12816246 |
org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ BrokerService[]
Task-702
org.apache.activemq.protobuf.InvalidProtocolBufferException: Protocol message
contained an invalid tag (zero).
at
org.apache.activemq.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:48)
at
org.apache.activemq.protobuf.CodedInputStream.readTag(CodedInputStream.java:75)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AMQ-6068) RAR - cannot reset clientId on pooled managed connection
[
https://issues.apache.org/jira/browse/AMQ-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully updated AMQ-6068:
Fix Version/s: 5.15.0
> RAR - cannot reset clientId on pooled managed connection
>
>
> Key: AMQ-6068
> URL: https://issues.apache.org/jira/browse/AMQ-6068
> Project: ActiveMQ
> Issue Type: Bug
> Components: RAR
>Affects Versions: 5.12.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.14.0, 5.15.0
>
>
> A managed connection returned to the pool has cleanup called, but cleanup is
> not releasing the underlying activemq connection info and clientid. On the
> second attempt to reuse the connection, setting the id fails due to the pre
> existing state in error.
> {code}[Server:eai01] 17:04:58,073 WARN
> [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool]
> (jmsListener-734) IJ000613: Throwable while trying to match managed
> connection, destroying connection:
> org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@3e11f08f[state=NORMAL
> managed
> connection=[org.apache.activemq.ra.ActiveMQManagedConnection@34bc339a,ActiveMQConnection
>
> {id=ID:macbookpro-2.local-54186-1448868251571-1463:1,clientId=xxx,started=false}]
> connection handles=0 lastUse=1448874205871 trackByTx=false
> pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@2f2f39fd pool
> internal
> context=SemaphoreArrayListManagedConnectionPool@51eff7c[pool=ActiveMQConnectionFactory]
>
> xaResource=XAResourceWrapperImpl@623206[xaResource=[org.apache.activemq.ra.ActiveMQManagedConnection$1@64df868a,TransactionContext{transactionId=null,connection=ActiveMQConnection
>
> {id=ID:macbookpro-2.local-54186-1448868251571-1463:1,clientId=xxx,started=false}}]
> pad=false overrideRmValue=null productName=ActiveMQ productVersion=5.12.1
> jndiName=java:/ra/activeMQ/ActiveMQConnectionFactory] txSync=null]:
> javax.resource.ResourceException: javax.jms.IllegalStateException: Setting
> clientID on a used Connection is not allowed
> [Server:eai01]at
> org.apache.activemq.ra.ActiveMQManagedConnectionFactory.matchManagedConnections(ActiveMQManagedConnectionFactory.java:217)
> [activemq-ra-5.12.1.jar:5.12.1]
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:314)
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:453)
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:425)
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:354)
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:510)
> [Server:eai01]at
> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94)
> [activemq-ra-5.12.1.jar:5.12.1]
> [Server:eai01]at
> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:78)
> [activemq-ra-5.12.1.jar:5.12.1]{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AMQ-6068) RAR - cannot reset clientId on pooled managed connection
[
https://issues.apache.org/jira/browse/AMQ-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996446#comment-15996446
]
Gary Tully commented on AMQ-6068:
-
thanks for the feedback. It may make sense to have this full cleanup
conditional on a user specifying a client id.
However, the old cleanup would remove any consumer info, so the advisory load
would be the same;
Usually only demand information, ie: consumer add/remove is propagated across a
network.
There may be something else in the mix that is leading to the excessive memory
usage.
> RAR - cannot reset clientId on pooled managed connection
>
>
> Key: AMQ-6068
> URL: https://issues.apache.org/jira/browse/AMQ-6068
> Project: ActiveMQ
> Issue Type: Bug
> Components: RAR
>Affects Versions: 5.12.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.14.0
>
>
> A managed connection returned to the pool has cleanup called, but cleanup is
> not releasing the underlying activemq connection info and clientid. On the
> second attempt to reuse the connection, setting the id fails due to the pre
> existing state in error.
> {code}[Server:eai01] 17:04:58,073 WARN
> [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool]
> (jmsListener-734) IJ000613: Throwable while trying to match managed
> connection, destroying connection:
> org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@3e11f08f[state=NORMAL
> managed
> connection=[org.apache.activemq.ra.ActiveMQManagedConnection@34bc339a,ActiveMQConnection
>
> {id=ID:macbookpro-2.local-54186-1448868251571-1463:1,clientId=xxx,started=false}]
> connection handles=0 lastUse=1448874205871 trackByTx=false
> pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@2f2f39fd pool
> internal
> context=SemaphoreArrayListManagedConnectionPool@51eff7c[pool=ActiveMQConnectionFactory]
>
> xaResource=XAResourceWrapperImpl@623206[xaResource=[org.apache.activemq.ra.ActiveMQManagedConnection$1@64df868a,TransactionContext{transactionId=null,connection=ActiveMQConnection
>
> {id=ID:macbookpro-2.local-54186-1448868251571-1463:1,clientId=xxx,started=false}}]
> pad=false overrideRmValue=null productName=ActiveMQ productVersion=5.12.1
> jndiName=java:/ra/activeMQ/ActiveMQConnectionFactory] txSync=null]:
> javax.resource.ResourceException: javax.jms.IllegalStateException: Setting
> clientID on a used Connection is not allowed
> [Server:eai01]at
> org.apache.activemq.ra.ActiveMQManagedConnectionFactory.matchManagedConnections(ActiveMQManagedConnectionFactory.java:217)
> [activemq-ra-5.12.1.jar:5.12.1]
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:314)
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:453)
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:425)
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:354)
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
> [Server:eai01]at
> org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:510)
> [Server:eai01]at
> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94)
> [activemq-ra-5.12.1.jar:5.12.1]
> [Server:eai01]at
> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:78)
> [activemq-ra-5.12.1.jar:5.12.1]{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AMQ-6667) Many instances of "duplicate message ... from cursor" redirecting to dlq
[
https://issues.apache.org/jira/browse/AMQ-6667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994642#comment-15994642
]
Gary Tully commented on AMQ-6667:
-
increasing the audit capacity; policyEntry maxProducersToAudit=200 ensures the
cursor audit tracks up to 200 producers and traps the duplicate from the store.
In addition, disabling concurrentStoreAndDispatch avoids the issue.
The root cause is enabling the cursor cache when there are pending sends
leaving a possible scenario where a sync send is lost behind sufficient async
sends that fill the cache. Reverting to the store in this case results in
duplicates.
> Many instances of "duplicate message ... from cursor" redirecting to dlq
>
>
> Key: AMQ-6667
> URL: https://issues.apache.org/jira/browse/AMQ-6667
> Project: ActiveMQ
> Issue Type: Bug
> Components: KahaDB
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> In a high throughput scenario on a single destination there can be many
> warnings of this kind:
> {code}
> [2017-03-21 09:51:37,548] WARN ActiveMQ BrokerService[localhost] Task-5 Queue
> - queue://TEST, subscriptions=10, memory=102%, size=399,
> pending=0, duplicate message ActiveMQTextMessage {commandId = 72,
> responseRequired = true,
> messageId = ID:Mac.fritz.box-64479-1490086286066-183:1:1:1:68,
> originalDestination = null, originalTransactionId = null,
> producerId = ID:Mac.fritz.box-64479-1490086286066-183:1:1:1, destination =
> queue://TEST, transactionId = null, expiration = 0,
> timestamp = 1490086297427, arrival = 0, brokerInTime = 1490086297427,
> brokerOutTime = 1490086297427, correlationId = null, replyTo = null,
> persistent = true,
> type = null, priority = 4, groupID = null, groupSequence = 0,
> targetConsumerId = null, compressed = false, userID = null,
> content = org.apache.activemq.util.ByteSequence@8672edc, marshalledProperties
> = null, dataStructure = null, redeliveryCounter = 0, size = 50216, properties
> = null,
> readOnlyProperties = false, readOnlyBody = false, droppable = false,
> jmsXGroupFirstForConsumer = false, text = ...}
> from cursor, is cursor audit disabled or too constrained? Redirecting to dlq
> {code}
> The broker loads messages twice into the cursor.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (AMQ-6667) Many instances of "duplicate message ... from cursor" redirecting to dlq
Gary Tully created AMQ-6667:
---
Summary: Many instances of "duplicate message ... from cursor"
redirecting to dlq
Key: AMQ-6667
URL: https://issues.apache.org/jira/browse/AMQ-6667
Project: ActiveMQ
Issue Type: Bug
Components: KahaDB
Affects Versions: 5.14.0
Reporter: Gary Tully
Assignee: Gary Tully
Fix For: 5.15.0
In a high throughput scenario on a single destination there can be many
warnings of this kind:
{code}
[2017-03-21 09:51:37,548] WARN ActiveMQ BrokerService[localhost] Task-5 Queue -
queue://TEST, subscriptions=10, memory=102%, size=399,
pending=0, duplicate message ActiveMQTextMessage {commandId = 72,
responseRequired = true,
messageId = ID:Mac.fritz.box-64479-1490086286066-183:1:1:1:68,
originalDestination = null, originalTransactionId = null,
producerId = ID:Mac.fritz.box-64479-1490086286066-183:1:1:1, destination =
queue://TEST, transactionId = null, expiration = 0,
timestamp = 1490086297427, arrival = 0, brokerInTime = 1490086297427,
brokerOutTime = 1490086297427, correlationId = null, replyTo = null, persistent
= true,
type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId
= null, compressed = false, userID = null,
content = org.apache.activemq.util.ByteSequence@8672edc, marshalledProperties =
null, dataStructure = null, redeliveryCounter = 0, size = 50216, properties =
null,
readOnlyProperties = false, readOnlyBody = false, droppable = false,
jmsXGroupFirstForConsumer = false, text = ...}
from cursor, is cursor audit disabled or too constrained? Redirecting to dlq
{code}
The broker loads messages twice into the cursor.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (AMQ-6665) certificate-based authentication on network bridge fails for nio+ssl protocol
[
https://issues.apache.org/jira/browse/AMQ-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully resolved AMQ-6665.
-
Resolution: Fixed
issue was nio+ssl transport not extending SslTtransport but since AMQ-6339 the
relevant getPeerCerts is pushed back to tcpTransport. Now checking for that in
the network bridge
> certificate-based authentication on network bridge fails for nio+ssl protocol
> -
>
> Key: AMQ-6665
> URL: https://issues.apache.org/jira/browse/AMQ-6665
> Project: ActiveMQ
> Issue Type: Bug
> Components: networkbridge
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> client certificate authentication works in the following scenario:
> {code}
> Broker A
> ...
> configuration="activemq"
> sslConfiguration="CertLogin" />
> ...
> uri="ssl://0.0.0.0:61618?needClientAuth=truewantClientAuth=true"/>
> {code}
> Broker B
> {code}
> ...
> networkConnector uri="static://(ssl://localhost:61618)"
> name="myNetworkConnector" duplex="true" consumerTTL="2" messageTTL="100"
> dynamicOnly="false">
> ...
> {code}
> But if you change the transport to nio+ssl, the network connector fails with
> the following message:
> {code}
> INFO | Stopping vm://localhosta#0 because Failed with SecurityException: User
> name [null] or password is invalid.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (AMQ-6665) certificate-based authentication on network bridge fails for nio+ssl protocol
Gary Tully created AMQ-6665:
---
Summary: certificate-based authentication on network bridge fails
for nio+ssl protocol
Key: AMQ-6665
URL: https://issues.apache.org/jira/browse/AMQ-6665
Project: ActiveMQ
Issue Type: Bug
Components: networkbridge
Affects Versions: 5.14.0
Reporter: Gary Tully
Assignee: Gary Tully
Fix For: 5.15.0
client certificate authentication works in the following scenario:
{code}
Broker A
...
...
{code}
Broker B
{code}
...
networkConnector uri="static://(ssl://localhost:61618)"
name="myNetworkConnector" duplex="true" consumerTTL="2" messageTTL="100"
dynamicOnly="false">
...
{code}
But if you change the transport to nio+ssl, the network connector fails with
the following message:
{code}
INFO | Stopping vm://localhosta#0 because Failed with SecurityException: User
name [null] or password is invalid.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AMQ-6654) Durable subscriber pendingQueueSize not flushed with KahaDB after force-kill of broker
[ https://issues.apache.org/jira/browse/AMQ-6654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963599#comment-15963599 ] Gary Tully commented on AMQ-6654: - this may be resolved via AMQ-6652 > Durable subscriber pendingQueueSize not flushed with KahaDB after force-kill > of broker > -- > > Key: AMQ-6654 > URL: https://issues.apache.org/jira/browse/AMQ-6654 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.14.4 > Environment: Reproducible in Linux and Windows >Reporter: Justin Reock > Attachments: localhost___Durable_Topic_Subscribers.jpg > > > This is related to AMQ-5960, marked as fixed, but the issue persists. > It is very easy to reproduce. > 1) Start up ActiveMQ > 2) Produce load into a topic > 3) Connect a few durable subscribers to the topic > 4) Force-terminate the running broker instance > 5) Restart the broker (without load) > 6) Allow durable subscribers to reconnect, and attempt to drain the durable > subscription queues > In almost every case, you will see the pending queue sizes of the durable > subscribers with lingering "messages." I say that in quotes, because I have > been able to prove that all the messages are in fact delivered to the > clients, there's no message loss, but, KahaDB still thinks that there are > messages waiting to be dispatched. > This causes KahaDB to be unable to clean up extents and ultimately will cause > the KahaDB store to grow out of control, hence the "Major" severity despite > no actual message loss occurring. > The only way to recover from the situation is to delete and recreate the > subscriber, which does allow KahaDB to clean itself up. > I have tried several things, including disabling the new ackCompation > functionality, significantly reducing the time between checkpoints, reducing > the size of the index cache to force more frequent flushes to disk, but none > of those completely eliminate the problem. > This does not happen with LevelDB, but, of course LevelDB has been > deprecated, so it's not a good solution to switch to that. Does not happen > with JDBC either, but, JDBC is as we know significantly slower than KahaDB, > so ideally we'd see this fixed in KahaDB. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6652) KahaDB checkpoint index page flush skips metadata page
Gary Tully created AMQ-6652:
---
Summary: KahaDB checkpoint index page flush skips metadata page
Key: AMQ-6652
URL: https://issues.apache.org/jira/browse/AMQ-6652
Project: ActiveMQ
Issue Type: Bug
Components: KahaDB
Affects Versions: 5.14.0
Reporter: Gary Tully
Assignee: Gary Tully
Fix For: 5.15.0
Disk image of the metadata is one checkpoint behind. This can surface on
restart after a kill -9 with the ackMessageFileMap referencing a deleted file
in error.
{code} Exception on start: java.io.IOException: Detected missing journal
files.{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (AMQ-6646) default blob strategy error reporting should include url
[ https://issues.apache.org/jira/browse/AMQ-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6646. - Resolution: Fixed > default blob strategy error reporting should include url > > > Key: AMQ-6646 > URL: https://issues.apache.org/jira/browse/AMQ-6646 > Project: ActiveMQ > Issue Type: Bug > Components: JMS client >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Minor > Fix For: 5.15.0 > > > Default blob upload strategy can do a little better at reporting errors by > including the url such that it is clear what url in in the mix. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6646) default blob strategy error reporting should include url
Gary Tully created AMQ-6646: --- Summary: default blob strategy error reporting should include url Key: AMQ-6646 URL: https://issues.apache.org/jira/browse/AMQ-6646 Project: ActiveMQ Issue Type: Bug Components: JMS client Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Priority: Minor Fix For: 5.15.0 Default blob upload strategy can do a little better at reporting errors by including the url such that it is clear what url in in the mix. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-5558) Make some activemq jar executable and able to send/receive messages
[
https://issues.apache.org/jira/browse/AMQ-5558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955269#comment-15955269
]
Gary Tully commented on AMQ-5558:
-
refinement
https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=commit;h=09acc504f18cdecbdf9f947ef3d07deafa2ec0b2
> Make some activemq jar executable and able to send/receive messages
> ---
>
> Key: AMQ-5558
> URL: https://issues.apache.org/jira/browse/AMQ-5558
> Project: ActiveMQ
> Issue Type: New Feature
>Affects Versions: 5.11.0
>Reporter: Dejan Bosanac
>Assignee: Dejan Bosanac
> Fix For: 5.12.0
>
>
> It would be nice to have basic verification/example tool builded directly in
> activemq-client.jar, so that folks can do something like
> {code}java -jar lib/activemq-client-xxx.jar producer
> java -jar lib/activemq-client-xxx.jar consumer{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (AMQ-6645) activemq:browse command does not display Header - OriginalDestination
[ https://issues.apache.org/jira/browse/AMQ-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6645. - Resolution: Fixed https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=commit;h=4a3d117d9693480c4a6564dc2c349317d16a14b3 > activemq:browse command does not display Header - OriginalDestination > - > > Key: AMQ-6645 > URL: https://issues.apache.org/jira/browse/AMQ-6645 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Minor > Fix For: 5.15.0 > > > The jmx view correctly exposes the originaDestination attribute but the > browse command does not map this property in error. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6645) activemq:browse command does not display Header - OriginalDestination
Gary Tully created AMQ-6645: --- Summary: activemq:browse command does not display Header - OriginalDestination Key: AMQ-6645 URL: https://issues.apache.org/jira/browse/AMQ-6645 Project: ActiveMQ Issue Type: Bug Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Priority: Minor Fix For: 5.15.0 The jmx view correctly exposes the originaDestination attribute but the browse command does not map this property in error. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-6494) Exceptions and timeouts during shutdown of connectors
[ https://issues.apache.org/jira/browse/AMQ-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955123#comment-15955123 ] Gary Tully commented on AMQ-6494: - reverted https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=commit;h=0f7561e85ab18912ce65120dde465aef5acda12a and added a fix to the vm transport such that it won't ignore exceptions. I think that is the root cause of the original test delay on stop via the shutdown hook. https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=commit;h=729766e4925ec05ec05410887df146dd27adbbf4 > Exceptions and timeouts during shutdown of connectors > - > > Key: AMQ-6494 > URL: https://issues.apache.org/jira/browse/AMQ-6494 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.0 >Reporter: Hadrian Zbarcea >Assignee: Hadrian Zbarcea > Fix For: 5.15.0 > > > During shutdown there are a number of BrokerStoppedException(s) thrown > unnecessarily and unnecessary timeouts. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AMQ-6644) Incorrect logging from KahaDB cleanup task when enableAckCompaction=true
[
https://issues.apache.org/jira/browse/AMQ-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully resolved AMQ-6644.
-
Resolution: Fixed
> Incorrect logging from KahaDB cleanup task when enableAckCompaction=true
>
>
> Key: AMQ-6644
> URL: https://issues.apache.org/jira/browse/AMQ-6644
> Project: ActiveMQ
> Issue Type: Bug
> Components: KahaDB
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
>Priority: Minor
> Fix For: 5.15.0
>
>
> When KahaDB is configured for enableAckCompaction=true, it moves acks into a
> new journal file. Such journal file will only contains the compacted acks, it
> won't be used to hold messages.
> If the actual journal (to which new messages are written to) has a lower
> number than the journal files that were created during ack compaction, the
> periodic cleanup task will not delete any journals that are higher than the
> actual journal file. So multiple journal files may remain active on disk
> although there is no single unconsumed message on the broker.
> This in itself is okay, however when trace logging for the cleanup task is
> enabled, it reports differently, namely that it is going to delete these
> journals, where in fact it is not deleting them.
> E.g. lets take the following example.
> The KahaDB folder on disk consists of
> {code}
> [kahadb]$ ls -alh
> total 54M
> drwxr-xr-x. 2 fuse fuse 128K Feb 1 15:50 .
> drwxr-xr-x. 13 fuse fuse 4.0K Nov 4 13:14 ..
> -rw-r--r--. 1 fuse fuse 32M Feb 1 16:26 db-65.log
> -rw-r--r--. 1 fuse fuse 4.6M Feb 1 15:24 db-66.log
> -rw-r--r--. 1 fuse fuse 4.5M Feb 1 15:29 db-67.log
> -rw-r--r--. 1 fuse fuse 4.6M Feb 1 15:34 db-68.log
> -rw-r--r--. 1 fuse fuse 4.5M Feb 1 15:39 db-69.log
> -rw-r--r--. 1 fuse fuse 2.5M Feb 1 16:26 db.data
> -rw-r--r--. 1 fuse fuse 32M Feb 1 14:51 db-log.template
> -rw-r--r--. 1 fuse fuse 1002K Feb 1 16:26 db.redo
> -rw-r--r--. 1 fuse fuse 8 Feb 1 14:51 lock
> {code}
> and the logging says:
> {code}
> Last update: 65:26636520, full gc candidates set: [65, 66, 67, 68, 69]
> gc candidates after producerSequenceIdTrackerLocation:65, [66, 67, 68, 69]
> gc candidates after ackMessageFileMapLocation:65, [66, 67, 68, 69]
> ...
> gc candidates: [66, 67, 68, 69]
> ackMessageFileMap: {65=[65]}
> Cleanup removing the data files: [66, 67, 68, 69]
> {code}
> In this example the actual journal file to which msgs are written is 65. The
> journal files 66-69 were created during ack compaction and have a higher
> number than 65.
> So KahaDB won't delete the journals 66-69 until the actual journal file is
> moved to journal 70, despite that there are no unconsumed messages on the
> broker.
> However the last log line suggests that it will remove the journals 66-69,
> but they will not get removed due to rule above.
> We should align the logging output with the logic used to determine which
> journals to delete.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (AMQ-6644) Incorrect logging from KahaDB cleanup task when enableAckCompaction=true
Gary Tully created AMQ-6644:
---
Summary: Incorrect logging from KahaDB cleanup task when
enableAckCompaction=true
Key: AMQ-6644
URL: https://issues.apache.org/jira/browse/AMQ-6644
Project: ActiveMQ
Issue Type: Bug
Components: KahaDB
Affects Versions: 5.14.0
Reporter: Gary Tully
Assignee: Gary Tully
Priority: Minor
Fix For: 5.15.0
When KahaDB is configured for enableAckCompaction=true, it moves acks into a
new journal file. Such journal file will only contains the compacted acks, it
won't be used to hold messages.
If the actual journal (to which new messages are written to) has a lower number
than the journal files that were created during ack compaction, the periodic
cleanup task will not delete any journals that are higher than the actual
journal file. So multiple journal files may remain active on disk although
there is no single unconsumed message on the broker.
This in itself is okay, however when trace logging for the cleanup task is
enabled, it reports differently, namely that it is going to delete these
journals, where in fact it is not deleting them.
E.g. lets take the following example.
The KahaDB folder on disk consists of
{code}
[kahadb]$ ls -alh
total 54M
drwxr-xr-x. 2 fuse fuse 128K Feb 1 15:50 .
drwxr-xr-x. 13 fuse fuse 4.0K Nov 4 13:14 ..
-rw-r--r--. 1 fuse fuse 32M Feb 1 16:26 db-65.log
-rw-r--r--. 1 fuse fuse 4.6M Feb 1 15:24 db-66.log
-rw-r--r--. 1 fuse fuse 4.5M Feb 1 15:29 db-67.log
-rw-r--r--. 1 fuse fuse 4.6M Feb 1 15:34 db-68.log
-rw-r--r--. 1 fuse fuse 4.5M Feb 1 15:39 db-69.log
-rw-r--r--. 1 fuse fuse 2.5M Feb 1 16:26 db.data
-rw-r--r--. 1 fuse fuse 32M Feb 1 14:51 db-log.template
-rw-r--r--. 1 fuse fuse 1002K Feb 1 16:26 db.redo
-rw-r--r--. 1 fuse fuse 8 Feb 1 14:51 lock
{code}
and the logging says:
{code}
Last update: 65:26636520, full gc candidates set: [65, 66, 67, 68, 69]
gc candidates after producerSequenceIdTrackerLocation:65, [66, 67, 68, 69]
gc candidates after ackMessageFileMapLocation:65, [66, 67, 68, 69]
...
gc candidates: [66, 67, 68, 69]
ackMessageFileMap: {65=[65]}
Cleanup removing the data files: [66, 67, 68, 69]
{code}
In this example the actual journal file to which msgs are written is 65. The
journal files 66-69 were created during ack compaction and have a higher number
than 65.
So KahaDB won't delete the journals 66-69 until the actual journal file is
moved to journal 70, despite that there are no unconsumed messages on the
broker.
However the last log line suggests that it will remove the journals 66-69, but
they will not get removed due to rule above.
We should align the logging output with the logic used to determine which
journals to delete.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (AMQ-6643) Shared virtual topic wildcard consumers receive spurious and duplicate messages
[ https://issues.apache.org/jira/browse/AMQ-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6643. - Resolution: Fixed > Shared virtual topic wildcard consumers receive spurious and duplicate > messages > --- > > Key: AMQ-6643 > URL: https://issues.apache.org/jira/browse/AMQ-6643 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > Virtual topic wildcard subscriber queues are physical queues, however the > matching of destinations to new subscriptions matches based on the wildcard > destination in error. Causing subs to get subscribed in error and causing > duplicate subscriptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-6643) Shared virtual topic wildcard consumers receive spurious and duplicate messages
[ https://issues.apache.org/jira/browse/AMQ-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954873#comment-15954873 ] Gary Tully commented on AMQ-6643: - The restrictions introduced in https://issues.apache.org/jira/browse/AMQ-5594 need to be more selective based on a more exact and consistent destination comparator. > Shared virtual topic wildcard consumers receive spurious and duplicate > messages > --- > > Key: AMQ-6643 > URL: https://issues.apache.org/jira/browse/AMQ-6643 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > Virtual topic wildcard subscriber queues are physical queues, however the > matching of destinations to new subscriptions matches based on the wildcard > destination in error. Causing subs to get subscribed in error and causing > duplicate subscriptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6643) Shared virtual topic wildcard consumers receive spurious and duplicate messages
Gary Tully created AMQ-6643: --- Summary: Shared virtual topic wildcard consumers receive spurious and duplicate messages Key: AMQ-6643 URL: https://issues.apache.org/jira/browse/AMQ-6643 Project: ActiveMQ Issue Type: Bug Components: Broker Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.15.0 Virtual topic wildcard subscriber queues are physical queues, however the matching of destinations to new subscriptions matches based on the wildcard destination in error. Causing subs to get subscribed in error and causing duplicate subscriptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-6640) network of brokers, duplex connector, network consumers not registered on destination - hung bridge
[
https://issues.apache.org/jira/browse/AMQ-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1594#comment-1594
]
Gary Tully commented on AMQ-6640:
-
network connector options suppressDuplicate[Queue|Topic]Subscriptions (which is
used in ring topologies (not the norm)) requires the default true value for
attribute dispatchAsync.
> network of brokers, duplex connector, network consumers not registered on
> destination - hung bridge
> ---
>
> Key: AMQ-6640
> URL: https://issues.apache.org/jira/browse/AMQ-6640
> Project: ActiveMQ
> Issue Type: Bug
> Components: networkbridge
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> Occasional hang in a network bridge under heavy load with large pending
> backlog across multiple destinations.
> Hung network bridge threads blocked through crosstalk between consumer demand
> and message forwarding traffic leading to blocked writes.
> Stack traces of the form, Broker1
> {code}
> ~~~
> "ActiveMQ Transport: tcp:///:51460@61616" daemon prio=10
> tid=0x7efbbc011000 nid=0x42be waiting on condition [0x7efbb1a58000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xc61a2118> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
> at
> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:48)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:914)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1187)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:772)
> - locked <0xc25ed058> (a java.net.URI)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:623)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:225)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:724)
> ~~~
> ~~~
> Dump 1 : "ActiveMQ VMTransport: vm://broker1#8-2" daemon prio=10
> tid=0x7efbbc11b800 nid=0x42c4 runnable [0x7efbb1554000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> at
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:334)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:316)
> at
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
> at
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:994)
> at
>
[jira] [Resolved] (AMQ-6640) network of brokers, duplex connector, network consumers not registered on destination - hung bridge
[
https://issues.apache.org/jira/browse/AMQ-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully resolved AMQ-6640.
-
Resolution: Fixed
network connector internal vm transports are a all sync with the exception of
the duplex end forwarding transport such that the vm dispatch thread can
compete with inbound responses for acks without blocking
> network of brokers, duplex connector, network consumers not registered on
> destination - hung bridge
> ---
>
> Key: AMQ-6640
> URL: https://issues.apache.org/jira/browse/AMQ-6640
> Project: ActiveMQ
> Issue Type: Bug
> Components: networkbridge
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> Occasional hang in a network bridge under heavy load with large pending
> backlog across multiple destinations.
> Hung network bridge threads blocked through crosstalk between consumer demand
> and message forwarding traffic leading to blocked writes.
> Stack traces of the form, Broker1
> {code}
> ~~~
> "ActiveMQ Transport: tcp:///:51460@61616" daemon prio=10
> tid=0x7efbbc011000 nid=0x42be waiting on condition [0x7efbb1a58000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xc61a2118> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
> at
> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:48)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:914)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1187)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:772)
> - locked <0xc25ed058> (a java.net.URI)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:623)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:225)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:724)
> ~~~
> ~~~
> Dump 1 : "ActiveMQ VMTransport: vm://broker1#8-2" daemon prio=10
> tid=0x7efbbc11b800 nid=0x42c4 runnable [0x7efbb1554000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> at
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:334)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:316)
> at
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
> at
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:994)
> at
>
[jira] [Reopened] (AMQ-6640) network of brokers, duplex connector, network consumers not registered on destination - hung bridge
[
https://issues.apache.org/jira/browse/AMQ-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully reopened AMQ-6640:
-
it is still not exactly right - need to rationalise the use of sync and async
vm transports on either end of the duplex pipe, there is a need for some async
processing because there is contention.
> network of brokers, duplex connector, network consumers not registered on
> destination - hung bridge
> ---
>
> Key: AMQ-6640
> URL: https://issues.apache.org/jira/browse/AMQ-6640
> Project: ActiveMQ
> Issue Type: Bug
> Components: networkbridge
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> Occasional hang in a network bridge under heavy load with large pending
> backlog across multiple destinations.
> Hung network bridge threads blocked through crosstalk between consumer demand
> and message forwarding traffic leading to blocked writes.
> Stack traces of the form, Broker1
> {code}
> ~~~
> "ActiveMQ Transport: tcp:///:51460@61616" daemon prio=10
> tid=0x7efbbc011000 nid=0x42be waiting on condition [0x7efbb1a58000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xc61a2118> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
> at
> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:48)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:914)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1187)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:772)
> - locked <0xc25ed058> (a java.net.URI)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:623)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:225)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:724)
> ~~~
> ~~~
> Dump 1 : "ActiveMQ VMTransport: vm://broker1#8-2" daemon prio=10
> tid=0x7efbbc11b800 nid=0x42c4 runnable [0x7efbb1554000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> at
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:334)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:316)
> at
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
> at
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:994)
> at
>
[jira] [Resolved] (AMQ-6640) network of brokers, duplex connector, network consumers not registered on destination - hung bridge
[
https://issues.apache.org/jira/browse/AMQ-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully resolved AMQ-6640.
-
Resolution: Fixed
> network of brokers, duplex connector, network consumers not registered on
> destination - hung bridge
> ---
>
> Key: AMQ-6640
> URL: https://issues.apache.org/jira/browse/AMQ-6640
> Project: ActiveMQ
> Issue Type: Bug
> Components: networkbridge
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> Occasional hang in a network bridge under heavy load with large pending
> backlog across multiple destinations.
> Hung network bridge threads blocked through crosstalk between consumer demand
> and message forwarding traffic leading to blocked writes.
> Stack traces of the form, Broker1
> {code}
> ~~~
> "ActiveMQ Transport: tcp:///:51460@61616" daemon prio=10
> tid=0x7efbbc011000 nid=0x42be waiting on condition [0x7efbb1a58000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xc61a2118> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
> at
> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:48)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:914)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1187)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:772)
> - locked <0xc25ed058> (a java.net.URI)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:623)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:225)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:724)
> ~~~
> ~~~
> Dump 1 : "ActiveMQ VMTransport: vm://broker1#8-2" daemon prio=10
> tid=0x7efbbc11b800 nid=0x42c4 runnable [0x7efbb1554000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> at
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:334)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:316)
> at
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
> at
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:994)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:207)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at
>
[jira] [Commented] (AMQ-6640) network of brokers, duplex connector, network consumers not registered on destination - hung bridge
[
https://issues.apache.org/jira/browse/AMQ-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945214#comment-15945214
]
Gary Tully commented on AMQ-6640:
-
AMQ-4328 fix needed to be confined to the async side, the duplex responder side.
> network of brokers, duplex connector, network consumers not registered on
> destination - hung bridge
> ---
>
> Key: AMQ-6640
> URL: https://issues.apache.org/jira/browse/AMQ-6640
> Project: ActiveMQ
> Issue Type: Bug
> Components: networkbridge
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0
>
>
> Occasional hang in a network bridge under heavy load with large pending
> backlog across multiple destinations.
> Hung network bridge threads blocked through crosstalk between consumer demand
> and message forwarding traffic leading to blocked writes.
> Stack traces of the form, Broker1
> {code}
> ~~~
> "ActiveMQ Transport: tcp:///:51460@61616" daemon prio=10
> tid=0x7efbbc011000 nid=0x42be waiting on condition [0x7efbb1a58000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xc61a2118> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
> at
> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:48)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:914)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1187)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:772)
> - locked <0xc25ed058> (a java.net.URI)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:623)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:225)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:724)
> ~~~
> ~~~
> Dump 1 : "ActiveMQ VMTransport: vm://broker1#8-2" daemon prio=10
> tid=0x7efbbc11b800 nid=0x42c4 runnable [0x7efbb1554000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> at
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:334)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:316)
> at
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
> at
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:116)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:994)
> at
> org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:207)
> at
>
[jira] [Created] (AMQ-6640) network of brokers, duplex connector, network consumers not registered on destination - hung bridge
Gary Tully created AMQ-6640:
---
Summary: network of brokers, duplex connector, network consumers
not registered on destination - hung bridge
Key: AMQ-6640
URL: https://issues.apache.org/jira/browse/AMQ-6640
Project: ActiveMQ
Issue Type: Bug
Components: networkbridge
Affects Versions: 5.14.0
Reporter: Gary Tully
Assignee: Gary Tully
Fix For: 5.15.0
Occasional hang in a network bridge under heavy load with large pending backlog
across multiple destinations.
Hung network bridge threads blocked through crosstalk between consumer demand
and message forwarding traffic leading to blocked writes.
Stack traces of the form, Broker1
{code}
~~~
"ActiveMQ Transport: tcp:///:51460@61616" daemon prio=10
tid=0x7efbbc011000 nid=0x42be waiting on condition [0x7efbb1a58000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xc61a2118> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
at
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:48)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
at
org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:914)
at
org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1187)
at
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:772)
- locked <0xc25ed058> (a java.net.URI)
at
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:623)
at
org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:225)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:724)
~~~
~~~
Dump 1 : "ActiveMQ VMTransport: vm://broker1#8-2" daemon prio=10
tid=0x7efbbc11b800 nid=0x42c4 runnable [0x7efbb1554000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:334)
at
org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:316)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:116)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:994)
at
org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:207)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:271)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
[jira] [Resolved] (AMQ-6628) [doc] Add details on message groups dispatch delay
[
https://issues.apache.org/jira/browse/AMQ-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully resolved AMQ-6628.
-
Resolution: Fixed
Fix Version/s: 5.15.0
update applied, thanks for the contribution.
> [doc] Add details on message groups dispatch delay
> --
>
> Key: AMQ-6628
> URL: https://issues.apache.org/jira/browse/AMQ-6628
> Project: ActiveMQ
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 5.14.4
>Reporter: Denis A.
>Assignee: Gary Tully
>Priority: Minor
> Labels: documentation
> Fix For: 5.15.0
>
>
> On the message groups documentation page
> (http://activemq.apache.org/message-groups.html), please add the following
> text in the "Adding New Consumers" section:
> {quote}
> When both consumersBeforeDispatchStarts and timeBeforeDispatchStarts are set
> to a value greater than zero, the dispatching will start as soon as the
> required number of consumers are present or the timeBeforeDispatchStarts
> timeout expires.
> If only consumersBeforeDispatchStarts is set then the timeout for consumers
> to connect is 1 second.
> If all consumers disconnect then message dispatch delay will be applied again
> at the next consumer connection.
> {quote}
> By the way, the link under the "As the appropriate test case" text need to be
> fixed to point to a valid location like
> https://github.com/apache/activemq/blob/master/activemq-unit-tests/src/test/java/org/apache/activemq/usecases/MessageGroupDelayedTest.java
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (AMQ-6628) [doc] Add details on message groups dispatch delay
[
https://issues.apache.org/jira/browse/AMQ-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully reassigned AMQ-6628:
---
Assignee: Gary Tully
> [doc] Add details on message groups dispatch delay
> --
>
> Key: AMQ-6628
> URL: https://issues.apache.org/jira/browse/AMQ-6628
> Project: ActiveMQ
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 5.14.4
>Reporter: Denis A.
>Assignee: Gary Tully
>Priority: Minor
> Labels: documentation
>
> On the message groups documentation page
> (http://activemq.apache.org/message-groups.html), please add the following
> text in the "Adding New Consumers" section:
> {quote}
> When both consumersBeforeDispatchStarts and timeBeforeDispatchStarts are set
> to a value greater than zero, the dispatching will start as soon as the
> required number of consumers are present or the timeBeforeDispatchStarts
> timeout expires.
> If only consumersBeforeDispatchStarts is set then the timeout for consumers
> to connect is 1 second.
> If all consumers disconnect then message dispatch delay will be applied again
> at the next consumer connection.
> {quote}
> By the way, the link under the "As the appropriate test case" text need to be
> fixed to point to a valid location like
> https://github.com/apache/activemq/blob/master/activemq-unit-tests/src/test/java/org/apache/activemq/usecases/MessageGroupDelayedTest.java
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (AMQ-6625) IOException in kahaDB needs to pause pending IOExceptionHandler intervention
[ https://issues.apache.org/jira/browse/AMQ-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-6625. - Resolution: Fixed Prior to the handIOException up-call the journal and index pause. The appender write thread won't get recreated and the index is marked unloaded. This ensures that subsequent writes will fail allowing auto recovery from a partial write to kick in on restart. The new kahaDBIOExceptionHandler implements a resume call on kahadb that allows the stopStartConnectors behaviour or ignore all errors behaviour to be chosen if necessary. > IOException in kahaDB needs to pause pending IOExceptionHandler intervention > > > Key: AMQ-6625 > URL: https://issues.apache.org/jira/browse/AMQ-6625 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > KahaDB can recover from kill -9 by replaying the journal from the last > checkpoint or by detecting and reapplying partial writes to the index. > However this activity is compromised if the journal or index accepts > subsequent writes. It an lead to skipped write batches or skipped partial > updates to the index. > The desirable behaviour of treating an IOException as fatal and stopping the > broker in the knowledge that it will restart and fully recover needs to treat > the first IO error as fatal and by default not accept any further writes. > A more advanced IOException handler can facilitate staying alive in more > specific scenarios and reactivate kahadb. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-6625) IOException in kahaDB needs to pause pending IOExceptionHandler intervention
[ https://issues.apache.org/jira/browse/AMQ-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907486#comment-15907486 ] Gary Tully commented on AMQ-6625: - If the broker needs to stay up, for sync traffic AMQ-6606 helps ensure the journal does not get corrupt batches. KahaDBIOExceptionHandler allow this sort of behaviour by resetting kahadb after the first error. > IOException in kahaDB needs to pause pending IOExceptionHandler intervention > > > Key: AMQ-6625 > URL: https://issues.apache.org/jira/browse/AMQ-6625 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > KahaDB can recover from kill -9 by replaying the journal from the last > checkpoint or by detecting and reapplying partial writes to the index. > However this activity is compromised if the journal or index accepts > subsequent writes. It an lead to skipped write batches or skipped partial > updates to the index. > The desirable behaviour of treating an IOException as fatal and stopping the > broker in the knowledge that it will restart and fully recover needs to treat > the first IO error as fatal and by default not accept any further writes. > A more advanced IOException handler can facilitate staying alive in more > specific scenarios and reactivate kahadb. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AMQ-6625) IOException in kahaDB needs to pause pending IOExceptionHandler intervention
[ https://issues.apache.org/jira/browse/AMQ-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907474#comment-15907474 ] Gary Tully commented on AMQ-6625: - AMQ-3725 feature is better suited to the jdbc store which is internally consistent. Kahadb is tolerant to a single write failure. > IOException in kahaDB needs to pause pending IOExceptionHandler intervention > > > Key: AMQ-6625 > URL: https://issues.apache.org/jira/browse/AMQ-6625 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.14.0 >Reporter: Gary Tully >Assignee: Gary Tully > Fix For: 5.15.0 > > > KahaDB can recover from kill -9 by replaying the journal from the last > checkpoint or by detecting and reapplying partial writes to the index. > However this activity is compromised if the journal or index accepts > subsequent writes. It an lead to skipped write batches or skipped partial > updates to the index. > The desirable behaviour of treating an IOException as fatal and stopping the > broker in the knowledge that it will restart and fully recover needs to treat > the first IO error as fatal and by default not accept any further writes. > A more advanced IOException handler can facilitate staying alive in more > specific scenarios and reactivate kahadb. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AMQ-6625) IOException in kahaDB needs to pause pending IOExceptionHandler intervention
Gary Tully created AMQ-6625: --- Summary: IOException in kahaDB needs to pause pending IOExceptionHandler intervention Key: AMQ-6625 URL: https://issues.apache.org/jira/browse/AMQ-6625 Project: ActiveMQ Issue Type: Bug Components: KahaDB Affects Versions: 5.14.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.15.0 KahaDB can recover from kill -9 by replaying the journal from the last checkpoint or by detecting and reapplying partial writes to the index. However this activity is compromised if the journal or index accepts subsequent writes. It an lead to skipped write batches or skipped partial updates to the index. The desirable behaviour of treating an IOException as fatal and stopping the broker in the knowledge that it will restart and fully recover needs to treat the first IO error as fatal and by default not accept any further writes. A more advanced IOException handler can facilitate staying alive in more specific scenarios and reactivate kahadb. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AMQ-6606) Journal partial write can result in batch corruption on restart
[
https://issues.apache.org/jira/browse/AMQ-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully resolved AMQ-6606.
-
Resolution: Fixed
the fix is refined to just sync batches such that index updates from async
writes do not get reused. In the sync case, the index is not updated till
completion.
> Journal partial write can result in batch corruption on restart
> ---
>
> Key: AMQ-6606
> URL: https://issues.apache.org/jira/browse/AMQ-6606
> Project: ActiveMQ
> Issue Type: Bug
> Components: KahaDB
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0, 5.14.4
>
>
> Recovery checking on kahadb will ignore a partial journal write at the end of
> the journal. However repeated write errors increment the write offset and if
> a subsequent write succeeds recovery fails reporting a corrupt block.
> {code}
> MessageDatabase | emq.store.kahadb.MessageDatabase | Detected corrupt
> journal files. [34:43883209 >= key < 34:47226069]
> {code}
> One scenario is write failure for no space followed by gc which allows
> subsequent writes to complete.
> A failed write or sync should fail with an exception and should revert any
> offset increment such that a subsequent write reuses that offset, avoiding a
> partial write corruption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (AMQ-6606) Journal partial write can result in batch corruption on restart
[
https://issues.apache.org/jira/browse/AMQ-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897168#comment-15897168
]
Gary Tully edited comment on AMQ-6606 at 3/7/17 12:32 PM:
--
This needs more work.. readjustment of offset needs to terminate next pending
write batch.
was (Author: gtully):
This needs more work.. readjustment of offset needs to percolate into all
queued writes... some more thought required to validate, but this current fix
is not suffice.
> Journal partial write can result in batch corruption on restart
> ---
>
> Key: AMQ-6606
> URL: https://issues.apache.org/jira/browse/AMQ-6606
> Project: ActiveMQ
> Issue Type: Bug
> Components: KahaDB
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0, 5.14.4
>
>
> Recovery checking on kahadb will ignore a partial journal write at the end of
> the journal. However repeated write errors increment the write offset and if
> a subsequent write succeeds recovery fails reporting a corrupt block.
> {code}
> MessageDatabase | emq.store.kahadb.MessageDatabase | Detected corrupt
> journal files. [34:43883209 >= key < 34:47226069]
> {code}
> One scenario is write failure for no space followed by gc which allows
> subsequent writes to complete.
> A failed write or sync should fail with an exception and should revert any
> offset increment such that a subsequent write reuses that offset, avoiding a
> partial write corruption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Reopened] (AMQ-6606) Journal partial write can result in batch corruption on restart
[
https://issues.apache.org/jira/browse/AMQ-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully reopened AMQ-6606:
-
This needs more work.. readjustment of offset needs to percolate into all
queued writes... some more thought required to validate, but this current fix
is not suffice.
> Journal partial write can result in batch corruption on restart
> ---
>
> Key: AMQ-6606
> URL: https://issues.apache.org/jira/browse/AMQ-6606
> Project: ActiveMQ
> Issue Type: Bug
> Components: KahaDB
>Affects Versions: 5.14.0
>Reporter: Gary Tully
>Assignee: Gary Tully
> Fix For: 5.15.0, 5.14.4
>
>
> Recovery checking on kahadb will ignore a partial journal write at the end of
> the journal. However repeated write errors increment the write offset and if
> a subsequent write succeeds recovery fails reporting a corrupt block.
> {code}
> MessageDatabase | emq.store.kahadb.MessageDatabase | Detected corrupt
> journal files. [34:43883209 >= key < 34:47226069]
> {code}
> One scenario is write failure for no space followed by gc which allows
> subsequent writes to complete.
> A failed write or sync should fail with an exception and should revert any
> offset increment such that a subsequent write reuses that offset, avoiding a
> partial write corruption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AMQ-6377) Introduce a periodic disk sync mode for KahaDB journal
[ https://issues.apache.org/jira/browse/AMQ-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894591#comment-15894591 ] Gary Tully commented on AMQ-6377: - [~cshannon] noticed the use of String for org.apache.activemq.store.kahadb.MessageDatabase#journalDiskSyncStrategy which makes that accessor a little expensive. Any good reason for that? isEnableJournalDiskSyncs is used on every kahadb transaction store! see: https://github.com/apache/activemq/blob/d9350912984f12356e9d51b0f00b5a28f5cfa58d/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBTransactionStore.java#L287 > Introduce a periodic disk sync mode for KahaDB journal > -- > > Key: AMQ-6377 > URL: https://issues.apache.org/jira/browse/AMQ-6377 > Project: ActiveMQ > Issue Type: New Feature > Components: Broker, KahaDB >Affects Versions: 5.13.4 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > Fix For: 5.14.0 > > > KahaDB has two modes for journal disk syncs, either always sync for each > write or never sync. I'm proposing that we add a third option, a period disk > sync. > The intended behavior of this would be to run a task in the file appender > that would sync the file (if necessary) at some periodic interval (such as > every 500 ms, or 1 second, etc) instead of every write. The file would also > be synced on close (on file rollover or shutdown) > In my testing, syncing every 1 second has been proven to be nearly > indistinguishable performance as never disk syncing but is a safer option as > you insure that a sync is performed at least once per interval. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
