[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jbertram commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @stanlyDoge, yes, I think so. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user stanlyDoge commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 Can I close this PR? ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @michaelandrepearce All thoughts and ideas are appreciated :-) I could achieve what we need (despite it violates JMS spec) by changing the durable flag of a QueueConfig in beforeCreateQueue via reflection to false but especially since the durable field is final (which I can still change by removing the final flag via reflection first) this feels like a risky operation. So if I could create the queue that I need programmatically in beforeCreateQueue() and the ActiveMQServerImpl would - instead of directly executing `queueFactory.createQueueWith(queueConfig);` after the broker plugins are called - check if the queue now already exists I would have all the flexibility I need. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 On the note of broker plugin, you do realise you can intercept beforeCreateAddress and beforeCreateQueue and a lot of other functions, this way you could do what you want without affecting the core broker. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 As I stated there is already a global flag, if there is an address level flag then this should have the exact same behaviour. Also Iâm not advocating changing the global flag behaviour. We provided you an option you can achieve what you needed using multicast addresses (jms topic). The plugin idea was just another that meant you donât need the change and donât affect other users of the broker as the plugin would be your own, if you are soo insistent on using but overriding the JMS queue logic in the amqp jms client. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @michaelandrepearce I cannot modify the durable flag of AMQP messages as they are meant to be immutable and it would prevent using features like digital signatures AFAIK. What we are looking for - no matter at which level it is configured - is a way to opt-out of message persistence for a queues in a certain namespace (or even globally) since the SLA/QoS of our service does not guarantee message delivery in case of broker failure. But if we receive more messages than what can be stored in the heap or receive very large messages we would still want/need paging/large messages support to ensure stable operation. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 -1 for this as well. The semantics gets blurry and confusing. Keep it simple principle broken IMO. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 -1 it having different or custom behaviour, if such flag exists it should have the same behaviour as the global just simply at the address level. If you want custom behaviour or to manipulate the message so itâs durable flag is set to false then you could create a custom brokerplugin ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @mtaylor In our case a `persistence=false` parameter in the address settings that only disables the persistent journal but not large message and paging support would work too. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user mtaylor commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 Sorry guys but I am -1 on this. I thought the idea of this was to behave in a similar way to default-max-consumers, or default-purge-on-no-consumers. I am not happy about overriding the protocol handlers behaviour in this way. I can see value in a default-queue-durability setting, for cases when it's not known. Can the original requirement not be addressed using auto-delete-queues and non-durable messages? If non-durable queues/temps are really required, how about a persistence=false address-setting instead? ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user clebertsuconic commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 I could easily rebase this also.. but if you could rebase it later please ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user mtaylor commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 Just to be clear here. I can see a use case for this. I'm +1 on the idea but not for overriding specific protocol / API behaviour. I Just wanted to clarify that as it was discussed here. This is more to do with what the default auto create settings are designed for vs any fundamental issue with the requirement. @jostbg It would be good to understand more if your use case. How about we have a chat, IRC #apache-activemq? We can go over your use case and discuss if what you want is possible or if we need a new feature. https://activemq.apache.org/artemis/community.html ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @mtaylor When I talked about durability in the context of queues I meant persistence (writing messages to disk) and not durable subscriptions. We want to allow clients to create their own queues under a given namespace but we do not want to persist these messages (so they will not survive a broker restart). How else if not by declaring these auto-created queues as non-durable can we achieve that? Maybe there is some other setting I am simply missing here. Then we don't need a flag on auto-created queues to specify their durability mode. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user mtaylor commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @jostbg The point is, if you want the broker to create something for you, then the broker needs to know what to create. I used JMS clients as an example as they are usually specific about what they want, the same thing can happen in other protocols. In the JMS case the client sends the broker the information it needs to create the appropriate queue with some additional bits of info. The JMS client doesn't say, hey I want a durable queue with these properties (except the CORE client as it's tied to the Artemis implementation). Instead, say the QPID AMQP client says, hey I want a JMS Durable Subscription. It does this by setting some special JMS AMQP properties. The AMQP protocol handler reads these properties and decides how to interpret them. In some cases these get translated to durable queues, in others they are translated to non-durable queues. Artemis has a underlying model that is protocol agnostic, so things are translated from API/protocol specifics down to the und erlying model. Coming back to your comments on https://issues.apache.org/jira/browse/ARTEMIS-1582. There are two meanings of durable. Durable in the context of a subscription (which is tied to the consumer connection life cycle), and durable in the context persistence (tied to the broker life cycle). The reason you are seeing some queues be durable and others not, is because this is how the JMS spec is implemented using the underlying Artemis model. An non-durable subscription means that the client only gets messages while it's connected. If the server crashes, the client isn't connected anymore. In this case, the spec says, its fine to lose all the messages. For this reason there's no point creating a durable queue. In the JMS Queue (ANYCAST) scenario we need a durable queue because the messages outlive the subscriber connection life cycle. This is also true for durable subscriptions. Regarding your comment on misconfiguration, yes you can do that, but you're specifically overriding broker behaviour to do something else. You're welcome to do this, but you have to be aware that you are breaking protocol/API contracts by doing so. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @mtaylor What is the difference between saying, "Artemis, create a non-durable queue named foo during server start" vs "Artemis create a non-durable queue named foo the moment it is first used." Why does it not seem to be an issue if I pre-create the non-durable anycast queue "foo" to which a JMS client can send a message but when that non-durable queue is auto-created the moment a message is send to it you suddenly talk about JMS spec violations. In both cases the result is the same, the JMS client sends a message to a non-durable queue. The only difference is the point in time when the queue is actually created. Following your argumentation, Artemis should generally prevent JMS clients trying to send a message to a non-persistent queue - no matter if it is pre-created or created on first use. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user mtaylor commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 A lot of discussion here :). This should not override what the client requests. There are a couple of ways addresses and queues can be set up in Artemis. The first way is to do what @michaelandrepearce has described. The client is explicit about what it wants. Typical example is JMS where the spec is very specific. In this case the client passes a description of what it wants to the broker. If the client requests a durable subscription, or similar then the broker will try to serve that request. Providing the client has the correct security permissions and address settings it's all good. The second way is for the broker to decide how to handle things. This can either be configured by being very specific about what resources can exist on the broker, for example, locking down all the addresses and pre-creating all queues and turning off any create permissions or auto-create settings or it can be configured to be open and auto-create things that aren't specified. This setting is for this 2nd case. With something like AMQP or STOMP the clients don't need to be aware of what type of resource they're sending or receiving from. They only require an address to subscribe from or send to. If the address the client asks for does not yet exist, the broker can create it automatically. This is what these auto-create settings are used for. They should not conflict with what clients ask for. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 I am too new to Artemis to implement that properly but I hope @stanlyDoge can address your points. Thanks for your feedback. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @jostbg as I said Iâm not a fan, but if you do want to implement could you address the points I raised. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 So when we pre-define non-durable anycast queues in Artemis and a JMS client publishes to that queue or subscribes to it, we are actually already violating the JMS spec. Therefore it would no be different if the non-durable queue is pre-defined or auto-created. In our case we do not really care about the JMS spec. We only use the Qpid AMQP JMS client because it seems to be the only actually working and easy to use AMQP 1.0 client for Java. We tried IBM's mqlight which we didn't get to work in our scenario and also it seems to be dead (no development since 2016). SwiftMQ has an odd non-OSS license/distribution model. And proton-j is just too bare bone without proper documentation. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 JMS Queue is durable by spec. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 I am not trying to annoying or anything, but how is a non-durable queue - that is auto created the moment the first client tries to use it (in whatever way) - less JMS compliant than a pre-created non-durable queue under the same name? ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @stanlyDoge as said earlier for this PR, if it is to go forwards id like to see - setting values should be enums, to cater for alternative way round so people can make them non-durable or durable - default should be null (which keeps todays behaviour) - it needs really good documentation with some big clear warnings about it. Including that setting these would break JMS specs. - needs cross protocol / client compatibility testing. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @michaelandrepearce What if the first client does not subscribe to any address but only sends an AMQP message to an address (either a queue or topic via QPid JMS). Based on what criterias should Artemis decide what to create? Currently even if the producer sends a non-durable message, durable queues are created. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @jostbg from the sounds of that ticket, what you hit was a bug thats been fixed, the AMQP client should be able to request a durable or non durable queue, and the AMQP JMS client should make this cleanly exposed to end users without needing to get into AMQP detail. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue:
https://github.com/apache/activemq-artemis/pull/1775
@jostbg there is implications especially for JMS users, basically if a
client requests a shared subscription, it should not marry up to a durable
shared subscription or vice versa.
"There is no restriction on durable subscriptions and shared non-durable
subscriptions having the same name and clientId (which may be unset). Such
subscriptions would be completely separate."
to avoid this due a prefix on the queue to separate the two, so on core
this is prefixed "non-durable", see code here
```
public static SimpleString createQueueNameForSubscription(final boolean
isDurable,
final String
clientID,
final String
subscriptionName) {
final String queueName;
if (clientID != null) {
if (isDurable) {
queueName = ActiveMQDestination.escape(clientID) + SEPARATOR +
ActiveMQDestination.escape(subscriptionName);
} else {
queueName = "nonDurable" + SEPARATOR +
ActiveMQDestination.escape(clientID) + SEPARATOR +
ActiveMQDestination.escape(subscriptionName);
}
} else {
if (isDurable) {
queueName = ActiveMQDestination.escape(subscriptionName);
} else {
queueName = "nonDurable" + SEPARATOR +
ActiveMQDestination.escape(subscriptionName);
}
}
return SimpleString.toSimpleString(queueName);
}
```
---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 @jostbg it can, look at the AMQP JMS client, when via JMS you make a shared subscription, or a shared durable subscription. if the On the Artemis side look at ProtonServerSenderContext, for how it maps this. But it looks at the source's TerminusDurability. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 I also don't see much of a difference if a client tries to send a message to a pre-created non-durable queue and but expects the queue to be durable over he a message sends to an auto-create address that then creates a non-durable queue on the fly while the client expects a durable queue. In both cases the client should know in advance what routing/durability properties he can use/expect for given address. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 The problem is, that at least when using AMQP the client cannot control if a durable or non-durable queue is created. The broker always auto-creates durable queues. We discussed that at https://issues.apache.org/jira/browse/ARTEMIS-1582 where the conclusion was it may make sense to - instead of hard-code the durable property - extend the address settings. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 This is what security settings are for e.g. you can set for address A users can only create non-durable queues. If a client try to create a durable queue they will get security error. This feature is there today, already. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user jostbg commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 I created the ARTEMIS issue as a followup to this https://issues.apache.org/jira/browse/ARTEMIS-1582. We actually have the use-case that for certain namespaces we want to dictate all properties of auto-created addresses/queues on the server-side. So the effective settings should not be guessed based on how a particular client tries to subscribes to a queue or sends a message to an address but based on a server side specification. The same way you can pre-create a queue and define it as non-durable. We need auto-created queues under certain namespaces to be non-durable no matter what one specific client tries to do. The reason is we only run the broker and do not control all the clients used by the different teams. ---
[GitHub] activemq-artemis issue #1775: ARTEMIS-1587 Add setting to control the queue ...
Github user michaelandrepearce commented on the issue: https://github.com/apache/activemq-artemis/pull/1775 Why is this needed? If an address is allowed for queues to be auto created, the client describes if the queue should be durable or not. e.g. See JMS Subscription (which maps onto queue) you create a non durable or durable via the client, and it is reflected in the queue it creates. ---
