date:20180302

Github user asfgit closed the pull request at:

https://github.com/apache/activemq-artemis/pull/1923


---

[GitHub] activemq-artemis pull request #1927: ARTEMIS-1720 put activemq.xsd in 'schem...

Github user asfgit closed the pull request at:

https://github.com/apache/activemq-artemis/pull/1927


---

[GitHub] activemq-artemis issue #1918: ARTEMIS-1722 Reduce pooled Netty ByteBuf usage

Github user tabish121 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1918
  
@franz1981 @clebertsuconic I've sent an alternate PR #1928 which performs 
the send without any buffer copy unless necessary due to message redelivery or 
existence of original delivery annotations.  Proton will be copying those bytes 
anyway so an intermediate isn't really needed on the common path.  

It could be argued that you could further change things beyond what I did 
and just keep a single ReadableBuffer instance around in the consumer and give 
it the outbound buffer to send as the proton send method will take that 
readable buffer and call get(array, offset, length) on it to get a copy which 
means netty should perform the most efficient write.  


---

[GitHub] activemq-artemis pull request #1928: ARTEMIS-1722 Don't copy message bytes u...

GitHub user tabish121 opened a pull request:

https://github.com/apache/activemq-artemis/pull/1928

ARTEMIS-1722 Don't copy message bytes unless needed

Alternate patch that doesn't copy the message bytes unless doing a
redelivery or skipping delivery annotations in the original version of
the message.  Proton-J will copy the bytes provided to the Sender's send
method so a copy isn't necessary on most common sends.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tabish121/activemq-artemis ARTEMIS-1722

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1928.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1928


commit 63b2e84fce76a9bf6c7cc3c9cef00b775457453c
Author: Timothy Bish 
Date:   2018-03-02T20:03:08Z

ARTEMIS-1722 Don't copy message bytes unless needed

Alternate patch that doesn't copy the message bytes unless doing a
redelivery or skipping delivery annotations in the original version of
the message.  Proton-J will copy the bytes provided to the Sender's send
method so a copy isn't necessary on most common sends.




---

[GitHub] activemq-artemis pull request #1927: ARTEMIS-1720 put activemq.xsd in 'schem...

GitHub user jbertram opened a pull request:

https://github.com/apache/activemq-artemis/pull/1927

ARTEMIS-1720 put activemq.xsd in 'schema' dir



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbertram/activemq-artemis ARTEMIS-1720

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1927.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1927


commit afda0b912eca0542f3d5c6029adfdd7f3bc49241
Author: Justin Bertram 
Date:   2018-03-02T19:43:19Z

ARTEMIS-1720 put activemq.xsd in 'schema' dir




---

[GitHub] activemq-artemis pull request #1925: ARTEMIS-1727 - Make sure transport is s...

Github user cshannon commented on a diff in the pull request:

https://github.com/apache/activemq-artemis/pull/1925#discussion_r171924792
  
--- Diff: 
artemis-protocols/artemis-openwire-protocol/src/main/java/org/apache/activemq/artemis/core/protocol/openwire/OpenWireConnection.java
 ---
@@ -641,13 +647,37 @@ public void fail(ActiveMQException me, String 
message) {
  }
   }
   try {
- protocolManager.removeConnection(this.getConnectionInfo(), me);
+ if (this.getConnectionInfo() != null) {
+protocolManager.removeConnection(this.getConnectionInfo(), me);
+ }
   } catch (InvalidClientIDException e) {
  ActiveMQServerLogger.LOGGER.warn("Couldn't close connection 
because invalid clientID", e);
   }
   shutdown(true);
}
 
+   private void delayedStop(final int waitTime, final String reason, 
Throwable cause) {
+  if (waitTime > 0) {
+ try {
+server.getExecutorFactory().getExecutor().execute(new 
Runnable() {
+   @Override
+   public void run() {
+  try {
+ Thread.sleep(waitTime);
--- End diff --

I won't have time today but I will fix my patch and push it up first thing 
monday morning.


---

[GitHub] activemq-artemis pull request #1925: ARTEMIS-1727 - Make sure transport is s...

Github user cshannon commented on a diff in the pull request:

https://github.com/apache/activemq-artemis/pull/1925#discussion_r171921837
  
--- Diff: 
artemis-protocols/artemis-openwire-protocol/src/main/java/org/apache/activemq/artemis/core/protocol/openwire/OpenWireConnection.java
 ---
@@ -641,13 +647,37 @@ public void fail(ActiveMQException me, String 
message) {
  }
   }
   try {
- protocolManager.removeConnection(this.getConnectionInfo(), me);
+ if (this.getConnectionInfo() != null) {
+protocolManager.removeConnection(this.getConnectionInfo(), me);
+ }
   } catch (InvalidClientIDException e) {
  ActiveMQServerLogger.LOGGER.warn("Couldn't close connection 
because invalid clientID", e);
   }
   shutdown(true);
}
 
+   private void delayedStop(final int waitTime, final String reason, 
Throwable cause) {
+  if (waitTime > 0) {
+ try {
+server.getExecutorFactory().getExecutor().execute(new 
Runnable() {
+   @Override
+   public void run() {
+  try {
+ Thread.sleep(waitTime);
--- End diff --

Agreed, I will make that change and use a scheduled executor instead.


---

[GitHub] activemq-artemis pull request #1925: ARTEMIS-1727 - Make sure transport is s...

Github user tabish121 commented on a diff in the pull request:

https://github.com/apache/activemq-artemis/pull/1925#discussion_r171918979
  
--- Diff: 
artemis-protocols/artemis-openwire-protocol/src/main/java/org/apache/activemq/artemis/core/protocol/openwire/OpenWireConnection.java
 ---
@@ -641,13 +647,37 @@ public void fail(ActiveMQException me, String 
message) {
  }
   }
   try {
- protocolManager.removeConnection(this.getConnectionInfo(), me);
+ if (this.getConnectionInfo() != null) {
+protocolManager.removeConnection(this.getConnectionInfo(), me);
+ }
   } catch (InvalidClientIDException e) {
  ActiveMQServerLogger.LOGGER.warn("Couldn't close connection 
because invalid clientID", e);
   }
   shutdown(true);
}
 
+   private void delayedStop(final int waitTime, final String reason, 
Throwable cause) {
+  if (waitTime > 0) {
+ try {
+server.getExecutorFactory().getExecutor().execute(new 
Runnable() {
+   @Override
+   public void run() {
+  try {
+ Thread.sleep(waitTime);
--- End diff --

Sleeping here seems like it might have a ripple on the other resources 
using the executor, maybe use a scheduled executor from the pool in 
ActiveMQServer ?


---

[GitHub] activemq-artemis pull request #1924: ARTEMIS-1696 doc updates

Github user asfgit closed the pull request at:

https://github.com/apache/activemq-artemis/pull/1924


---

[GitHub] activemq-artemis pull request #1922: [ARTEMIS-1724] Create Artemis Openwire ...

Github user jbertram commented on a diff in the pull request:

https://github.com/apache/activemq-artemis/pull/1922#discussion_r171909862
  
--- Diff: tests/karaf-client-integration-tests/pom.xml ---
@@ -36,6 +36,18 @@

 

+  
+ org.apache.activemq
+ activemq-client
+ test
+ 
+
+   
--- End diff --

Sorry about that. I didn't notice the exclusion. Makes perfect sense.


---

[GitHub] activemq-artemis pull request #1922: [ARTEMIS-1724] Create Artemis Openwire ...

2018-03-02 Thread johnpoth

Github user johnpoth commented on a diff in the pull request:

https://github.com/apache/activemq-artemis/pull/1922#discussion_r171908429
  
--- Diff: tests/karaf-client-integration-tests/pom.xml ---
@@ -36,6 +36,18 @@

 

+  
+ org.apache.activemq
+ activemq-client
+ test
+ 
+
+   
--- End diff --

Hi @jbertram thanks for reviewing! 

I'm excluding the JMS 1.1 artifact, so we are compiling against JMS 2 
(which is brought on the classpath transitively by artemis-jms-client). It was 
causing compiling issues because javax.jms.Connection doesn't implement 
AutoCloseable in JMS 1.1 whereas it does in JMS 2 which makes it possible to 
use in try-with-resources statements.

Thanks!


---

[GitHub] activemq-artemis pull request #1926: ARTEMIS-1721 fix broken doc link

Github user asfgit closed the pull request at:

https://github.com/apache/activemq-artemis/pull/1926


---

[GitHub] activemq-artemis pull request #1922: [ARTEMIS-1724] Create Artemis Openwire ...

Github user jbertram commented on a diff in the pull request:

https://github.com/apache/activemq-artemis/pull/1922#discussion_r171901488
  
--- Diff: tests/karaf-client-integration-tests/pom.xml ---
@@ -36,6 +36,18 @@

 

+  
+ org.apache.activemq
+ activemq-client
+ test
+ 
+
+   
--- End diff --

The comment says "JMS 2," but the artifactId references the 1.1 spec. This 
seems wrong.


---

[GitHub] activemq-artemis pull request #1926: ARTEMIS-1721 fix broken doc link

GitHub user jbertram opened a pull request:

https://github.com/apache/activemq-artemis/pull/1926

ARTEMIS-1721 fix broken doc link



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbertram/activemq-artemis ARTEMIS-1721

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1926.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1926


commit cf9640bf3508b72b5cf04478e5b46ba7295ff37c
Author: Justin Bertram 
Date:   2018-03-02T16:39:37Z

ARTEMIS-1721 fix broken doc link




---

[GitHub] activemq-artemis pull request #1925: ARTEMIS-1727 - Make sure transport is s...

GitHub user cshannon opened a pull request:

https://github.com/apache/activemq-artemis/pull/1925

ARTEMIS-1727 - Make sure transport is stopped on failed OpenWire

connection

To prevent a socket from hanging open by a bad client the broker should
make sure to stop the transport if a connection attempt fails by an
OpenWire client

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cshannon/activemq-artemis ARTEMIS-1727

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1925.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1925


commit 8de2fccc883aa3a532e540e5e700e2c668e4e289
Author: Christopher L. Shannon (cshannon) 
Date:   2018-03-02T16:38:07Z

ARTEMIS-1727 - Make sure transport is stopped on failed OpenWire
connection

To prevent a socket from hanging open by a bad client the broker should
make sure to stop the transport if a connection attempt fails by an
OpenWire client




---

[GitHub] activemq-artemis pull request #1924: ARTEMIS-1696 doc updates

GitHub user jbertram opened a pull request:

https://github.com/apache/activemq-artemis/pull/1924

ARTEMIS-1696 doc updates



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbertram/activemq-artemis ARTEMIS-1696

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1924.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1924


commit 3d4b52df9dfd1ac95ffd59dd06c813fa770f8989
Author: Justin Bertram 
Date:   2018-02-22T16:15:52Z

ARTEMIS-1696 doc updates




---

[GitHub] activemq-artemis issue #1918: ARTEMIS-1722 Reduce pooled Netty ByteBuf usage

Github user franz1981 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1918
  
@clebertsuconic 
> SoftReference sucks. It usually let you OME before itâs released.

I know and i was too hurried to say that it could be used :P you're right 
:+1: 

> What about having a pool somewhere else? Or a smaller pool from Netty?

It could be a solution: TBH if we found together a solution that doesn't 
require much (Netty) config from the user pov, would be even better!

> Iâm out today and Monday. Can we talk about this when I get back on 
Tuesday?

Sure buddy, this is not a fix so we can arrive on it taking the right time

@tabish121 

> It doesn't look like the buffer could be contended 

On Tuesday with @clebertsuconic will be confirmed, but thanks...the code 
was really too complex for just a simple behaviour ie reuse a buffer.

> It isn't entirely uncommon for consumers to be subscribed to low volume 
queues or topics where they might sit idle for a longer period of time so 
having each consumer hold onto memory like that indefinitely could lead to 
quick resource exhaustion on the broker in the case of 1000s of connections 
(think IoT)

That's important to know, yes, it makes sense! 
So a solution could be to use a scheduled sweeper that check the alive 
consumers, but idle for "too long" and deallocate their buffers or maybe making 
use of something that the AMQP protocol itself probably provide (a keepalive, 
tick or similar, don't know) to do the same. 
Just thinking loud; anyway, I will let this PR rest a bit and we could have 
a chat together next week if something here could be done :+1: 










---

[GitHub] activemq-artemis issue #1918: ARTEMIS-1722 Reduce pooled Netty ByteBuf usage

Github user tabish121 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1918
  
@franz1981 It doesn't look like the buffer could be contended unless 
perhaps the consumer was able to be subscribed to more than one queue at a time 
via some sort of composite type subscription but I don't think that is possible 
?  otherwise the broker should be dispatching serially if I understand 
correctly but I'd defer to @clebertsuconic on that one.  

It isn't entirely uncommon for consumers to be subscribed to low volume 
queues or topics where they might sit idle for a longer period of time so 
having each consumer hold onto memory like that indefinitely could lead to 
quick resource exhaustion on the broker in the case of 1000s of connections 
(think IoT)




---

[GitHub] activemq-artemis issue #1918: ARTEMIS-1722 Reduce pooled Netty ByteBuf usage

2018-03-02 Thread clebertsuconic

Github user clebertsuconic commented on the issue:

https://github.com/apache/activemq-artemis/pull/1918
  
SoftReference sucks.  It usually let you OME before itâs released. 


What about having a pool somewhere else? Or a smaller pool from Netty?


Iâm out today and Monday.  Can we talk about this when I get back on 
Tuesday?


---

[GitHub] activemq-artemis pull request #1923: ARTEMIS-1726 - check proper permissions...

GitHub user cshannon opened a pull request:

https://github.com/apache/activemq-artemis/pull/1923

ARTEMIS-1726 - check proper permissions when using OpenWire

Ensure that on queue creation and deletion that the proper permissions
are checked

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cshannon/activemq-artemis ARTEMIS-1726

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1923.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1923


commit 17b0e6547ab848fd0f61b109abddd8c2e8c506e0
Author: Christopher L. Shannon (cshannon) 
Date:   2018-02-28T21:28:55Z

ARTEMIS-1726 - check proper permissions when using OpenWire

Ensure that on queue creation and deletion that the proper permissions
are checked




---

[GitHub] activemq-artemis issue #1918: ARTEMIS-1722 Reduce pooled Netty ByteBuf usage

Github user franz1981 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1918

@clebertsuconic Yep , don't worry I'm glad that you have taken a look on
it! Will answer inline..

> i - There's a buffer sitting on the ProtonSenderContext now that may be
unused for a while. What if you had 1000 connections at one time.. and then all
of them go inactive? I don't understand how this will be Reducing footprint. I
see quite the contrary here.

I think that optimizing for the common case would be the bigger fish, but I
don't know if what you have described is the common pattern...it is? If yes
you're right, but consider that although Netty will release it to the pool, the
memory footprint won't goes away. To solve it I will use `SoftReference` that
will react on memory pressure demands, just to be safe.

Re memory footprint, it helps due to how Netty heap pools work: just
allocating 10 bytes on it from any thread will trigger all the netty heap
arenas to be allocated: that means to have more than (using the default config)
16 MB * 16 = 250 MB of allocated heaps just sitting to perform buffer copies.
And the sad fact is that it will add a constant memory footprint to Artemis
when the broker is idle for real too.
Let me show you it...
That's on master (with the patch of Tim), after forcing a Full GC: 250 MB
of Netty arena remaining allocated

![image](https://user-images.githubusercontent.com/13125299/36903689-ba230bc2-1e2e-11e8-8976-a79b87813411.png)

That's is with this patch (and Tim commit too): all the allocated bytes
disappear

![image](https://user-images.githubusercontent.com/13125299/36903735-dcbab05e-1e2e-11e8-8e21-2af7ff8c9427.png)

For the graph is not visible the throughput improvement and we can just
focus just on memory: it is pretty visbile that the memory used is lower than
with pooled Netty ByteBuf too and that's because with too many threads
(producers/consumers) the Netty pool isn't able to scale and will fallback to
produce garbage.
I hope to have explained my concerns.

> ii - It is not clear to me when the buffer would be busy, since the
ProtonSenderContext represents a single consumer. Maybe it would.. but it
shouldn't. if that's happening you will be sending a large heap buffer to GC.

That's why I needed you and @tabish121 :P : if it is uncontended the logic
could be made much simpler.
From my tests (with 64 pairs of producers/consumers) seems uncontended so I
have supposed that the "busy" case is the uncommon one and just having one
byte[] created that will die very soon isn't a big deal for any GC (if is
uncommon). Re how much big, with G1 (our defualt GC) `-XX:G1HeapRegionSize` is
2 MB AFAIK and any byte[] allocation > 50% of it ie 1MB is "less optimized".

Let me re-write it with `SoftReference` and without 'AtomicReference` (it
isn't contended) and it could address all your point, thanks!!

---

[GitHub] activemq-artemis issue #1918: ARTEMIS-1722 Reduce pooled Netty ByteBuf usage

2018-03-02 Thread clebertsuconic

Github user clebertsuconic commented on the issue:

https://github.com/apache/activemq-artemis/pull/1918
  
I won't be able to look into this before Tuesday.. Please don't merge this 
without me... :)


---

[GitHub] activemq-artemis issue #1918: ARTEMIS-1722 Reduce pooled Netty ByteBuf usage

2018-03-02 Thread clebertsuconic

Github user clebertsuconic commented on the issue:

https://github.com/apache/activemq-artemis/pull/1918
  
I see two problems here..

i - There's a buffer sitting on the ProtonSenderContext now that may be 
unused for a while. What if you had 1000 connections at one time.. and then all 
of them go inactive?  I don't understand how this will be Reducing footprint. I 
see quite the contrary here.


ii - It is not clear to me when the buffer would be busy, since the 
ProtonSenderContext represents a single consumer. Maybe it would.. but it 
shouldn't. if that's happening you will be sending a large heap buffer to GC.


---

[GitHub] activemq-artemis pull request #1916: ARTEMIS-1504 Update Qpid JMS to 0.30.0 ...

Github user asfgit closed the pull request at:

https://github.com/apache/activemq-artemis/pull/1916


---

[GitHub] activemq-artemis pull request #1922: [ARTEMIS-1724] Create Artemis Openwire ...

2018-03-02 Thread johnpoth

GitHub user johnpoth opened a pull request:

https://github.com/apache/activemq-artemis/pull/1922

[ARTEMIS-1724] Create Artemis Openwire client Karaf feature

https://issues.apache.org/jira/browse/ARTEMIS-1724

Thanks!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/johnpoth/activemq-artemis ARTEMIS-1724

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1922.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1922


commit f9b0469fd8c5f1be1e05749ab73b788d0c2a6895
Author: jpoth 
Date:   2018-03-01T19:42:07Z

[ARTEMIS-1724] Create Artemis Openwire client Karaf feature




---

[GitHub] activemq-artemis pull request #1921: ARTEMIS-1725 fix browsing non-listing t...

2018-03-02 Thread stanlyDoge

GitHub user stanlyDoge opened a pull request:

https://github.com/apache/activemq-artemis/pull/1921

ARTEMIS-1725 fix browsing non-listing tabs under JMX



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/stanlyDoge/activemq-artemis ARTEMIS-1725

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1921.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1921


commit 34d9daa809b8948dd8957598796da98301ecbd75
Author: Stanislav Knot 
Date:   2018-03-02T13:18:19Z

ARTEMIS-1725 fix browsing non-listing tabs under JMX




---

[GitHub] activemq-artemis pull request #1920: Added acknowledgements for messages exp...

2018-03-02 Thread ilkkavi

GitHub user ilkkavi opened a pull request:

https://github.com/apache/activemq-artemis/pull/1920

Added acknowledgements for messages expired without expiry address bindings

Referring to a resource leak and JMX reporting expired messages as 
in-delivery as discussed in the user mailing list between 2018-02-21 and 
2018-03-01. I was unable to reproduce the OOM after this change. Would you care 
to comment on this, if possible?


http://activemq.2283324.n4.nabble.com/Artemis-2-4-0-Issues-with-memory-leaks-and-JMS-message-redistribution-td4736891.html

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ilkkavi/activemq-artemis expiry-memoryleak

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1920.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1920


commit c9e544b19e8990cf28798df239f83248d2aae07d
Author: Ilkka Virolainen 
Date:   2018-03-02T12:18:40Z

Added acknowledgements for messages expired without expiry address bindings




---

[GitHub] activemq-artemis pull request #1919: ARTEMIS-1723 Migrate AMQP C# examples t...

2018-03-02 Thread dmvolod

GitHub user dmvolod opened a pull request:

https://github.com/apache/activemq-artemis/pull/1919

ARTEMIS-1723 Migrate AMQP C# examples to the common .NET version



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dmvolod/activemq-artemis ARTEMIS-1723

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1919.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1919


commit 45bbef06e1bdad3e95ceea4ed9bf0996db9263f9
Author: Dmitry Volodin 
Date:   2018-03-02T10:50:26Z

ARTEMIS-1723 Migrate AMQP C# examples to the common .NET version




---

[GitHub] activemq-artemis issue #1918: ARTEMIS-1722 Reduce pooled Netty ByteBuf usage

Github user franz1981 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1918
  
This one, together with 
https://github.com/apache/activemq-artemis/pull/1916 aims to reduce the memoty 
footprint (improving scalability by different angles) of AMQP on the broker.
I need a review from @clebertsuconic  and @tabish121 for the mechanics that 
I've used to reuse the same send buffer: the mechanics is solid but if 
`ProtonServerSenderContext.deliverMessage` would be already 
uncontended/single-threaded it could be simplified avoiding the 
`AtomicReference` usage.


---

[GitHub] activemq-artemis pull request #1918: ARTEMIS-1722 Reduce pooled Netty ByteBu...