Re: Compiling proton-c for Linux Alpine

2017-11-23 Thread Alan Conway 
On Thu, Nov 23, 2017 at 5:48 AM, Chris Richardson  wrote:

> [snip: discussion of issues building on Linux Alpine]
>
> Another alternative is that we make a more official version of the source
> code hack I previously mentioned to make Proton compatible with the reduced
> (I'm not judging) musl libc. Any thoughts, Messrs. Stitcher, Conway et al?
>
>
In general I like the idea of making the core more portable and reducing
dependencies, of course the devil is in the details. If you can put forward
something that works for you as a JIRA  we can figure out how/if to adopt
the changes into the core.

I haven't followed the ins and outs but I have a couple comments:

> > tested it at all, but if glibc isn't an option you could change
> > the PTHREAD_MUTEX_ADAPTIVE_NP to PTHREAD_MUTEX_NORMAL. I _think_ the only
> > side effect would be that some sub-optimal system calls


I suspect we can produce a more portable version of epoll.c that uses the
more modern pthread stuff conditionally (or not at all) - Cliff is the
expert.

Another option is to use a different proactor - although of you have epoll
then making the epoll proactor work is probably the best option. We have
been considering whether or not we need a plain Unix poll() or select()
proactor but we are hoping the libuv proactor will fill that gap. I don't
know if libuv is an option for you, but since you want less dependencies
rather than more it probably isn't.

Cheers,
Alan.

Re: Java Broker performance

2017-11-23 Thread Robbie Gemmell 
Could you answer my question around confirming what client you are
using for the comparison below, the C++ one or the JMS one?

I'm guessing it is the C++ one. In which case, Rob's thoughts are the
likely explanation for why the Java broker isnt any faster with it
than you are seeing, and testing with the JMS client code given will
not be able to demonstrate that same difference between the two
brokers as it is doing a synchronous publish while the C++ code isnt.

Robbie

On 23 November 2017 at 15:33, tomas.soltys  wrote:
> Hi Keith,
>
> I'm still getting huge differences, but I still hope it is related to how I
> configured my brokers. Please find attached file  brokers.gz
>    containing
> setup of my two brokers (cpp and java). Both I tried to setup to be as
> similar as possible.
>
> With my test send tool I sent 1000 messages each 102400 bytes. Tool sends
> messages asynchronously and settle after each 100-th message.
>
> Output (2001 - cpp, 2002 - java):
>
> $ time amq_send.sh --host=localhost --port=20001 --user=BE --pass="BE"
> --node-name=broadcast --subject="broadcast.PublicRejectStream"
> --message-count=1000 --print-message=0 --message-size=102400
> --settle-rate=100
>   100 messages sent
>   200 messages sent
>   300 messages sent
>   400 messages sent
>   500 messages sent
>   600 messages sent
>   700 messages sent
>   800 messages sent
>   900 messages sent
>  1000 messages sent
> Time to send = 1 [seconds]
>
> real0m1.796s
> user0m0.125s
> sys 0m0.050s
>
> $ time amq_send.sh --host=localhost --port=20002 --user=BE --pass="BE"
> --node-name=broadcast --subject="broadcast.PublicRejectStream"
> --message-count=1000 --print-message=0 --message-size=102400
> --settle-rate=100
>   100 messages sent
>   200 messages sent
>   300 messages sent
>   400 messages sent
>   500 messages sent
>   600 messages sent
>   700 messages sent
>   800 messages sent
>   900 messages sent
>  1000 messages sent
> Time to send = 109 [seconds]
>
> real1m48.865s
> user0m1.504s
> sys 0m1.558s
>
> Best regards,
> Tomas
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

2017-11-23 Thread tomas.soltys 
Hi Keith,

I'm still getting huge differences, but I still hope it is related to how I
configured my brokers. Please find attached file  brokers.gz
   containing
setup of my two brokers (cpp and java). Both I tried to setup to be as
similar as possible.

With my test send tool I sent 1000 messages each 102400 bytes. Tool sends
messages asynchronously and settle after each 100-th message.

Output (2001 - cpp, 2002 - java):

$ time amq_send.sh --host=localhost --port=20001 --user=BE --pass="BE"
--node-name=broadcast --subject="broadcast.PublicRejectStream"
--message-count=1000 --print-message=0 --message-size=102400
--settle-rate=100
  100 messages sent
  200 messages sent
  300 messages sent
  400 messages sent
  500 messages sent
  600 messages sent
  700 messages sent
  800 messages sent
  900 messages sent
 1000 messages sent
Time to send = 1 [seconds]

real0m1.796s
user0m0.125s
sys 0m0.050s

$ time amq_send.sh --host=localhost --port=20002 --user=BE --pass="BE"
--node-name=broadcast --subject="broadcast.PublicRejectStream"
--message-count=1000 --print-message=0 --message-size=102400
--settle-rate=100
  100 messages sent
  200 messages sent
  300 messages sent
  400 messages sent
  500 messages sent
  600 messages sent
  700 messages sent
  800 messages sent
  900 messages sent
 1000 messages sent
Time to send = 109 [seconds]

real1m48.865s
user0m1.504s
sys 0m1.558s

Best regards,
Tomas



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: [VOTE] Release Apache Qpid Python 1.37.0

2017-11-23 Thread Jakub Scholz 
+1 ... I run my tests against different versions of Qpid C++ broker

On Wed, Nov 22, 2017 at 9:10 PM, Robbie Gemmell 
wrote:

> Hi folks,
>
> I have put together a spin for a Qpid Python 1.37.0 release, please
> give it a test out and vote accordingly.
>
> The source archive can be grabbed from:
> https://dist.apache.org/repos/dist/dev/qpid/python/1.37.0-rc1/
>
> The JIRAs currently assigned are:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12310520&version=12339780
>
> It is tagged as 1.37.0-rc1.
>
> Regards,
> Robbie
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: [VOTE] Release Apache Qpid Python 1.37.0

2017-11-23 Thread Gordon Sim 

On 22/11/17 20:10, Robbie Gemmell wrote:

Hi folks,

I have put together a spin for a Qpid Python 1.37.0 release, please
give it a test out and vote accordingly.

The source archive can be grabbed from:
https://dist.apache.org/repos/dist/dev/qpid/python/1.37.0-rc1/

The JIRAs currently assigned are:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310520&version=12339780

It is tagged as 1.37.0-rc1.


+1 (verified signature and checksum, installed and ran tests from 
installation against the c++ broker).


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Proton 0.18.1 - libuv.c

2017-11-23 Thread Venkat B 
Good day Roddie,

Thanks for your reply & the pointers to the information.

Regards,
Venkat

On Thu, Nov 23, 2017 at 6:42 AM, Roddie Kieley  wrote:

> Hi Venkat, there are indeed issues on Mac OS X with the mutex
> initialization and
> destruction as you indicate. A pull request with the required fixes,
> including the
> update below is located here [1] which was generated from the work on
> PROTON-522 and commented upon at [2].
>
> While the changes have not made it to master yet, there is now an issue
> logged,
> PROTON-1684, documenting the problem at [3] with a target fix version of
> 0.19.0.
>
>
> Roddie
> ---
> [1] -
> https://github.com/apache/qpid-proton/pull/129/commits/
> 63e2a052e5ebf9b29ea64f30f952fd56adb9b9d1#diff-
> 68220cdc99750102b47b42d421079707
> [2] -
> https://issues.apache.org/jira/browse/PROTON-522?
> focusedCommentId=16171590&page=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-16171590
> [3] - https://issues.apache.org/jira/browse/PROTON-1694
>
> On Wed, Nov 22, 2017 at 1:00 AM, Venkat B  wrote:
>
> > Good day,
> >
> > In the proactor implementation,  uv_mutex_lock is called without first
> > initializing the mutex with uv_mutex_init.  This works fine on linux but
> on
> > Mac OS X, the uv_mutex_lock fails & calls abort().
> > Below patch fixes the issue, I will appreciate if same can be applied.
> >
> > Regards,
> > Venkat
> >
> > diff -ru qpid-proton-0.18.1_new/proton-c/src/proactor/libuv.c
> > qpid-proton-0.18.1/proton-c/src/proactor/libuv.c
> > --- qpid-proton-0.18.1_new/proton-c/src/proactor/libuv.c2017-10-31
> > 18:16:02.0 +0530
> > +++ qpid-proton-0.18.1/proton-c/src/proactor/libuv.c2017-11-20
> > 13:08:32.0 +0530
> > @@ -305,6 +305,7 @@
> >work_init(&pc->work, p,  T_CONNECTION);
> >pc->next = pconnection_unqueued;
> >pc->write.data = &pc->work;
> > +  uv_mutex_init(&pc->lock);
> >if (server) {
> >  pn_transport_set_server(pc->driver.transport);
> >}
> >
>

Re: [VOTE] Release Apache Qpid Python 1.37.0

2017-11-23 Thread Robbie Gemmell 
On 22 November 2017 at 20:10, Robbie Gemmell  wrote:
> Hi folks,
>
> I have put together a spin for a Qpid Python 1.37.0 release, please
> give it a test out and vote accordingly.
>
> The source archive can be grabbed from:
> https://dist.apache.org/repos/dist/dev/qpid/python/1.37.0-rc1/
>
> The JIRAs currently assigned are:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310520&version=12339780
>
> It is tagged as 1.37.0-rc1.
>
> Regards,
> Robbie

+1

I checked things over as follows:
- Verified the signature and checksum files.
- Checked LICENCE+NOTICE files present.
- Did a user local install of the package.
- Ran the hello example against Broker-J 7.0.0 and 1.37.0 RC1 C++ broker.
- Ran the tests against the 1.37.0 RC1 C++ broker, and Broker-J 7.0.0
with the 0-10 excludes file from its master.

Robbie

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Compiling proton-c for Linux Alpine

2017-11-23 Thread Chris Richardson 
On 21 November 2017 at 17:41, Alessio Gottardo 
wrote:

>  Thank you Chris,
>
> I've been reading about this new world (to me) of the reasons behind glibc
> and ulibc. In some way there are similar issues in "toy languages / higher
> level languages" like java or javascript (node.js) when shipping the
> equivalent of executables with tons of dependencies (and transitive
> dependencies) making the artefact huge in terms of disk space.
>
> Anyway, I've added the following to the docker file (using RUN statements)
> in order to include the binary package of glibc to Linux Alpine (cf.
> https://github.com/sgerrand/alpine-pkg-glibc#installing):
>
>
> apk --no-cache add ca-certificates wget
> wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://raw.githubusercontent.
> com/sgerrand/alpine-pkg-glibc/master/sgerrand.rsa.pub
> wget https://github.com/sgerrand/alpine-pkg-glibc/releases/
> download/2.25-r0/glibc-2.25-r0.apk
> apk add glibc-2.25-r0.apk
>
> However I get the same compilation error.
>
> I believe that's because somehow I need to point the cmake/make stuff to
> glibc (instead of the default/system ulibc) hardcoding somewhere the path?
> If that's the case then I am not sure where I should change this stuff and
> which path I should add.
> Please let me know if this rings any bell.
>

I've followed these installation steps and now realise that the glibc
package above is runtime only and does not support local compilation - were
it to do so it would probably be called "glibc-dev" by convention. This
means that you can only run packages that were compiled elsewhere, which is
presumably the case for JDKs etc. In retrospect an obvious choice when the
target is a minimalist OS so apologies for leading you down a bit of a
blind alley.

In terms of progressing further I would suggest you might look into
extending a build system such as sgerrand's (the one that generates the
glibc.apk package) in order to either build a glibc-dev package (this will
probably result in a bloated Alpine image) or to build qpid-proton and
export it as a .apk file. The latter might be the most satisfactory
solution since you would then not require many of the build dependencies
you already have (git, cmake, gcc etc) in order to install qpid-proton in
the live environment. The down-side is that it's quite and involved process
and certainly not one for the faint of heart!

Another alternative is that we make a more official version of the source
code hack I previously mentioned to make Proton compatible with the reduced
(I'm not judging) musl libc. Any thoughts, Messrs. Stitcher, Conway et al?

Thanks
> Alessio
>
>
>
>
>
> On Monday, 20 November 2017, 19:34:50 GMT, Chris Richardson <
> c...@fourc.eu> wrote:
>
>  Hi Alessio,
> The problem here appears to be that qpid-proton assumes glibc is installed
> on your system (it is on "most"(tm) linuces), but alpine doesn't appear to
> contain it, nor yet is it available in the default apk repositories. The
> pthread-stub package which provides /usr/include/pthread.h does not include
> the expected PTHREAD_MUTEX_ADAPTIVE_NP definition.
> The easiest option might be to use a docker image which does include
> glibc, try one of the documented solutions here http://wiki.alpinelinux.
> org/wiki/Running_glibc_programs or here https://github.com/sgerrand.
> This is a bit of a hack and comes with a huge disclaimer as I haven't
> tested it at all, but if glibc isn't an option you could change
> the PTHREAD_MUTEX_ADAPTIVE_NP to PTHREAD_MUTEX_NORMAL. I _think_ the only
> side effect would be that some sub-optimal system calls might take place so
> the thread locking would not be as efficient.I've attached a patch to do
> this which you can apply withpatch -p1 < qpid-proton-0.18.0-pthread-
> stub.patch
> I hope this helps to point you in the right direction.
> Chris
>
> On 20 November 2017 at 17:11, Alessio Gottardo 
> wrote:
>
> Hi there,
> I am trying to compile Qpid Proton on Linux Alpine (within a docker image).
> I am getting this error:
> /go/src/qpid-proton/proton-c/ src/proactor/epoll.c: In function
> 'pmutex_init':/go/src/qpid- proton/proton-c/src/proactor/ epoll.c:104:36:
> error: 'PTHREAD_MUTEX_ADAPTIVE_NP' undeclared (first use in this function)
>  pthread_mutexattr_settype(& attr, PTHREAD_MUTEX_ADAPTIVE_NP);
>   ^/go/
> src/qpid-proton/proton-c/src/ proactor/epoll.c:104:36: note: each
> undeclared identifier is reported only once for each function it appears
> inmake[2]: *** [proton-c/CMakeFiles/qpid- proton.dir/build.make:695:
> proton-c/CMakeFiles/qpid- proton.dir/src/proactor/epoll. c.o] Error
> 1make[1]: *** [CMakeFiles/Makefile2:1069: proton-c/CMakeFiles/qpid-
> proton.dir/all] Error 2make: *** [Makefile:141: all] Error 2
> The relevant part of the Dockerfile (FROM golang:1.8.3-alpine3.6):
> ```# download Apache QPIDWORKDIR /go/src/RUN git clone --progress
> --verbose http://git.apache.org/qpid- proton.git
> # dependencies for Apache QPID (check