Re: [qpid-proton-cpp] default_container does not implement thread safety at all?

[Andrew Stitcher]

On Tue, 2017-03-14 at 01:23 +0200, Fj wrote:
> ... (elided somewhat inflammatory comments)

> Am I missing something obvious, 

That's difficult to say (obvious is so subjective!)

The code you are looking at implements the multithreaded API, but is
not multithreaded itself, and yes it is unfortunate that it doesn't do
a thread safe inject, but C++03 has no threading capability and the st
code will compile with 03. Since it's important for us to minimise
dependencies for the proton libraries, this code can't do much better.

In the same code drop you will find multithreaded implementations of
the container API too - This is in the examples/cpp/mt directory, this
code relies on external thread capabilities.

Note that we use the threading abilities of the C++11 and later
standard library, so you will need to compile with a modern compiler to
get mt support.

Note that this does mean that if you want to support *any* variant of
multithreading with the C++ binding (either multiple worker threads for
the container, or running the container in a separate thread to the
rest of your application) you will need C++11 or later to compile the
proton C++ binding library. I don't consider this to be a real problem
at this point. C++11 is supported well enough for our purposes in all
popular C++ compilers, and has been for quite some time - I think over
5 years.

> or is the documentation there, and in
> other
> places, and the example, chemically pure wishful thinking, like, it
> would
> be pretty nice if we supported multithreading, and it would work in
> such
> and such ways then, but unfortunately we don't, as a matter of fact?
> 
> If so, maybe you gals and guys should fix the documentation, and not
> just
> with "experimental" but with "DOESN'T WORK AT ALL" prominent on the
> page?

Experimental means that the API is not stable, but is provided so you
can get something done, and hopefully let us know how it goes, and give
us feedback so we can stabilise the API so that it can be released as
no longer experimental.

All this code is being replaced as we speak with a proactor based
implementation of the container API; this should support multiple
worker threads in the container as long as you have C++11 or later. but
only a single thread with C++03.

> On more constructive notes:
> 
> 1) do people maybe use some custom containers, similar to the
> epoll_container in the examples (which doesn't support schedule()
> though, I
> have to point out)? If so, I much desire to see one.

Custom containers essentially go away in the latest version of the API.
Instead of suggesting the user might want to implement their own
container we are providing a proactor based container.

We are providing proactor implementations based in libuv, epoll (for
Linux), iocp (for Windows). It will be possible to write your own
proactor if necessary, but the ones we will provide cover a lot of the
usual implementation landscape.

> 
> 2) why do you attempt to bolt on your event_loop thing to Connection
> and
> below when the only thing that has the run() method is Container,
> therefore
> that's the scope that any worker thread would have? It's the
> Container that
> should have the inject() method. Underlying C API notwithstanding.

I think you may have a misunderstanding here. Although the run loops
are owned by the container, the unit of serialisation is the
connection. The "bolting on" is to give access from whichever object
you have to a thread safe context to inject work. Even if the inject
happens at the container level, it's convenient to be able to inject
from an event handler based on the object contained in the event,
rather than having to navigate up to the container and then injecting. 
You *must* specify the connection that is to serialised, as
*everything* happens per connection, so there is no value in running
code in the run loop without having some way to safely run code in the
connection context.

The container schedule API does run code only in the container context,
with no connection serialisation, but it's not safe to use to refer to
any connection (or lower ) state without injecting into the relevant
connection.

The "event_loop" concept is to indicate the unit of serialisation. In
theory this could be any level of the object tree. I agree that
"event_loop" is a somewhat confusing name to indicate the serialisation
unit, but we couldn't think of a better one - probably we need to make
the doc clearer.


> 
> 3) What's the best way forward if I really have to have a threadsafe
> container that I can inject some event into threadsafely:

For immediate trying the example mt container is probably your best
bet. In the near future the proactor based container is "the future"
(tm).

> 
>   3a) does the C API support thread safety? Like, the whole problem
> is
> touching the reactor's job queue or whatever it has with taking an
> appropriate lock, and it must take that same lock itself for it to
> work.

I don't think the C 

[RESULT] [VOTE] Release Qpid Python 1.36.0

[Justin Ross]

The vote has passed with 4 binding votes and 1 non-binding.

I'll setup the artifacts for distribution right away and update the website
once they are propagated to the mirrors.

Re: [VOTE] Release Qpid Python 1.36.0

[Justin Ross]

+1

I tested install and ran the test suite against recent qpidd master on
Fedora 25.

On Tue, Mar 7, 2017 at 10:24 AM, Robbie Gemmell 
wrote:

> On 6 March 2017 at 11:08, Robbie Gemmell  wrote:
> > On 4 March 2017 at 01:33, Justin Ross  wrote:
> >> Hi, everyone.  This our first release candidate for the Qpid Python
> 1.36.0
> >> release.  Please test in your environment, report what you find, and
> then
> >> vote to approve or reject the release.
> >>
> >> Source distribution:
> >> https://dist.apache.org/repos/dist/dev/qpid/python/1.36.0-rc1/
> >>
> >> Tag:
> >> https://git-wip-us.apache.org/repos/asf?p=qpid-python.git;a=
> shortlog;h=refs/tags/1.36.0-rc1
> >>
> >> Branch:
> >> https://git-wip-us.apache.org/repos/asf?p=qpid-python.git;a=
> shortlog;h=refs/heads/1.36.x
> >>
> >> Issues:
> >> https://issues.apache.org/jira/issues/?jql=fixVersion%
> 20%3D%20qpid-python-1.36.0%20AND%20project%20%3D%20QPID
> >
> > +1
> >
> > I tested the RC out as follows:
> >  - Verified the signature and checksum files (note the .sha changed to
> SHA-256).
> >  - Checked LICENCE+NOTICE files present, year updated in the latter.
> >  - Ran RAT on the extracted archive.
> >  - Did a user local install of the package.
> >  - Ran the hello example against Qpid Broker for Java 6.1.1.
> >
> > Robbie
>
> Based on discussion in thread "[DISCUSS] release checksum filename
> extension", I think maybe we should consider changing the .sha
> checksum to use SHA512 rather than SHA256.
>
> Robbie
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: Jira components

[Justin Ross]

Er, correction.  -legacy -> -legacy-jmx-console

On Tue, Mar 14, 2017 at 10:11 AM, Justin Ross  wrote:

> Here's the updated list of components.
>
> https://issues.apache.org/jira/browse/QPID/?selectedTab=
> com.atlassian.jira.jira-projects-plugin:components-panel
>
> On Mon, Mar 13, 2017 at 6:28 AM, Justin Ross 
> wrote:
>
>> There are some components that I'd like us to consider removing (close
>> all issues, unset component) or warning off users (prefix with "OBSOLETE"
>> or similar in the component description).
>>
>>   - WCF/C++ Client - 3 open issues - Qpid no longer offers this
>>   - JCA - 16 open issues - This is 0-10 based, we don't maintain it, and
>> it's no longer offered on our website
>>   - JMS AMQP 1.0 Client - 1 open issue - Not sure about this one -
>> potential for confusion with Qpid JMS
>>
>
> I added special labels for each case and then removed the component.
> Based on Rob's feedback, I also did this for the "Java Management : JMX
> Console" component.  I left the JCA component alone for now.
>
> The labels:
>
>  - legacy-amqp-1.0-jms-client
>  - legacy
>  - legacy-wcf-client
>
>

Re: Jira components

[Justin Ross]

On Mon, Mar 13, 2017 at 4:48 PM, Keith W  wrote:

> On 13 March 2017 at 15:06, Rob Godfrey  wrote:
> > On 13 March 2017 at 15:34, Robbie Gemmell 
> wrote:
> >
> >> On 13 March 2017 at 13:28, Justin Ross  wrote:
> >> > Hi, everyone.
> >> >
> >> > We have some components that we have historically shared across Java
> and
> >> > C++.  Because we've formally split these in the source tree, I'd like
> to
> >> > reflect that into our jira components.  I'd like to make the following
> >> name
> >> > changes.
> >> >
> >> >   Documentation -> Java Documentation
> >> >   ("C++ Documentation" now contains all C++-related doc issues.)
> >> >
> >> >   Build Tools -> C++ Build
> >> >   ("Java Build" now contains all Java-related build issues.)
> >> >
> >> > There are some components that I'd like us to consider removing (close
> >> all
> >> > issues, unset component) or warning off users (prefix with "OBSOLETE"
> or
> >> > similar in the component description).
> >> >
> >> >   - WCF/C++ Client - 3 open issues - Qpid no longer offers this
> >> >   - JCA - 16 open issues - This is 0-10 based, we don't maintain it,
> and
> >> > it's no longer offered on our website
> >>
> >
> > We still appear to be pushing the jca/ra jars/poms out to maven (and it
> is
> > still in the codebase).  Are we proposing to remove the component from
> the
> > codebase?  I'd not remove the JIRA component while we are still (although
> > not shouting about it) releasing the libraries.
>
> We are currently still publish them to Maven.   If we want to drop
> them, then I'd suggest now is moment to propose their removal.
> There are still a few tidy up actions related to them under QPID-7622
> which could them be dropped.
>

I think we should propose removing them.  I will raise this as its own
thread.


> >> I would probably also rename a related one you didn't mention, "Java
> >> Client", to make it clearer it is the AMQP 0-x JMS client.
> >> Particularly as it will be moving to an independent release named
> >> along those lines.
> >>
> >>
> > +1
>
> +1
>
> I'd suggest "qpid-jms-client-amqp-0-x".
>

Lowercase with hyphens is out of step with the prevailing style.  "JMS
Client (AMQP 0-x)" would match the patterns in use here:

https://issues.apache.org/jira/browse/QPID/?selectedTab=com.atlassian.jira.jira-projects-plugin:components-panel

Re: Jira components

[Justin Ross]

On Mon, Mar 13, 2017 at 8:06 AM, Rob Godfrey 
wrote:

> Given the restructure of the Java codebase, I'd suggest that maybe we'd
> want to cut down the Java components to simply Java Broker and AMQP 0-x JMS
> client and that all other components simply be turned into labels.
>

I have no objection, but I didn't feel quite that ambitious, so I left
things largely as they have been.


> There are also some components that have been long ago removed from the
> codebase, eg Java Management : JMX Console, and some where it's unclear
> what they are referring to (e.g. Java Tools ... which possibly now only
> refers to the QMF2 plugin which lives outwith the java broker/client
> trees).
>

I also left Java Tools in place.  I wasn't sure what to do with it.  As you
hint at, it has a wide assortment of things in it.

Re: Jira components

[Justin Ross]

Here's the updated list of components.

https://issues.apache.org/jira/browse/QPID/?selectedTab=com.atlassian.jira.jira-projects-plugin:components-panel

On Mon, Mar 13, 2017 at 6:28 AM, Justin Ross  wrote:

> There are some components that I'd like us to consider removing (close all
> issues, unset component) or warning off users (prefix with "OBSOLETE" or
> similar in the component description).
>
>   - WCF/C++ Client - 3 open issues - Qpid no longer offers this
>   - JCA - 16 open issues - This is 0-10 based, we don't maintain it, and
> it's no longer offered on our website
>   - JMS AMQP 1.0 Client - 1 open issue - Not sure about this one -
> potential for confusion with Qpid JMS
>

I added special labels for each case and then removed the component.  Based
on Rob's feedback, I also did this for the "Java Management : JMX Console"
component.  I left the JCA component alone for now.

The labels:

 - legacy-amqp-1.0-jms-client
 - legacy
 - legacy-wcf-client

Re: [Java Broker] Port 0 range

[Adel Boutros]

Hello Rob,


I think I wasn't clear enough. Sorry for that.


As referenced here[1], there are registered ports which are dynamic ports 
however they identify a know service (5672 is one of them).

What I am talking about are private dynamic ports (ephemeral ports) which are 
not registered and to be used internally.


As this is not a requirement useful for all users of the Broker, I was 
wondering if there was a way to specify a certain port range for the broker to 
get an available port from it and which is a lot more restrictive than the full 
dynamic range.


[1]: https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers



From: Rob Godfrey 
Sent: Tuesday, March 14, 2017 5:25:56 PM
To: users@qpid.apache.org
Subject: Re: [Java Broker] Port 0 range

So, the Broker is simply using the Java mechanism... and the Java mechanism
is (I presume) just obeying the settings in your operating system.  Which
operating system are you seeing the broker pick "low" ports on, and how is
that operating system configured with respect to the "dynamic" port
range[1]?  What sort of "low" numbers are you getting... which operating
system are you seeing this on... and does the port number you are seeing
lay outside the OS settings for dynamic port assignment?

-- Rob

[1] According to this (
http://stackoverflow.com/questions/913501/how-to-let-kernel-choose-a-port-number-in-the-range-1024-5000-in-tcp-socket-pr)
StackOverflow question, the following commands can be used to get the
operating system settings:

Linux:
  cat /proc/sys/net/ipv4/ip_local_port_range
Windows:
  netsh int ipv4 show dynamicport tcp
OS X:
  sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last
Solaris:
  /usr/sbin/ndd /dev/tcp tcp_smallest_anon_port tcp_largest_anon_port

On 14 March 2017 at 16:56, Adel Boutros  wrote:

> Hello,
>
>
> We are asked to deploy broker on random ports. So we thought about using
> Port 0 and let the Broker find available ports.
>
> This works as expected however we have a concern with the port range.
>
>
> It seems by default Java will take any port outside the well known ports
> and assign it. However, in large environments, services are requested to
> use private/dynamic ports (range 49152 to 65535 as specified by the
> Internet Assigned Numbers Authority).
>
>
> So I was wondering if there was a way to make the broker respect this port
> range when it is passed a port value of 0?
>
>
> Maybe allow the user to pass a property defining the range of ports
> available.
>
>
> What do you think?
>
>
> Regards,
>
> Adel
>

Re: [Java Broker] Port 0 range

[Rob Godfrey]

So, the Broker is simply using the Java mechanism... and the Java mechanism
is (I presume) just obeying the settings in your operating system.  Which
operating system are you seeing the broker pick "low" ports on, and how is
that operating system configured with respect to the "dynamic" port
range[1]?  What sort of "low" numbers are you getting... which operating
system are you seeing this on... and does the port number you are seeing
lay outside the OS settings for dynamic port assignment?

-- Rob

[1] According to this (
http://stackoverflow.com/questions/913501/how-to-let-kernel-choose-a-port-number-in-the-range-1024-5000-in-tcp-socket-pr)
StackOverflow question, the following commands can be used to get the
operating system settings:

Linux:
  cat /proc/sys/net/ipv4/ip_local_port_range
Windows:
  netsh int ipv4 show dynamicport tcp
OS X:
  sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last
Solaris:
  /usr/sbin/ndd /dev/tcp tcp_smallest_anon_port tcp_largest_anon_port

On 14 March 2017 at 16:56, Adel Boutros  wrote:

> Hello,
>
>
> We are asked to deploy broker on random ports. So we thought about using
> Port 0 and let the Broker find available ports.
>
> This works as expected however we have a concern with the port range.
>
>
> It seems by default Java will take any port outside the well known ports
> and assign it. However, in large environments, services are requested to
> use private/dynamic ports (range 49152 to 65535 as specified by the
> Internet Assigned Numbers Authority).
>
>
> So I was wondering if there was a way to make the broker respect this port
> range when it is passed a port value of 0?
>
>
> Maybe allow the user to pass a property defining the range of ports
> available.
>
>
> What do you think?
>
>
> Regards,
>
> Adel
>

[Java Broker] Port 0 range

[Adel Boutros]

Hello,


We are asked to deploy broker on random ports. So we thought about using Port 0 
and let the Broker find available ports.

This works as expected however we have a concern with the port range.


It seems by default Java will take any port outside the well known ports and 
assign it. However, in large environments, services are requested to use 
private/dynamic ports (range 49152 to 65535 as specified by the Internet 
Assigned Numbers Authority).


So I was wondering if there was a way to make the broker respect this port 
range when it is passed a port value of 0?


Maybe allow the user to pass a property defining the range of ports available.


What do you think?


Regards,

Adel

[Qpid-Proton C++] detecting only connection errors

[Rabih M]

Hello,

I am trying to implement a retry mechanism using proton. If the connection
between the amqp peers is down or the remote peer itself is down, i would
like to retry the current operation (send, receive...). A bit similar to
the failover Url in JMS...

Therefore i would like to capture only the errors related to connection
problems to trigger the retry.
If I implement on_transport_error(), will I get only the connection error
here? or will I receive other types of undesired errors too?

Best regards,
Rabih

[RESULT] [VOTE] Release Apache Qpid JMS 0.21.0

[Robbie Gemmell]

There were 4 binding +1 votes, with no other votes received. The vote
has passed.

I will add the archives to the dist release repo and release the maven
staging repo shortly. I will update the website tomorrow once the
artifacts have had time to sync to the mirrors and maven central.

Robbie

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org