Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?

[Axel Voitier]

The use case is true (although I don't feel it personally so far). But if
its implementation is too complicated in its current form maybe it could be
done in another way.

Like a kind of muxing-proxy socket that can connect to any kind of sockets
on one side (on ipc or inproc most likely are they are the local sockets
one wants to gather I guess), and to another demuxing-proxy socket on the
other side. It could actually be only one socket type that can both do
muxing and demuxing.
It would have to somehow tag each message coming from one socket on one
side to be mapped to the right socket on the other side. A bit like we can
do with a router socket, but that can work with any other socket types.


Axel

2016-02-10 14:14 GMT+01:00 Pieter Hintjens :

> There was something nice about writing the resource as a path, yet as
> no-one has implemented it, in any language, I'm probably going to
> remove it from the 3.1 RFC.
>
> On Wed, Feb 10, 2016 at 8:49 AM, Tom Quarendon
>  wrote:
> > Yes, I see the issue now.
> > So when you bind it will have to check whether you've already bound to
> something on the same endpoint and if so ensure the security configuration
> is the same, as it has to evaluate the security before it can uncover the
> resource to which the connection it being routed.
> >
> > It introduces an ambiguity in the XRAP URI syntax. So at the moment you
> have {transport}://{endpoint}/{xrap_resource}. You can write a nice cURL
> style command line tool that does generic XRAP operations. Once ZMTP
> resources are added, the URI becomes
> {transport}://{endpoint}/{zmtp_resource}/{xrap_resource}, and this becomes
> impossible to parse as you don't know where the zmtp resource name ends and
> the xrap resource name starts.
> >
> >
> > -Original Message-
> > From: zeromq-dev-boun...@lists.zeromq.org [mailto:
> zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter Hintjens
> > Sent: 09 February 2016 15:30
> > To: ZeroMQ development list 
> > Subject: Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?
> >
> > Resources act like an internal proxy with a single front-facing socket
> (with security) and multiple back-end sockets. It's not a trivial change to
> libzmq, which is why we've not made it yet.
> >
> > On Tue, Feb 9, 2016 at 4:20 PM, Tom Quarendon <
> tom.quaren...@teamwpc.co.uk> wrote:
> >> You're right, I hadn't noticed that I was looking at ZMTP 3.1 rather
> than 3.0. I think I'd read the "resources" section ages ago and thought
> recently, yes, you can do that with zmq, that's exactly what I want.
> >> They don't look trivial to implement, since you presumably have to
> exhibit the same behaviour of connecting to something what hasn't been
> bound to yet. I also didn't understand the section on "Since resource
> evaluation happens after security handshaking, all services that use the
> same endpoint SHALL use the same security credentials".  It's unclear
> whether the intention is that resources are multiplexed over a single TCP
> connection, or whether actually each resource has its own connection. The
> fact that it says that all services using the same endpoint use the same
> security credentials suggests that they are multiplexed over the same
> connection?
> >>
> >> Anyway. Sounds like I need a new design.
> >>
> >> -Original Message-
> >> From: zeromq-dev-boun...@lists.zeromq.org
> >> [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter
> >> Hintjens
> >> Sent: 09 February 2016 14:11
> >> To: ZeroMQ development list 
> >> Subject: Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?
> >>
> >> Resources are not implemented in libzmq master, and are specified in
> the ZMTP 3.1 draft.
> >>
> >> So the stable 4.1 release of libzmq supports the stable ZMTP 3.0 RFC.
> >> Master supports other new pieces of 3.1 like heartbeating. If no-one
> volunteers to implement (or pay for) resources, they will eventually be
> moved to a further draft or dropped.
> >>
> >> The only way to tell what a given ZMTP stack supports is to test it
> from the outside and via its API, I think. We don't have such test tools
> yet. I'd like to make them at some point.
> >>
> >> -Pieter
> >>
> >> On Tue, Feb 9, 2016 at 12:39 PM, Tom Quarendon <
> tom.quaren...@teamwpc.co.uk> wrote:
> >>> I thought I’d just have an experiment with the “resource” facility in
> >>> ZMTP 3.0, but as far as I can tell there’s no implementation of it,
> >>> either on master, or in any of the releases.
> >>>
> >>> In general, how do I tell what version of ZMTP a release supports?
> >>>
> >>> Is the “resource” facility implemented anywhere? It was a rather
> >>> integral part of a design I’d been working on L
> >>>
> >>>
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> ___
> >>> zeromq-dev mailing list
> >>> zeromq-dev@lists.zeromq.org
> >>> 

Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?

[Pieter Hintjens]

Actually, taking your idea, we can use a single (thread safe) server
socket in a process that is shared by many threads. This kills the use
case.

On Wed, Feb 10, 2016 at 2:44 PM, Axel Voitier  wrote:
> The use case is true (although I don't feel it personally so far). But if
> its implementation is too complicated in its current form maybe it could be
> done in another way.
>
> Like a kind of muxing-proxy socket that can connect to any kind of sockets
> on one side (on ipc or inproc most likely are they are the local sockets one
> wants to gather I guess), and to another demuxing-proxy socket on the other
> side. It could actually be only one socket type that can both do muxing and
> demuxing.
> It would have to somehow tag each message coming from one socket on one side
> to be mapped to the right socket on the other side. A bit like we can do
> with a router socket, but that can work with any other socket types.
>
>
> Axel
>
> 2016-02-10 14:14 GMT+01:00 Pieter Hintjens :
>>
>> There was something nice about writing the resource as a path, yet as
>> no-one has implemented it, in any language, I'm probably going to
>> remove it from the 3.1 RFC.
>>
>> On Wed, Feb 10, 2016 at 8:49 AM, Tom Quarendon
>>  wrote:
>> > Yes, I see the issue now.
>> > So when you bind it will have to check whether you've already bound to
>> > something on the same endpoint and if so ensure the security configuration
>> > is the same, as it has to evaluate the security before it can uncover the
>> > resource to which the connection it being routed.
>> >
>> > It introduces an ambiguity in the XRAP URI syntax. So at the moment you
>> > have {transport}://{endpoint}/{xrap_resource}. You can write a nice cURL
>> > style command line tool that does generic XRAP operations. Once ZMTP
>> > resources are added, the URI becomes
>> > {transport}://{endpoint}/{zmtp_resource}/{xrap_resource}, and this becomes
>> > impossible to parse as you don't know where the zmtp resource name ends and
>> > the xrap resource name starts.
>> >
>> >
>> > -Original Message-
>> > From: zeromq-dev-boun...@lists.zeromq.org
>> > [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter Hintjens
>> > Sent: 09 February 2016 15:30
>> > To: ZeroMQ development list 
>> > Subject: Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?
>> >
>> > Resources act like an internal proxy with a single front-facing socket
>> > (with security) and multiple back-end sockets. It's not a trivial change to
>> > libzmq, which is why we've not made it yet.
>> >
>> > On Tue, Feb 9, 2016 at 4:20 PM, Tom Quarendon
>> >  wrote:
>> >> You're right, I hadn't noticed that I was looking at ZMTP 3.1 rather
>> >> than 3.0. I think I'd read the "resources" section ages ago and thought
>> >> recently, yes, you can do that with zmq, that's exactly what I want.
>> >> They don't look trivial to implement, since you presumably have to
>> >> exhibit the same behaviour of connecting to something what hasn't been 
>> >> bound
>> >> to yet. I also didn't understand the section on "Since resource evaluation
>> >> happens after security handshaking, all services that use the same 
>> >> endpoint
>> >> SHALL use the same security credentials".  It's unclear whether the
>> >> intention is that resources are multiplexed over a single TCP connection, 
>> >> or
>> >> whether actually each resource has its own connection. The fact that it 
>> >> says
>> >> that all services using the same endpoint use the same security 
>> >> credentials
>> >> suggests that they are multiplexed over the same connection?
>> >>
>> >> Anyway. Sounds like I need a new design.
>> >>
>> >> -Original Message-
>> >> From: zeromq-dev-boun...@lists.zeromq.org
>> >> [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter
>> >> Hintjens
>> >> Sent: 09 February 2016 14:11
>> >> To: ZeroMQ development list 
>> >> Subject: Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?
>> >>
>> >> Resources are not implemented in libzmq master, and are specified in
>> >> the ZMTP 3.1 draft.
>> >>
>> >> So the stable 4.1 release of libzmq supports the stable ZMTP 3.0 RFC.
>> >> Master supports other new pieces of 3.1 like heartbeating. If no-one
>> >> volunteers to implement (or pay for) resources, they will eventually be
>> >> moved to a further draft or dropped.
>> >>
>> >> The only way to tell what a given ZMTP stack supports is to test it
>> >> from the outside and via its API, I think. We don't have such test tools
>> >> yet. I'd like to make them at some point.
>> >>
>> >> -Pieter
>> >>
>> >> On Tue, Feb 9, 2016 at 12:39 PM, Tom Quarendon
>> >>  wrote:
>> >>> I thought I’d just have an experiment with the “resource” facility in
>> >>> ZMTP 3.0, but as far as I can tell there’s no implementation of it,
>> >>> either on master, or in any 

Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?

[Tom Quarendon]

My scenario was that I wanted to expose multiple services from a single port, 
which I think was the original intention of this. So I was imagining having a 
broker process binding , say, service1 to tcp://*:/service1 and service2 to 
tcp://*:/service2. The actual implementations of service1 and service2 
would then run either within the same process or in another process, and the 
broker just running a select loop to connect the relevant sockets up. 

You could do this at the application level but it would appear to require 
having a simple channel protocol such that each message that arrives at the 
broker has the service ID, so that the broker knows the correct destination 
(and ditto responses). This is essentially what SSH does with its channel 
mechanism (if you just take the transport layer and the connection layer, this 
is what you have, a secure multiplexed socket).

The trouble with doing it at the application level though is that you can't 
then use any of the other protocols (majordomo, lazy pirate, xrap etc) on top, 
at least not without writing the whole thing manually. The advantage of just 
being able to bind to "tcp://*:/service1" is that then any library that has 
been written to use zmq sockets, and just takes a connection string as 
parameter will continue to work.


-Original Message-
From: zeromq-dev-boun...@lists.zeromq.org 
[mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter Hintjens
Sent: 10 February 2016 14:10
To: ZeroMQ development list 
Subject: Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?

Actually, taking your idea, we can use a single (thread safe) server socket in 
a process that is shared by many threads. This kills the use case.

On Wed, Feb 10, 2016 at 2:44 PM, Axel Voitier  wrote:
> The use case is true (although I don't feel it personally so far). But 
> if its implementation is too complicated in its current form maybe it 
> could be done in another way.
>
> Like a kind of muxing-proxy socket that can connect to any kind of 
> sockets on one side (on ipc or inproc most likely are they are the 
> local sockets one wants to gather I guess), and to another 
> demuxing-proxy socket on the other side. It could actually be only one 
> socket type that can both do muxing and demuxing.
> It would have to somehow tag each message coming from one socket on 
> one side to be mapped to the right socket on the other side. A bit 
> like we can do with a router socket, but that can work with any other socket 
> types.
>
>
> Axel
>
> 2016-02-10 14:14 GMT+01:00 Pieter Hintjens :
>>
>> There was something nice about writing the resource as a path, yet as 
>> no-one has implemented it, in any language, I'm probably going to 
>> remove it from the 3.1 RFC.
>>
>> On Wed, Feb 10, 2016 at 8:49 AM, Tom Quarendon 
>>  wrote:
>> > Yes, I see the issue now.
>> > So when you bind it will have to check whether you've already bound 
>> > to something on the same endpoint and if so ensure the security 
>> > configuration is the same, as it has to evaluate the security 
>> > before it can uncover the resource to which the connection it being routed.
>> >
>> > It introduces an ambiguity in the XRAP URI syntax. So at the moment 
>> > you have {transport}://{endpoint}/{xrap_resource}. You can write a 
>> > nice cURL style command line tool that does generic XRAP 
>> > operations. Once ZMTP resources are added, the URI becomes 
>> > {transport}://{endpoint}/{zmtp_resource}/{xrap_resource}, and this 
>> > becomes impossible to parse as you don't know where the zmtp 
>> > resource name ends and the xrap resource name starts.
>> >
>> >
>> > -Original Message-
>> > From: zeromq-dev-boun...@lists.zeromq.org
>> > [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter 
>> > Hintjens
>> > Sent: 09 February 2016 15:30
>> > To: ZeroMQ development list 
>> > Subject: Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?
>> >
>> > Resources act like an internal proxy with a single front-facing 
>> > socket (with security) and multiple back-end sockets. It's not a 
>> > trivial change to libzmq, which is why we've not made it yet.
>> >
>> > On Tue, Feb 9, 2016 at 4:20 PM, Tom Quarendon 
>> >  wrote:
>> >> You're right, I hadn't noticed that I was looking at ZMTP 3.1 
>> >> rather than 3.0. I think I'd read the "resources" section ages ago 
>> >> and thought recently, yes, you can do that with zmq, that's exactly what 
>> >> I want.
>> >> They don't look trivial to implement, since you presumably have to 
>> >> exhibit the same behaviour of connecting to something what hasn't 
>> >> been bound to yet. I also didn't understand the section on "Since 
>> >> resource evaluation happens after security handshaking, all 
>> >> services that use the same endpoint SHALL use the same security 
>> >> credentials".  

[zeromq-dev] Access to underlying Linux socket?

[Mark Gillott]

[I'm sure this must have been asked before, but can't find anything]

Is there a facility or method to "map" a 0MQ socket to its supporting
Linux socket?

Working with a Linux kernel that has some new networking facility
together with socket extensions to control the facility. With vanilla
sockets this results in calls to setsockopt(sockfd, SOL_SOCKET, ...).

But how do I manage 0MQ sockets wanting to enable/control this facility?
Can I obtain the Linux socket and thus use native setsockopt() calls? Or
is the only option to generate (& maintain) zmq_set_xyz() & zmq_xyz()
patches for the base ZMQ library (together with the various language
binding - C, Python, Perl, etc)?

Cheers,

Mark

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] Multiplexing a TCP endpoint

[Alex Bligh]

I am investigating building a zeromq-based application which runs over TCP. It 
will run many separate services over zeromq, mostly REP/REQ (via proxies), but 
also PUB/SUB.

The normal way to do this as far as I can tell is to use one TCP port per 
service. I want to avoid doing this, as the services can be dynamically created 
and deleted and I don't want to be bothered with different firewall rules, port 
management etc.

For inproc communication that's fine, as I can specify a unique name. However, 
the TCP endpoint does not have this. Only one socket can bind to a tcp port.

What I'd really like to do is simply to multiplex (by service name) multiple 
services (i.e. multiple sockets) onto the same TCP port. They would then pass 
like shadows in the night. I would like this to transparently work with 
encryption too.

I think this would be no harder than adding an additional frame (the service 
name) on send, and stripping it on receive. However, I can't immediately see 
how to do this using standard Router / Dealer code (partly because I'd have to 
somehow route the messages to the correct endpoints that had 'connected'), 
though I may be missing something. In many ways it would seem to be easier to 
modify the endpoint protocol.

Am I missing something? Am I approaching this the right way?

-- 
Alex Bligh




___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Jerry Scharf]

Mark,

As a comment from the peanut gallery, it is generally a bad thing to 
leak that information up the protocol stack. Rather than getting that 
information back, it might be better to be able to attach user callout 
hooks at the places where you would want to make changes. That way there 
is still a unified code base and you can write and install your hook 
code to do your special work. It also allows people to do other things 
without needing to invent another mechanism.

That's the way I have liked it the best when these things came up with 
other systems.

jerry

On 02/10/2016 07:50 AM, Mark Gillott wrote:
> [I'm sure this must have been asked before, but can't find anything]
>
> Is there a facility or method to "map" a 0MQ socket to its supporting
> Linux socket?
>
> Working with a Linux kernel that has some new networking facility
> together with socket extensions to control the facility. With vanilla
> sockets this results in calls to setsockopt(sockfd, SOL_SOCKET, ...).
>
> But how do I manage 0MQ sockets wanting to enable/control this facility?
> Can I obtain the Linux socket and thus use native setsockopt() calls? Or
> is the only option to generate (& maintain) zmq_set_xyz() & zmq_xyz()
> patches for the base ZMQ library (together with the various language
> binding - C, Python, Perl, etc)?
>
> Cheers,
>
> Mark
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Pieter Hintjens]

You can't do this really, since one ZeroMQ socket can map to 0..n
system sockets.

There is a new option on libzmq master that lets you pre-configure a
FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).

A custom hook to configure new sockets is a nice idea.

-Pieter

On Wed, Feb 10, 2016 at 6:29 PM, Jerry Scharf
 wrote:
> Mark,
>
> As a comment from the peanut gallery, it is generally a bad thing to
> leak that information up the protocol stack. Rather than getting that
> information back, it might be better to be able to attach user callout
> hooks at the places where you would want to make changes. That way there
> is still a unified code base and you can write and install your hook
> code to do your special work. It also allows people to do other things
> without needing to invent another mechanism.
>
> That's the way I have liked it the best when these things came up with
> other systems.
>
> jerry
>
> On 02/10/2016 07:50 AM, Mark Gillott wrote:
>> [I'm sure this must have been asked before, but can't find anything]
>>
>> Is there a facility or method to "map" a 0MQ socket to its supporting
>> Linux socket?
>>
>> Working with a Linux kernel that has some new networking facility
>> together with socket extensions to control the facility. With vanilla
>> sockets this results in calls to setsockopt(sockfd, SOL_SOCKET, ...).
>>
>> But how do I manage 0MQ sockets wanting to enable/control this facility?
>> Can I obtain the Linux socket and thus use native setsockopt() calls? Or
>> is the only option to generate (& maintain) zmq_set_xyz() & zmq_xyz()
>> patches for the base ZMQ library (together with the various language
>> binding - C, Python, Perl, etc)?
>>
>> Cheers,
>>
>> Mark
>>
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Multiplexing a TCP endpoint

[Tom Quarendon]

This is what "resources" in ZMTP 3.1 were designed for. No implementation yet 
though (see other discussion on his list) :-)

You would have to write the proxy loop yourself. So do a select on the external 
socket and all the internal sockets and know that when you pull a message from 
the external socket that you peel off the first frame after the delimiter, then 
use the service name it includes to pick which internal socket to pass the rest 
of the message to. The services then run independently on their own inproc (or 
even out of process on ipc or tcp) sockets. It's what I would end up doing to 
do the same thing. 

Hope that helps.

-Original Message-
From: zeromq-dev-boun...@lists.zeromq.org 
[mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Alex Bligh
Sent: 10 February 2016 17:24
To: zeromq-dev@lists.zeromq.org
Cc: Alex Bligh 
Subject: [zeromq-dev] Multiplexing a TCP endpoint

I am investigating building a zeromq-based application which runs over TCP. It 
will run many separate services over zeromq, mostly REP/REQ (via proxies), but 
also PUB/SUB.

The normal way to do this as far as I can tell is to use one TCP port per 
service. I want to avoid doing this, as the services can be dynamically created 
and deleted and I don't want to be bothered with different firewall rules, port 
management etc.

For inproc communication that's fine, as I can specify a unique name. However, 
the TCP endpoint does not have this. Only one socket can bind to a tcp port.

What I'd really like to do is simply to multiplex (by service name) multiple 
services (i.e. multiple sockets) onto the same TCP port. They would then pass 
like shadows in the night. I would like this to transparently work with 
encryption too.

I think this would be no harder than adding an additional frame (the service 
name) on send, and stripping it on receive. However, I can't immediately see 
how to do this using standard Router / Dealer code (partly because I'd have to 
somehow route the messages to the correct endpoints that had 'connected'), 
though I may be missing something. In many ways it would seem to be easier to 
modify the endpoint protocol.

Am I missing something? Am I approaching this the right way?

-- 
Alex Bligh




___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Multiplexing a TCP endpoint

[Michal Vyskocil]

Hi,

In case you want to play even more, there is malamute broker. It can listen
in tcp socket and you can filter incoming messages per stream or subject.

The disadvantage is you must use mlm client API to make it work. However
sounds like right tool for your use case.
Dne 10. 2. 2016 8:12 PM napsal uživatel "Tom Quarendon" <
tom.quaren...@teamwpc.co.uk>:

> This is what "resources" in ZMTP 3.1 were designed for. No implementation
> yet though (see other discussion on his list) :-)
>
> You would have to write the proxy loop yourself. So do a select on the
> external socket and all the internal sockets and know that when you pull a
> message from the external socket that you peel off the first frame after
> the delimiter, then use the service name it includes to pick which internal
> socket to pass the rest of the message to. The services then run
> independently on their own inproc (or even out of process on ipc or tcp)
> sockets. It's what I would end up doing to do the same thing.
>
> Hope that helps.
>
> -Original Message-
> From: zeromq-dev-boun...@lists.zeromq.org [mailto:
> zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Alex Bligh
> Sent: 10 February 2016 17:24
> To: zeromq-dev@lists.zeromq.org
> Cc: Alex Bligh 
> Subject: [zeromq-dev] Multiplexing a TCP endpoint
>
> I am investigating building a zeromq-based application which runs over
> TCP. It will run many separate services over zeromq, mostly REP/REQ (via
> proxies), but also PUB/SUB.
>
> The normal way to do this as far as I can tell is to use one TCP port per
> service. I want to avoid doing this, as the services can be dynamically
> created and deleted and I don't want to be bothered with different firewall
> rules, port management etc.
>
> For inproc communication that's fine, as I can specify a unique name.
> However, the TCP endpoint does not have this. Only one socket can bind to a
> tcp port.
>
> What I'd really like to do is simply to multiplex (by service name)
> multiple services (i.e. multiple sockets) onto the same TCP port. They
> would then pass like shadows in the night. I would like this to
> transparently work with encryption too.
>
> I think this would be no harder than adding an additional frame (the
> service name) on send, and stripping it on receive. However, I can't
> immediately see how to do this using standard Router / Dealer code (partly
> because I'd have to somehow route the messages to the correct endpoints
> that had 'connected'), though I may be missing something. In many ways it
> would seem to be easier to modify the endpoint protocol.
>
> Am I missing something? Am I approaching this the right way?
>
> --
> Alex Bligh
>
>
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Mark Gillott]

On Wed, 2016-02-10 at 20:45 +0100, Pieter Hintjens wrote:
> You can't do this really, since one ZeroMQ socket can map to 0..n
> system sockets.
> 

Had a feeling that was going to be the case.

> There is a new option on libzmq master that lets you pre-configure a
> FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).
> 

Care to expand a bit more? Is there something (test code? source module)
you can point me at?

> A custom hook to configure new sockets is a nice idea.
> 

So you register a hook with a newly created socket and you would get a
callback just prior to any bind/connect. Is that the idea? In czmq
terms, something like:

s = zsock_new(ZMQ_REP)
zsock_configure(s, myfunc, myarg)
zsock_bind(s)

The callback would be provided with the base socket? 

Thanks,

Mark

> -Pieter
> 
> On Wed, Feb 10, 2016 at 6:29 PM, Jerry Scharf
>  wrote:
> > Mark,
> >
> > As a comment from the peanut gallery, it is generally a bad thing to
> > leak that information up the protocol stack. Rather than getting that
> > information back, it might be better to be able to attach user callout
> > hooks at the places where you would want to make changes. That way there
> > is still a unified code base and you can write and install your hook
> > code to do your special work. It also allows people to do other things
> > without needing to invent another mechanism.
> >
> > That's the way I have liked it the best when these things came up with
> > other systems.
> >
> > jerry
> >
> > On 02/10/2016 07:50 AM, Mark Gillott wrote:
> >> [I'm sure this must have been asked before, but can't find anything]
> >>
> >> Is there a facility or method to "map" a 0MQ socket to its supporting
> >> Linux socket?
> >>
> >> Working with a Linux kernel that has some new networking facility
> >> together with socket extensions to control the facility. With vanilla
> >> sockets this results in calls to setsockopt(sockfd, SOL_SOCKET, ...).
> >>
> >> But how do I manage 0MQ sockets wanting to enable/control this facility?
> >> Can I obtain the Linux socket and thus use native setsockopt() calls? Or
> >> is the only option to generate (& maintain) zmq_set_xyz() & zmq_xyz()
> >> patches for the base ZMQ library (together with the various language
> >> binding - C, Python, Perl, etc)?
> >>
> >> Cheers,
> >>
> >> Mark
> >>
> >> ___
> >> zeromq-dev mailing list
> >> zeromq-dev@lists.zeromq.org
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
> >>  
> >
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
> >  
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
>  

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Pieter Hintjens]

There is a test case (and a man page) on libzmq/master, search for
ZMQ_USE_FD in zmq_setsockopt.

On Wed, Feb 10, 2016 at 9:38 PM, Mark Gillott  wrote:
> On Wed, 2016-02-10 at 20:45 +0100, Pieter Hintjens wrote:
>> You can't do this really, since one ZeroMQ socket can map to 0..n
>> system sockets.
>>
>
> Had a feeling that was going to be the case.
>
>> There is a new option on libzmq master that lets you pre-configure a
>> FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).
>>
>
> Care to expand a bit more? Is there something (test code? source module)
> you can point me at?
>
>> A custom hook to configure new sockets is a nice idea.
>>
>
> So you register a hook with a newly created socket and you would get a
> callback just prior to any bind/connect. Is that the idea? In czmq
> terms, something like:
>
> s = zsock_new(ZMQ_REP)
> zsock_configure(s, myfunc, myarg)
> zsock_bind(s)
>
> The callback would be provided with the base socket?
>
> Thanks,
>
> Mark
>
>> -Pieter
>>
>> On Wed, Feb 10, 2016 at 6:29 PM, Jerry Scharf
>>  wrote:
>> > Mark,
>> >
>> > As a comment from the peanut gallery, it is generally a bad thing to
>> > leak that information up the protocol stack. Rather than getting that
>> > information back, it might be better to be able to attach user callout
>> > hooks at the places where you would want to make changes. That way there
>> > is still a unified code base and you can write and install your hook
>> > code to do your special work. It also allows people to do other things
>> > without needing to invent another mechanism.
>> >
>> > That's the way I have liked it the best when these things came up with
>> > other systems.
>> >
>> > jerry
>> >
>> > On 02/10/2016 07:50 AM, Mark Gillott wrote:
>> >> [I'm sure this must have been asked before, but can't find anything]
>> >>
>> >> Is there a facility or method to "map" a 0MQ socket to its supporting
>> >> Linux socket?
>> >>
>> >> Working with a Linux kernel that has some new networking facility
>> >> together with socket extensions to control the facility. With vanilla
>> >> sockets this results in calls to setsockopt(sockfd, SOL_SOCKET, ...).
>> >>
>> >> But how do I manage 0MQ sockets wanting to enable/control this facility?
>> >> Can I obtain the Linux socket and thus use native setsockopt() calls? Or
>> >> is the only option to generate (& maintain) zmq_set_xyz() & zmq_xyz()
>> >> patches for the base ZMQ library (together with the various language
>> >> binding - C, Python, Perl, etc)?
>> >>
>> >> Cheers,
>> >>
>> >> Mark
>> >>
>> >> ___
>> >> zeromq-dev mailing list
>> >> zeromq-dev@lists.zeromq.org
>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
>> >
>> > ___
>> > zeromq-dev mailing list
>> > zeromq-dev@lists.zeromq.org
>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] Defaulting to tweetnacl?

[Pieter Hintjens]

Hi all,

I'd like to start moving to tweetnacl as the default when building
libzmq. This means, no separate install of libsodium, and encryption
built in by default. We can still have a --with-libsodium and
--without-curve at configure time.

Does anyone have a problem with this? It will not change anything
significant in terms of performance nor interoperability. Just easier
builds.

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Multiplexing a TCP endpoint

[ozamiatin]

Hi, Alex

10.02.16 19:23, Alex Bligh пишет:
> I am investigating building a zeromq-based application which runs over TCP. 
> It will run many separate services over zeromq, mostly REP/REQ (via proxies), 
> but also PUB/SUB.
>
> The normal way to do this as far as I can tell is to use one TCP port per 
> service. I want to avoid doing this, as the services can be dynamically 
> created and deleted and I don't want to be bothered with different firewall 
> rules, port management etc.
Have you considered random port binding? You can specify ports range so 
that makes easier to configure the firewall. You will need some kind of 
a directory service (Redis is mostly used) to tell the clients (who 
connects) about the running services and their ports. The benefit of 
this approach is that you are not limited by only one host (in case of 
IPC you are) but can scale your services to any number of hosts.
>
> For inproc communication that's fine, as I can specify a unique name. 
> However, the TCP endpoint does not have this. Only one socket can bind to a 
> tcp port.
>
> What I'd really like to do is simply to multiplex (by service name) multiple 
> services (i.e. multiple sockets) onto the same TCP port. They would then pass 
> like shadows in the night. I would like this to transparently work with 
> encryption too.
>
> I think this would be no harder than adding an additional frame (the service 
> name) on send, and stripping it on receive. However, I can't immediately see 
> how to do this using standard Router / Dealer code (partly because I'd have 
> to somehow route the messages to the correct endpoints that had 'connected'), 
> though I may be missing something. In many ways it would seem to be easier to 
> modify the endpoint protocol.
>
> Am I missing something? Am I approaching this the right way?
>
The solution here is to have some kind of a proxy process which binds to 
a TCP port and then redirects messages to the target services by IPC 
(You can have the same in a thread and use INPROC as well). zmq_proxy 
device exists for such reason.

Thanks,
Oleksii
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?

[Pieter Hintjens]

There was something nice about writing the resource as a path, yet as
no-one has implemented it, in any language, I'm probably going to
remove it from the 3.1 RFC.

On Wed, Feb 10, 2016 at 8:49 AM, Tom Quarendon
 wrote:
> Yes, I see the issue now.
> So when you bind it will have to check whether you've already bound to 
> something on the same endpoint and if so ensure the security configuration is 
> the same, as it has to evaluate the security before it can uncover the 
> resource to which the connection it being routed.
>
> It introduces an ambiguity in the XRAP URI syntax. So at the moment you have 
> {transport}://{endpoint}/{xrap_resource}. You can write a nice cURL style 
> command line tool that does generic XRAP operations. Once ZMTP resources are 
> added, the URI becomes 
> {transport}://{endpoint}/{zmtp_resource}/{xrap_resource}, and this becomes 
> impossible to parse as you don't know where the zmtp resource name ends and 
> the xrap resource name starts.
>
>
> -Original Message-
> From: zeromq-dev-boun...@lists.zeromq.org 
> [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter Hintjens
> Sent: 09 February 2016 15:30
> To: ZeroMQ development list 
> Subject: Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?
>
> Resources act like an internal proxy with a single front-facing socket (with 
> security) and multiple back-end sockets. It's not a trivial change to libzmq, 
> which is why we've not made it yet.
>
> On Tue, Feb 9, 2016 at 4:20 PM, Tom Quarendon  
> wrote:
>> You're right, I hadn't noticed that I was looking at ZMTP 3.1 rather than 
>> 3.0. I think I'd read the "resources" section ages ago and thought recently, 
>> yes, you can do that with zmq, that's exactly what I want.
>> They don't look trivial to implement, since you presumably have to exhibit 
>> the same behaviour of connecting to something what hasn't been bound to yet. 
>> I also didn't understand the section on "Since resource evaluation happens 
>> after security handshaking, all services that use the same endpoint SHALL 
>> use the same security credentials".  It's unclear whether the intention is 
>> that resources are multiplexed over a single TCP connection, or whether 
>> actually each resource has its own connection. The fact that it says that 
>> all services using the same endpoint use the same security credentials 
>> suggests that they are multiplexed over the same connection?
>>
>> Anyway. Sounds like I need a new design.
>>
>> -Original Message-
>> From: zeromq-dev-boun...@lists.zeromq.org
>> [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter
>> Hintjens
>> Sent: 09 February 2016 14:11
>> To: ZeroMQ development list 
>> Subject: Re: [zeromq-dev] ZMTP 3.0 "Resources". Implemented?
>>
>> Resources are not implemented in libzmq master, and are specified in the 
>> ZMTP 3.1 draft.
>>
>> So the stable 4.1 release of libzmq supports the stable ZMTP 3.0 RFC.
>> Master supports other new pieces of 3.1 like heartbeating. If no-one 
>> volunteers to implement (or pay for) resources, they will eventually be 
>> moved to a further draft or dropped.
>>
>> The only way to tell what a given ZMTP stack supports is to test it from the 
>> outside and via its API, I think. We don't have such test tools yet. I'd 
>> like to make them at some point.
>>
>> -Pieter
>>
>> On Tue, Feb 9, 2016 at 12:39 PM, Tom Quarendon  
>> wrote:
>>> I thought I’d just have an experiment with the “resource” facility in
>>> ZMTP 3.0, but as far as I can tell there’s no implementation of it,
>>> either on master, or in any of the releases.
>>>
>>> In general, how do I tell what version of ZMTP a release supports?
>>>
>>> Is the “resource” facility implemented anywhere? It was a rather
>>> integral part of a design I’d been working on L
>>>
>>>
>>>
>>> Thanks.
>>>
>>>
>>> ___
>>> zeromq-dev mailing list
>>> zeromq-dev@lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Luca Boccassi]

On Feb 10, 2016 20:39, "Mark Gillott"  wrote:
>
> On Wed, 2016-02-10 at 20:45 +0100, Pieter Hintjens wrote:
> > You can't do this really, since one ZeroMQ socket can map to 0..n
> > system sockets.
> >
>
> Had a feeling that was going to be the case.
>
> > There is a new option on libzmq master that lets you pre-configure a
> > FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).
> >
>
> Care to expand a bit more? Is there something (test code? source module)
> you can point me at?

I can point you to the guy who developed that option. He sits a few meters
from you in the same office and he has an awesome beard. :-)

> > A custom hook to configure new sockets is a nice idea.
> >
>
> So you register a hook with a newly created socket and you would get a
> callback just prior to any bind/connect. Is that the idea? In czmq
> terms, something like:
>
> s = zsock_new(ZMQ_REP)
> zsock_configure(s, myfunc, myarg)
> zsock_bind(s)
>
> The callback would be provided with the base socket?

Nice. That could work because all actual system sockets are created after
bind/connect. And in CZMQ we could add a global zys switch that applies it
automatically to all sockets to make it more convenient.

But the problem is that such callback should be executed for all sockets,
be they TCP or IPC or inproc. So, there needs to be a way to discriminate
somehow, as what you can do with the FD varies wildly.

It could be left to the application developer to be careful, but I think
that's confusing and likely to cause troubles.

Kind regards,
Luca Boccassi
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Defaulting to tweetnacl?

[Luca Boccassi]

On Feb 10, 2016 22:41, "Pieter Hintjens"  wrote:
>
> Hi all,
>
> I'd like to start moving to tweetnacl as the default when building
> libzmq. This means, no separate install of libsodium, and encryption
> built in by default. We can still have a --with-libsodium and
> --without-curve at configure time.
>
> Does anyone have a problem with this? It will not change anything
> significant in terms of performance nor interoperability. Just easier
> builds.

Hi,

As long as libsodium remains supported I don't think it is a problem. Bear
in mind that distributions like Debian will most likely use libsodium,
because even though it is allowed, it is strongly discouraged to ship
statically linked binaries, and it makes the mainteiners life harder. See:
https://wiki.debian.org/StaticLinking

Kind regards,
Luca Boccassi
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev