Re: [zeromq-dev] Access to underlying Linux socket?

[Mark Gillott]

On Wed, 2016-02-10 at 23:33 +, Luca Boccassi wrote:
> On Feb 10, 2016 20:39, "Mark Gillott"  wrote:
> >
> > On Wed, 2016-02-10 at 20:45 +0100, Pieter Hintjens wrote:
> > > You can't do this really, since one ZeroMQ socket can map to 0..n
> > > system sockets.
> > >
> >
> > Had a feeling that was going to be the case.
> >
> > > There is a new option on libzmq master that lets you pre-configure
> a
> > > FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).
> > >
> >
> > Care to expand a bit more? Is there something (test code? source
> module)
> > you can point me at?
> 
> I can point you to the guy who developed that option. He sits a few
> meters from you in the same office and he has an awesome beard. :-)
> 

Is that the Italian guy who likes to add chicken to his pizza? :-).

> > > A custom hook to configure new sockets is a nice idea.
> > >
> >
> > So you register a hook with a newly created socket and you would get
> a
> > callback just prior to any bind/connect. Is that the idea? In czmq
> > terms, something like:
> >
> > s = zsock_new(ZMQ_REP)
> > zsock_configure(s, myfunc, myarg)
> > zsock_bind(s)
> >
> > The callback would be provided with the base socket?
> 
> Nice. That could work because all actual system sockets are created
> after bind/connect. And in CZMQ we could add a global zys switch that
> applies it automatically to all sockets to make it more convenient.
> 
> But the problem is that such callback should be executed for all
> sockets, be they TCP or IPC or inproc. So, there needs to be a way to
> discriminate somehow, as what you can do with the FD varies wildly.
> 
> It could be left to the application developer to be careful, but I
> think that's confusing and likely to cause troubles.
> 

Rather than a global callback, couldn't you attach the callback to
individual sockets?

Catch you later (if you're still talking to me!),

Mark

> Kind regards,
> Luca Boccassi
> 

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Doron Somech]

You can actually use monitoring for this. Event socket accepted. Maybe a
socket option that accept function that create the underlying socket.
On Feb 11, 2016 18:50, "Luca Boccassi"  wrote:

> On 11 February 2016 at 15:27, Doron Somech  wrote:
> > What is the specific option you need? Just make a socket option and apply
> > before connect. Like tcp keep alive and ipv6.
>
> Hi Doron,
>
> The problem is that, for the use case we have, we use it with an
> internal build of the Linux kernel, with additional options that are
> not merged upstream yet.
>
> We could simply maintain an internal patch, but we are thinking that
> if we can find a nice, general solution that could be useful for
> everyone, it would be better :-)
>
> Kind regards,
> Luca Boccassi
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Luca Boccassi]

On 11 February 2016 at 09:32, Mark Gillott  wrote:
> On Wed, 2016-02-10 at 23:33 +, Luca Boccassi wrote:
>> On Feb 10, 2016 20:39, "Mark Gillott"  wrote:
>> >
>> > On Wed, 2016-02-10 at 20:45 +0100, Pieter Hintjens wrote:
>> > > You can't do this really, since one ZeroMQ socket can map to 0..n
>> > > system sockets.
>> > >
>> >
>> > Had a feeling that was going to be the case.
>> >
>> > > There is a new option on libzmq master that lets you pre-configure
>> a
>> > > FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).
>> > >
>> >
>> > Care to expand a bit more? Is there something (test code? source
>> module)
>> > you can point me at?
>>
>> I can point you to the guy who developed that option. He sits a few
>> meters from you in the same office and he has an awesome beard. :-)
>>
>
> Is that the Italian guy who likes to add chicken to his pizza? :-).

Still better than deep fried mars bars!

>> > > A custom hook to configure new sockets is a nice idea.
>> > >
>> >
>> > So you register a hook with a newly created socket and you would get
>> a
>> > callback just prior to any bind/connect. Is that the idea? In czmq
>> > terms, something like:
>> >
>> > s = zsock_new(ZMQ_REP)
>> > zsock_configure(s, myfunc, myarg)
>> > zsock_bind(s)
>> >
>> > The callback would be provided with the base socket?
>>
>> Nice. That could work because all actual system sockets are created
>> after bind/connect. And in CZMQ we could add a global zys switch that
>> applies it automatically to all sockets to make it more convenient.
>>
>> But the problem is that such callback should be executed for all
>> sockets, be they TCP or IPC or inproc. So, there needs to be a way to
>> discriminate somehow, as what you can do with the FD varies wildly.
>>
>> It could be left to the application developer to be careful, but I
>> think that's confusing and likely to cause troubles.
>>
>
> Rather than a global callback, couldn't you attach the callback to
> individual sockets?
>
> Catch you later (if you're still talking to me!),

I propose both: the low-level library could have the basic feature of
per-socket callback, and the higher level binding in addition could
provide a global setting for convenience.

Kind regards,
Luca Boccassi
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Mark Gillott]

On Wed, 2016-02-10 at 23:33 +, Luca Boccassi wrote:
> > > There is a new option on libzmq master that lets you pre-configure
> a
> > > FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).
> > >
> >
> > Care to expand a bit more? Is there something (test code? source
> module)
> > you can point me at?
> 
> I can point you to the guy who developed that option. He sits a few
> meters from you in the same office and he has an awesome beard. :-)
> 

Many thanks Luca.

To close the loop on this. Luca & I talked and the ZMQ_USE_FD option
solves the issue for listener (bind) sockets. Unfortunately there needs
to be a different solution for when an application wants to set an
option prior to a ZeroMQ connect (need the option to be persistent
across "reconnects"). Kicking around a few ideas.

Mark
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Doron Somech]

What is the specific option you need? Just make a socket option and apply
before connect. Like tcp keep alive and ipv6.
On Feb 11, 2016 15:12, "Mark Gillott"  wrote:

> On Wed, 2016-02-10 at 23:33 +, Luca Boccassi wrote:
> > > > There is a new option on libzmq master that lets you pre-configure
> > a
> > > > FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).
> > > >
> > >
> > > Care to expand a bit more? Is there something (test code? source
> > module)
> > > you can point me at?
> >
> > I can point you to the guy who developed that option. He sits a few
> > meters from you in the same office and he has an awesome beard. :-)
> >
>
> Many thanks Luca.
>
> To close the loop on this. Luca & I talked and the ZMQ_USE_FD option
> solves the issue for listener (bind) sockets. Unfortunately there needs
> to be a different solution for when an application wants to set an
> option prior to a ZeroMQ connect (need the option to be persistent
> across "reconnects"). Kicking around a few ideas.
>
> Mark
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Luca Boccassi]

On 11 February 2016 at 15:27, Doron Somech  wrote:
> What is the specific option you need? Just make a socket option and apply
> before connect. Like tcp keep alive and ipv6.

Hi Doron,

The problem is that, for the use case we have, we use it with an
internal build of the Linux kernel, with additional options that are
not merged upstream yet.

We could simply maintain an internal patch, but we are thinking that
if we can find a nice, general solution that could be useful for
everyone, it would be better :-)

Kind regards,
Luca Boccassi
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Jerry Scharf]

Mark,

As a comment from the peanut gallery, it is generally a bad thing to 
leak that information up the protocol stack. Rather than getting that 
information back, it might be better to be able to attach user callout 
hooks at the places where you would want to make changes. That way there 
is still a unified code base and you can write and install your hook 
code to do your special work. It also allows people to do other things 
without needing to invent another mechanism.

That's the way I have liked it the best when these things came up with 
other systems.

jerry

On 02/10/2016 07:50 AM, Mark Gillott wrote:
> [I'm sure this must have been asked before, but can't find anything]
>
> Is there a facility or method to "map" a 0MQ socket to its supporting
> Linux socket?
>
> Working with a Linux kernel that has some new networking facility
> together with socket extensions to control the facility. With vanilla
> sockets this results in calls to setsockopt(sockfd, SOL_SOCKET, ...).
>
> But how do I manage 0MQ sockets wanting to enable/control this facility?
> Can I obtain the Linux socket and thus use native setsockopt() calls? Or
> is the only option to generate (& maintain) zmq_set_xyz() & zmq_xyz()
> patches for the base ZMQ library (together with the various language
> binding - C, Python, Perl, etc)?
>
> Cheers,
>
> Mark
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Pieter Hintjens]

You can't do this really, since one ZeroMQ socket can map to 0..n
system sockets.

There is a new option on libzmq master that lets you pre-configure a
FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).

A custom hook to configure new sockets is a nice idea.

-Pieter

On Wed, Feb 10, 2016 at 6:29 PM, Jerry Scharf
 wrote:
> Mark,
>
> As a comment from the peanut gallery, it is generally a bad thing to
> leak that information up the protocol stack. Rather than getting that
> information back, it might be better to be able to attach user callout
> hooks at the places where you would want to make changes. That way there
> is still a unified code base and you can write and install your hook
> code to do your special work. It also allows people to do other things
> without needing to invent another mechanism.
>
> That's the way I have liked it the best when these things came up with
> other systems.
>
> jerry
>
> On 02/10/2016 07:50 AM, Mark Gillott wrote:
>> [I'm sure this must have been asked before, but can't find anything]
>>
>> Is there a facility or method to "map" a 0MQ socket to its supporting
>> Linux socket?
>>
>> Working with a Linux kernel that has some new networking facility
>> together with socket extensions to control the facility. With vanilla
>> sockets this results in calls to setsockopt(sockfd, SOL_SOCKET, ...).
>>
>> But how do I manage 0MQ sockets wanting to enable/control this facility?
>> Can I obtain the Linux socket and thus use native setsockopt() calls? Or
>> is the only option to generate (& maintain) zmq_set_xyz() & zmq_xyz()
>> patches for the base ZMQ library (together with the various language
>> binding - C, Python, Perl, etc)?
>>
>> Cheers,
>>
>> Mark
>>
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Mark Gillott]

On Wed, 2016-02-10 at 20:45 +0100, Pieter Hintjens wrote:
> You can't do this really, since one ZeroMQ socket can map to 0..n
> system sockets.
> 

Had a feeling that was going to be the case.

> There is a new option on libzmq master that lets you pre-configure a
> FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).
> 

Care to expand a bit more? Is there something (test code? source module)
you can point me at?

> A custom hook to configure new sockets is a nice idea.
> 

So you register a hook with a newly created socket and you would get a
callback just prior to any bind/connect. Is that the idea? In czmq
terms, something like:

s = zsock_new(ZMQ_REP)
zsock_configure(s, myfunc, myarg)
zsock_bind(s)

The callback would be provided with the base socket? 

Thanks,

Mark

> -Pieter
> 
> On Wed, Feb 10, 2016 at 6:29 PM, Jerry Scharf
>  wrote:
> > Mark,
> >
> > As a comment from the peanut gallery, it is generally a bad thing to
> > leak that information up the protocol stack. Rather than getting that
> > information back, it might be better to be able to attach user callout
> > hooks at the places where you would want to make changes. That way there
> > is still a unified code base and you can write and install your hook
> > code to do your special work. It also allows people to do other things
> > without needing to invent another mechanism.
> >
> > That's the way I have liked it the best when these things came up with
> > other systems.
> >
> > jerry
> >
> > On 02/10/2016 07:50 AM, Mark Gillott wrote:
> >> [I'm sure this must have been asked before, but can't find anything]
> >>
> >> Is there a facility or method to "map" a 0MQ socket to its supporting
> >> Linux socket?
> >>
> >> Working with a Linux kernel that has some new networking facility
> >> together with socket extensions to control the facility. With vanilla
> >> sockets this results in calls to setsockopt(sockfd, SOL_SOCKET, ...).
> >>
> >> But how do I manage 0MQ sockets wanting to enable/control this facility?
> >> Can I obtain the Linux socket and thus use native setsockopt() calls? Or
> >> is the only option to generate (& maintain) zmq_set_xyz() & zmq_xyz()
> >> patches for the base ZMQ library (together with the various language
> >> binding - C, Python, Perl, etc)?
> >>
> >> Cheers,
> >>
> >> Mark
> >>
> >> ___
> >> zeromq-dev mailing list
> >> zeromq-dev@lists.zeromq.org
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
> >>  
> >
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
> >  
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
>  

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Pieter Hintjens]

There is a test case (and a man page) on libzmq/master, search for
ZMQ_USE_FD in zmq_setsockopt.

On Wed, Feb 10, 2016 at 9:38 PM, Mark Gillott  wrote:
> On Wed, 2016-02-10 at 20:45 +0100, Pieter Hintjens wrote:
>> You can't do this really, since one ZeroMQ socket can map to 0..n
>> system sockets.
>>
>
> Had a feeling that was going to be the case.
>
>> There is a new option on libzmq master that lets you pre-configure a
>> FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).
>>
>
> Care to expand a bit more? Is there something (test code? source module)
> you can point me at?
>
>> A custom hook to configure new sockets is a nice idea.
>>
>
> So you register a hook with a newly created socket and you would get a
> callback just prior to any bind/connect. Is that the idea? In czmq
> terms, something like:
>
> s = zsock_new(ZMQ_REP)
> zsock_configure(s, myfunc, myarg)
> zsock_bind(s)
>
> The callback would be provided with the base socket?
>
> Thanks,
>
> Mark
>
>> -Pieter
>>
>> On Wed, Feb 10, 2016 at 6:29 PM, Jerry Scharf
>>  wrote:
>> > Mark,
>> >
>> > As a comment from the peanut gallery, it is generally a bad thing to
>> > leak that information up the protocol stack. Rather than getting that
>> > information back, it might be better to be able to attach user callout
>> > hooks at the places where you would want to make changes. That way there
>> > is still a unified code base and you can write and install your hook
>> > code to do your special work. It also allows people to do other things
>> > without needing to invent another mechanism.
>> >
>> > That's the way I have liked it the best when these things came up with
>> > other systems.
>> >
>> > jerry
>> >
>> > On 02/10/2016 07:50 AM, Mark Gillott wrote:
>> >> [I'm sure this must have been asked before, but can't find anything]
>> >>
>> >> Is there a facility or method to "map" a 0MQ socket to its supporting
>> >> Linux socket?
>> >>
>> >> Working with a Linux kernel that has some new networking facility
>> >> together with socket extensions to control the facility. With vanilla
>> >> sockets this results in calls to setsockopt(sockfd, SOL_SOCKET, ...).
>> >>
>> >> But how do I manage 0MQ sockets wanting to enable/control this facility?
>> >> Can I obtain the Linux socket and thus use native setsockopt() calls? Or
>> >> is the only option to generate (& maintain) zmq_set_xyz() & zmq_xyz()
>> >> patches for the base ZMQ library (together with the various language
>> >> binding - C, Python, Perl, etc)?
>> >>
>> >> Cheers,
>> >>
>> >> Mark
>> >>
>> >> ___
>> >> zeromq-dev mailing list
>> >> zeromq-dev@lists.zeromq.org
>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
>> >
>> > ___
>> > zeromq-dev mailing list
>> > zeromq-dev@lists.zeromq.org
>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev=CwICAg=IL_XqQWOjubgfqINi2jTzg=jvQi-CKjLvh8eMz9WSgpXPemqlgP9vG7H0zwS3acfHk=0hDDdkf0P3ySnqh5QzyjSsxZ80Vc3GGXl0hFFQsq1SU=2Xu9Jr_yHb-drSVHhsw4TPO6LNFJdyYqqH6EkCnsNHo=
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Access to underlying Linux socket?

[Luca Boccassi]

On Feb 10, 2016 20:39, "Mark Gillott"  wrote:
>
> On Wed, 2016-02-10 at 20:45 +0100, Pieter Hintjens wrote:
> > You can't do this really, since one ZeroMQ socket can map to 0..n
> > system sockets.
> >
>
> Had a feeling that was going to be the case.
>
> > There is a new option on libzmq master that lets you pre-configure a
> > FD and give it to ZeroMQ to use for its first pipe. (ZMQ_USE_FD).
> >
>
> Care to expand a bit more? Is there something (test code? source module)
> you can point me at?

I can point you to the guy who developed that option. He sits a few meters
from you in the same office and he has an awesome beard. :-)

> > A custom hook to configure new sockets is a nice idea.
> >
>
> So you register a hook with a newly created socket and you would get a
> callback just prior to any bind/connect. Is that the idea? In czmq
> terms, something like:
>
> s = zsock_new(ZMQ_REP)
> zsock_configure(s, myfunc, myarg)
> zsock_bind(s)
>
> The callback would be provided with the base socket?

Nice. That could work because all actual system sockets are created after
bind/connect. And in CZMQ we could add a global zys switch that applies it
automatically to all sockets to make it more convenient.

But the problem is that such callback should be executed for all sockets,
be they TCP or IPC or inproc. So, there needs to be a way to discriminate
somehow, as what you can do with the FD varies wildly.

It could be left to the application developer to be careful, but I think
that's confusing and likely to cause troubles.

Kind regards,
Luca Boccassi
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev