Re: [zeromq-dev] Protocol Plugins
On Wed, Feb 12, 2014 at 3:05 AM, Lyle Thompson lthomp...@ixiacom.com wrote: I guess nobody is interested in pluggable protocols. Thanks anyway. Making pluggable protocols in the core library is just a very hard thing. It would require some major redesigns. People have added transports, and it's doable. They're not pluggable though, in the sense we usually mean. I built a simple framework for pluggable protocols over inproc bridges, some years ago: https://github.com/hintjens/vtx. It didn't get much attention. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Protocol Plugins
OK. Thanks for your feedback, Pieter. I think, then, that I will just do it the simple way. Regards, Lyle -Original Message- From: zeromq-dev-boun...@lists.zeromq.org [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter Hintjens Sent: Thursday, February 13, 2014 12:30 AM To: ZeroMQ development list Subject: Re: [zeromq-dev] Protocol Plugins On Wed, Feb 12, 2014 at 3:05 AM, Lyle Thompson lthomp...@ixiacom.com wrote: I guess nobody is interested in pluggable protocols. Thanks anyway. Making pluggable protocols in the core library is just a very hard thing. It would require some major redesigns. People have added transports, and it's doable. They're not pluggable though, in the sense we usually mean. I built a simple framework for pluggable protocols over inproc bridges, some years ago: https://github.com/hintjens/vtx. It didn't get much attention. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Protocol Plugins
I think it's a worthwhile goal. In fact, I think a lot of aspects of libzmq which are currently hard-coded should be made more modular. There are two issues: 1) It's a big task, as mentioned, and someone just has to do it. 2) As a big task, it's also a big stability risk. There's a greater potential to introduce bugs than there would be with smaller changes. On Thu, Feb 13, 2014 at 12:34 PM, Lyle Thompson lthomp...@ixiacom.comwrote: OK. Thanks for your feedback, Pieter. I think, then, that I will just do it the simple way. Regards, Lyle -Original Message- From: zeromq-dev-boun...@lists.zeromq.org [mailto: zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter Hintjens Sent: Thursday, February 13, 2014 12:30 AM To: ZeroMQ development list Subject: Re: [zeromq-dev] Protocol Plugins On Wed, Feb 12, 2014 at 3:05 AM, Lyle Thompson lthomp...@ixiacom.com wrote: I guess nobody is interested in pluggable protocols. Thanks anyway. Making pluggable protocols in the core library is just a very hard thing. It would require some major redesigns. People have added transports, and it's doable. They're not pluggable though, in the sense we usually mean. I built a simple framework for pluggable protocols over inproc bridges, some years ago: https://github.com/hintjens/vtx. It didn't get much attention. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Protocol Plugins
Well, large changes can always be grown incrementally. If we try a plugin architecture there's no reason to try to replace what we have now with new code. It can grow alongside it, without risk, then if/when it's mature, we use it. On Thu, Feb 13, 2014 at 7:45 PM, Lindley French lindl...@gmail.com wrote: I think it's a worthwhile goal. In fact, I think a lot of aspects of libzmq which are currently hard-coded should be made more modular. There are two issues: 1) It's a big task, as mentioned, and someone just has to do it. 2) As a big task, it's also a big stability risk. There's a greater potential to introduce bugs than there would be with smaller changes. On Thu, Feb 13, 2014 at 12:34 PM, Lyle Thompson lthomp...@ixiacom.com wrote: OK. Thanks for your feedback, Pieter. I think, then, that I will just do it the simple way. Regards, Lyle -Original Message- From: zeromq-dev-boun...@lists.zeromq.org [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Pieter Hintjens Sent: Thursday, February 13, 2014 12:30 AM To: ZeroMQ development list Subject: Re: [zeromq-dev] Protocol Plugins On Wed, Feb 12, 2014 at 3:05 AM, Lyle Thompson lthomp...@ixiacom.com wrote: I guess nobody is interested in pluggable protocols. Thanks anyway. Making pluggable protocols in the core library is just a very hard thing. It would require some major redesigns. People have added transports, and it's doable. They're not pluggable though, in the sense we usually mean. I built a simple framework for pluggable protocols over inproc bridges, some years ago: https://github.com/hintjens/vtx. It didn't get much attention. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Protocol Plugins
you can do this quite easily I believe with existing patterns. one auth server takes in requests. that server checks the request and if it is valid forwards the request to the outside world, sending back the response. here is the RFC for socks5: http://www.ietf.org/rfc/rfc1928.txt I assume you only want to connect to TCP/IP, not UPD? On Wed, Feb 12, 2014 at 3:05 AM, Lyle Thompson lthomp...@ixiacom.comwrote: I guess nobody is interested in pluggable protocols. Thanks anyway. *From:* zeromq-dev-boun...@lists.zeromq.org [mailto: zeromq-dev-boun...@lists.zeromq.org] *On Behalf Of *Lyle Thompson *Sent:* Monday, February 10, 2014 3:11 PM *To:* ZeroMQ development list *Subject:* [zeromq-dev] Protocol Plugins Hi ZeroMQ experts, I am investigating how one might add SOCKS proxy support to 0mq, and of course there are several options. But I was thinking that the most elegant would be to add support for a socks protocol to 0mq. I took a look at the source code, and I was a bit surprised to find that the protocols are not pluggable. At the first level, I don't think this would be very hard to do (at least for **some** of the protocols); Just add a registration mechanism and change the code that is executed as the result of hard-coded protocol checks into calls through a table of available plugins. This would make it a bit easier to add new protocols (especially if they are similar or running on top of existing ones, as is the case for SOCKS). Presumably, one would just need to compile the module and link it into the code via the make file. Of course it would be nicer if it were dynamically linked (i.e. the second level), but I don't know of a platform independent way to do this that would make sense for 0mq. Of course it could be done with #ifdefs and what not, but would it be worth it? My motivation is that my company uses 0mq in a commercial product. And while we don't mind contributing, we'd rather contribute a general improvement than a very specific one (SOCKS), that many people would not need or want. I know this is a very high-level idea at this point, so blast away and let's see if there's anything left that worth implementing J Thanks, Lyle ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Protocol Plugins
P.S. as I understand the tao of 0mq, the idea is to get rid of all these idiosyncratic patterns. for example in the socks5 RFC you have this, see below. the document was written in 1996. now its 2014, so it should be possible to do this in better ways, unless you have to talk some other server which only speaks that language (in that case you can write a mapper). The values currently defined for METHOD are: o X'00' NO AUTHENTICATION REQUIRED o X'01' GSSAPI o X'02' USERNAME/PASSWORD o X'03' to X'7F' IANA ASSIGNED o X'80' to X'FE' RESERVED FOR PRIVATE METHODS o X'FF' NO ACCEPTABLE METHODS On Wed, Feb 12, 2014 at 9:01 AM, Benjamin Cordes benjamin.l.cor...@gmail.com wrote: you can do this quite easily I believe with existing patterns. one auth server takes in requests. that server checks the request and if it is valid forwards the request to the outside world, sending back the response. here is the RFC for socks5: http://www.ietf.org/rfc/rfc1928.txt I assume you only want to connect to TCP/IP, not UPD? On Wed, Feb 12, 2014 at 3:05 AM, Lyle Thompson lthomp...@ixiacom.comwrote: I guess nobody is interested in pluggable protocols. Thanks anyway. *From:* zeromq-dev-boun...@lists.zeromq.org [mailto: zeromq-dev-boun...@lists.zeromq.org] *On Behalf Of *Lyle Thompson *Sent:* Monday, February 10, 2014 3:11 PM *To:* ZeroMQ development list *Subject:* [zeromq-dev] Protocol Plugins Hi ZeroMQ experts, I am investigating how one might add SOCKS proxy support to 0mq, and of course there are several options. But I was thinking that the most elegant would be to add support for a socks protocol to 0mq. I took a look at the source code, and I was a bit surprised to find that the protocols are not pluggable. At the first level, I don't think this would be very hard to do (at least for **some** of the protocols); Just add a registration mechanism and change the code that is executed as the result of hard-coded protocol checks into calls through a table of available plugins. This would make it a bit easier to add new protocols (especially if they are similar or running on top of existing ones, as is the case for SOCKS). Presumably, one would just need to compile the module and link it into the code via the make file. Of course it would be nicer if it were dynamically linked (i.e. the second level), but I don't know of a platform independent way to do this that would make sense for 0mq. Of course it could be done with #ifdefs and what not, but would it be worth it? My motivation is that my company uses 0mq in a commercial product. And while we don't mind contributing, we'd rather contribute a general improvement than a very specific one (SOCKS), that many people would not need or want. I know this is a very high-level idea at this point, so blast away and let's see if there's anything left that worth implementing J Thanks, Lyle ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Protocol Plugins
Hi Guys,
Thanks for your responses. I definitely appreciate it. But maybe I said SOCKS
too early in the email :) The main idea was to make protocols pluggable (i.e.
introduce protocol drivers) so the main code base wouldn't be burdened with
odd protocols that few people use anymore (like SOCKS), and 0mq wouldn't have
to be modified to support them. This looks pretty easy to do at link time, but
I would be more interested to do a run-time binding. I don't think there is a
generic dll/so framework that would make sense to use within 0mq, so I'm
thinking about introducing an api, something like (this is just pseudo code
because I'm not on the machine where I checked out the code, and I'm too lazy
to look up the specific details):
ECODE addProtocol(ZMQ_CONTEXT context, IProtocol Descriptor *pProtoDescr);
Where IProtocol would be an interface/struct with something like:
char* getProtocolId();
char* getProtocolDisplayName();
IConnectHandler connectHandler;
IDisconnectHandler disconnectHandler;
IReadHandler readHandler;
IWriteHandler writeHandler;
IErrorHandler errorHandler;
Then, wherever the current code uses if (protocol == xxx') { do something; }
it would instead do something like:
IProtocolDescriptor protoDescr = context.protoRegistry.getProtocol(protocol);
If (protoDescr == null) {
Throw exception;
}
protoDescr.connectHandler.connect(); // or whatever it was about to do.
Of course the handler could be cached so the lookup wouldn't have to happen
every time (maybe even cached on the context).
With this, the user (programmer) would then just have to write a protocol
handler (e.g. SOCKS plugin) and call addProtocol after initializing 0mq and
he'd be off and running without having to modify the 0mq code base.
Thanks!
Lyle
From: zeromq-dev-boun...@lists.zeromq.org
[mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Benjamin Cordes
Sent: Wednesday, February 12, 2014 12:08 AM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Protocol Plugins
P.S. as I understand the tao of 0mq, the idea is to get rid of all these
idiosyncratic patterns. for example in the socks5 RFC you have this, see below.
the document was written in 1996. now its 2014, so it should be possible to do
this in better ways, unless you have to talk some other server which only
speaks that language (in that case you can write a mapper).
The values currently defined for METHOD are:
o X'00' NO AUTHENTICATION REQUIRED
o X'01' GSSAPI
o X'02' USERNAME/PASSWORD
o X'03' to X'7F' IANA ASSIGNED
o X'80' to X'FE' RESERVED FOR PRIVATE METHODS
o X'FF' NO ACCEPTABLE METHODS
On Wed, Feb 12, 2014 at 9:01 AM, Benjamin Cordes
benjamin.l.cor...@gmail.commailto:benjamin.l.cor...@gmail.com wrote:
you can do this quite easily I believe with existing patterns. one auth server
takes in requests. that server checks the request and if it is valid forwards
the request to the outside world, sending back the response. here is the RFC
for socks5: http://www.ietf.org/rfc/rfc1928.txt I assume you only want to
connect to TCP/IP, not UPD?
On Wed, Feb 12, 2014 at 3:05 AM, Lyle Thompson
lthomp...@ixiacom.commailto:lthomp...@ixiacom.com wrote:
I guess nobody is interested in pluggable protocols. Thanks anyway.
From:
zeromq-dev-boun...@lists.zeromq.orgmailto:zeromq-dev-boun...@lists.zeromq.org
[mailto:zeromq-dev-boun...@lists.zeromq.orgmailto:zeromq-dev-boun...@lists.zeromq.org]
On Behalf Of Lyle Thompson
Sent: Monday, February 10, 2014 3:11 PM
To: ZeroMQ development list
Subject: [zeromq-dev] Protocol Plugins
Hi ZeroMQ experts,
I am investigating how one might add SOCKS proxy support to 0mq, and of course
there are several options. But I was thinking that the most elegant would be to
add support for a socks protocol to 0mq. I took a look at the source code,
and I was a bit surprised to find that the protocols are not pluggable. At the
first level, I don't think this would be very hard to do (at least for *some*
of the protocols); Just add a registration mechanism and change the code that
is executed as the result of hard-coded protocol checks into calls through a
table of available plugins.
This would make it a bit easier to add new protocols (especially if they are
similar or running on top of existing ones, as is the case for SOCKS).
Presumably, one would just need to compile the module and link it into the code
via the make file.
Of course it would be nicer if it were dynamically linked (i.e. the second
level), but I don't know of a platform independent way to do this that would
make sense for 0mq. Of course it could be done with #ifdefs and what not, but
would it be worth it?
My motivation is that my company uses 0mq in a commercial product. And while we
don't mind contributing, we'd rather contribute a general improvement than a
very specific one (SOCKS), that many people would not need or want.
I know this is a very high-level idea
Re: [zeromq-dev] Protocol Plugins
I very much like this idea and would like to see this in. If a proper spec
is made, I'll likely be adding this to NetMQ.
On Wed, Feb 12, 2014 at 6:03 PM, Lyle Thompson lthomp...@ixiacom.comwrote:
Hi Guys,
Thanks for your responses. I definitely appreciate it. But maybe I said
SOCKS too early in the email J The main idea was to make protocols
pluggable (i.e. introduce protocol drivers) so the main code base
wouldn't be burdened with odd protocols that few people use anymore (like
SOCKS), and 0mq wouldn't have to be modified to support them. This looks
pretty easy to do at link time, but I would be more interested to do a
run-time binding. I don't think there is a generic dll/so framework that
would make sense to use within 0mq, so I'm thinking about introducing an
api, something like (this is just pseudo code because I'm not on the
machine where I checked out the code, and I'm too lazy to look up the
specific details):
ECODE addProtocol(ZMQ_CONTEXT context, IProtocol Descriptor *pProtoDescr);
Where IProtocol would be an interface/struct with something like:
char* getProtocolId();
char* getProtocolDisplayName();
IConnectHandler connectHandler;
IDisconnectHandler disconnectHandler;
IReadHandler readHandler;
IWriteHandler writeHandler;
IErrorHandler errorHandler;
Then, wherever the current code uses if (protocol == xxx') { do
something; } it would instead do something like:
IProtocolDescriptor protoDescr =
context.protoRegistry.getProtocol(protocol);
If (protoDescr == null) {
Throw exception;
}
protoDescr.connectHandler.connect(); // or whatever it was about to do.
Of course the handler could be cached so the lookup wouldn't have to
happen every time (maybe even cached on the context).
With this, the user (programmer) would then just have to write a protocol
handler (e.g. SOCKS plugin) and call addProtocol after initializing 0mq and
he'd be off and running without having to modify the 0mq code base.
Thanks!
Lyle
*From:* zeromq-dev-boun...@lists.zeromq.org [mailto:
zeromq-dev-boun...@lists.zeromq.org] *On Behalf Of *Benjamin Cordes
*Sent:* Wednesday, February 12, 2014 12:08 AM
*To:* ZeroMQ development list
*Subject:* Re: [zeromq-dev] Protocol Plugins
P.S. as I understand the tao of 0mq, the idea is to get rid of all these
idiosyncratic patterns. for example in the socks5 RFC you have this, see
below. the document was written in 1996. now its 2014, so it should be
possible to do this in better ways, unless you have to talk some other
server which only speaks that language (in that case you can write a
mapper).
The values currently defined for METHOD are:
o X'00' NO AUTHENTICATION REQUIRED
o X'01' GSSAPI
o X'02' USERNAME/PASSWORD
o X'03' to X'7F' IANA ASSIGNED
o X'80' to X'FE' RESERVED FOR PRIVATE METHODS
o X'FF' NO ACCEPTABLE METHODS
On Wed, Feb 12, 2014 at 9:01 AM, Benjamin Cordes
benjamin.l.cor...@gmail.com wrote:
you can do this quite easily I believe with existing patterns. one auth
server takes in requests. that server checks the request and if it is valid
forwards the request to the outside world, sending back the response. here
is the RFC for socks5: http://www.ietf.org/rfc/rfc1928.txt I assume you
only want to connect to TCP/IP, not UPD?
On Wed, Feb 12, 2014 at 3:05 AM, Lyle Thompson lthomp...@ixiacom.com
wrote:
I guess nobody is interested in pluggable protocols. Thanks anyway.
*From:* zeromq-dev-boun...@lists.zeromq.org [mailto:
zeromq-dev-boun...@lists.zeromq.org] *On Behalf Of *Lyle Thompson
*Sent:* Monday, February 10, 2014 3:11 PM
*To:* ZeroMQ development list
*Subject:* [zeromq-dev] Protocol Plugins
Hi ZeroMQ experts,
I am investigating how one might add SOCKS proxy support to 0mq, and of
course there are several options. But I was thinking that the most elegant
would be to add support for a socks protocol to 0mq. I took a look at the
source code, and I was a bit surprised to find that the protocols are not
pluggable. At the first level, I don't think this would be very hard to do
(at least for **some** of the protocols); Just add a registration
mechanism and change the code that is executed as the result of hard-coded
protocol checks into calls through a table of available plugins.
This would make it a bit easier to add new protocols (especially if they
are similar or running on top of existing ones, as is the case for SOCKS).
Presumably, one would just need to compile the module and link it into the
code via the make file.
Of course it would be nicer if it were dynamically linked (i.e. the
second level), but I don't know of a platform independent way to do this
that would make sense for 0mq. Of course it could be done with #ifdefs and
what not, but would it be worth it?
My motivation is that my company uses 0mq in a commercial product. And
while we don't
Re: [zeromq-dev] Protocol Plugins
I guess nobody is interested in pluggable protocols. Thanks anyway. From: zeromq-dev-boun...@lists.zeromq.org [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Lyle Thompson Sent: Monday, February 10, 2014 3:11 PM To: ZeroMQ development list Subject: [zeromq-dev] Protocol Plugins Hi ZeroMQ experts, I am investigating how one might add SOCKS proxy support to 0mq, and of course there are several options. But I was thinking that the most elegant would be to add support for a socks protocol to 0mq. I took a look at the source code, and I was a bit surprised to find that the protocols are not pluggable. At the first level, I don't think this would be very hard to do (at least for *some* of the protocols); Just add a registration mechanism and change the code that is executed as the result of hard-coded protocol checks into calls through a table of available plugins. This would make it a bit easier to add new protocols (especially if they are similar or running on top of existing ones, as is the case for SOCKS). Presumably, one would just need to compile the module and link it into the code via the make file. Of course it would be nicer if it were dynamically linked (i.e. the second level), but I don't know of a platform independent way to do this that would make sense for 0mq. Of course it could be done with #ifdefs and what not, but would it be worth it? My motivation is that my company uses 0mq in a commercial product. And while we don't mind contributing, we'd rather contribute a general improvement than a very specific one (SOCKS), that many people would not need or want. I know this is a very high-level idea at this point, so blast away and let's see if there's anything left that worth implementing :) Thanks, Lyle ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
