subject:"Re\: \[zeromq\-dev\] Notes from a hackathon"

On Thu, Feb 5, 2015 at 8:41 PM, AJ Lewis aj.le...@quantum.com wrote:

 Is the plan really to drop multi-hop routing?

I did want, and am glad to get, stories from people using multihop routing.

I think the situation here is that we're using the multihop
request-reply pattern, it depends on multipart, and that's fine. It's
a standard, it's implemented, and it's stable.

So we can focus on new patterns (like client/server) quite separately,
and if we make threadsafe sockets, perhaps use them only in these new
cases.

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread MinRK

If I recall correctly, libzmq-3.0 (or was it the old 4.0 experimental
branch?) preserved multi-hop while separating routing from content by
making the routing id a sequence rather than a single value. That way, it
seems that the same push/pop behavior that happens on a ROUTER-DEALER
device today could still work on CLIENT-SERVER. The push/pop would be
happening on the routing stack instead of message content.

If a message were something like:

{
   frame_t frames[];
   int nframes;
   routing_id_t route[];
   int nroutes;
}

it would seem that you could have multi-part content (selfishly, in the way
that affects me - messages composed of non-contiguous memory), multi-hop
routing, and send it all with a single `zmq_send_newfangled` call, allowing
future attempts at making `zmq_send_newfangled` a threadsafe call.

There may be reasons this would be super gross and horrible, but it's an
idea, anyway.

-MinRK


On Fri, Feb 6, 2015 at 9:02 AM, Thomas Rodgers rodg...@twrodgers.com
wrote:

 Adding a mutex, even one that is never contended, to the socket will
 essentially triple this (one atomic CAS to acquire the mutex, one atomic
 CAS to put the message on the pipe, one atomic CAS to release the mutex).


 This is a bit of a blue sky view of the cost of acquiring a mutex. For
 the adventurous of spirit, chase down the call path of pthread_mutex
 sometime in GLIBC. It is substantially more involved than a single pair of
 'lock; cmpxchg' instructions, but it tries really hard to make that the
 rough cost of the happy path.

 On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 Having thought about this for a couple of more days, I want to at least
 take a stab at arguing against threadsafe sockets -

 libzmq's thread safety guarantees, to me anyway, are very clear,
 unsurprising and non-controversial - I cannot share a socket with another
 thread without a full fence.

 The kinds of systems I generally build have very strict requirements on
 overall latency, to the point that most of my networking IO is done through
 kernel-bypass libraries and NICs that support this, for raw TCP and UDP
 multicast. The latency sensitive code that does IO is in it's own thread,
 with exclusive access to the NICs which are accessed via kernel bypass.
 Coordination with other threads in the same process is done via inproc pair
 sockets. Pair sockets + very small messages (small enough that libzmq does
 not need to perform allocation) provide a very nice interface to a lock
 free queue with low overhead using a single atomic CAS operation. Atomic
 operations are cheap, but they are not free (~30 clocks on x86). Adding a
 mutex, even one that is never contended, to the socket will essentially
 triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the
 message on the pipe, one atomic CAS to release the mutex). I would like to
 have the option to avoid this.

 If a wrapper wants thread safe sockets to enable certain use-cases that
 may be more idiomatic for the language in question, it can provide the full
 fence. AZMQ https://github.com/zeromq/azmq does exactly this by
 default, but you have the option to opt out of it. It does this because
 Boost Asio by default allows it's sockets to be used from multiple threads
 for async IO and I need to guard more than just exclusive access to the
 ZeroMQ socket a the fence in this case. Putting a mutex inside of the
 libzmq socket, essentially doubles the overhead for no gain in useful
 functionality and runs completely counter to one of C and C++'s overarching
 principles: don't pay for what you don't use.

 If a class of apps really demands short lived exclusive access to a
 socket, provide a pool abstraction. The pool is thread safe, obtain a
 socket, perform I/O in a single thread, return the socket to the pool.

 On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier 
 pelletier.mic...@gmail.com wrote:



 On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens p...@imatix.com wrote:

 The discussion about thread safety was quite short iirc, though that
 contributor did discuss other things... at length. I merged his
 thread safe socket change rapidly, then we reverted it after a few
 days, and he disappeared. It was rather brute force and I suspect did
 not work at all, it simply wrapped all accesses to the socket
 structure in mutexes. No discussion at the time of multipart data and
 atomic send/recv.


 My memory of the conversation at the time is pretty dim, I agree the
 changes were ugly and untested and the contributor was difficult to reason
 with and seemed to want to make the changes based on no real need at all.



 As for socket safety, I've no strong opinion. I see that many people
 expect that to work and hit errors when it doesn't. I see that nanomsg
 has threadsafe sockets and no multipart. I see that sharing sockets
 across threads would make some actor models simpler, which is nice.


 This is the classic problem with thread safe anything.

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread Michel Pelletier

Something else that occurred to me today, perhaps incorrectly, is that
there was mention that nanomsg doesn't have multipart messages, but it does
still preserve multipart messages in a way, but supporting scatter and
gather io vectors.  Perhaps emulating this well known API pattern is the
right approach to moving forward that still keep everyone happy and brings
us closer to the ability to have thread safety.

-Michel

On Fri, Feb 6, 2015 at 12:28 PM, MinRK benjami...@gmail.com wrote:

 If I recall correctly, libzmq-3.0 (or was it the old 4.0 experimental
 branch?) preserved multi-hop while separating routing from content by
 making the routing id a sequence rather than a single value. That way, it
 seems that the same push/pop behavior that happens on a ROUTER-DEALER
 device today could still work on CLIENT-SERVER. The push/pop would be
 happening on the routing stack instead of message content.

 If a message were something like:

 {
frame_t frames[];
int nframes;
routing_id_t route[];
int nroutes;
 }

 it would seem that you could have multi-part content (selfishly, in the
 way that affects me - messages composed of non-contiguous memory),
 multi-hop routing, and send it all with a single `zmq_send_newfangled`
 call, allowing future attempts at making `zmq_send_newfangled` a threadsafe
 call.

 There may be reasons this would be super gross and horrible, but it's an
 idea, anyway.

 -MinRK


 On Fri, Feb 6, 2015 at 9:02 AM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 Adding a mutex, even one that is never contended, to the socket will
 essentially triple this (one atomic CAS to acquire the mutex, one atomic
 CAS to put the message on the pipe, one atomic CAS to release the mutex).


 This is a bit of a blue sky view of the cost of acquiring a mutex. For
 the adventurous of spirit, chase down the call path of pthread_mutex
 sometime in GLIBC. It is substantially more involved than a single pair of
 'lock; cmpxchg' instructions, but it tries really hard to make that the
 rough cost of the happy path.

 On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 Having thought about this for a couple of more days, I want to at least
 take a stab at arguing against threadsafe sockets -

 libzmq's thread safety guarantees, to me anyway, are very clear,
 unsurprising and non-controversial - I cannot share a socket with another
 thread without a full fence.

 The kinds of systems I generally build have very strict requirements on
 overall latency, to the point that most of my networking IO is done through
 kernel-bypass libraries and NICs that support this, for raw TCP and UDP
 multicast. The latency sensitive code that does IO is in it's own thread,
 with exclusive access to the NICs which are accessed via kernel bypass.
 Coordination with other threads in the same process is done via inproc pair
 sockets. Pair sockets + very small messages (small enough that libzmq does
 not need to perform allocation) provide a very nice interface to a lock
 free queue with low overhead using a single atomic CAS operation. Atomic
 operations are cheap, but they are not free (~30 clocks on x86). Adding a
 mutex, even one that is never contended, to the socket will essentially
 triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the
 message on the pipe, one atomic CAS to release the mutex). I would like to
 have the option to avoid this.

 If a wrapper wants thread safe sockets to enable certain use-cases that
 may be more idiomatic for the language in question, it can provide the full
 fence. AZMQ https://github.com/zeromq/azmq does exactly this by
 default, but you have the option to opt out of it. It does this because
 Boost Asio by default allows it's sockets to be used from multiple threads
 for async IO and I need to guard more than just exclusive access to the
 ZeroMQ socket a the fence in this case. Putting a mutex inside of the
 libzmq socket, essentially doubles the overhead for no gain in useful
 functionality and runs completely counter to one of C and C++'s overarching
 principles: don't pay for what you don't use.

 If a class of apps really demands short lived exclusive access to a
 socket, provide a pool abstraction. The pool is thread safe, obtain a
 socket, perform I/O in a single thread, return the socket to the pool.

 On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier 
 pelletier.mic...@gmail.com wrote:



 On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens p...@imatix.com wrote:

 The discussion about thread safety was quite short iirc, though that
 contributor did discuss other things... at length. I merged his
 thread safe socket change rapidly, then we reverted it after a few
 days, and he disappeared. It was rather brute force and I suspect did
 not work at all, it simply wrapped all accesses to the socket
 structure in mutexes. No discussion at the time of multipart data and
 atomic send/recv.


 My memory of the conversation at the time is

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread Thomas Rodgers

libzmq has had io_vec like send/receive for a while (zmq_sendiov,
zmq_recviov), there is no documentation for them however.

On Fri, Feb 6, 2015 at 2:40 PM, Michel Pelletier pelletier.mic...@gmail.com
 wrote:

 Something else that occurred to me today, perhaps incorrectly, is that
 there was mention that nanomsg doesn't have multipart messages, but it does
 still preserve multipart messages in a way, but supporting scatter and
 gather io vectors.  Perhaps emulating this well known API pattern is the
 right approach to moving forward that still keep everyone happy and brings
 us closer to the ability to have thread safety.

 -Michel

 On Fri, Feb 6, 2015 at 12:28 PM, MinRK benjami...@gmail.com wrote:

 If I recall correctly, libzmq-3.0 (or was it the old 4.0 experimental
 branch?) preserved multi-hop while separating routing from content by
 making the routing id a sequence rather than a single value. That way, it
 seems that the same push/pop behavior that happens on a ROUTER-DEALER
 device today could still work on CLIENT-SERVER. The push/pop would be
 happening on the routing stack instead of message content.

 If a message were something like:

 {
frame_t frames[];
int nframes;
routing_id_t route[];
int nroutes;
 }

 it would seem that you could have multi-part content (selfishly, in the
 way that affects me - messages composed of non-contiguous memory),
 multi-hop routing, and send it all with a single `zmq_send_newfangled`
 call, allowing future attempts at making `zmq_send_newfangled` a threadsafe
 call.

 There may be reasons this would be super gross and horrible, but it's an
 idea, anyway.

 -MinRK


 On Fri, Feb 6, 2015 at 9:02 AM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 Adding a mutex, even one that is never contended, to the socket will
 essentially triple this (one atomic CAS to acquire the mutex, one atomic
 CAS to put the message on the pipe, one atomic CAS to release the mutex).


 This is a bit of a blue sky view of the cost of acquiring a mutex. For
 the adventurous of spirit, chase down the call path of pthread_mutex
 sometime in GLIBC. It is substantially more involved than a single pair of
 'lock; cmpxchg' instructions, but it tries really hard to make that the
 rough cost of the happy path.

 On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 Having thought about this for a couple of more days, I want to at least
 take a stab at arguing against threadsafe sockets -

 libzmq's thread safety guarantees, to me anyway, are very clear,
 unsurprising and non-controversial - I cannot share a socket with another
 thread without a full fence.

 The kinds of systems I generally build have very strict requirements on
 overall latency, to the point that most of my networking IO is done through
 kernel-bypass libraries and NICs that support this, for raw TCP and UDP
 multicast. The latency sensitive code that does IO is in it's own thread,
 with exclusive access to the NICs which are accessed via kernel bypass.
 Coordination with other threads in the same process is done via inproc pair
 sockets. Pair sockets + very small messages (small enough that libzmq does
 not need to perform allocation) provide a very nice interface to a lock
 free queue with low overhead using a single atomic CAS operation. Atomic
 operations are cheap, but they are not free (~30 clocks on x86). Adding a
 mutex, even one that is never contended, to the socket will essentially
 triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the
 message on the pipe, one atomic CAS to release the mutex). I would like to
 have the option to avoid this.

 If a wrapper wants thread safe sockets to enable certain use-cases that
 may be more idiomatic for the language in question, it can provide the full
 fence. AZMQ https://github.com/zeromq/azmq does exactly this by
 default, but you have the option to opt out of it. It does this because
 Boost Asio by default allows it's sockets to be used from multiple threads
 for async IO and I need to guard more than just exclusive access to the
 ZeroMQ socket a the fence in this case. Putting a mutex inside of the
 libzmq socket, essentially doubles the overhead for no gain in useful
 functionality and runs completely counter to one of C and C++'s overarching
 principles: don't pay for what you don't use.

 If a class of apps really demands short lived exclusive access to a
 socket, provide a pool abstraction. The pool is thread safe, obtain a
 socket, perform I/O in a single thread, return the socket to the pool.

 On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier 
 pelletier.mic...@gmail.com wrote:



 On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens p...@imatix.com
 wrote:

 The discussion about thread safety was quite short iirc, though that
 contributor did discuss other things... at length. I merged his
 thread safe socket change rapidly, then we reverted it after a few
 days, and he disappeared. It was rather brute force

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread AJ Lewis

On Fri, Feb 06, 2015 at 10:19:28AM +0100, Pieter Hintjens wrote:
 Actually one could just hold a stack of 16-bit routing IDs at the
 start of a message and push/pop these while forwarding.

A stack of (some length) routing IDs at the start of message would work
well for this I think and would address the multihop problem.

This doesn't address the use of having messages that don't need to be
decoded at each hop - I'm using multipart for this now because the first
tier of routing handles one message on the stack and ignores
everything else - it decides what to do with the rest of the message
stack based on that and sends it on.

This could be handled by sticking a blob in the first tier's message,
but it means that needs to be deserialized to read it every time, which
has potential performance impacts if the blob can be large (of course,
the blob still needs to be pushed around even with multipart, but at
least it doesn't have to be actively worked on).

 Let's slowly explore the client-server pattern, and perhaps over time
 this will become a usable replacement for existing socket patterns, or
 perhaps not. No stress.

Sure.
 
 Thanks to everyone for the feedback, this has been useful for me at
 least, and I hope to others.

Sounds good. 

AJ

 On Fri, Feb 6, 2015 at 9:53 AM, Pieter Hintjens p...@imatix.com wrote:
  We should not touch this, if it's working and stable and people
  actually use it. The nice thing about multiframing is you can push and
  pop addresses without decoding the rest of the message. Honestly I
  don't see a cleaner way to do multihop, except perhaps to reduce
  message size (integer routing IDs instead of the 5-byte or longer IDs
  used today).
 
  -Pieter
 
 
 
  On Fri, Feb 6, 2015 at 9:43 AM, Doron Somech somdo...@gmail.com wrote:
  Regarding multihope on ZProto we can have a stack of routing id (that will
  be serialized to the wire) and a method called reply and a method call
  forward. What do you think?
 
  On Feb 6, 2015 10:09 AM, Pieter Hintjens p...@imatix.com wrote:
 
  On Thu, Feb 5, 2015 at 8:41 PM, AJ Lewis aj.le...@quantum.com wrote:
 
   Is the plan really to drop multi-hop routing?
 
  I did want, and am glad to get, stories from people using multihop
  routing.
 
  I think the situation here is that we're using the multihop
  request-reply pattern, it depends on multipart, and that's fine. It's
  a standard, it's implemented, and it's stable.
 
  So we can focus on new patterns (like client/server) quite separately,
  and if we make threadsafe sockets, perhaps use them only in these new
  cases.
 
  -Pieter
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  https://urldefense.proofpoint.com/v1/url?u=http://lists.zeromq.org/mailman/listinfo/zeromq-devk=8F5TVnBDKF32UabxXsxZiA%3D%3D%0Ar=R2zWNFHLVImAo2qABlJKsHZpXtrP7rtLmoGaa3CV1do%3D%0Am=1VxxpdoYw9sa54If4BFOz1KTfg3AedlQ5kwDLjqupFk%3D%0As=963e3a75b9d55a2f32b970a98f3123ca29aa90dc4045b360c191db6ea2041658
 
 
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  https://urldefense.proofpoint.com/v1/url?u=http://lists.zeromq.org/mailman/listinfo/zeromq-devk=8F5TVnBDKF32UabxXsxZiA%3D%3D%0Ar=R2zWNFHLVImAo2qABlJKsHZpXtrP7rtLmoGaa3CV1do%3D%0Am=1VxxpdoYw9sa54If4BFOz1KTfg3AedlQ5kwDLjqupFk%3D%0As=963e3a75b9d55a2f32b970a98f3123ca29aa90dc4045b360c191db6ea2041658
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 https://urldefense.proofpoint.com/v1/url?u=http://lists.zeromq.org/mailman/listinfo/zeromq-devk=8F5TVnBDKF32UabxXsxZiA%3D%3D%0Ar=R2zWNFHLVImAo2qABlJKsHZpXtrP7rtLmoGaa3CV1do%3D%0Am=1VxxpdoYw9sa54If4BFOz1KTfg3AedlQ5kwDLjqupFk%3D%0As=963e3a75b9d55a2f32b970a98f3123ca29aa90dc4045b360c191db6ea2041658

-- 
AJ Lewis
Software Engineer
Quantum Corporation

Work:651 688-4346
email:   aj.le...@quantum.com

--
The information contained in this transmission may be confidential. Any 
disclosure, copying, or further distribution of confidential information is not 
permitted unless such privilege is explicitly granted in writing by Quantum. 
Quantum reserves the right to have electronic communications, including email 
and attachments, sent across its networks filtered through anti virus and spam 
software programs and retain such messages in order to comply with applicable 
data security and retention requirements. Quantum is not responsible for the 
proper and complete transmission of the substance of this communication or for 
any delay in its receipt.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread MinRK

On Fri, Feb 6, 2015 at 1:08 PM, Thomas Rodgers rodg...@twrodgers.com
wrote:

 libzmq has had io_vec like send/receive for a while (zmq_sendiov,
 zmq_recviov), there is no documentation for them however.


Yes, though with it's current implementation, using sendiov is the same as
several calls to zmq_send(..., SNDMORE), making it no better or worse than
making those several calls yourself.

-MinRK



 On Fri, Feb 6, 2015 at 2:40 PM, Michel Pelletier 
 pelletier.mic...@gmail.com wrote:

 Something else that occurred to me today, perhaps incorrectly, is that
 there was mention that nanomsg doesn't have multipart messages, but it does
 still preserve multipart messages in a way, but supporting scatter and
 gather io vectors.  Perhaps emulating this well known API pattern is the
 right approach to moving forward that still keep everyone happy and brings
 us closer to the ability to have thread safety.

 -Michel

 On Fri, Feb 6, 2015 at 12:28 PM, MinRK benjami...@gmail.com wrote:

 If I recall correctly, libzmq-3.0 (or was it the old 4.0 experimental
 branch?) preserved multi-hop while separating routing from content by
 making the routing id a sequence rather than a single value. That way, it
 seems that the same push/pop behavior that happens on a ROUTER-DEALER
 device today could still work on CLIENT-SERVER. The push/pop would be
 happening on the routing stack instead of message content.

 If a message were something like:

 {
frame_t frames[];
int nframes;
routing_id_t route[];
int nroutes;
 }

 it would seem that you could have multi-part content (selfishly, in the
 way that affects me - messages composed of non-contiguous memory),
 multi-hop routing, and send it all with a single `zmq_send_newfangled`
 call, allowing future attempts at making `zmq_send_newfangled` a threadsafe
 call.

 There may be reasons this would be super gross and horrible, but it's an
 idea, anyway.

 -MinRK


 On Fri, Feb 6, 2015 at 9:02 AM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 Adding a mutex, even one that is never contended, to the socket will
 essentially triple this (one atomic CAS to acquire the mutex, one atomic
 CAS to put the message on the pipe, one atomic CAS to release the mutex).


 This is a bit of a blue sky view of the cost of acquiring a mutex.
 For the adventurous of spirit, chase down the call path of pthread_mutex
 sometime in GLIBC. It is substantially more involved than a single pair of
 'lock; cmpxchg' instructions, but it tries really hard to make that the
 rough cost of the happy path.

 On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 Having thought about this for a couple of more days, I want to at
 least take a stab at arguing against threadsafe sockets -

 libzmq's thread safety guarantees, to me anyway, are very clear,
 unsurprising and non-controversial - I cannot share a socket with another
 thread without a full fence.

 The kinds of systems I generally build have very strict requirements
 on overall latency, to the point that most of my networking IO is done
 through kernel-bypass libraries and NICs that support this, for raw TCP 
 and
 UDP multicast. The latency sensitive code that does IO is in it's own
 thread, with exclusive access to the NICs which are accessed via kernel
 bypass. Coordination with other threads in the same process is done via
 inproc pair sockets. Pair sockets + very small messages (small enough that
 libzmq does not need to perform allocation) provide a very nice interface
 to a lock free queue with low overhead using a single atomic CAS 
 operation.
 Atomic operations are cheap, but they are not free (~30 clocks on x86).
 Adding a mutex, even one that is never contended, to the socket will
 essentially triple this (one atomic CAS to acquire the mutex, one atomic
 CAS to put the message on the pipe, one atomic CAS to release the mutex). 
 I
 would like to have the option to avoid this.

 If a wrapper wants thread safe sockets to enable certain use-cases
 that may be more idiomatic for the language in question, it can provide 
 the
 full fence. AZMQ https://github.com/zeromq/azmq does exactly this
 by default, but you have the option to opt out of it. It does this because
 Boost Asio by default allows it's sockets to be used from multiple threads
 for async IO and I need to guard more than just exclusive access to the
 ZeroMQ socket a the fence in this case. Putting a mutex inside of the
 libzmq socket, essentially doubles the overhead for no gain in useful
 functionality and runs completely counter to one of C and C++'s 
 overarching
 principles: don't pay for what you don't use.

 If a class of apps really demands short lived exclusive access to a
 socket, provide a pool abstraction. The pool is thread safe, obtain a
 socket, perform I/O in a single thread, return the socket to the pool.

 On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier 
 pelletier.mic...@gmail.com wrote:



 On Wed, Feb 4, 2015 at 10:51 AM, Pieter

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread Martin Sustrik

On 2015-02-06 10:11, Joe McIlvain wrote:

 Integer routing ids sounds nice to me, quite honestly.

Just a small comment on this:

I've used fixed-sized integer IDs in nanomsg and, in retrospect, I think 
it was an error.

In particular, think of UDP or any other unconnected protocol as a 
transport. If you want to stay true to the nature of the protocol (and 
thus allow single socket to communicate with unrestricted number of 
Internet hosts), you can't accumulate state, i.e. any kind of 
pseudo-connections, at the endpoints. Therefore, there's nothing for the 
ID to point to. The only option seems to be to store the IP address 
inside of the routing record in the message. And while IPv4 address may 
fit into 32bit ID, IPv6 address certainly wouldn't.

Martin
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

I'm not sure what staying true to UDP means. UDP's lack of
connection does not mean it is not addressable. I suspect what you
mean is that you would like replies to go back along arbitrary paths,
rather than rely on their original paths still existing. This assumes
the problem that proxies are liable to fail or disappear.

If that is the problem, then IP addresses won't help you. NAT, for one.

The existing solution to that problem is mesh routing, e.g. Babel,
where nodes have unique IDs that are separate from the physical
network, and routing tables to know how to get a message or reply to a
specific node. It's rather slow and complex.

There is an inherent weakness in any mesh design. Either your fabric
is stable, or it's not. If it's stable, you can do stateful
request/reply routing as we do today, and the choice of encoding
determines ID size. If it's not stable, you cannot rely on the
endpoints existing *at all*, and so the very concept of transient
request-reply becomes fragile.

-Pieter

On Fri, Feb 6, 2015 at 11:37 AM, Martin Sustrik sust...@250bpm.com wrote:
 On 2015-02-06 10:11, Joe McIlvain wrote:

 Integer routing ids sounds nice to me, quite honestly.

 Just a small comment on this:

 I've used fixed-sized integer IDs in nanomsg and, in retrospect, I think
 it was an error.

 In particular, think of UDP or any other unconnected protocol as a
 transport. If you want to stay true to the nature of the protocol (and
 thus allow single socket to communicate with unrestricted number of
 Internet hosts), you can't accumulate state, i.e. any kind of
 pseudo-connections, at the endpoints. Therefore, there's nothing for the
 ID to point to. The only option seems to be to store the IP address
 inside of the routing record in the message. And while IPv4 address may
 fit into 32bit ID, IPv6 address certainly wouldn't.

 Martin
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread Doron Somech

Regarding multihope on ZProto we can have a stack of routing id (that will
be serialized to the wire) and a method called reply and a method call
forward. What do you think?
On Feb 6, 2015 10:09 AM, Pieter Hintjens p...@imatix.com wrote:

 On Thu, Feb 5, 2015 at 8:41 PM, AJ Lewis aj.le...@quantum.com wrote:

  Is the plan really to drop multi-hop routing?

 I did want, and am glad to get, stories from people using multihop routing.

 I think the situation here is that we're using the multihop
 request-reply pattern, it depends on multipart, and that's fine. It's
 a standard, it's implemented, and it's stable.

 So we can focus on new patterns (like client/server) quite separately,
 and if we make threadsafe sockets, perhaps use them only in these new
 cases.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread Joe McIlvain

Pieter,
Integer routing ids sounds nice to me, quite honestly.

I wouldn't mind having the routing ids being semantically separated
from the content frames by more than just the empty delimiter, so
representing them with separate accessors as integers is a possibly
appealing id.  Arguably, it adds more complexity to the API, but that
may be outweighed by increased separation of concerns.

Doron,
If you or others are interested in multi-hop support for zproto, I'm
happy to share what I've been doing.  It's not terribly efficient
though, I'm afraid, although in my application I am not really pushing
the speed limits of zeromq at all so speed is not my biggest concern.

I only wanted to maintain a patch of the message codec generator gsl
script (and not of the client or server generators), so I needed a
scheme that only alters that.  Because the zproto server expects a
single zframe for a routing_id, in the message object when the
routing_id is requested I am serializing all of the frames of the
envelope into a single frame and making this accessible on the message
object as the routing_id, but I also provide the original group of
frames accessible on the message object as the envelope.  I also
provide a few other helper methods to do various operations on the
envelope.

Again, this strategy is not really ideal for high-throughput systems
so I didn't want to weigh down zproto for others by merging my patch
in.  However, if there is enough interest, perhaps we could find a way
for zproto users to opt-in to this kind of behavior.  For now, I'm
happy to maintain it privately until I'm nudged enough by interested
parties to share it.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

We should not touch this, if it's working and stable and people
actually use it. The nice thing about multiframing is you can push and
pop addresses without decoding the rest of the message. Honestly I
don't see a cleaner way to do multihop, except perhaps to reduce
message size (integer routing IDs instead of the 5-byte or longer IDs
used today).

-Pieter



On Fri, Feb 6, 2015 at 9:43 AM, Doron Somech somdo...@gmail.com wrote:
 Regarding multihope on ZProto we can have a stack of routing id (that will
 be serialized to the wire) and a method called reply and a method call
 forward. What do you think?

 On Feb 6, 2015 10:09 AM, Pieter Hintjens p...@imatix.com wrote:

 On Thu, Feb 5, 2015 at 8:41 PM, AJ Lewis aj.le...@quantum.com wrote:

  Is the plan really to drop multi-hop routing?

 I did want, and am glad to get, stories from people using multihop
 routing.

 I think the situation here is that we're using the multihop
 request-reply pattern, it depends on multipart, and that's fine. It's
 a standard, it's implemented, and it's stable.

 So we can focus on new patterns (like client/server) quite separately,
 and if we make threadsafe sockets, perhaps use them only in these new
 cases.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev


 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

Actually one could just hold a stack of 16-bit routing IDs at the
start of a message and push/pop these while forwarding.

Let's slowly explore the client-server pattern, and perhaps over time
this will become a usable replacement for existing socket patterns, or
perhaps not. No stress.

Thanks to everyone for the feedback, this has been useful for me at
least, and I hope to others.


On Fri, Feb 6, 2015 at 9:53 AM, Pieter Hintjens p...@imatix.com wrote:
 We should not touch this, if it's working and stable and people
 actually use it. The nice thing about multiframing is you can push and
 pop addresses without decoding the rest of the message. Honestly I
 don't see a cleaner way to do multihop, except perhaps to reduce
 message size (integer routing IDs instead of the 5-byte or longer IDs
 used today).

 -Pieter



 On Fri, Feb 6, 2015 at 9:43 AM, Doron Somech somdo...@gmail.com wrote:
 Regarding multihope on ZProto we can have a stack of routing id (that will
 be serialized to the wire) and a method called reply and a method call
 forward. What do you think?

 On Feb 6, 2015 10:09 AM, Pieter Hintjens p...@imatix.com wrote:

 On Thu, Feb 5, 2015 at 8:41 PM, AJ Lewis aj.le...@quantum.com wrote:

  Is the plan really to drop multi-hop routing?

 I did want, and am glad to get, stories from people using multihop
 routing.

 I think the situation here is that we're using the multihop
 request-reply pattern, it depends on multipart, and that's fine. It's
 a standard, it's implemented, and it's stable.

 So we can focus on new patterns (like client/server) quite separately,
 and if we make threadsafe sockets, perhaps use them only in these new
 cases.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev


 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread Charles Remes

I agree with this. The final paragraph is key.

cr

 On Feb 6, 2015, at 9:41 AM, Thomas Rodgers rodg...@twrodgers.com wrote:
 
 Having thought about this for a couple of more days, I want to at least take 
 a stab at arguing against threadsafe sockets -
 
 libzmq's thread safety guarantees, to me anyway, are very clear, unsurprising 
 and non-controversial - I cannot share a socket with another thread without a 
 full fence.
 
 The kinds of systems I generally build have very strict requirements on 
 overall latency, to the point that most of my networking IO is done through 
 kernel-bypass libraries and NICs that support this, for raw TCP and UDP 
 multicast. The latency sensitive code that does IO is in it's own thread, 
 with exclusive access to the NICs which are accessed via kernel bypass. 
 Coordination with other threads in the same process is done via inproc pair 
 sockets. Pair sockets + very small messages (small enough that libzmq does 
 not need to perform allocation) provide a very nice interface to a lock free 
 queue with low overhead using a single atomic CAS operation. Atomic 
 operations are cheap, but they are not free (~30 clocks on x86). Adding a 
 mutex, even one that is never contended, to the socket will essentially 
 triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the 
 message on the pipe, one atomic CAS to release the mutex). I would like to 
 have the option to avoid this.
 
 If a wrapper wants thread safe sockets to enable certain use-cases that may 
 be more idiomatic for the language in question, it can provide the full 
 fence. AZMQ https://github.com/zeromq/azmq does exactly this by default, 
 but you have the option to opt out of it. It does this because Boost Asio by 
 default allows it's sockets to be used from multiple threads for async IO and 
 I need to guard more than just exclusive access to the ZeroMQ socket a the 
 fence in this case. Putting a mutex inside of the libzmq socket, essentially 
 doubles the overhead for no gain in useful functionality and runs completely 
 counter to one of C and C++'s overarching principles: don't pay for what you 
 don't use.
 
 If a class of apps really demands short lived exclusive access to a socket, 
 provide a pool abstraction. The pool is thread safe, obtain a socket, perform 
 I/O in a single thread, return the socket to the pool.
 
 On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier pelletier.mic...@gmail.com 
 mailto:pelletier.mic...@gmail.com wrote:
 
 
 On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens p...@imatix.com 
 mailto:p...@imatix.com wrote:
 The discussion about thread safety was quite short iirc, though that
 contributor did discuss other things... at length. I merged his
 thread safe socket change rapidly, then we reverted it after a few
 days, and he disappeared. It was rather brute force and I suspect did
 not work at all, it simply wrapped all accesses to the socket
 structure in mutexes. No discussion at the time of multipart data and
 atomic send/recv.
 
 My memory of the conversation at the time is pretty dim, I agree the changes 
 were ugly and untested and the contributor was difficult to reason with and 
 seemed to want to make the changes based on no real need at all.
  
 
 As for socket safety, I've no strong opinion. I see that many people
 expect that to work and hit errors when it doesn't. I see that nanomsg
 has threadsafe sockets and no multipart. I see that sharing sockets
 across threads would make some actor models simpler, which is nice.
 
 This is the classic problem with thread safe anything.  Threads are hard, and 
 there is a balance between the complexity of making a thread safe construct 
 and the skill required of a programmer to use unsafe construct in a safe 
 manner.  I still think if the concrete problem is very short lived threads 
 causing slow joiner problems, then the simple solution is pools (troupes of 
 actors?).
 
 -Michel
  
 
 
 
 
 
 
 On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
 pelletier.mic...@gmail.com mailto:pelletier.mic...@gmail.com wrote:
  I think Brian has some good points here, there are numerous unrelated issues
  being discussed in this thread.
 
  A few points that I have:
 
  Multi part messages have also bothered me.  However as a Python programmer i
  see Min's points about the expense of buffer creation.  To my knowledge
  zproto does not (yet) have Python generation support either, or maybe
  something like generated cffi or ctypes wrappers around the zproto generated
  C code.  That being said there are a variety of serialization libraries for
  Python.  With some ctypes and mmap magic they can also be done zero copy
  but it's not pretty:
 
  https://gist.github.com/michelp/7522179 
  https://gist.github.com/michelp/7522179
 
  Multi part envelops are also how multi-hop routing is done.  I don't see how
  the new ideas handle that.  I don't think we can just say multi hop routing
  is bad and get rid of it.
 
  Thread

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread Thomas Rodgers

Having thought about this for a couple of more days, I want to at least
take a stab at arguing against threadsafe sockets -

libzmq's thread safety guarantees, to me anyway, are very clear,
unsurprising and non-controversial - I cannot share a socket with another
thread without a full fence.

The kinds of systems I generally build have very strict requirements on
overall latency, to the point that most of my networking IO is done through
kernel-bypass libraries and NICs that support this, for raw TCP and UDP
multicast. The latency sensitive code that does IO is in it's own thread,
with exclusive access to the NICs which are accessed via kernel bypass.
Coordination with other threads in the same process is done via inproc pair
sockets. Pair sockets + very small messages (small enough that libzmq does
not need to perform allocation) provide a very nice interface to a lock
free queue with low overhead using a single atomic CAS operation. Atomic
operations are cheap, but they are not free (~30 clocks on x86). Adding a
mutex, even one that is never contended, to the socket will essentially
triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the
message on the pipe, one atomic CAS to release the mutex). I would like to
have the option to avoid this.

If a wrapper wants thread safe sockets to enable certain use-cases that may
be more idiomatic for the language in question, it can provide the full
fence. AZMQ https://github.com/zeromq/azmq does exactly this by default,
but you have the option to opt out of it. It does this because Boost Asio
by default allows it's sockets to be used from multiple threads for async
IO and I need to guard more than just exclusive access to the ZeroMQ socket
a the fence in this case. Putting a mutex inside of the libzmq socket,
essentially doubles the overhead for no gain in useful functionality and
runs completely counter to one of C and C++'s overarching principles:
don't pay for what you don't use.

If a class of apps really demands short lived exclusive access to a socket,
provide a pool abstraction. The pool is thread safe, obtain a socket,
perform I/O in a single thread, return the socket to the pool.

On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier pelletier.mic...@gmail.com
 wrote:



 On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens p...@imatix.com wrote:

 The discussion about thread safety was quite short iirc, though that
 contributor did discuss other things... at length. I merged his
 thread safe socket change rapidly, then we reverted it after a few
 days, and he disappeared. It was rather brute force and I suspect did
 not work at all, it simply wrapped all accesses to the socket
 structure in mutexes. No discussion at the time of multipart data and
 atomic send/recv.


 My memory of the conversation at the time is pretty dim, I agree the
 changes were ugly and untested and the contributor was difficult to reason
 with and seemed to want to make the changes based on no real need at all.



 As for socket safety, I've no strong opinion. I see that many people
 expect that to work and hit errors when it doesn't. I see that nanomsg
 has threadsafe sockets and no multipart. I see that sharing sockets
 across threads would make some actor models simpler, which is nice.


 This is the classic problem with thread safe anything.  Threads are hard,
 and there is a balance between the complexity of making a thread safe
 construct and the skill required of a programmer to use unsafe construct
 in a safe manner.  I still think if the concrete problem is very short
 lived threads causing slow joiner problems, then the simple solution is
 pools (troupes of actors?).

 -Michel








 On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
 pelletier.mic...@gmail.com wrote:
  I think Brian has some good points here, there are numerous unrelated
 issues
  being discussed in this thread.
 
  A few points that I have:
 
  Multi part messages have also bothered me.  However as a Python
 programmer i
  see Min's points about the expense of buffer creation.  To my knowledge
  zproto does not (yet) have Python generation support either, or maybe
  something like generated cffi or ctypes wrappers around the zproto
 generated
  C code.  That being said there are a variety of serialization libraries
 for
  Python.  With some ctypes and mmap magic they can also be done zero
 copy
  but it's not pretty:
 
  https://gist.github.com/michelp/7522179
 
  Multi part envelops are also how multi-hop routing is done.  I don't
 see how
  the new ideas handle that.  I don't think we can just say multi hop
 routing
  is bad and get rid of it.
 
  Thread safe sockets do not sound appealing to me.  We did that, had a
 long
  and contentious discussion with the person championing them, merged it,
 then
  reverted it and that person is now no longer in the community.  Pieter
 was
  the most vocal opponent to them then and now he wants them back.  Of
 course,
  anyone can change their mind, but

Re: [zeromq-dev] Notes from a hackathon

2015-02-06 Thread Thomas Rodgers


 Adding a mutex, even one that is never contended, to the socket will
 essentially triple this (one atomic CAS to acquire the mutex, one atomic
 CAS to put the message on the pipe, one atomic CAS to release the mutex).


This is a bit of a blue sky view of the cost of acquiring a mutex. For
the adventurous of spirit, chase down the call path of pthread_mutex
sometime in GLIBC. It is substantially more involved than a single pair of
'lock; cmpxchg' instructions, but it tries really hard to make that the
rough cost of the happy path.

On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers rodg...@twrodgers.com
wrote:

 Having thought about this for a couple of more days, I want to at least
 take a stab at arguing against threadsafe sockets -

 libzmq's thread safety guarantees, to me anyway, are very clear,
 unsurprising and non-controversial - I cannot share a socket with another
 thread without a full fence.

 The kinds of systems I generally build have very strict requirements on
 overall latency, to the point that most of my networking IO is done through
 kernel-bypass libraries and NICs that support this, for raw TCP and UDP
 multicast. The latency sensitive code that does IO is in it's own thread,
 with exclusive access to the NICs which are accessed via kernel bypass.
 Coordination with other threads in the same process is done via inproc pair
 sockets. Pair sockets + very small messages (small enough that libzmq does
 not need to perform allocation) provide a very nice interface to a lock
 free queue with low overhead using a single atomic CAS operation. Atomic
 operations are cheap, but they are not free (~30 clocks on x86). Adding a
 mutex, even one that is never contended, to the socket will essentially
 triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the
 message on the pipe, one atomic CAS to release the mutex). I would like to
 have the option to avoid this.

 If a wrapper wants thread safe sockets to enable certain use-cases that
 may be more idiomatic for the language in question, it can provide the full
 fence. AZMQ https://github.com/zeromq/azmq does exactly this by
 default, but you have the option to opt out of it. It does this because
 Boost Asio by default allows it's sockets to be used from multiple threads
 for async IO and I need to guard more than just exclusive access to the
 ZeroMQ socket a the fence in this case. Putting a mutex inside of the
 libzmq socket, essentially doubles the overhead for no gain in useful
 functionality and runs completely counter to one of C and C++'s overarching
 principles: don't pay for what you don't use.

 If a class of apps really demands short lived exclusive access to a
 socket, provide a pool abstraction. The pool is thread safe, obtain a
 socket, perform I/O in a single thread, return the socket to the pool.

 On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier 
 pelletier.mic...@gmail.com wrote:



 On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens p...@imatix.com wrote:

 The discussion about thread safety was quite short iirc, though that
 contributor did discuss other things... at length. I merged his
 thread safe socket change rapidly, then we reverted it after a few
 days, and he disappeared. It was rather brute force and I suspect did
 not work at all, it simply wrapped all accesses to the socket
 structure in mutexes. No discussion at the time of multipart data and
 atomic send/recv.


 My memory of the conversation at the time is pretty dim, I agree the
 changes were ugly and untested and the contributor was difficult to reason
 with and seemed to want to make the changes based on no real need at all.



 As for socket safety, I've no strong opinion. I see that many people
 expect that to work and hit errors when it doesn't. I see that nanomsg
 has threadsafe sockets and no multipart. I see that sharing sockets
 across threads would make some actor models simpler, which is nice.


 This is the classic problem with thread safe anything.  Threads are hard,
 and there is a balance between the complexity of making a thread safe
 construct and the skill required of a programmer to use unsafe construct
 in a safe manner.  I still think if the concrete problem is very short
 lived threads causing slow joiner problems, then the simple solution is
 pools (troupes of actors?).

 -Michel








 On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
 pelletier.mic...@gmail.com wrote:
  I think Brian has some good points here, there are numerous unrelated
 issues
  being discussed in this thread.
 
  A few points that I have:
 
  Multi part messages have also bothered me.  However as a Python
 programmer i
  see Min's points about the expense of buffer creation.  To my knowledge
  zproto does not (yet) have Python generation support either, or maybe
  something like generated cffi or ctypes wrappers around the zproto
 generated
  C code.  That being said there are a variety of serialization
 libraries for
  Python.  With some ctypes and

Re: [zeromq-dev] Notes from a hackathon

2015-02-05 Thread Pieter Hintjens

On Thu, Feb 5, 2015 at 10:21 AM, Andreas Schultz aschu...@tpip.net wrote:

 Doing multipart messages with an atomic, sendmsg like API, would be
 much simpler. The 0MQ library could convert the 0MQ socket API call
 into an real sendmsg call without having to copy the parts around.

Sure. Then we're just talking about framing, and copying costs.

So be clear, the problem for thread safety is not multipart framing on
the wire, it is the current multipart API.

The main problem for complexity is (IME) carrying the multipart
concept up to higher layers like brokers and APIs.

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-05 Thread Doron Somech

Andreas - what you are saying is exactly like saying let's remove
multipart, use send that sends a blob and have a serialization solution
that support framing. I think that is the plan anyway.

On Thu, Feb 5, 2015 at 11:21 AM, Andreas Schultz aschu...@tpip.net wrote:

 Hi,

 - On 2 Feb, 2015, at 10:23, Pieter Hintjens p...@imatix.com wrote:

 [...]

  Multipart messages are the main reason we can't make threadsafe sockets.

 Well, yes and no. The API the 0MQ use for multipart messages is the main
 reason that you can't make threadsafe sockets, because the API
 does not have a atomic send/recv operation for multipart messages.

 Having to call the send/recv operation multiple times to process
 multipart messages creates a serialization problem and also necessitates
 in socket buffering, thus creating a concurrency problem for multi
 part messages (not that it is impossible to solve, adding a per thread
 buffers shouldn't be too difficult). The socket buffering is also likely
 the reason that the performance drops (because of the additional copying).

 Doing multipart messages with an atomic, sendmsg like API, would be
 much simpler. The 0MQ library could convert the 0MQ socket API call
 into an real sendmsg call without having to copy the parts around.

 Andreas

 [...]

  Cheers
  Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-05 Thread Andreas Schultz

Hi,

- On 2 Feb, 2015, at 10:23, Pieter Hintjens p...@imatix.com wrote:

[...]

 Multipart messages are the main reason we can't make threadsafe sockets.

Well, yes and no. The API the 0MQ use for multipart messages is the main
reason that you can't make threadsafe sockets, because the API
does not have a atomic send/recv operation for multipart messages.

Having to call the send/recv operation multiple times to process
multipart messages creates a serialization problem and also necessitates
in socket buffering, thus creating a concurrency problem for multi
part messages (not that it is impossible to solve, adding a per thread
buffers shouldn't be too difficult). The socket buffering is also likely
the reason that the performance drops (because of the additional copying).

Doing multipart messages with an atomic, sendmsg like API, would be
much simpler. The 0MQ library could convert the 0MQ socket API call
into an real sendmsg call without having to copy the parts around.

Andreas

[...]

 Cheers
 Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-05 Thread Joe McIlvain

FWIW, we are currently using a patched version of zproto message codec
to support multi-hop messages by tracking routing info as an envelope
of zero or more frames instead of as zero or one frames.  I'm not
afraid of change, but I'm still trying to work out how we are going to
change our architecture to not use multi-hop request/reply multi-frame
semantics.

One thing I thought I'd bring up here is zmq_proxy, which we are using
extensively to funnel traffic from many client (DEALER) sockets on a
single machine so that we can redirect them all to a different server
(ROUTER) at will.  This is part of why we support a variable number of
hops between client and server.  It's not clear to me whether
zmq_proxy could work correctly or at all with CLIENT/SERVER, or what
our course of action would be if we could not have any zmq_proxy
layers between client and server.  Our redirector would probably have
to become an actor that has sockets that connect are connected to each
client and are able to instruct them to disconnect from the current
server and connect to another (specific) one.  We'd also have to
figure out a way to synchronize this so that all clients are fully
disconnected from the first server before any start connecting to the
second.  Also, it would sound like this layer would have to speak a
different zproto protocol than the main one we have implemented for
the main flow of client/server messages, adding another zproto model
to maintain.

Again, I'm not afraid of change, but I just wanted to bring up a case
where multi-hop request/reply was (IMO) an elegant and simple solution
and the alternative would actually be adding complexity.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-05 Thread Pieter Hintjens

OK, noted.

On Thu, Feb 5, 2015 at 6:55 PM, Joe McIlvain joe.eli@gmail.com wrote:
 FWIW, we are currently using a patched version of zproto message codec
 to support multi-hop messages by tracking routing info as an envelope
 of zero or more frames instead of as zero or one frames.  I'm not
 afraid of change, but I'm still trying to work out how we are going to
 change our architecture to not use multi-hop request/reply multi-frame
 semantics.

 One thing I thought I'd bring up here is zmq_proxy, which we are using
 extensively to funnel traffic from many client (DEALER) sockets on a
 single machine so that we can redirect them all to a different server
 (ROUTER) at will.  This is part of why we support a variable number of
 hops between client and server.  It's not clear to me whether
 zmq_proxy could work correctly or at all with CLIENT/SERVER, or what
 our course of action would be if we could not have any zmq_proxy
 layers between client and server.  Our redirector would probably have
 to become an actor that has sockets that connect are connected to each
 client and are able to instruct them to disconnect from the current
 server and connect to another (specific) one.  We'd also have to
 figure out a way to synchronize this so that all clients are fully
 disconnected from the first server before any start connecting to the
 second.  Also, it would sound like this layer would have to speak a
 different zproto protocol than the main one we have implemented for
 the main flow of client/server messages, adding another zproto model
 to maintain.

 Again, I'm not afraid of change, but I just wanted to bring up a case
 where multi-hop request/reply was (IMO) an elegant and simple solution
 and the alternative would actually be adding complexity.
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-05 Thread AJ Lewis

On Thu, Feb 05, 2015 at 09:55:10AM -0800, Joe McIlvain wrote:
FWIW, we are currently using a patched version of zproto message codec
to support multi-hop messages by tracking routing info as an envelope
of zero or more frames instead of as zero or one frames. I'm not
afraid of change, but I'm still trying to work out how we are going to
change our architecture to not use multi-hop request/reply multi-frame
semantics.

Is the plan really to drop multi-hop routing? That seems like a bad
idea if that's what's going on...we're using multi-hop routing quite
extensively. What's the replacement? I haven't been able to figure
that out from this thread.

Can't the non-multipart message routing headers handle multiple hops?

Thanks,
AJ

One thing I thought I'd bring up here is zmq_proxy, which we are using
extensively to funnel traffic from many client (DEALER) sockets on a
single machine so that we can redirect them all to a different server
(ROUTER) at will. This is part of why we support a variable number of
hops between client and server. It's not clear to me whether
zmq_proxy could work correctly or at all with CLIENT/SERVER, or what
our course of action would be if we could not have any zmq_proxy
layers between client and server. Our redirector would probably have
to become an actor that has sockets that connect are connected to each
client and are able to instruct them to disconnect from the current
server and connect to another (specific) one. We'd also have to
figure out a way to synchronize this so that all clients are fully
disconnected from the first server before any start connecting to the
second. Also, it would sound like this layer would have to speak a
different zproto protocol than the main one we have implemented for
the main flow of client/server messages, adding another zproto model
to maintain.

Again, I'm not afraid of change, but I just wanted to bring up a case
where multi-hop request/reply was (IMO) an elegant and simple solution
and the alternative would actually be adding complexity.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://urldefense.proofpoint.com/v1/url?u=http://lists.zeromq.org/mailman/listinfo/zeromq-devk=8F5TVnBDKF32UabxXsxZiA%3D%3D%0Ar=R2zWNFHLVImAo2qABlJKsHZpXtrP7rtLmoGaa3CV1do%3D%0Am=tk9%2BVPVImoQCUleK2mCioAxfwpoubrnPjTpb%2FekLDM4%3D%0As=5be95c7bb0d7cd94b67d0afede00fa794b6c6b041e1abd7f464f84acfb2d9e2c

--
AJ Lewis
Software Engineer
Quantum Corporation

Work:651 688-4346
email: aj.le...@quantum.com

--
The information contained in this transmission may be confidential. Any
disclosure, copying, or further distribution of confidential information is not
permitted unless such privilege is explicitly granted in writing by Quantum.
Quantum reserves the right to have electronic communications, including email
and attachments, sent across its networks filtered through anti virus and spam
software programs and retain such messages in order to comply with applicable
data security and retention requirements. Quantum is not responsible for the
proper and complete transmission of the substance of this communication or for
any delay in its receipt.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-04 Thread Gregg Irwin

Hi Ed,

EG 1) language independance...

This was another thing that made 0MQ very attractive to me when I
found it. Multipart framing aids in this, as your point #4 makes.
Zproto as a replacement will define protocols more exactly, which is a
good thing, but it's also something new to add to your mix.

-- Gregg

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-04 Thread Ed Griffiths

Hi Gregg,


 Hi Ed,

 EG 1) language independance...

 This was another thing that made 0MQ very attractive to me when I
 found it. Multipart framing aids in this, as your point #4 makes.
 Zproto as a replacement will define protocols more exactly, which is a
 good thing, but it's also something new to add to your mix.

We will certainly take a look at that. From my quick skim I like the sound of 
tools that take models and turn them into perfect code;-)

I'll be out of job before I know it !


Ed
-- 
  
| Ed Griffiths, Acedb/ZMap development, Computational Genomics Group,|
|  The Sulston Building, Sanger Institute, Wellcome Trust Genome Campus, |
|  Hinxton, Cambridge CB10 1HH   |
||
| email: edg...@sanger.ac.uk  Tel: +44-1223-496844  Fax: +44-1223-494919 |
  


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-04 Thread Michel Pelletier

I think Brian has some good points here, there are numerous unrelated
issues being discussed in this thread.

A few points that I have:

Multi part messages have also bothered me.  However as a Python programmer
i see Min's points about the expense of buffer creation.  To my knowledge
zproto does not (yet) have Python generation support either, or maybe
something like generated cffi or ctypes wrappers around the zproto
generated C code.  That being said there are a variety of serialization
libraries for Python.  With some ctypes and mmap magic they can also be
done zero copy but it's not pretty:

https://gist.github.com/michelp/7522179

Multi part envelops are also how multi-hop routing is done.  I don't see
how the new ideas handle that.  I don't think we can just say multi hop
routing is bad and get rid of it.

Thread safe sockets do not sound appealing to me.  We did that, had a
long and contentious discussion with the person championing them, merged
it, then reverted it and that person is now no longer in the community.
Pieter was the most vocal opponent to them then and now he wants them
back.  Of course, anyone can change their mind, but the only current
argument I hear now for them though is improving the performance of short
lived threads, but that can be solved, more correctly in my opinion, with
thread or connection pools.  If you creating and tearing down threads that
rapidly then you have two problems.

-Michel

On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox bk...@digitalocean.com wrote:

 After catching up on this thread, I feel like at least three problems are
 being conflated into one problem.  I'll state what I see being discussed
 from my perspective:

 1. Using multi part messages as a way to route to clients from a router
 socket is overly complicated and not how new users expect things to work

 2. Using multi part messages for message serialization is costly, and
 potentially confusing to others.

 3. ZeroMQ sockets are not thread safe.

 While on an implementation level these three problems may be related, on a
 conceptual level I don't see them as related.  I may agree with some of
 these problem statements and not others.

 For me, my first priority is to always have the ability to get back a nice
 agnostic blob of bytes from ZeroMQ.   This makes it easy to make ZeroMQ
 socket use compatible with standard io interfaces in Go.  Structure for
 what is contained in those bytes is a concern of a different layer.
 Sometimes I use zproto for this (which I like), and other times I don't.

 As a demonstration that the problems are different problems, I solved #1
 for myself in goczmq without addressing anything else.

 I would assert some of the confusion in this discussion is that we're
 talking about multiple problem statements at the same time.

 Cheers - and it was great meeting people this week!

 Brian




 On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens p...@imatix.com wrote:

 Ironically, in my testing of high message rate), allowing multipart
 creates significant costs. Multipart is just one way of getting
 zero-copy, and even then only works on writing, not reading.

 For high performance brokers like Malamute I'd *really* like to be
 moving blobs around instead of lists of blobs.


 On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin gr...@pointillistic.com
 wrote:
  M Perhaps it is because I spend my days in a higher level language
  M like Python, but zproto is not an attractive option.
 
  Same here. I will read in detail about it shortly, but it may not make
  it into my toolbox as a multipart replacement. Multipart looked very
  cool when I found 0MQ, but I've ended up not using it much. I'm not
  doing high performance stuff though. Simplicity and ease of use are
  tops on my list.
 
  -- Gregg
 
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev



 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev


___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-04 Thread Michel Pelletier

On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens p...@imatix.com wrote:

 The discussion about thread safety was quite short iirc, though that
 contributor did discuss other things... at length. I merged his
 thread safe socket change rapidly, then we reverted it after a few
 days, and he disappeared. It was rather brute force and I suspect did
 not work at all, it simply wrapped all accesses to the socket
 structure in mutexes. No discussion at the time of multipart data and
 atomic send/recv.


My memory of the conversation at the time is pretty dim, I agree the
changes were ugly and untested and the contributor was difficult to reason
with and seemed to want to make the changes based on no real need at all.



 As for socket safety, I've no strong opinion. I see that many people
 expect that to work and hit errors when it doesn't. I see that nanomsg
 has threadsafe sockets and no multipart. I see that sharing sockets
 across threads would make some actor models simpler, which is nice.


This is the classic problem with thread safe anything.  Threads are hard,
and there is a balance between the complexity of making a thread safe
construct and the skill required of a programmer to use unsafe construct
in a safe manner.  I still think if the concrete problem is very short
lived threads causing slow joiner problems, then the simple solution is
pools (troupes of actors?).

-Michel








 On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
 pelletier.mic...@gmail.com wrote:
  I think Brian has some good points here, there are numerous unrelated
 issues
  being discussed in this thread.
 
  A few points that I have:
 
  Multi part messages have also bothered me.  However as a Python
 programmer i
  see Min's points about the expense of buffer creation.  To my knowledge
  zproto does not (yet) have Python generation support either, or maybe
  something like generated cffi or ctypes wrappers around the zproto
 generated
  C code.  That being said there are a variety of serialization libraries
 for
  Python.  With some ctypes and mmap magic they can also be done zero
 copy
  but it's not pretty:
 
  https://gist.github.com/michelp/7522179
 
  Multi part envelops are also how multi-hop routing is done.  I don't see
 how
  the new ideas handle that.  I don't think we can just say multi hop
 routing
  is bad and get rid of it.
 
  Thread safe sockets do not sound appealing to me.  We did that, had a
 long
  and contentious discussion with the person championing them, merged it,
 then
  reverted it and that person is now no longer in the community.  Pieter
 was
  the most vocal opponent to them then and now he wants them back.  Of
 course,
  anyone can change their mind, but the only current argument I hear now
 for
  them though is improving the performance of short lived threads, but that
  can be solved, more correctly in my opinion, with thread or connection
  pools.  If you creating and tearing down threads that rapidly then you
 have
  two problems.
 
  -Michel
 
  On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox bk...@digitalocean.com
 wrote:
 
  After catching up on this thread, I feel like at least three problems
 are
  being conflated into one problem.  I'll state what I see being discussed
  from my perspective:
 
  1. Using multi part messages as a way to route to clients from a router
  socket is overly complicated and not how new users expect things to
 work
 
  2. Using multi part messages for message serialization is costly, and
  potentially confusing to others.
 
  3. ZeroMQ sockets are not thread safe.
 
  While on an implementation level these three problems may be related,
 on a
  conceptual level I don't see them as related.  I may agree with some of
  these problem statements and not others.
 
  For me, my first priority is to always have the ability to get back a
 nice
  agnostic blob of bytes from ZeroMQ.   This makes it easy to make ZeroMQ
  socket use compatible with standard io interfaces in Go.  Structure for
 what
  is contained in those bytes is a concern of a different layer.
 Sometimes I
  use zproto for this (which I like), and other times I don't.
 
  As a demonstration that the problems are different problems, I solved #1
  for myself in goczmq without addressing anything else.
 
  I would assert some of the confusion in this discussion is that we're
  talking about multiple problem statements at the same time.
 
  Cheers - and it was great meeting people this week!
 
  Brian
 
 
 
 
  On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens p...@imatix.com wrote:
 
  Ironically, in my testing of high message rate), allowing multipart
  creates significant costs. Multipart is just one way of getting
  zero-copy, and even then only works on writing, not reading.
 
  For high performance brokers like Malamute I'd *really* like to be
  moving blobs around instead of lists of blobs.
 
 
  On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin gr...@pointillistic.com
  wrote:
   M Perhaps it is because I spend

Re: [zeromq-dev] Notes from a hackathon

The 'string' type is very useful for reducing memory allocations in
languages like C, since it's length limited to 255 bytes, so can live
on the heap or inside another object.

Removing octets, perhaps. That's an API concept. On the wire, octets
and chunks and longstrings are encoded the same way.

On Wed, Feb 4, 2015 at 7:10 PM, Doron Somech somdo...@gmail.com wrote:
 In way of simplify ZProto, specially if zeromq is going to be compatible
 with it. What do you think about deprecate the string (staying only with
 long string) and octets (staying with chunk)?
 If the size is really important we can use variable length instead of
 different types of field for some language type which can confuse and cause
 errors.

 On Wed, Feb 4, 2015 at 8:03 PM, Pieter Hintjens p...@imatix.com wrote:

 Making models instead of code gives remarkable leverage.

 However rather than make zproto a mandatory tool, I'd like to do what
 we do in CZMQ, which is to provide API serialization methods (like
 zsock_send/recv) which are compatible with the zproto serialization.



 On Wed, Feb 4, 2015 at 5:35 PM, Ed Griffiths edg...@sanger.ac.uk wrote:
  Hi Gregg,
 
 
  Hi Ed,
 
  EG 1) language independance...
 
  This was another thing that made 0MQ very attractive to me when I
  found it. Multipart framing aids in this, as your point #4 makes.
  Zproto as a replacement will define protocols more exactly, which is a
  good thing, but it's also something new to add to your mix.
 
  We will certainly take a look at that. From my quick skim I like the
  sound of
  tools that take models and turn them into perfect code;-)
 
  I'll be out of job before I know it !
 
 
  Ed
  --
 
  
  | Ed Griffiths, Acedb/ZMap development, Computational Genomics Group,
  |
  |  The Sulston Building, Sanger Institute, Wellcome Trust Genome Campus,
  |
  |  Hinxton, Cambridge CB10 1HH
  |
  |
  |
  | email: edg...@sanger.ac.uk  Tel: +44-1223-496844  Fax: +44-1223-494919
  |
 
  
 
 
  --
   The Wellcome Trust Sanger Institute is operated by Genome Research
   Limited, a charity registered in England with number 1021457 and a
   company registered in England with number 2742969, whose registered
   office is 215 Euston Road, London, NW1 2BE.
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev



 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

Incidentally, threads like these are the main reason I prefer small
problem driven change over planning and consensus.

I'm going to in any case start removing multipart support from some
higher layers like Malamute and Zyre, as a matter of experimentation,
and probably play with zclient and zserver classes in CZMQ, based on
the new client and server socket types, to explore the session
management.



On Wed, Feb 4, 2015 at 7:51 PM, Pieter Hintjens p...@imatix.com wrote:
 The discussion about thread safety was quite short iirc, though that
 contributor did discuss other things... at length. I merged his
 thread safe socket change rapidly, then we reverted it after a few
 days, and he disappeared. It was rather brute force and I suspect did
 not work at all, it simply wrapped all accesses to the socket
 structure in mutexes. No discussion at the time of multipart data and
 atomic send/recv.

 As for socket safety, I've no strong opinion. I see that many people
 expect that to work and hit errors when it doesn't. I see that nanomsg
 has threadsafe sockets and no multipart. I see that sharing sockets
 across threads would make some actor models simpler, which is nice.






 On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
 pelletier.mic...@gmail.com wrote:
 I think Brian has some good points here, there are numerous unrelated issues
 being discussed in this thread.

 A few points that I have:

 Multi part messages have also bothered me.  However as a Python programmer i
 see Min's points about the expense of buffer creation.  To my knowledge
 zproto does not (yet) have Python generation support either, or maybe
 something like generated cffi or ctypes wrappers around the zproto generated
 C code.  That being said there are a variety of serialization libraries for
 Python.  With some ctypes and mmap magic they can also be done zero copy
 but it's not pretty:

 https://gist.github.com/michelp/7522179

 Multi part envelops are also how multi-hop routing is done.  I don't see how
 the new ideas handle that.  I don't think we can just say multi hop routing
 is bad and get rid of it.

 Thread safe sockets do not sound appealing to me.  We did that, had a long
 and contentious discussion with the person championing them, merged it, then
 reverted it and that person is now no longer in the community.  Pieter was
 the most vocal opponent to them then and now he wants them back.  Of course,
 anyone can change their mind, but the only current argument I hear now for
 them though is improving the performance of short lived threads, but that
 can be solved, more correctly in my opinion, with thread or connection
 pools.  If you creating and tearing down threads that rapidly then you have
 two problems.

 -Michel

 On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox bk...@digitalocean.com wrote:

 After catching up on this thread, I feel like at least three problems are
 being conflated into one problem.  I'll state what I see being discussed
 from my perspective:

 1. Using multi part messages as a way to route to clients from a router
 socket is overly complicated and not how new users expect things to work

 2. Using multi part messages for message serialization is costly, and
 potentially confusing to others.

 3. ZeroMQ sockets are not thread safe.

 While on an implementation level these three problems may be related, on a
 conceptual level I don't see them as related.  I may agree with some of
 these problem statements and not others.

 For me, my first priority is to always have the ability to get back a nice
 agnostic blob of bytes from ZeroMQ.   This makes it easy to make ZeroMQ
 socket use compatible with standard io interfaces in Go.  Structure for what
 is contained in those bytes is a concern of a different layer.  Sometimes I
 use zproto for this (which I like), and other times I don't.

 As a demonstration that the problems are different problems, I solved #1
 for myself in goczmq without addressing anything else.

 I would assert some of the confusion in this discussion is that we're
 talking about multiple problem statements at the same time.

 Cheers - and it was great meeting people this week!

 Brian




 On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens p...@imatix.com wrote:

 Ironically, in my testing of high message rate), allowing multipart
 creates significant costs. Multipart is just one way of getting
 zero-copy, and even then only works on writing, not reading.

 For high performance brokers like Malamute I'd *really* like to be
 moving blobs around instead of lists of blobs.


 On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin gr...@pointillistic.com
 wrote:
  M Perhaps it is because I spend my days in a higher level language
  M like Python, but zproto is not an attractive option.
 
  Same here. I will read in detail about it shortly, but it may not make
  it into my toolbox as a multipart replacement. Multipart looked very
  cool when I found 0MQ, but I've ended up not using it much. I'm not
  doing

Re: [zeromq-dev] Notes from a hackathon

The discussion about thread safety was quite short iirc, though that
contributor did discuss other things... at length. I merged his
thread safe socket change rapidly, then we reverted it after a few
days, and he disappeared. It was rather brute force and I suspect did
not work at all, it simply wrapped all accesses to the socket
structure in mutexes. No discussion at the time of multipart data and
atomic send/recv.

As for socket safety, I've no strong opinion. I see that many people
expect that to work and hit errors when it doesn't. I see that nanomsg
has threadsafe sockets and no multipart. I see that sharing sockets
across threads would make some actor models simpler, which is nice.






On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
pelletier.mic...@gmail.com wrote:
 I think Brian has some good points here, there are numerous unrelated issues
 being discussed in this thread.

 A few points that I have:

 Multi part messages have also bothered me.  However as a Python programmer i
 see Min's points about the expense of buffer creation.  To my knowledge
 zproto does not (yet) have Python generation support either, or maybe
 something like generated cffi or ctypes wrappers around the zproto generated
 C code.  That being said there are a variety of serialization libraries for
 Python.  With some ctypes and mmap magic they can also be done zero copy
 but it's not pretty:

 https://gist.github.com/michelp/7522179

 Multi part envelops are also how multi-hop routing is done.  I don't see how
 the new ideas handle that.  I don't think we can just say multi hop routing
 is bad and get rid of it.

 Thread safe sockets do not sound appealing to me.  We did that, had a long
 and contentious discussion with the person championing them, merged it, then
 reverted it and that person is now no longer in the community.  Pieter was
 the most vocal opponent to them then and now he wants them back.  Of course,
 anyone can change their mind, but the only current argument I hear now for
 them though is improving the performance of short lived threads, but that
 can be solved, more correctly in my opinion, with thread or connection
 pools.  If you creating and tearing down threads that rapidly then you have
 two problems.

 -Michel

 On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox bk...@digitalocean.com wrote:

 After catching up on this thread, I feel like at least three problems are
 being conflated into one problem.  I'll state what I see being discussed
 from my perspective:

 1. Using multi part messages as a way to route to clients from a router
 socket is overly complicated and not how new users expect things to work

 2. Using multi part messages for message serialization is costly, and
 potentially confusing to others.

 3. ZeroMQ sockets are not thread safe.

 While on an implementation level these three problems may be related, on a
 conceptual level I don't see them as related.  I may agree with some of
 these problem statements and not others.

 For me, my first priority is to always have the ability to get back a nice
 agnostic blob of bytes from ZeroMQ.   This makes it easy to make ZeroMQ
 socket use compatible with standard io interfaces in Go.  Structure for what
 is contained in those bytes is a concern of a different layer.  Sometimes I
 use zproto for this (which I like), and other times I don't.

 As a demonstration that the problems are different problems, I solved #1
 for myself in goczmq without addressing anything else.

 I would assert some of the confusion in this discussion is that we're
 talking about multiple problem statements at the same time.

 Cheers - and it was great meeting people this week!

 Brian




 On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens p...@imatix.com wrote:

 Ironically, in my testing of high message rate), allowing multipart
 creates significant costs. Multipart is just one way of getting
 zero-copy, and even then only works on writing, not reading.

 For high performance brokers like Malamute I'd *really* like to be
 moving blobs around instead of lists of blobs.


 On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin gr...@pointillistic.com
 wrote:
  M Perhaps it is because I spend my days in a higher level language
  M like Python, but zproto is not an attractive option.
 
  Same here. I will read in detail about it shortly, but it may not make
  it into my toolbox as a multipart replacement. Multipart looked very
  cool when I found 0MQ, but I've ended up not using it much. I'm not
  doing high performance stuff though. Simplicity and ease of use are
  tops on my list.
 
  -- Gregg
 
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev



 ___
 zeromq-dev mailing list

Re: [zeromq-dev] Notes from a hackathon

Making models instead of code gives remarkable leverage.

However rather than make zproto a mandatory tool, I'd like to do what
we do in CZMQ, which is to provide API serialization methods (like
zsock_send/recv) which are compatible with the zproto serialization.



On Wed, Feb 4, 2015 at 5:35 PM, Ed Griffiths edg...@sanger.ac.uk wrote:
 Hi Gregg,


 Hi Ed,

 EG 1) language independance...

 This was another thing that made 0MQ very attractive to me when I
 found it. Multipart framing aids in this, as your point #4 makes.
 Zproto as a replacement will define protocols more exactly, which is a
 good thing, but it's also something new to add to your mix.

 We will certainly take a look at that. From my quick skim I like the sound of
 tools that take models and turn them into perfect code;-)

 I'll be out of job before I know it !


 Ed
 --
   
 | Ed Griffiths, Acedb/ZMap development, Computational Genomics Group,|
 |  The Sulston Building, Sanger Institute, Wellcome Trust Genome Campus, |
 |  Hinxton, Cambridge CB10 1HH   |
 ||
 | email: edg...@sanger.ac.uk  Tel: +44-1223-496844  Fax: +44-1223-494919 |
   


 --
  The Wellcome Trust Sanger Institute is operated by Genome Research
  Limited, a charity registered in England with number 1021457 and a
  company registered in England with number 2742969, whose registered
  office is 215 Euston Road, London, NW1 2BE.
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-04 Thread Doron Somech

In way of simplify ZProto, specially if zeromq is going to be compatible
with it. What do you think about deprecate the string (staying only with
long string) and octets (staying with chunk)?
If the size is really important we can use variable length instead of
different types of field for some language type which can confuse and cause
errors.

On Wed, Feb 4, 2015 at 8:03 PM, Pieter Hintjens p...@imatix.com wrote:

 Making models instead of code gives remarkable leverage.

 However rather than make zproto a mandatory tool, I'd like to do what
 we do in CZMQ, which is to provide API serialization methods (like
 zsock_send/recv) which are compatible with the zproto serialization.



 On Wed, Feb 4, 2015 at 5:35 PM, Ed Griffiths edg...@sanger.ac.uk wrote:
  Hi Gregg,
 
 
  Hi Ed,
 
  EG 1) language independance...
 
  This was another thing that made 0MQ very attractive to me when I
  found it. Multipart framing aids in this, as your point #4 makes.
  Zproto as a replacement will define protocols more exactly, which is a
  good thing, but it's also something new to add to your mix.
 
  We will certainly take a look at that. From my quick skim I like the
 sound of
  tools that take models and turn them into perfect code;-)
 
  I'll be out of job before I know it !
 
 
  Ed
  --
 
  
  | Ed Griffiths, Acedb/ZMap development, Computational Genomics Group,
 |
  |  The Sulston Building, Sanger Institute, Wellcome Trust Genome Campus,
 |
  |  Hinxton, Cambridge CB10 1HH
  |
  |
 |
  | email: edg...@sanger.ac.uk  Tel: +44-1223-496844  Fax: +44-1223-494919
 |
 
  
 
 
  --
   The Wellcome Trust Sanger Institute is operated by Genome Research
   Limited, a charity registered in England with number 1021457 and a
   company registered in England with number 2742969, whose registered
   office is 215 Euston Road, London, NW1 2BE.
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-04 Thread Maurice Barnum

On Tuesday, February 3, 2015 2:05 AM, Pieter Hintjens p...@imatix.com wrote:

 I've written up my understanding of this thread: http://hintjens.com/blog:84

 Thread safety would solve the perennial problem of sending data via
 ZeroMQ, from short-lived threads. Performance in many cases would go
 up, not down. We see way too many apps that create/connect a socket
 for every message they send.

 Apart from that I agree that thread safety is a weird thing and only
 worth doing in very specific cases (ZeroMQ contexts and sockets seem
 to be good candidates).

For sockets, I've several times have wanted to integrate with a library that
uses callbacks on an implementation-defined thread (aka, not mine).  I want
to take the data supplied with the callback and enqueue it to a worker that
uses minimal (preferably no) shared state with the rest of the application.
Since ZeroMQ sockets are not thread safe, I have to build my own queuing
mechanism and either not use ZeroMQ for this part of the application or live
with the complexity of a hybrid solution.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-04 Thread Colin Ingarfield

On 02/04/2015 02:34 PM, Maurice Barnum wrote:
 On Tuesday, February 3, 2015 2:05 AM, Pieter Hintjens p...@imatix.com wrote:

 I've written up my understanding of this thread: http://hintjens.com/blog:84
 Thread safety would solve the perennial problem of sending data via
 ZeroMQ, from short-lived threads. Performance in many cases would go
 up, not down. We see way too many apps that create/connect a socket
 for every message they send.
 Apart from that I agree that thread safety is a weird thing and only
 worth doing in very specific cases (ZeroMQ contexts and sockets seem
 to be good candidates).
 For sockets, I've several times have wanted to integrate with a library that
 uses callbacks on an implementation-defined thread (aka, not mine).  I want
 to take the data supplied with the callback and enqueue it to a worker that
 uses minimal (preferably no) shared state with the rest of the application.
 Since ZeroMQ sockets are not thread safe, I have to build my own queuing
 mechanism and either not use ZeroMQ for this part of the application or live
 with the complexity of a hybrid solution.
 ___


I have run into the same issue with library callbacks.  Also, thread 
safe sockets would be useful when sharing a single socket within a web 
container (eg, Tomcat, etc.)  A common case is to want to dispatch a 
message to a socket from a container thread.  Since the socket is not 
thread safe I have to hack around it (typically creating an iproc socket 
per request).  This is probably not very efficient and it's definitely 
kind of ugly.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-04 Thread Brian Knox

After catching up on this thread, I feel like at least three problems are
being conflated into one problem.  I'll state what I see being discussed
from my perspective:

1. Using multi part messages as a way to route to clients from a router
socket is overly complicated and not how new users expect things to work

2. Using multi part messages for message serialization is costly, and
potentially confusing to others.

3. ZeroMQ sockets are not thread safe.

While on an implementation level these three problems may be related, on a
conceptual level I don't see them as related.  I may agree with some of
these problem statements and not others.

For me, my first priority is to always have the ability to get back a nice
agnostic blob of bytes from ZeroMQ.   This makes it easy to make ZeroMQ
socket use compatible with standard io interfaces in Go.  Structure for
what is contained in those bytes is a concern of a different layer.
Sometimes I use zproto for this (which I like), and other times I don't.

As a demonstration that the problems are different problems, I solved #1
for myself in goczmq without addressing anything else.

I would assert some of the confusion in this discussion is that we're
talking about multiple problem statements at the same time.

Cheers - and it was great meeting people this week!

Brian




On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens p...@imatix.com wrote:

 Ironically, in my testing of high message rate), allowing multipart
 creates significant costs. Multipart is just one way of getting
 zero-copy, and even then only works on writing, not reading.

 For high performance brokers like Malamute I'd *really* like to be
 moving blobs around instead of lists of blobs.


 On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin gr...@pointillistic.com
 wrote:
  M Perhaps it is because I spend my days in a higher level language
  M like Python, but zproto is not an attractive option.
 
  Same here. I will read in detail about it shortly, but it may not make
  it into my toolbox as a multipart replacement. Multipart looked very
  cool when I found 0MQ, but I've ended up not using it much. I'm not
  doing high performance stuff though. Simplicity and ease of use are
  tops on my list.
 
  -- Gregg
 
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-04 Thread Ed Griffiths

Hi Pieter and others,

Just to give some information about why we chose zeromq for our project...


We have a project where two programs need to communicate with each other to 
allow them to operate seemlessly as one from the point of view of the user. To 
do this we have requests and replies flowing between the two with the 
requirement that each request must be serviced and replied to before the next 
request can be handled. Requests and replies flow from either side, the two 
programs are equal peers.

We had one mechanism that was quite hard to implement and a colleague suggested 
we give zeromq a try. In particular he was keen on REQ-REP as it would relieve 
us of the need to check that messages were getting though and would help us 
enforce our strict one request at a time protocol.

Here are the main reasons we chose to try zeromq:

1) language independance, for good solid reasons one of our programs is in C, 
the other in perl.

2) simplicity of interface, zeromq provided out of the box a REQ-REP protocol.

3) robustness, we felt we could rely on zeromq to the extent that we would not 
need to implement the kind of acknowledging of every single message in the way 
that TCP/IP does under the covers.

4) Multipart Messages, we used this facility to very easily partition our 
messages into a header giving type of request, request id, retry attempt and 
time of request and a body which was the actual application level of request.


As REQ-REP and Multipart Messages are under discussion I hope this will be 
useful.

While I understand the need to move some things out of zeromq if there is no 
replacement in the form of some kind of layer above zeromq then this may 
deter 
some from using this excellent library.


thanks Ed
-- 
  
| Ed Griffiths, Acedb/ZMap development, Computational Genomics Group,|
|  The Sulston Building, Sanger Institute, Wellcome Trust Genome Campus, |
|  Hinxton, Cambridge CB10 1HH   |
||
| email: edg...@sanger.ac.uk  Tel: +44-1223-496844  Fax: +44-1223-494919 |
  


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Doron Somech

I agree that the best place for session management is at the transport.
However implementing the session management at the server socket level
should be pretty simple, even as POC. Only issue is how to make sure user
is not sending the same messages as the socket sending, but we might ignore
the problem at the beginning. Do you think it worth a try or to leave it to
the transport?

On Tue, Feb 3, 2015 at 3:06 PM, Pieter Hintjens p...@imatix.com wrote:

 On Tue, Feb 3, 2015 at 1:01 PM, Doron Somech somdo...@gmail.com wrote:

  Pieter - regarding the blog post, I'm not sure about setting the routing
 id
  on the socket

 Indeed, that isn't atomic with sending and so can't work with
 threadsafe sockets. I've changed the article.

 I think we can either get the routing id as a message property, and/or
 add a reply to message semantic that does this implicitly. The
 second is easier for simple servers, the first is better for realistic
 servers.

 Since it only applies to server sockets, I think we would benefit from
 a zserver_reply () method (or similar).

 We might also think of moving the picture send/recv API into libzmq at
 some point since it's the immediate replacement for framing.

  I also have a question regarding the session management (really liked
 it),
  but how would know it is a socket level message and not a application
 level?
  Also do you think it should be enabled by default or only if an option is
  set?

 To make session management work we'd have to implement it using ZMTP
 control commands, invisibly to the application. Same as heartbeating
 at that level.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Luna Duclos

I'm not sure if I like having picture as a libzmq level thing.
It really only makes sense for C language bindings in my book. most other
bindings use their own serialization mechanisms.

On Tue, Feb 3, 2015 at 2:06 PM, Pieter Hintjens p...@imatix.com wrote:

 On Tue, Feb 3, 2015 at 1:01 PM, Doron Somech somdo...@gmail.com wrote:

  Pieter - regarding the blog post, I'm not sure about setting the routing
 id
  on the socket

 Indeed, that isn't atomic with sending and so can't work with
 threadsafe sockets. I've changed the article.

 I think we can either get the routing id as a message property, and/or
 add a reply to message semantic that does this implicitly. The
 second is easier for simple servers, the first is better for realistic
 servers.

 Since it only applies to server sockets, I think we would benefit from
 a zserver_reply () method (or similar).

 We might also think of moving the picture send/recv API into libzmq at
 some point since it's the immediate replacement for framing.

  I also have a question regarding the session management (really liked
 it),
  but how would know it is a socket level message and not a application
 level?
  Also do you think it should be enabled by default or only if an option is
  set?

 To make session management work we'd have to implement it using ZMTP
 control commands, invisibly to the application. Same as heartbeating
 at that level.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

Thanks for the questions. I'd try to flatter by saying how great these
questions are, except I know you're all way too smart to fall for
that. :)

Let me answer a bunch of questions in one go:

* Sorry for not posting this here. I've written a more detailed
analysis of the motivation for this thread (and experiments):
http://hintjens.com/blog:84.

* The main goal with this is simplicity and accuracy. I'm tired of
teaching people ZeroMQ constructs that are clearly over-designed and
disconnected from reality in subtle ways. Each apology for friction
(sorry, but don't use REQ-REP, they are kind of weird, sorry,
IDENTITY is not really identity, sorry, message sometimes means
frame and sometimes message) really signals costs up and down the
stacks.

* Sessions require protocol commands that are invisible to normal
use. That means either protocol support (e.g. command frames) or some
layering on top of ZeroMQ blobs.

* Picture sending solves a cross-language problem, namely how to
construct zproto-style commands without using dedicated codecs. While
the API is C-ish, the functionality is not. I'd expect that to be
wrapped in bindings.

* Per-language serialization is an anti-pattern. It's common, and fine
as such, yet ZeroMQ should IMO always strive to be cross-language.

* REQ-REP between threads may make sense. We certainly use it (ZAP,
for authentication).

* Is this discussion C4 compatible? Yes, no, this is chatter, not a
plan or roadmap. Until pull requests hit the road, it's all vapour.
However the problems I restated in my blog article are real, to many
people, and should be aired.

* Multipart does easy serialization, yes. Not in C though. We've
learned how to do variable list serialization with zproto-compatible
picture sending in CZMQ. Is this a sufficient alternative? I think so.
zsock_send() was essential to making actors cheap enough to use
widely. egrep zsock_send *.c in malamute-core/src.

* Locking on socketsL: we did in fact have a pull request for this
once. It was... nasty. We reverted it. We breathed a sigh of relief.
However looking back, the problem was valid and has never been solved
since.

* If you're doing multipart a lot in an application, you can replace
that with zproto. You already have contracts then, why not make them
explicit and documented? That's always a win in my experience.

* Multi-hop request reply is a strange beast and I'm keen to
understand those architectures that use it. My suspicion is that we've
shifted our vision of what ZeroMQ is over the years. My own
experience seems plausible, yet I distrust it. So, the stories in my
article are please for falsification, not attempts to convince.

* I think we have learned to not break existing code. It's going to be
interesting to run multiple models at once. We do this in CZMQ and it
works OK. There isn't much overhead.

* Authentication and routing information should be (MUST be) two
totally separate layers. CURVE certainly does not use the routing ID
(even if it's still misnamed as IDENTITY). You can have the same
client on N connections (obviously). You authenticate a client using a
mechanism-specific key: PLAIN user id, or CURVE public key.

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Thomas Rodgers


 * Sessions require protocol commands that are invisible to normal
 use. That means either protocol support (e.g. command frames) or some
 layering on top of ZeroMQ blobs.


What would the argument be for not just using command frames for this?

On Tue, Feb 3, 2015 at 4:04 PM, Pieter Hintjens p...@imatix.com wrote:

 Thanks for the questions. I'd try to flatter by saying how great these
 questions are, except I know you're all way too smart to fall for
 that. :)

 Let me answer a bunch of questions in one go:

 * Sorry for not posting this here. I've written a more detailed
 analysis of the motivation for this thread (and experiments):
 http://hintjens.com/blog:84.

 * The main goal with this is simplicity and accuracy. I'm tired of
 teaching people ZeroMQ constructs that are clearly over-designed and
 disconnected from reality in subtle ways. Each apology for friction
 (sorry, but don't use REQ-REP, they are kind of weird, sorry,
 IDENTITY is not really identity, sorry, message sometimes means
 frame and sometimes message) really signals costs up and down the
 stacks.

 * Sessions require protocol commands that are invisible to normal
 use. That means either protocol support (e.g. command frames) or some
 layering on top of ZeroMQ blobs.

 * Picture sending solves a cross-language problem, namely how to
 construct zproto-style commands without using dedicated codecs. While
 the API is C-ish, the functionality is not. I'd expect that to be
 wrapped in bindings.

 * Per-language serialization is an anti-pattern. It's common, and fine
 as such, yet ZeroMQ should IMO always strive to be cross-language.

 * REQ-REP between threads may make sense. We certainly use it (ZAP,
 for authentication).

 * Is this discussion C4 compatible? Yes, no, this is chatter, not a
 plan or roadmap. Until pull requests hit the road, it's all vapour.
 However the problems I restated in my blog article are real, to many
 people, and should be aired.

 * Multipart does easy serialization, yes. Not in C though. We've
 learned how to do variable list serialization with zproto-compatible
 picture sending in CZMQ. Is this a sufficient alternative? I think so.
 zsock_send() was essential to making actors cheap enough to use
 widely. egrep zsock_send *.c in malamute-core/src.

 * Locking on socketsL: we did in fact have a pull request for this
 once. It was... nasty. We reverted it. We breathed a sigh of relief.
 However looking back, the problem was valid and has never been solved
 since.

 * If you're doing multipart a lot in an application, you can replace
 that with zproto. You already have contracts then, why not make them
 explicit and documented? That's always a win in my experience.

 * Multi-hop request reply is a strange beast and I'm keen to
 understand those architectures that use it. My suspicion is that we've
 shifted our vision of what ZeroMQ is over the years. My own
 experience seems plausible, yet I distrust it. So, the stories in my
 article are please for falsification, not attempts to convince.

 * I think we have learned to not break existing code. It's going to be
 interesting to run multiple models at once. We do this in CZMQ and it
 works OK. There isn't much overhead.

 * Authentication and routing information should be (MUST be) two
 totally separate layers. CURVE certainly does not use the routing ID
 (even if it's still misnamed as IDENTITY). You can have the same
 client on N connections (obviously). You authenticate a client using a
 mechanism-specific key: PLAIN user id, or CURVE public key.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

On Tue, Feb 3, 2015 at 1:01 PM, Doron Somech somdo...@gmail.com wrote:

 Pieter - regarding the blog post, I'm not sure about setting the routing id
 on the socket

Indeed, that isn't atomic with sending and so can't work with
threadsafe sockets. I've changed the article.

I think we can either get the routing id as a message property, and/or
add a reply to message semantic that does this implicitly. The
second is easier for simple servers, the first is better for realistic
servers.

Since it only applies to server sockets, I think we would benefit from
a zserver_reply () method (or similar).

We might also think of moving the picture send/recv API into libzmq at
some point since it's the immediate replacement for framing.

 I also have a question regarding the session management (really liked it),
 but how would know it is a socket level message and not a application level?
 Also do you think it should be enabled by default or only if an option is
 set?

To make session management work we'd have to implement it using ZMTP
control commands, invisibly to the application. Same as heartbeating
at that level.

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Doron Somech

Pieter - regarding the blog post, I'm not sure about setting the routing id
on the socket, the problem is that it's not thread safe (2 calls to the
socket) and could lead to complexity, we can use the thread local storage
for the routing id but I think it is still complicated.

Why not have it as part of the send method when sending messages, like:

zsock_send(void *self, int routing_id, const char *picture, ...)

when you don't need it pass zero or use another overload without the
routing id.

I also have a question regarding the session management (really liked it),
but how would know it is a socket level message and not a application
level? Also do you think it should be enabled by default or only if an
option is set?


On Tue, Feb 3, 2015 at 12:04 PM, Pieter Hintjens p...@imatix.com wrote:

 There has been a zmq_sendiov/recviov call for ages, undocumented and
 rarely used. This would indeed be a good alternative to multipart for
 the zerocopy use case.

 On Mon, Feb 2, 2015 at 10:59 PM, Maurice Barnum m...@yahoo-inc.com wrote:
  Is there any thought to adding a writev() analogue like zmq_msg_sendv()?
 
  One of the reasons I send send multiple frames is to avoid stitching
 together
  buffers just to call send, which something like writev() would address
 that use
  without the complexity of the socket buffering the parts.
 
  On Monday, February 2, 2015 8:36 AM, Joe McIlvain joe.eli@gmail.com
 wrote:
 
 
 
  In my opinion, maintaining some kind of multi-frame abstraction at the
 API level is in fact very useful to applications, even if they are sent
 atomically on the implementation level as you describe.
 
  For example, in CZMQ It's quite useful in program flow to be able to
 push or pop a single zframe onto or off of a zmsg, then send that zmsg as
 an entire message.  Rarely do I find myself using zframe_send or
 zframe_recv with the send more and receive more flags in a CZMQ
 application.  The CZMQ picture messages are nice, but they are not always
 the easiest API to integrate into an application.  Having zmsg and zframe
 remain (or something like them), while removing zframe_send and zframe_recv
 could ease applications relying on multi-frame semantics into the
 transition.  The implementation for zmsg would simply serialize the frames
 in some zmsg-determined way into a single-part zmq_msg to send and
 deserialize according to the same scheme to receive.  This multi-frame
 serialization essentially be at the application level rather than at the
 ZMTP level.
 
  In short, I think a multi-frame abstraction is important for
 applications, but if we remove the ability to send one frame at a time at
 the fundamental level, this abstraction can remain at a higher level
 (whether in libzmq, or in CZMQ and other wrappers) and coexist happily with
 the proposed changes.
 
 
 
 
  On Mon, Feb 2, 2015 at 7:37 AM, Doron Somech somdo...@gmail.com wrote:
 
  So the new sockets are in the github master, you can take a loot at the
 test to see how to use the new routing id field:
 
 
 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp
 
 
 Few of the reasons I didn't like multi frames:
 
 * I tried in the past to make both zeromq and NetMQ thread safe, think
 of sending from multiple threads or receiving from multiple threads. we
 cannot do that with Multipart.
 
 * There are a lot of good transport that already implement message
 framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required
 it is own framing it was not easy to add them.
 
 * Multipart, router usage (routing id frame) and not being thread safe
 make the learning curve of zeromq hard to beginners. Without three of them
 zeromq become much simpler.
 
 * At least with NetMQ single part message is much faster than multipart.
 
 * New stacks, multipart is complicated to implement, with the new API it
 will much more easier to implement new stacks (like JSMQ) or any native
 stack.
 
 
 
 Pieter I'm looking forward to see how you expose the routing id in the
 czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.
 
 
 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.
 
 
 As ZMTP 3.1 is still in raw status, what do you think of removing the
 multipart from it? maybe the 3.1 will only support the new socket types.
 
 
 
 Anyway I really excited about this change.
 
 
 
 
 
 
 
 
 
 On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:
 
 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.
 
 
 Currently working on that :)
 
 
 Having two layers that both
 carry per-message data is... wrong IMO.
 
 Protocols supporting 'out of band' data aren't exactly uncommon.

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Arnaud Kapp

Hello everyone,

I have read this thread and the blog post from Pieter.
I am looking forward to some of those things, but I have some reserve
about other points. I'll try to explain what I think, and why.

I agree that ZMQ_IDENTITY is a bit confusing. Allowing remote client
to set routing information is a bit counterintuitive. Also there is
risk for impersonation. If a peer disconnect, an other one could
steal it's identity. I believe that relying on the socket identity
to identify peer is not a robust enough approach, even when using
CURVE.
Storing the peer routing id in the message's properties seems like a good idea.

I think thread-safe socket would be cool. As in great to have in some
particular situation, but most of the case it shouldn't be needed.


Aiming for a cleaner and simpler ZMQ internal code is good. We must be
careful however that this will not make the end-user's life harder. I
believe I see 2 points that would make my life harder as a ZMQ user
today if we had what we is discussed here:
+ The removal (or depreciation) of the REQ-REP socket. While
fragile over the network, I find them very useful for sending
synchronous messages between threads.
+ Removal / depreciation of multipart messages. Read below.

The next paragraph is about multipart messages. Let me be honest and
state that I have no idea of the real performance penalty induced by
multipart messages. What I want to defend here is their usefulness.
I *really* like multipart messages. To be honest, this is one of the
many features that made me try ZMQ. This is also the feature that
helps me most: I my current project, I mainly pass messages around
between thread, so I have less use for other awesome features like
auto reconnect or security.

IMO framing is very good because it allows you to send (and receive)
various kind of data *easily*. I like doing this kind of stuff:

zmqpp::message msg;
msg  HELLO WORLD  42  13;
s.send(msg);

And on the receiving end:
   std::string cmd;
   int value_1, value_2;
   msg  cmd  value_1  value_2;

Sure, there is no verification of the message content, the application
would even crash if a message was badly formatted. However, this is
not a concern when messages are flowing on inproc:// : I know that I
send and I know what to receive. I believe that the picture, which is
great in CZMQ, is not really appropriate for C++ and maybe other
language. My point is (for c++) that stream operator are a more
apprioriate / natural way of doing what the picture does.

ZProto is mentioned as a possible replacement to framing. Disclaimer:
I've taken a look at zproto, but I have never used it.
I think it's a great project with some strong advantages and real
world use case. However, it comes at the cost of more complexity. This
is what I meant when talking about making the end-user's life harder.
Do I want to use zproto to design complex, client-server protocol?
Yes, I probably should give it a try.
Do I really want to use zproto when sending 3-4 frames on an inproc
socket? I'd rather avoid it and keep it as simple as possible.

 In my opinion, maintaining some kind of multi-frame abstraction at the API 
 level is in fact very useful to applications, even if they are sent 
 atomically on the implementation level as you describe.
I totally agree with Joe on this point.


One last thing about multithreaded socket. Maybe this is completely
stupid as I know little about libzmq's internal, but wouldn't it be
possible to make explicit thread safe socket by providing a
zmq_socket_lock()/unlock() function that would allow ZMQ internal to
serialize multiple frames properly ?
Sure, this is a not as classy as properly thread-safe socket, but
would work in most case: grab the socket from shared memory, lock it,
write/receive, unlock it.



On Tue, Feb 3, 2015 at 3:52 PM, Luna Duclos
luna.duc...@palmstonegames.com wrote:
 I'm not sure if I like having picture as a libzmq level thing.
 It really only makes sense for C language bindings in my book. most other
 bindings use their own serialization mechanisms.

 On Tue, Feb 3, 2015 at 2:06 PM, Pieter Hintjens p...@imatix.com wrote:

 On Tue, Feb 3, 2015 at 1:01 PM, Doron Somech somdo...@gmail.com wrote:

  Pieter - regarding the blog post, I'm not sure about setting the routing
  id
  on the socket

 Indeed, that isn't atomic with sending and so can't work with
 threadsafe sockets. I've changed the article.

 I think we can either get the routing id as a message property, and/or
 add a reply to message semantic that does this implicitly. The
 second is easier for simple servers, the first is better for realistic
 servers.

 Since it only applies to server sockets, I think we would benefit from
 a zserver_reply () method (or similar).

 We might also think of moving the picture send/recv API into libzmq at
 some point since it's the immediate replacement for framing.

  I also have a question regarding the session management (really liked
  it),
  but how

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread MinRK

I happily use multipart rather a lot, to the point that I recommend that
pyzmq users exclusively use send/recv_multipart, and never single-frame
send/recv. I think it enables a really nice API. I find it very useful to
be able to send a large buffer plus a header of message information without
having to copy both blocks into a new buffer before handing off to zeromq.
I find this particularly valuable in Python, where building that new buffer
can be quite expensive.

In Python, and I suspect other higher level wrappers, where most messages
are bytes or buffer objects (not wrapped in Message APIs), moving the
routing information to message properties is not really an improvement,
because I'm not aware of any pyzmq users accessing message properties, so
the standard way to use pyzmq is to discard the `zmq_msg_t` object and only
deal with bytes. But that's more of an API question for pyzmq than libzmq.

As far as I can tell, I also use the multihop case that Pieter's blog post
suggests does not seem to happen in practice every day, so maybe my use
of zeromq is not typical.

From the never break things stance of libzmq and the discussion above, it
seems that there's no plan for multipart to ever be removed. Deprecated
even seems like too strong a word for a useful feature that will continue
to be supported indefinitely. Or is anyone proposing the eventual removal
of multipart?

-MinRK



On Tue, Feb 3, 2015 at 9:37 AM, Arnaud Kapp kapp.a...@gmail.com wrote:

 Hello everyone,

 I have read this thread and the blog post from Pieter.
 I am looking forward to some of those things, but I have some reserve
 about other points. I'll try to explain what I think, and why.

 I agree that ZMQ_IDENTITY is a bit confusing. Allowing remote client
 to set routing information is a bit counterintuitive. Also there is
 risk for impersonation. If a peer disconnect, an other one could
 steal it's identity. I believe that relying on the socket identity
 to identify peer is not a robust enough approach, even when using
 CURVE.
 Storing the peer routing id in the message's properties seems like a good
 idea.

 I think thread-safe socket would be cool. As in great to have in some
 particular situation, but most of the case it shouldn't be needed.


 Aiming for a cleaner and simpler ZMQ internal code is good. We must be
 careful however that this will not make the end-user's life harder. I
 believe I see 2 points that would make my life harder as a ZMQ user
 today if we had what we is discussed here:
 + The removal (or depreciation) of the REQ-REP socket. While
 fragile over the network, I find them very useful for sending
 synchronous messages between threads.
 + Removal / depreciation of multipart messages. Read below.

 The next paragraph is about multipart messages. Let me be honest and
 state that I have no idea of the real performance penalty induced by
 multipart messages. What I want to defend here is their usefulness.
 I *really* like multipart messages. To be honest, this is one of the
 many features that made me try ZMQ. This is also the feature that
 helps me most: I my current project, I mainly pass messages around
 between thread, so I have less use for other awesome features like
 auto reconnect or security.

 IMO framing is very good because it allows you to send (and receive)
 various kind of data *easily*. I like doing this kind of stuff:

 zmqpp::message msg;
 msg  HELLO WORLD  42  13;
 s.send(msg);

 And on the receiving end:
std::string cmd;
int value_1, value_2;
msg  cmd  value_1  value_2;

 Sure, there is no verification of the message content, the application
 would even crash if a message was badly formatted. However, this is
 not a concern when messages are flowing on inproc:// : I know that I
 send and I know what to receive. I believe that the picture, which is
 great in CZMQ, is not really appropriate for C++ and maybe other
 language. My point is (for c++) that stream operator are a more
 apprioriate / natural way of doing what the picture does.

 ZProto is mentioned as a possible replacement to framing. Disclaimer:
 I've taken a look at zproto, but I have never used it.
 I think it's a great project with some strong advantages and real
 world use case. However, it comes at the cost of more complexity. This
 is what I meant when talking about making the end-user's life harder.
 Do I want to use zproto to design complex, client-server protocol?
 Yes, I probably should give it a try.
 Do I really want to use zproto when sending 3-4 frames on an inproc
 socket? I'd rather avoid it and keep it as simple as possible.

  In my opinion, maintaining some kind of multi-frame abstraction at the
 API level is in fact very useful to applications, even if they are sent
 atomically on the implementation level as you describe.
 I totally agree with Joe on this point.


 One last thing about multithreaded socket. Maybe this is completely
 stupid as I know little about libzmq's internal,

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Gregg Irwin

AK So *maybe* a documentation effort is what is needed to better
AK explain those thing.

The guide explains things in great detail, and an engaging manner. How
do you improve on it, aside from removing the things that need
explaining?

-- Gregg

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

On Tue, Feb 3, 2015 at 11:27 PM, Thomas Rodgers rodg...@twrodgers.com wrote:

 * Sessions require protocol commands that are invisible to normal
 use. That means either protocol support (e.g. command frames) or some
 layering on top of ZeroMQ blobs.

For sure, command frames are designed for this and that's what we'd
use if we implemented sessions in libzmq. If we implemented them above
libzmq, e.g. in CZMQ/zserver, zclient then we'd need some other
layering.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

I like the idea of using ZAP as a test case. It's using multipart now
and that does indeed work well.

On Wed, Feb 4, 2015 at 12:35 AM, Gregg Irwin gr...@pointillistic.com wrote:
 AK So *maybe* a documentation effort is what is needed to better
 AK explain those thing.

 The guide explains things in great detail, and an engaging manner. How
 do you improve on it, aside from removing the things that need
 explaining?

 -- Gregg

 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Gregg Irwin

M Perhaps it is because I spend my days in a higher level language
M like Python, but zproto is not an attractive option.

Same here. I will read in detail about it shortly, but it may not make
it into my toolbox as a multipart replacement. Multipart looked very
cool when I found 0MQ, but I've ended up not using it much. I'm not
doing high performance stuff though. Simplicity and ease of use are
tops on my list.

-- Gregg

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Arnaud Kapp

@Doron: Yes I like multi part because its a very simple way build
multiple-fields message. I once considered using protobuff + raw
socket for a pet project of mine, and I ended using ZMQ framing
because it was simpler.
The problem if we pick an other serialization solution is that it
needs to built-in libzmq. Per-binding serialization would be a
disaster IMO.


 Arnaud - you can do that today, just use synchronization object, however this 
 is not really work, what if you call receive and there is no message 
 currently? so you block and the socket is still locked, maybe for long time. 
 Now you want to send from another thread but socket is locked.

Ah yes that's true, you'd have to use non-blocking read and such. I
reckon this is clearly not as practical as a fully thread safe socket
would be.


 Multi-part does easy serialization, yes. Not in C though. We've
 learned how to do variable list serialization with zproto-compatible
 picture sending in CZMQ. Is this a sufficient alternative? I think so.
 zsock_send() was essential to making actors cheap enough to use
 widely. egrep zsock_send *.c in malamute-core/src.

I'd argue that using zmsg_addstr() / zmsg_add() / zmsg_pop() are fine
for serialization. Agreed, when using the raw libzmq API this is not
as nice.


 If you're doing multipart a lot in an application, you can replace
 that with zproto. You already have contracts then, why not make them
 explicit and documented? That's always a win in my experience.

Yes I'm using multi-part quite a lot. I manually documented those
contracts and they are enforced by assert() (for anything that does'nt
come from the external world, obviously).
I'd argue that using c++ stream operator is easier, but I'd really
have to convert part of my code to zproto and give it an honest try.


 Picture sending solves a cross-language problem, namely how to
 construct zproto-style commands without using dedicated codecs. While
 the API is C-ish, the functionality is not. I'd expect that to be
 wrapped in bindings.

Yes but I believe pictures are sometime not practical. (I think) they
require that the picture be known when the message is being created.
I have some code where I do the following (if it's that weird / bad
please tell me):
   Some code create a message, push the identity frame and then some
application-level header.
   Control flow is handed to some other function, along with a
reference to the message. This function push a command frame and
more data (type and how many are unkown to the previous piece of
code).
   Control flow return to initial code, and send the message.

On the other side of the socket, I do something similar:
   Pop application-level header. Pop the command frame.
   Based on those 2 piece of data, call a function (again, along with
a reference to the message).
   Function will expect some kind of data layout, will extract it and
perform work.

In this scenario, the first part of the code doesn't know (or care,
really) about the real content of the message. In case my
explanation sucked, this is a bit similar to the ZAP rfc: the request
contains some must be there frames, and then some zero or more
opaque frames. In case of ZAP, those opaque frames only make sense to
mechanism specific code. In my example, they only make sense to some
specific message handling code. I do not really see how ZAP could be
that simple and nice to work w/ if it wasn't for multi-part messages.


 The main goal with this is simplicity and accuracy. I'm tired of
 teaching people ZeroMQ constructs that are clearly over-designed and
 disconnected from reality in subtle ways. Each apology for friction
 (sorry, but don't use REQ-REP, they are kind of weird, sorry,
 IDENTITY is not really identity, sorry, message sometimes means
 frame and sometimes message) really signals costs up and down the
 stacks.

This is a valid point. However, a good read through the guide really
help understanding how things work. I wouldn't be that surprised if
a lot of people having trouble understanding frames vs message or
REQ-REP problem didn't really read the zguide.
So *maybe* a documentation effort is what is needed to better explain
those thing. I agree about the IDENTITY tho.
I do not know, I don't have much experience teaching, and none teaching ZMQ.

Anyway, no matter what it's going to be interesting.

On Tue, Feb 3, 2015 at 11:04 PM, Pieter Hintjens p...@imatix.com wrote:
 Thanks for the questions. I'd try to flatter by saying how great these
 questions are, except I know you're all way too smart to fall for
 that. :)

 Let me answer a bunch of questions in one go:

 * Sorry for not posting this here. I've written a more detailed
 analysis of the motivation for this thread (and experiments):
 http://hintjens.com/blog:84.

 * The main goal with this is simplicity and accuracy. I'm tired of
 teaching people ZeroMQ constructs that are clearly over-designed and
 disconnected from reality in subtle ways. Each apology for friction
 (sorry,

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread frank

Hi,

Please excuse for writing this on my phone

I do not see any usecase nor detailed motivation what the problem is that 
causes this somewhat fundamental proposal. 

I think this approach is somewhat surprising and against the C4 thing. Not sure 
if this is a problem or not, but there is a lesson there somehow.

Could you please (re-)publish the problem descriptions? Is the problem in czmq 
, netmq, go-bindings or really in libzmq? 

Apart from this 'style' related thing, i have a technical question:

When a router (API4.x)received a curve-encrypted message, i thought that the 
identity field was the indented way to identify which of the many possible 
clients send the message.

In fact I was thinking of signing the identity field using the client secret 
key to have a robuster way of stopping clients which woul like to steal an 
identity. 

So with an 'int' as ID how is it possible to distinguish client1 from client2 
cryptografically?

Or did I misunderstand how this should have been done in API4.x? And is this 
possible with api5.x?



kind regards
  Frank





--  
unterwegs

Am 02.02.2015 um 10:23 schrieb Pieter Hintjens p...@imatix.com:

 Hi folks,
 
 We had an interesting pre-FOSDEM hackathon on Thursday and Friday, in 
 Brussels.
 
 One of the threads that came out, thanks mainly to Doron Somech
 (NetMQ) was a strategy for simplifying ZeroMQ. This started as a
 discussion of Nanomsg's dropping of multipart messages.
 
 Overall, multipart messages add a lot of complexity and confusion to
 the library and bindings. In case we forget:
 
 - frame vs, message vs. part
 - routing id frames
 - request-reply envelopes
 - router sockets
 - identities
 
 Multipart messages are the main reason we can't make threadsafe sockets.
 
 So, we're going to experiment with shifting ZeroMQ (libzmq, NetMQ,
 CZMQ, and other bindings) in this direction:
 
 - deprecate multipart messages from the API - when we need framing, we
 can use zproto
 - deprecate ROUTER and DEALER and slowly replace with SERVER / CLIENT
 sockets that refuse multipart data
 - deprecate REQ and REP
 - SERVER has get/set routing ID on message
 - routing ID is an integer
 - routing ID cannot be set by peer, so we deprecate ZMQ_IDENTITY
 - start to aim for threadsafe sockets
 - deprecate the ability to build request-reply chains
 
 ...
 
 This is not a roadmap, nor is this a proposal, it's just a realization
 by several of us that multipart messages create complexity, which we
 dislike, and which causes cost and irritation.
 
 Cheers
 Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev


smime.p7s
Description: S/MIME cryptographic signature
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Trevor Bernard

The single biggest problem I see people having in the JVM community is
their lack of understanding of the concurrency model. I haven't had
time to internalize the proposal just yet but if it helps people write
better/simpler software, I'm all for it.

-Trev

On Tue, Feb 3, 2015 at 4:33 PM, frank sound...@gmx.net wrote:
 Hi,

 Please excuse for writing this on my phone

 I do not see any usecase nor detailed motivation what the problem is that 
 causes this somewhat fundamental proposal.

 I think this approach is somewhat surprising and against the C4 thing. Not 
 sure if this is a problem or not, but there is a lesson there somehow.

 Could you please (re-)publish the problem descriptions? Is the problem in 
 czmq , netmq, go-bindings or really in libzmq?

 Apart from this 'style' related thing, i have a technical question:

 When a router (API4.x)received a curve-encrypted message, i thought that the 
 identity field was the indented way to identify which of the many possible 
 clients send the message.

 In fact I was thinking of signing the identity field using the client secret 
 key to have a robuster way of stopping clients which woul like to steal an 
 identity.

 So with an 'int' as ID how is it possible to distinguish client1 from client2 
 cryptografically?

 Or did I misunderstand how this should have been done in API4.x? And is this 
 possible with api5.x?



 kind regards
   Frank





 --
 unterwegs

 Am 02.02.2015 um 10:23 schrieb Pieter Hintjens p...@imatix.com:

 Hi folks,

 We had an interesting pre-FOSDEM hackathon on Thursday and Friday, in 
 Brussels.

 One of the threads that came out, thanks mainly to Doron Somech
 (NetMQ) was a strategy for simplifying ZeroMQ. This started as a
 discussion of Nanomsg's dropping of multipart messages.

 Overall, multipart messages add a lot of complexity and confusion to
 the library and bindings. In case we forget:

 - frame vs, message vs. part
 - routing id frames
 - request-reply envelopes
 - router sockets
 - identities

 Multipart messages are the main reason we can't make threadsafe sockets.

 So, we're going to experiment with shifting ZeroMQ (libzmq, NetMQ,
 CZMQ, and other bindings) in this direction:

 - deprecate multipart messages from the API - when we need framing, we
 can use zproto
 - deprecate ROUTER and DEALER and slowly replace with SERVER / CLIENT
 sockets that refuse multipart data
 - deprecate REQ and REP
 - SERVER has get/set routing ID on message
 - routing ID is an integer
 - routing ID cannot be set by peer, so we deprecate ZMQ_IDENTITY
 - start to aim for threadsafe sockets
 - deprecate the ability to build request-reply chains

 ...

 This is not a roadmap, nor is this a proposal, it's just a realization
 by several of us that multipart messages create complexity, which we
 dislike, and which causes cost and irritation.

 Cheers
 Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Ben Gray

Can I ask why people think thread safety around sockets is in any way
important? or even beneficial at all if it impacts performance negatively?
Most libraries I've found aren't thread safe because it's cheaper to be
that way and let the application handle any concurrency in a sensible way
for whatever they are trying to solve.


On 2 February 2015 at 20:36, Thomas Rodgers rodg...@twrodgers.com wrote:

 My eventual suggestion was to look at using metadata for this, it is
 already there and is per-connection. But there are (currently) a few
 problems with using it this way and I wanted to think those through a bit
 before making the suggestion.

 I am not sure why file_desc was placed where it is.

 On Mon, Feb 2, 2015 at 2:25 PM, Doron Somech somdo...@gmail.com wrote:

 Currently the router_id is in the last 4 bytes of the 64 bytes message
 size, could this cause alignment issue?

 Regarding the idea to combine it with the file descriptor, I think it can
 cause portability issues in the future, but it might be platform specific.

 Why wasn't it originally in end of the union which can solve the
 alignment issue?

 Anyway in the future we might not need it, because we can query the
 socket with the routing id to get metadata.

 On Mon, Feb 2, 2015 at 10:02 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 Some observations on adding a discrete router_id field to the message -

 Depending on alignment, this reduces the size of a VSM message by 4 or 8
 bytes.

 Does this field really need to be a member of the union? Currently
 msg_t::file_desc is 64 bit to force alignment, but FDs are 32bit values
 (even on Windows, where the FD is 64 bits, it will never give values 
 2^32). Could we not just pack these two together, and make use of the
 currently unused bits? It gets rid of the combinatorial expansion of adding
 fields to each member of the union (at least for this case).

 On Mon, Feb 2, 2015 at 1:06 PM, Doron Somech somdo...@gmail.com wrote:

 Brian -

 The issue is that with the new api and to enable thread safety the
 routing id is part of the message. So czmq new api should somehow expose
 routing id on the message and/or part of the send message.
 On Feb 2, 2015 7:39 PM, Brian Knox bk...@digitalocean.com wrote:

 Doron -

 I did a test implementation around this idea in goczmq today and I'm
 liking the semantics.  I wrote an interface that conforms to io.Reader and
 io.Writer where you pass a call a byte buffer and you get the message
 placed into it.

 When the socket is a router socket, it silently drops the ID frame
 under the hood, but sets a lastClientID in the socket struct, that if you
 need, you can get with a GetLastClientID call.

 For reference:

 https://github.com/zeromq/goczmq/blob/master/sock.go#L35 (note
 strings in go are just immutable byte arrays)

 https://github.com/zeromq/goczmq/blob/master/sock.go#L333

 https://github.com/zeromq/goczmq/blob/master/sock.go#L366

 So if the socket is a router, the Write call just transparently uses
 the lastClientID value.  In a one to one request / reply, you then don't
 need to think about it at all, and if you need to do async work, you can
 get and set the id as needed.

 I'll just move this to use the new API later - just wanted to try it
 out and see how I liked it, and I say thumbs up to dropping frames on the
 new Server / Client sockets.



 On Mon, Feb 2, 2015 at 4:37 PM, Doron Somech somdo...@gmail.com
 wrote:

 So the new sockets are in the github master, you can take a loot at
 the test to see how to use the new routing id field:


 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe,
 think of sending from multiple threads or receiving from multiple 
 threads.
 we cannot do that with Multipart.
 * There are a lot of good transport that already implement message
 framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq 
 required
 it is own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread
 safe make the learning curve of zeromq hard to beginners. Without three 
 of
 them zeromq become much simpler.
 * At least with NetMQ single part message is much faster than
 multipart.
 * New stacks, multipart is complicated to implement, with the new API
 it will much more easier to implement new stacks (like JSMQ) or any 
 native
 stack.

 Pieter I'm looking forward to see how you expose the routing id in
 the czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.

 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.

 As ZMTP 3.1 is still in raw status, what do you think of removing

Re: [zeromq-dev] Notes from a hackathon

I've written up my understanding of this thread: http://hintjens.com/blog:84

Thread safety would solve the perennial problem of sending data via
ZeroMQ, from short-lived threads. Performance in many cases would go
up, not down. We see way too many apps that create/connect a socket
for every message they send.

Apart from that I agree that thread safety is a weird thing and only
worth doing in very specific cases (ZeroMQ contexts and sockets seem
to be good candidates).



On Tue, Feb 3, 2015 at 10:52 AM, Ben Gray b...@benjamg.com wrote:
 Can I ask why people think thread safety around sockets is in any way
 important? or even beneficial at all if it impacts performance negatively?
 Most libraries I've found aren't thread safe because it's cheaper to be that
 way and let the application handle any concurrency in a sensible way for
 whatever they are trying to solve.


 On 2 February 2015 at 20:36, Thomas Rodgers rodg...@twrodgers.com wrote:

 My eventual suggestion was to look at using metadata for this, it is
 already there and is per-connection. But there are (currently) a few
 problems with using it this way and I wanted to think those through a bit
 before making the suggestion.

 I am not sure why file_desc was placed where it is.

 On Mon, Feb 2, 2015 at 2:25 PM, Doron Somech somdo...@gmail.com wrote:

 Currently the router_id is in the last 4 bytes of the 64 bytes message
 size, could this cause alignment issue?

 Regarding the idea to combine it with the file descriptor, I think it can
 cause portability issues in the future, but it might be platform specific.

 Why wasn't it originally in end of the union which can solve the
 alignment issue?

 Anyway in the future we might not need it, because we can query the
 socket with the routing id to get metadata.

 On Mon, Feb 2, 2015 at 10:02 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 Some observations on adding a discrete router_id field to the message -

 Depending on alignment, this reduces the size of a VSM message by 4 or 8
 bytes.

 Does this field really need to be a member of the union? Currently
 msg_t::file_desc is 64 bit to force alignment, but FDs are 32bit values
 (even on Windows, where the FD is 64 bits, it will never give values 
 2^32). Could we not just pack these two together, and make use of the
 currently unused bits? It gets rid of the combinatorial expansion of adding
 fields to each member of the union (at least for this case).

 On Mon, Feb 2, 2015 at 1:06 PM, Doron Somech somdo...@gmail.com wrote:

 Brian -

 The issue is that with the new api and to enable thread safety the
 routing id is part of the message. So czmq new api should somehow expose
 routing id on the message and/or part of the send message.

 On Feb 2, 2015 7:39 PM, Brian Knox bk...@digitalocean.com wrote:

 Doron -

 I did a test implementation around this idea in goczmq today and I'm
 liking the semantics.  I wrote an interface that conforms to io.Reader 
 and
 io.Writer where you pass a call a byte buffer and you get the message 
 placed
 into it.

 When the socket is a router socket, it silently drops the ID frame
 under the hood, but sets a lastClientID in the socket struct, that if you
 need, you can get with a GetLastClientID call.

 For reference:

 https://github.com/zeromq/goczmq/blob/master/sock.go#L35 (note
 strings in go are just immutable byte arrays)

 https://github.com/zeromq/goczmq/blob/master/sock.go#L333

 https://github.com/zeromq/goczmq/blob/master/sock.go#L366

 So if the socket is a router, the Write call just transparently uses
 the lastClientID value.  In a one to one request / reply, you then don't
 need to think about it at all, and if you need to do async work, you can 
 get
 and set the id as needed.

 I'll just move this to use the new API later - just wanted to try it
 out and see how I liked it, and I say thumbs up to dropping frames on the
 new Server / Client sockets.



 On Mon, Feb 2, 2015 at 4:37 PM, Doron Somech somdo...@gmail.com
 wrote:

 So the new sockets are in the github master, you can take a loot at
 the test to see how to use the new routing id field:


 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe,
 think of sending from multiple threads or receiving from multiple 
 threads.
 we cannot do that with Multipart.
 * There are a lot of good transport that already implement message
 framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq 
 required
 it is own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread
 safe make the learning curve of zeromq hard to beginners. Without three 
 of
 them zeromq become much simpler.
 * At least with NetMQ single part message is much faster than
 multipart.
 * New stacks, multipart is complicated to implement, with the new API
 it will much more easier to implement new

Re: [zeromq-dev] Notes from a hackathon

There has been a zmq_sendiov/recviov call for ages, undocumented and
rarely used. This would indeed be a good alternative to multipart for
the zerocopy use case.

On Mon, Feb 2, 2015 at 10:59 PM, Maurice Barnum m...@yahoo-inc.com wrote:
 Is there any thought to adding a writev() analogue like zmq_msg_sendv()?

 One of the reasons I send send multiple frames is to avoid stitching together
 buffers just to call send, which something like writev() would address that 
 use
 without the complexity of the socket buffering the parts.

 On Monday, February 2, 2015 8:36 AM, Joe McIlvain joe.eli@gmail.com 
 wrote:



 In my opinion, maintaining some kind of multi-frame abstraction at the API 
 level is in fact very useful to applications, even if they are sent 
 atomically on the implementation level as you describe.

 For example, in CZMQ It's quite useful in program flow to be able to push or 
 pop a single zframe onto or off of a zmsg, then send that zmsg as an entire 
 message.  Rarely do I find myself using zframe_send or zframe_recv with the 
 send more and receive more flags in a CZMQ application.  The CZMQ 
 picture messages are nice, but they are not always the easiest API to 
 integrate into an application.  Having zmsg and zframe remain (or something 
 like them), while removing zframe_send and zframe_recv could ease 
 applications relying on multi-frame semantics into the transition.  The 
 implementation for zmsg would simply serialize the frames in some 
 zmsg-determined way into a single-part zmq_msg to send and deserialize 
 according to the same scheme to receive.  This multi-frame serialization 
 essentially be at the application level rather than at the ZMTP level.

 In short, I think a multi-frame abstraction is important for applications, 
 but if we remove the ability to send one frame at a time at the fundamental 
 level, this abstraction can remain at a higher level (whether in libzmq, or 
 in CZMQ and other wrappers) and coexist happily with the proposed changes.




 On Mon, Feb 2, 2015 at 7:37 AM, Doron Somech somdo...@gmail.com wrote:

 So the new sockets are in the github master, you can take a loot at the test 
 to see how to use the new routing id field:

https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp


Few of the reasons I didn't like multi frames:

* I tried in the past to make both zeromq and NetMQ thread safe, think of 
sending from multiple threads or receiving from multiple threads. we cannot 
do that with Multipart.

* There are a lot of good transport that already implement message framing 
(UDP, WebSockets, SCTP and even HTTP), but because zeromq required it is own 
framing it was not easy to add them.

* Multipart, router usage (routing id frame) and not being thread safe make 
the learning curve of zeromq hard to beginners. Without three of them zeromq 
become much simpler.

* At least with NetMQ single part message is much faster than multipart.

* New stacks, multipart is complicated to implement, with the new API it will 
much more easier to implement new stacks (like JSMQ) or any native stack.



Pieter I'm looking forward to see how you expose the routing id in the czmq 
API.
I also like the czmq API for sending mutlipart messages (the picture feature) 
so maybe we can use that to generate single frame which is also compatible 
with zproto.


About the implementation, none of new sockets support any option now. server 
behave like mandatory router, so when router is not reachable or highwater 
mark is reached an error will be returned.


As ZMTP 3.1 is still in raw status, what do you think of removing the 
multipart from it? maybe the 3.1 will only support the new socket types.



Anyway I really excited about this change.









On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com wrote:

What we really want IMO is per-peer metadata, and an API to get/set
this. Using messages is a hack.


Currently working on that :)


Having two layers that both
carry per-message data is... wrong IMO.

Protocols supporting 'out of band' data aren't exactly uncommon.



However the key thing is, what's the problem. Then we can discuss
solutions to that.


I don't have an immediate use-case justifying it. I noted it, mostly because 
it has come up a few times since I started paying attention.


On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

 It seems to me that in order to remove multi-part messages and introduce 
 new
 socket types (e.g. SERVER/CLIENT) that would
 necessitate a revision of the wire protocol. If we are going to do that, 
 it
 might be worth considering per-message
 metadata -

We'd have to be very clear about the problem that per-message metadata
is aiming for. There is an elegance to delivering blobs and nothing
more. Metadata can be added on top using zproto.

What we really want IMO is per-peer metadata, and an API to get/set
this. Using messages is a hack. If we are sending/receiving data on

Re: [zeromq-dev] Notes from a hackathon

2015-02-03 Thread Doron Somech

The thing is that you like multi-part because it simple serialization tool
and not because mutli access to socket,

I think that removing mutli-part doesn't necessary mean removing
serialization solution.

For java and .Net we can use fluent api for sending:

Message.New().AddString(Hello).AddString(World).Send(socket);

The advantage of this kind of API is that we can calculate the needed
buffer in advance (only the send will serialize everything).

Receiving can be very close to the current API, but you don't really have
frames, you can only pop.

Any serialization solution we will pick should in my opinion be compatible
with ZProto.

Arnaud - you can do that today, just use synchronization object, however
this is not really work, what if you call receive and there is no message
currently? so you block and the socket is still locked, maybe for long
time. Now you want to send from another thread but socket is locked.

MinRK - the new CLIENT and  SERVER probably won't support multi-part (and
maybe keep-alive and heartbeat).






On Tue, Feb 3, 2015 at 10:00 PM, MinRK benjami...@gmail.com wrote:

 I happily use multipart rather a lot, to the point that I recommend that
 pyzmq users exclusively use send/recv_multipart, and never single-frame
 send/recv. I think it enables a really nice API. I find it very useful to
 be able to send a large buffer plus a header of message information without
 having to copy both blocks into a new buffer before handing off to zeromq.
 I find this particularly valuable in Python, where building that new buffer
 can be quite expensive.

 In Python, and I suspect other higher level wrappers, where most messages
 are bytes or buffer objects (not wrapped in Message APIs), moving the
 routing information to message properties is not really an improvement,
 because I'm not aware of any pyzmq users accessing message properties, so
 the standard way to use pyzmq is to discard the `zmq_msg_t` object and only
 deal with bytes. But that's more of an API question for pyzmq than libzmq.

 As far as I can tell, I also use the multihop case that Pieter's blog post
 suggests does not seem to happen in practice every day, so maybe my use
 of zeromq is not typical.

 From the never break things stance of libzmq and the discussion above,
 it seems that there's no plan for multipart to ever be removed. Deprecated
 even seems like too strong a word for a useful feature that will continue
 to be supported indefinitely. Or is anyone proposing the eventual removal
 of multipart?

 -MinRK



 On Tue, Feb 3, 2015 at 9:37 AM, Arnaud Kapp kapp.a...@gmail.com wrote:

 Hello everyone,

 I have read this thread and the blog post from Pieter.
 I am looking forward to some of those things, but I have some reserve
 about other points. I'll try to explain what I think, and why.

 I agree that ZMQ_IDENTITY is a bit confusing. Allowing remote client
 to set routing information is a bit counterintuitive. Also there is
 risk for impersonation. If a peer disconnect, an other one could
 steal it's identity. I believe that relying on the socket identity
 to identify peer is not a robust enough approach, even when using
 CURVE.
 Storing the peer routing id in the message's properties seems like a good
 idea.

 I think thread-safe socket would be cool. As in great to have in some
 particular situation, but most of the case it shouldn't be needed.


 Aiming for a cleaner and simpler ZMQ internal code is good. We must be
 careful however that this will not make the end-user's life harder. I
 believe I see 2 points that would make my life harder as a ZMQ user
 today if we had what we is discussed here:
 + The removal (or depreciation) of the REQ-REP socket. While
 fragile over the network, I find them very useful for sending
 synchronous messages between threads.
 + Removal / depreciation of multipart messages. Read below.

 The next paragraph is about multipart messages. Let me be honest and
 state that I have no idea of the real performance penalty induced by
 multipart messages. What I want to defend here is their usefulness.
 I *really* like multipart messages. To be honest, this is one of the
 many features that made me try ZMQ. This is also the feature that
 helps me most: I my current project, I mainly pass messages around
 between thread, so I have less use for other awesome features like
 auto reconnect or security.

 IMO framing is very good because it allows you to send (and receive)
 various kind of data *easily*. I like doing this kind of stuff:

 zmqpp::message msg;
 msg  HELLO WORLD  42  13;
 s.send(msg);

 And on the receiving end:
std::string cmd;
int value_1, value_2;
msg  cmd  value_1  value_2;

 Sure, there is no verification of the message content, the application
 would even crash if a message was badly formatted. However, this is
 not a concern when messages are flowing on inproc:// : I know that I
 send and I know what to receive. I believe that the picture, which is

Re: [zeromq-dev] Notes from a hackathon

Currently the router_id is in the last 4 bytes of the 64 bytes message
size, could this cause alignment issue?

Regarding the idea to combine it with the file descriptor, I think it can
cause portability issues in the future, but it might be platform specific.

Why wasn't it originally in end of the union which can solve the alignment
issue?

Anyway in the future we might not need it, because we can query the socket
with the routing id to get metadata.

On Mon, Feb 2, 2015 at 10:02 PM, Thomas Rodgers rodg...@twrodgers.com
wrote:

 Some observations on adding a discrete router_id field to the message -

 Depending on alignment, this reduces the size of a VSM message by 4 or 8
 bytes.

 Does this field really need to be a member of the union? Currently
 msg_t::file_desc is 64 bit to force alignment, but FDs are 32bit values
 (even on Windows, where the FD is 64 bits, it will never give values 
 2^32). Could we not just pack these two together, and make use of the
 currently unused bits? It gets rid of the combinatorial expansion of adding
 fields to each member of the union (at least for this case).

 On Mon, Feb 2, 2015 at 1:06 PM, Doron Somech somdo...@gmail.com wrote:

 Brian -

 The issue is that with the new api and to enable thread safety the
 routing id is part of the message. So czmq new api should somehow expose
 routing id on the message and/or part of the send message.
 On Feb 2, 2015 7:39 PM, Brian Knox bk...@digitalocean.com wrote:

 Doron -

 I did a test implementation around this idea in goczmq today and I'm
 liking the semantics.  I wrote an interface that conforms to io.Reader and
 io.Writer where you pass a call a byte buffer and you get the message
 placed into it.

 When the socket is a router socket, it silently drops the ID frame under
 the hood, but sets a lastClientID in the socket struct, that if you need,
 you can get with a GetLastClientID call.

 For reference:

 https://github.com/zeromq/goczmq/blob/master/sock.go#L35 (note
 strings in go are just immutable byte arrays)

 https://github.com/zeromq/goczmq/blob/master/sock.go#L333

 https://github.com/zeromq/goczmq/blob/master/sock.go#L366

 So if the socket is a router, the Write call just transparently uses the
 lastClientID value.  In a one to one request / reply, you then don't need
 to think about it at all, and if you need to do async work, you can get and
 set the id as needed.

 I'll just move this to use the new API later - just wanted to try it out
 and see how I liked it, and I say thumbs up to dropping frames on the new
 Server / Client sockets.



 On Mon, Feb 2, 2015 at 4:37 PM, Doron Somech somdo...@gmail.com wrote:

 So the new sockets are in the github master, you can take a loot at the
 test to see how to use the new routing id field:


 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe, think
 of sending from multiple threads or receiving from multiple threads. we
 cannot do that with Multipart.
 * There are a lot of good transport that already implement message
 framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required
 it is own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread safe
 make the learning curve of zeromq hard to beginners. Without three of them
 zeromq become much simpler.
 * At least with NetMQ single part message is much faster than multipart.
 * New stacks, multipart is complicated to implement, with the new API
 it will much more easier to implement new stacks (like JSMQ) or any native
 stack.

 Pieter I'm looking forward to see how you expose the routing id in the
 czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.

 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.

 As ZMTP 3.1 is still in raw status, what do you think of removing the
 multipart from it? maybe the 3.1 will only support the new socket types.

 Anyway I really excited about this change.





 On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


 Currently working on that :)

 Having two layers that both
 carry per-message data is... wrong IMO.


 Protocols supporting 'out of band' data aren't exactly uncommon.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.


 I don't have an immediate use-case justifying it. I noted it, mostly
 because it has come up a few times since I started paying attention.

 On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

  It seems to

Re: [zeromq-dev] Notes from a hackathon

2015-02-02 Thread Brian Knox

Doron - yup, this was just an implementation for the current router
sockets  that makes them simpler to work with.

In the case of a single frame message containing the id, as long as the id
is a consistent length then I can slice the id and payload out of the
[]byte trivially.

My one ask on this is we do NOT have the ability to set their own socket id
on the SERVER and CLIENT sockets.  Right now the default id length a ROUTER
assigns is small, but a user can set an id of up to 255 bytes - having a
single consistent length for an id would be much simpler.

Cheers,
Brian

On Mon, Feb 2, 2015 at 8:06 PM, Doron Somech somdo...@gmail.com wrote:

 Brian -

 The issue is that with the new api and to enable thread safety the routing
 id is part of the message. So czmq new api should somehow expose routing id
 on the message and/or part of the send message.
 On Feb 2, 2015 7:39 PM, Brian Knox bk...@digitalocean.com wrote:

 Doron -

 I did a test implementation around this idea in goczmq today and I'm
 liking the semantics.  I wrote an interface that conforms to io.Reader and
 io.Writer where you pass a call a byte buffer and you get the message
 placed into it.

 When the socket is a router socket, it silently drops the ID frame under
 the hood, but sets a lastClientID in the socket struct, that if you need,
 you can get with a GetLastClientID call.

 For reference:

 https://github.com/zeromq/goczmq/blob/master/sock.go#L35 (note strings
 in go are just immutable byte arrays)

 https://github.com/zeromq/goczmq/blob/master/sock.go#L333

 https://github.com/zeromq/goczmq/blob/master/sock.go#L366

 So if the socket is a router, the Write call just transparently uses the
 lastClientID value.  In a one to one request / reply, you then don't need
 to think about it at all, and if you need to do async work, you can get and
 set the id as needed.

 I'll just move this to use the new API later - just wanted to try it out
 and see how I liked it, and I say thumbs up to dropping frames on the new
 Server / Client sockets.



 On Mon, Feb 2, 2015 at 4:37 PM, Doron Somech somdo...@gmail.com wrote:

 So the new sockets are in the github master, you can take a loot at the
 test to see how to use the new routing id field:

 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe, think
 of sending from multiple threads or receiving from multiple threads. we
 cannot do that with Multipart.
 * There are a lot of good transport that already implement message
 framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required
 it is own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread safe
 make the learning curve of zeromq hard to beginners. Without three of them
 zeromq become much simpler.
 * At least with NetMQ single part message is much faster than multipart.
 * New stacks, multipart is complicated to implement, with the new API it
 will much more easier to implement new stacks (like JSMQ) or any native
 stack.

 Pieter I'm looking forward to see how you expose the routing id in the
 czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.

 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.

 As ZMTP 3.1 is still in raw status, what do you think of removing the
 multipart from it? maybe the 3.1 will only support the new socket types.

 Anyway I really excited about this change.





 On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


 Currently working on that :)

 Having two layers that both
 carry per-message data is... wrong IMO.


 Protocols supporting 'out of band' data aren't exactly uncommon.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.


 I don't have an immediate use-case justifying it. I noted it, mostly
 because it has come up a few times since I started paying attention.

 On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

  It seems to me that in order to remove multi-part messages and
 introduce new
  socket types (e.g. SERVER/CLIENT) that would
  necessitate a revision of the wire protocol. If we are going to do
 that, it
  might be worth considering per-message
  metadata -

 We'd have to be very clear about the problem that per-message metadata
 is aiming for. There is an elegance to delivering blobs and nothing
 more. Metadata can be added on top using zproto.

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.

Re: [zeromq-dev] Notes from a hackathon

Some observations on adding a discrete router_id field to the message -

Depending on alignment, this reduces the size of a VSM message by 4 or 8
bytes.

Does this field really need to be a member of the union? Currently
msg_t::file_desc is 64 bit to force alignment, but FDs are 32bit values
(even on Windows, where the FD is 64 bits, it will never give values 
2^32). Could we not just pack these two together, and make use of the
currently unused bits? It gets rid of the combinatorial expansion of adding
fields to each member of the union (at least for this case).

On Mon, Feb 2, 2015 at 1:06 PM, Doron Somech somdo...@gmail.com wrote:

 Brian -

 The issue is that with the new api and to enable thread safety the routing
 id is part of the message. So czmq new api should somehow expose routing id
 on the message and/or part of the send message.
 On Feb 2, 2015 7:39 PM, Brian Knox bk...@digitalocean.com wrote:

 Doron -

 I did a test implementation around this idea in goczmq today and I'm
 liking the semantics.  I wrote an interface that conforms to io.Reader and
 io.Writer where you pass a call a byte buffer and you get the message
 placed into it.

 When the socket is a router socket, it silently drops the ID frame under
 the hood, but sets a lastClientID in the socket struct, that if you need,
 you can get with a GetLastClientID call.

 For reference:

 https://github.com/zeromq/goczmq/blob/master/sock.go#L35 (note strings
 in go are just immutable byte arrays)

 https://github.com/zeromq/goczmq/blob/master/sock.go#L333

 https://github.com/zeromq/goczmq/blob/master/sock.go#L366

 So if the socket is a router, the Write call just transparently uses the
 lastClientID value.  In a one to one request / reply, you then don't need
 to think about it at all, and if you need to do async work, you can get and
 set the id as needed.

 I'll just move this to use the new API later - just wanted to try it out
 and see how I liked it, and I say thumbs up to dropping frames on the new
 Server / Client sockets.



 On Mon, Feb 2, 2015 at 4:37 PM, Doron Somech somdo...@gmail.com wrote:

 So the new sockets are in the github master, you can take a loot at the
 test to see how to use the new routing id field:

 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe, think
 of sending from multiple threads or receiving from multiple threads. we
 cannot do that with Multipart.
 * There are a lot of good transport that already implement message
 framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required
 it is own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread safe
 make the learning curve of zeromq hard to beginners. Without three of them
 zeromq become much simpler.
 * At least with NetMQ single part message is much faster than multipart.
 * New stacks, multipart is complicated to implement, with the new API it
 will much more easier to implement new stacks (like JSMQ) or any native
 stack.

 Pieter I'm looking forward to see how you expose the routing id in the
 czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.

 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.

 As ZMTP 3.1 is still in raw status, what do you think of removing the
 multipart from it? maybe the 3.1 will only support the new socket types.

 Anyway I really excited about this change.





 On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


 Currently working on that :)

 Having two layers that both
 carry per-message data is... wrong IMO.


 Protocols supporting 'out of band' data aren't exactly uncommon.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.


 I don't have an immediate use-case justifying it. I noted it, mostly
 because it has come up a few times since I started paying attention.

 On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

  It seems to me that in order to remove multi-part messages and
 introduce new
  socket types (e.g. SERVER/CLIENT) that would
  necessitate a revision of the wire protocol. If we are going to do
 that, it
  might be worth considering per-message
  metadata -

 We'd have to be very clear about the problem that per-message metadata
 is aiming for. There is an elegance to delivering blobs and nothing
 more. Metadata can be added on top using zproto.

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack. If we

Re: [zeromq-dev] Notes from a hackathon

My eventual suggestion was to look at using metadata for this, it is
already there and is per-connection. But there are (currently) a few
problems with using it this way and I wanted to think those through a bit
before making the suggestion.

I am not sure why file_desc was placed where it is.

On Mon, Feb 2, 2015 at 2:25 PM, Doron Somech somdo...@gmail.com wrote:

 Currently the router_id is in the last 4 bytes of the 64 bytes message
 size, could this cause alignment issue?

 Regarding the idea to combine it with the file descriptor, I think it can
 cause portability issues in the future, but it might be platform specific.

 Why wasn't it originally in end of the union which can solve the alignment
 issue?

 Anyway in the future we might not need it, because we can query the socket
 with the routing id to get metadata.

 On Mon, Feb 2, 2015 at 10:02 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 Some observations on adding a discrete router_id field to the message -

 Depending on alignment, this reduces the size of a VSM message by 4 or 8
 bytes.

 Does this field really need to be a member of the union? Currently
 msg_t::file_desc is 64 bit to force alignment, but FDs are 32bit values
 (even on Windows, where the FD is 64 bits, it will never give values 
 2^32). Could we not just pack these two together, and make use of the
 currently unused bits? It gets rid of the combinatorial expansion of adding
 fields to each member of the union (at least for this case).

 On Mon, Feb 2, 2015 at 1:06 PM, Doron Somech somdo...@gmail.com wrote:

 Brian -

 The issue is that with the new api and to enable thread safety the
 routing id is part of the message. So czmq new api should somehow expose
 routing id on the message and/or part of the send message.
 On Feb 2, 2015 7:39 PM, Brian Knox bk...@digitalocean.com wrote:

 Doron -

 I did a test implementation around this idea in goczmq today and I'm
 liking the semantics.  I wrote an interface that conforms to io.Reader and
 io.Writer where you pass a call a byte buffer and you get the message
 placed into it.

 When the socket is a router socket, it silently drops the ID frame
 under the hood, but sets a lastClientID in the socket struct, that if you
 need, you can get with a GetLastClientID call.

 For reference:

 https://github.com/zeromq/goczmq/blob/master/sock.go#L35 (note
 strings in go are just immutable byte arrays)

 https://github.com/zeromq/goczmq/blob/master/sock.go#L333

 https://github.com/zeromq/goczmq/blob/master/sock.go#L366

 So if the socket is a router, the Write call just transparently uses
 the lastClientID value.  In a one to one request / reply, you then don't
 need to think about it at all, and if you need to do async work, you can
 get and set the id as needed.

 I'll just move this to use the new API later - just wanted to try it
 out and see how I liked it, and I say thumbs up to dropping frames on the
 new Server / Client sockets.



 On Mon, Feb 2, 2015 at 4:37 PM, Doron Somech somdo...@gmail.com
 wrote:

 So the new sockets are in the github master, you can take a loot at
 the test to see how to use the new routing id field:


 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe, think
 of sending from multiple threads or receiving from multiple threads. we
 cannot do that with Multipart.
 * There are a lot of good transport that already implement message
 framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required
 it is own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread safe
 make the learning curve of zeromq hard to beginners. Without three of them
 zeromq become much simpler.
 * At least with NetMQ single part message is much faster than
 multipart.
 * New stacks, multipart is complicated to implement, with the new API
 it will much more easier to implement new stacks (like JSMQ) or any native
 stack.

 Pieter I'm looking forward to see how you expose the routing id in the
 czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.

 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.

 As ZMTP 3.1 is still in raw status, what do you think of removing the
 multipart from it? maybe the 3.1 will only support the new socket types.

 Anyway I really excited about this change.





 On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


 Currently working on that :)

 Having two layers that both
 carry per-message data is... wrong IMO.

Re: [zeromq-dev] Notes from a hackathon

2015-02-02 Thread Maurice Barnum

Is there any thought to adding a writev() analogue like zmq_msg_sendv()?

One of the reasons I send send multiple frames is to avoid stitching together 
buffers just to call send, which something like writev() would address that use
without the complexity of the socket buffering the parts.

On Monday, February 2, 2015 8:36 AM, Joe McIlvain joe.eli@gmail.com wrote:



In my opinion, maintaining some kind of multi-frame abstraction at the API 
level is in fact very useful to applications, even if they are sent atomically 
on the implementation level as you describe.

For example, in CZMQ It's quite useful in program flow to be able to push or 
pop a single zframe onto or off of a zmsg, then send that zmsg as an entire 
message.  Rarely do I find myself using zframe_send or zframe_recv with the 
send more and receive more flags in a CZMQ application.  The CZMQ picture 
messages are nice, but they are not always the easiest API to integrate into an 
application.  Having zmsg and zframe remain (or something like them), while 
removing zframe_send and zframe_recv could ease applications relying on 
multi-frame semantics into the transition.  The implementation for zmsg would 
simply serialize the frames in some zmsg-determined way into a single-part 
zmq_msg to send and deserialize according to the same scheme to receive.  This 
multi-frame serialization essentially be at the application level rather than 
at the ZMTP level.

In short, I think a multi-frame abstraction is important for applications, but 
if we remove the ability to send one frame at a time at the fundamental level, 
this abstraction can remain at a higher level (whether in libzmq, or in CZMQ 
and other wrappers) and coexist happily with the proposed changes.




On Mon, Feb 2, 2015 at 7:37 AM, Doron Somech somdo...@gmail.com wrote:

So the new sockets are in the github master, you can take a loot at the test to 
see how to use the new routing id field: 

https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp


Few of the reasons I didn't like multi frames:

* I tried in the past to make both zeromq and NetMQ thread safe, think of 
sending from multiple threads or receiving from multiple threads. we cannot do 
that with Multipart.

* There are a lot of good transport that already implement message framing 
(UDP, WebSockets, SCTP and even HTTP), but because zeromq required it is own 
framing it was not easy to add them.

* Multipart, router usage (routing id frame) and not being thread safe make 
the learning curve of zeromq hard to beginners. Without three of them zeromq 
become much simpler. 

* At least with NetMQ single part message is much faster than multipart.

* New stacks, multipart is complicated to implement, with the new API it will 
much more easier to implement new stacks (like JSMQ) or any native stack.



Pieter I'm looking forward to see how you expose the routing id in the czmq 
API. 
I also like the czmq API for sending mutlipart messages (the picture feature) 
so maybe we can use that to generate single frame which is also compatible 
with zproto.


About the implementation, none of new sockets support any option now. server 
behave like mandatory router, so when router is not reachable or highwater 
mark is reached an error will be returned.


As ZMTP 3.1 is still in raw status, what do you think of removing the 
multipart from it? maybe the 3.1 will only support the new socket types.



Anyway I really excited about this change. 









On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com wrote:

What we really want IMO is per-peer metadata, and an API to get/set
this. Using messages is a hack. 


Currently working on that :)


Having two layers that both
carry per-message data is... wrong IMO.
 
Protocols supporting 'out of band' data aren't exactly uncommon.



However the key thing is, what's the problem. Then we can discuss
solutions to that.


I don't have an immediate use-case justifying it. I noted it, mostly because 
it has come up a few times since I started paying attention. 


On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

 It seems to me that in order to remove multi-part messages and introduce new
 socket types (e.g. SERVER/CLIENT) that would
 necessitate a revision of the wire protocol. If we are going to do that, it
 might be worth considering per-message
 metadata -

We'd have to be very clear about the problem that per-message metadata
is aiming for. There is an elegance to delivering blobs and nothing
more. Metadata can be added on top using zproto.

What we really want IMO is per-peer metadata, and an API to get/set
this. Using messages is a hack. If we are sending/receiving data on a
per-message basis, that is the message. Having two layers that both
carry per-message data is... wrong IMO.

However the key thing is, what's the problem. Then we can discuss
solutions to that.

-Pieter

___

Re: [zeromq-dev] Notes from a hackathon

2015-02-02 Thread Pieter Hintjens

 It seems to me that in order to remove multi-part messages and introduce new
 socket types (e.g. SERVER/CLIENT) that would
 necessitate a revision of the wire protocol. If we are going to do that, it
 might be worth considering per-message
 metadata -

We'd have to be very clear about the problem that per-message metadata
is aiming for. There is an elegance to delivering blobs and nothing
more. Metadata can be added on top using zproto.

What we really want IMO is per-peer metadata, and an API to get/set
this. Using messages is a hack. If we are sending/receiving data on a
per-message basis, that is the message. Having two layers that both
carry per-message data is... wrong IMO.

However the key thing is, what's the problem. Then we can discuss
solutions to that.

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon


 - deprecate REQ and REP


It would seem, given their limited utility, that marking these DEPRECATED
in the next version of libzmq would be an easy step to take.

It seems to me that in order to remove multi-part messages and introduce
new socket types (e.g. SERVER/CLIENT) that would
necessitate a revision of the wire protocol. If we are going to do that, it
might be worth considering per-message
metadata -

* There have been at least a few questions/requests in this direction, the
wire protocol doesn't support it.
* The metadata would have the same basic wire format as connection
metadata, and read as part of a single message.
* It might serve to eliminate some uses of multi-part messages without
resorting to some form of in-message framing.


On Mon, Feb 2, 2015 at 3:23 AM, Pieter Hintjens p...@imatix.com wrote:

 Hi folks,

 We had an interesting pre-FOSDEM hackathon on Thursday and Friday, in
 Brussels.

 One of the threads that came out, thanks mainly to Doron Somech
 (NetMQ) was a strategy for simplifying ZeroMQ. This started as a
 discussion of Nanomsg's dropping of multipart messages.

 Overall, multipart messages add a lot of complexity and confusion to
 the library and bindings. In case we forget:

 - frame vs, message vs. part
 - routing id frames
 - request-reply envelopes
 - router sockets
 - identities

 Multipart messages are the main reason we can't make threadsafe sockets.

 So, we're going to experiment with shifting ZeroMQ (libzmq, NetMQ,
 CZMQ, and other bindings) in this direction:

 - deprecate multipart messages from the API - when we need framing, we
 can use zproto
 - deprecate ROUTER and DEALER and slowly replace with SERVER / CLIENT
 sockets that refuse multipart data
 - deprecate REQ and REP
 - SERVER has get/set routing ID on message
 - routing ID is an integer
 - routing ID cannot be set by peer, so we deprecate ZMQ_IDENTITY
 - start to aim for threadsafe sockets
 - deprecate the ability to build request-reply chains

 ...

 This is not a roadmap, nor is this a proposal, it's just a realization
 by several of us that multipart messages create complexity, which we
 dislike, and which causes cost and irritation.

 Cheers
 Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon


 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


Currently working on that :)

Having two layers that both
 carry per-message data is... wrong IMO.


Protocols supporting 'out of band' data aren't exactly uncommon.

However the key thing is, what's the problem. Then we can discuss
 solutions to that.


I don't have an immediate use-case justifying it. I noted it, mostly
because it has come up a few times since I started paying attention.

On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

  It seems to me that in order to remove multi-part messages and introduce
 new
  socket types (e.g. SERVER/CLIENT) that would
  necessitate a revision of the wire protocol. If we are going to do that,
 it
  might be worth considering per-message
  metadata -

 We'd have to be very clear about the problem that per-message metadata
 is aiming for. There is an elegance to delivering blobs and nothing
 more. Metadata can be added on top using zproto.

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack. If we are sending/receiving data on a
 per-message basis, that is the message. Having two layers that both
 carry per-message data is... wrong IMO.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-02 Thread Pieter Hintjens

Ironically I was writing a blog post decrying version numbers. But
yes, this would perhaps be a V5 API. The thing is, we want to make the
API contract work across many stacks, including libzmq, JeroMQ, CZMQ,
NetMQ, etc. So we have evolved past software version numbers, perhaps.

Let's consider this as a new generation of the API and protocol
contract, document it as such (new RFCs, eventually), and eventually
move the software stacks to support it, while also supporting older
stable versions of the contracts.

Brian Knox points out that SERVER and CLIENT also remove all stress of
REQ / REP state machines. The caller decides: if you do straight recv
request, send reply on SERVER, it'll work automatically. And if you do
recv, recv, recv then you manage the routing IDs yourself... :-) Same
for CLIENT replacing REQ.

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

So the new sockets are in the github master, you can take a loot at the
test to see how to use the new routing id field:

https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

Few of the reasons I didn't like multi frames:
* I tried in the past to make both zeromq and NetMQ thread safe, think of
sending from multiple threads or receiving from multiple threads. we cannot
do that with Multipart.
* There are a lot of good transport that already implement message framing
(UDP, WebSockets, SCTP and even HTTP), but because zeromq required it is
own framing it was not easy to add them.
* Multipart, router usage (routing id frame) and not being thread safe make
the learning curve of zeromq hard to beginners. Without three of them
zeromq become much simpler.
* At least with NetMQ single part message is much faster than multipart.
* New stacks, multipart is complicated to implement, with the new API it
will much more easier to implement new stacks (like JSMQ) or any native
stack.

Pieter I'm looking forward to see how you expose the routing id in the czmq
API.
I also like the czmq API for sending mutlipart messages (the picture
feature) so maybe we can use that to generate single frame which is also
compatible with zproto.

About the implementation, none of new sockets support any option now.
server behave like mandatory router, so when router is not reachable or
highwater mark is reached an error will be returned.

As ZMTP 3.1 is still in raw status, what do you think of removing the
multipart from it? maybe the 3.1 will only support the new socket types.

Anyway I really excited about this change.





On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
wrote:

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


 Currently working on that :)

 Having two layers that both
 carry per-message data is... wrong IMO.


 Protocols supporting 'out of band' data aren't exactly uncommon.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.


 I don't have an immediate use-case justifying it. I noted it, mostly
 because it has come up a few times since I started paying attention.

 On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

  It seems to me that in order to remove multi-part messages and
 introduce new
  socket types (e.g. SERVER/CLIENT) that would
  necessitate a revision of the wire protocol. If we are going to do
 that, it
  might be worth considering per-message
  metadata -

 We'd have to be very clear about the problem that per-message metadata
 is aiming for. There is an elegance to delivering blobs and nothing
 more. Metadata can be added on top using zproto.

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack. If we are sending/receiving data on a
 per-message basis, that is the message. Having two layers that both
 carry per-message data is... wrong IMO.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev



 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev


___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

2015-02-02 Thread Joe McIlvain

In my opinion, maintaining some kind of multi-frame abstraction at the API
level is in fact very useful to applications, even if they are sent
atomically on the implementation level as you describe.

For example, in CZMQ It's quite useful in program flow to be able to push
or pop a single zframe onto or off of a zmsg, then send that zmsg as an
entire message.  Rarely do I find myself using zframe_send or zframe_recv
with the send more and receive more flags in a CZMQ application.  The
CZMQ picture messages are nice, but they are not always the easiest API
to integrate into an application.  Having zmsg and zframe remain (or
something like them), while removing zframe_send and zframe_recv could ease
applications relying on multi-frame semantics into the transition.  The
implementation for zmsg would simply serialize the frames in some
zmsg-determined way into a single-part zmq_msg to send and deserialize
according to the same scheme to receive.  This multi-frame serialization
essentially be at the application level rather than at the ZMTP level.

In short, I think a multi-frame abstraction is important for applications,
but if we remove the ability to send one frame at a time at the fundamental
level, this abstraction can remain at a higher level (whether in libzmq, or
in CZMQ and other wrappers) and coexist happily with the proposed changes.



On Mon, Feb 2, 2015 at 7:37 AM, Doron Somech somdo...@gmail.com wrote:

 So the new sockets are in the github master, you can take a loot at the
 test to see how to use the new routing id field:

 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe, think of
 sending from multiple threads or receiving from multiple threads. we cannot
 do that with Multipart.
 * There are a lot of good transport that already implement message framing
 (UDP, WebSockets, SCTP and even HTTP), but because zeromq required it is
 own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread safe
 make the learning curve of zeromq hard to beginners. Without three of them
 zeromq become much simpler.
 * At least with NetMQ single part message is much faster than multipart.
 * New stacks, multipart is complicated to implement, with the new API it
 will much more easier to implement new stacks (like JSMQ) or any native
 stack.

 Pieter I'm looking forward to see how you expose the routing id in the
 czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.

 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.

 As ZMTP 3.1 is still in raw status, what do you think of removing the
 multipart from it? maybe the 3.1 will only support the new socket types.

 Anyway I really excited about this change.





 On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


 Currently working on that :)

 Having two layers that both
 carry per-message data is... wrong IMO.


 Protocols supporting 'out of band' data aren't exactly uncommon.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.


 I don't have an immediate use-case justifying it. I noted it, mostly
 because it has come up a few times since I started paying attention.

 On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

  It seems to me that in order to remove multi-part messages and
 introduce new
  socket types (e.g. SERVER/CLIENT) that would
  necessitate a revision of the wire protocol. If we are going to do
 that, it
  might be worth considering per-message
  metadata -

 We'd have to be very clear about the problem that per-message metadata
 is aiming for. There is an elegance to delivering blobs and nothing
 more. Metadata can be added on top using zproto.

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack. If we are sending/receiving data on a
 per-message basis, that is the message. Having two layers that both
 carry per-message data is... wrong IMO.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev



 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev



 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org

Re: [zeromq-dev] Notes from a hackathon

Joe the problem with the api of zmsg for single frame is that you cannot
calculate the frame size in advance causing a lot of re allocation and
copying data around (it's ok for receiving, the problem is sending).



On Mon, Feb 2, 2015 at 6:35 PM, Joe McIlvain joe.eli@gmail.com wrote:

 In my opinion, maintaining some kind of multi-frame abstraction at the API
 level is in fact very useful to applications, even if they are sent
 atomically on the implementation level as you describe.

 For example, in CZMQ It's quite useful in program flow to be able to push
 or pop a single zframe onto or off of a zmsg, then send that zmsg as an
 entire message.  Rarely do I find myself using zframe_send or zframe_recv
 with the send more and receive more flags in a CZMQ application.  The
 CZMQ picture messages are nice, but they are not always the easiest API
 to integrate into an application.  Having zmsg and zframe remain (or
 something like them), while removing zframe_send and zframe_recv could ease
 applications relying on multi-frame semantics into the transition.  The
 implementation for zmsg would simply serialize the frames in some
 zmsg-determined way into a single-part zmq_msg to send and deserialize
 according to the same scheme to receive.  This multi-frame serialization
 essentially be at the application level rather than at the ZMTP level.

 In short, I think a multi-frame abstraction is important for applications,
 but if we remove the ability to send one frame at a time at the fundamental
 level, this abstraction can remain at a higher level (whether in libzmq, or
 in CZMQ and other wrappers) and coexist happily with the proposed changes.



 On Mon, Feb 2, 2015 at 7:37 AM, Doron Somech somdo...@gmail.com wrote:

 So the new sockets are in the github master, you can take a loot at the
 test to see how to use the new routing id field:

 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe, think of
 sending from multiple threads or receiving from multiple threads. we cannot
 do that with Multipart.
 * There are a lot of good transport that already implement message
 framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required
 it is own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread safe
 make the learning curve of zeromq hard to beginners. Without three of them
 zeromq become much simpler.
 * At least with NetMQ single part message is much faster than multipart.
 * New stacks, multipart is complicated to implement, with the new API it
 will much more easier to implement new stacks (like JSMQ) or any native
 stack.

 Pieter I'm looking forward to see how you expose the routing id in the
 czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.

 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.

 As ZMTP 3.1 is still in raw status, what do you think of removing the
 multipart from it? maybe the 3.1 will only support the new socket types.

 Anyway I really excited about this change.





 On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


 Currently working on that :)

 Having two layers that both
 carry per-message data is... wrong IMO.


 Protocols supporting 'out of band' data aren't exactly uncommon.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.


 I don't have an immediate use-case justifying it. I noted it, mostly
 because it has come up a few times since I started paying attention.

 On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

  It seems to me that in order to remove multi-part messages and
 introduce new
  socket types (e.g. SERVER/CLIENT) that would
  necessitate a revision of the wire protocol. If we are going to do
 that, it
  might be worth considering per-message
  metadata -

 We'd have to be very clear about the problem that per-message metadata
 is aiming for. There is an elegance to delivering blobs and nothing
 more. Metadata can be added on top using zproto.

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack. If we are sending/receiving data on a
 per-message basis, that is the message. Having two layers that both
 carry per-message data is... wrong IMO.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org

Re: [zeromq-dev] Notes from a hackathon

2015-02-02 Thread Joe McIlvain

Doron,

You're right - there's a cost there I wasn't considering.  I'm not yet
entirely convinced that the cost makes the abstraction no longer worth it
in certain applications, but it at least shows that it should not be the
most advocated pattern in general.


On Mon, Feb 2, 2015 at 8:55 AM, Doron Somech somdo...@gmail.com wrote:

 Joe the problem with the api of zmsg for single frame is that you cannot
 calculate the frame size in advance causing a lot of re allocation and
 copying data around (it's ok for receiving, the problem is sending).



 On Mon, Feb 2, 2015 at 6:35 PM, Joe McIlvain joe.eli@gmail.com
 wrote:

 In my opinion, maintaining some kind of multi-frame abstraction at the
 API level is in fact very useful to applications, even if they are sent
 atomically on the implementation level as you describe.

 For example, in CZMQ It's quite useful in program flow to be able to push
 or pop a single zframe onto or off of a zmsg, then send that zmsg as an
 entire message.  Rarely do I find myself using zframe_send or zframe_recv
 with the send more and receive more flags in a CZMQ application.  The
 CZMQ picture messages are nice, but they are not always the easiest API
 to integrate into an application.  Having zmsg and zframe remain (or
 something like them), while removing zframe_send and zframe_recv could ease
 applications relying on multi-frame semantics into the transition.  The
 implementation for zmsg would simply serialize the frames in some
 zmsg-determined way into a single-part zmq_msg to send and deserialize
 according to the same scheme to receive.  This multi-frame serialization
 essentially be at the application level rather than at the ZMTP level.

 In short, I think a multi-frame abstraction is important for
 applications, but if we remove the ability to send one frame at a time at
 the fundamental level, this abstraction can remain at a higher level
 (whether in libzmq, or in CZMQ and other wrappers) and coexist happily with
 the proposed changes.



 On Mon, Feb 2, 2015 at 7:37 AM, Doron Somech somdo...@gmail.com wrote:

 So the new sockets are in the github master, you can take a loot at the
 test to see how to use the new routing id field:

 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe, think
 of sending from multiple threads or receiving from multiple threads. we
 cannot do that with Multipart.
 * There are a lot of good transport that already implement message
 framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required
 it is own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread safe
 make the learning curve of zeromq hard to beginners. Without three of them
 zeromq become much simpler.
 * At least with NetMQ single part message is much faster than multipart.
 * New stacks, multipart is complicated to implement, with the new API it
 will much more easier to implement new stacks (like JSMQ) or any native
 stack.

 Pieter I'm looking forward to see how you expose the routing id in the
 czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.

 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.

 As ZMTP 3.1 is still in raw status, what do you think of removing the
 multipart from it? maybe the 3.1 will only support the new socket types.

 Anyway I really excited about this change.





 On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


 Currently working on that :)

 Having two layers that both
 carry per-message data is... wrong IMO.


 Protocols supporting 'out of band' data aren't exactly uncommon.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.


 I don't have an immediate use-case justifying it. I noted it, mostly
 because it has come up a few times since I started paying attention.

 On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

  It seems to me that in order to remove multi-part messages and
 introduce new
  socket types (e.g. SERVER/CLIENT) that would
  necessitate a revision of the wire protocol. If we are going to do
 that, it
  might be worth considering per-message
  metadata -

 We'd have to be very clear about the problem that per-message metadata
 is aiming for. There is an elegance to delivering blobs and nothing
 more. Metadata can be added on top using zproto.

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack. If we are

Re: [zeromq-dev] Notes from a hackathon

2015-02-02 Thread Pieter Hintjens

Ironically the original argument for multiple frames was zero-copy
performance -- you can send a message without copying, with a prefix
frame for routing. In practice multiframe messages appear to punish
performance.

On Mon, Feb 2, 2015 at 6:11 PM, Joe McIlvain joe.eli@gmail.com wrote:
 Doron,

 You're right - there's a cost there I wasn't considering.  I'm not yet
 entirely convinced that the cost makes the abstraction no longer worth it in
 certain applications, but it at least shows that it should not be the most
 advocated pattern in general.


 On Mon, Feb 2, 2015 at 8:55 AM, Doron Somech somdo...@gmail.com wrote:

 Joe the problem with the api of zmsg for single frame is that you cannot
 calculate the frame size in advance causing a lot of re allocation and
 copying data around (it's ok for receiving, the problem is sending).



 On Mon, Feb 2, 2015 at 6:35 PM, Joe McIlvain joe.eli@gmail.com
 wrote:

 In my opinion, maintaining some kind of multi-frame abstraction at the
 API level is in fact very useful to applications, even if they are sent
 atomically on the implementation level as you describe.

 For example, in CZMQ It's quite useful in program flow to be able to push
 or pop a single zframe onto or off of a zmsg, then send that zmsg as an
 entire message.  Rarely do I find myself using zframe_send or zframe_recv
 with the send more and receive more flags in a CZMQ application.  The
 CZMQ picture messages are nice, but they are not always the easiest API to
 integrate into an application.  Having zmsg and zframe remain (or something
 like them), while removing zframe_send and zframe_recv could ease
 applications relying on multi-frame semantics into the transition.  The
 implementation for zmsg would simply serialize the frames in some
 zmsg-determined way into a single-part zmq_msg to send and deserialize
 according to the same scheme to receive.  This multi-frame serialization
 essentially be at the application level rather than at the ZMTP level.

 In short, I think a multi-frame abstraction is important for
 applications, but if we remove the ability to send one frame at a time at
 the fundamental level, this abstraction can remain at a higher level
 (whether in libzmq, or in CZMQ and other wrappers) and coexist happily with
 the proposed changes.



 On Mon, Feb 2, 2015 at 7:37 AM, Doron Somech somdo...@gmail.com wrote:

 So the new sockets are in the github master, you can take a loot at the
 test to see how to use the new routing id field:


 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe, think
 of sending from multiple threads or receiving from multiple threads. we
 cannot do that with Multipart.
 * There are a lot of good transport that already implement message
 framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required
 it is own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread safe
 make the learning curve of zeromq hard to beginners. Without three of them
 zeromq become much simpler.
 * At least with NetMQ single part message is much faster than multipart.
 * New stacks, multipart is complicated to implement, with the new API it
 will much more easier to implement new stacks (like JSMQ) or any native
 stack.

 Pieter I'm looking forward to see how you expose the routing id in the
 czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.

 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.

 As ZMTP 3.1 is still in raw status, what do you think of removing the
 multipart from it? maybe the 3.1 will only support the new socket types.

 Anyway I really excited about this change.





 On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


 Currently working on that :)

 Having two layers that both
 carry per-message data is... wrong IMO.


 Protocols supporting 'out of band' data aren't exactly uncommon.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.


 I don't have an immediate use-case justifying it. I noted it, mostly
 because it has come up a few times since I started paying attention.

 On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

  It seems to me that in order to remove multi-part messages and
  introduce new
  socket types (e.g. SERVER/CLIENT) that would
  necessitate a revision of the wire protocol. If we are going to do
  that, it
  might be worth considering per-message
  metadata -

 We'd have to be

Re: [zeromq-dev] Notes from a hackathon

2015-02-02 Thread Brian Knox

Doron -

I did a test implementation around this idea in goczmq today and I'm liking
the semantics.  I wrote an interface that conforms to io.Reader and
io.Writer where you pass a call a byte buffer and you get the message
placed into it.

When the socket is a router socket, it silently drops the ID frame under
the hood, but sets a lastClientID in the socket struct, that if you need,
you can get with a GetLastClientID call.

For reference:

https://github.com/zeromq/goczmq/blob/master/sock.go#L35 (note strings in
go are just immutable byte arrays)

https://github.com/zeromq/goczmq/blob/master/sock.go#L333

https://github.com/zeromq/goczmq/blob/master/sock.go#L366

So if the socket is a router, the Write call just transparently uses the
lastClientID value.  In a one to one request / reply, you then don't need
to think about it at all, and if you need to do async work, you can get and
set the id as needed.

I'll just move this to use the new API later - just wanted to try it out
and see how I liked it, and I say thumbs up to dropping frames on the new
Server / Client sockets.



On Mon, Feb 2, 2015 at 4:37 PM, Doron Somech somdo...@gmail.com wrote:

 So the new sockets are in the github master, you can take a loot at the
 test to see how to use the new routing id field:

 https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp

 Few of the reasons I didn't like multi frames:
 * I tried in the past to make both zeromq and NetMQ thread safe, think of
 sending from multiple threads or receiving from multiple threads. we cannot
 do that with Multipart.
 * There are a lot of good transport that already implement message framing
 (UDP, WebSockets, SCTP and even HTTP), but because zeromq required it is
 own framing it was not easy to add them.
 * Multipart, router usage (routing id frame) and not being thread safe
 make the learning curve of zeromq hard to beginners. Without three of them
 zeromq become much simpler.
 * At least with NetMQ single part message is much faster than multipart.
 * New stacks, multipart is complicated to implement, with the new API it
 will much more easier to implement new stacks (like JSMQ) or any native
 stack.

 Pieter I'm looking forward to see how you expose the routing id in the
 czmq API.
 I also like the czmq API for sending mutlipart messages (the picture
 feature) so maybe we can use that to generate single frame which is also
 compatible with zproto.

 About the implementation, none of new sockets support any option now.
 server behave like mandatory router, so when router is not reachable or
 highwater mark is reached an error will be returned.

 As ZMTP 3.1 is still in raw status, what do you think of removing the
 multipart from it? maybe the 3.1 will only support the new socket types.

 Anyway I really excited about this change.





 On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com
 wrote:

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack.


 Currently working on that :)

 Having two layers that both
 carry per-message data is... wrong IMO.


 Protocols supporting 'out of band' data aren't exactly uncommon.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.


 I don't have an immediate use-case justifying it. I noted it, mostly
 because it has come up a few times since I started paying attention.

 On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote:

  It seems to me that in order to remove multi-part messages and
 introduce new
  socket types (e.g. SERVER/CLIENT) that would
  necessitate a revision of the wire protocol. If we are going to do
 that, it
  might be worth considering per-message
  metadata -

 We'd have to be very clear about the problem that per-message metadata
 is aiming for. There is an elegance to delivering blobs and nothing
 more. Metadata can be added on top using zproto.

 What we really want IMO is per-peer metadata, and an API to get/set
 this. Using messages is a hack. If we are sending/receiving data on a
 per-message basis, that is the message. Having two layers that both
 carry per-message data is... wrong IMO.

 However the key thing is, what's the problem. Then we can discuss
 solutions to that.

 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev



 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev



 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev


___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon