Re: [zeromq-dev] Rare feature combination possible?

[Luca Boccassi]

On August 29, 2017 9:41:50 PM GMT+01:00, Peter Krey  wrote:
>Have you tried to build zmq with the NORM protocol? Sounds like that
>will
>give you almost what you need.
>
>On Tue, Aug 29, 2017 at 11:36 AM, Patrik VV.  wrote:
>
>> Just my 5 cents: AFAIK the RADIO/DISH sockets are the newer,
>thread-safe
>> sockets, which don't support multi part messages (ZMQ_SNDMORE option
>is not
>> accepted).
>>
>> On 29 Aug 2017, at 14:35, Stephan Opfer 
>wrote:
>>
>> >> Note that multi-part does _not_ imply contiguous memory location
>on the
>> >> receive end (it might happen but there's no guarantee anywhere).
>So if
>> >> that's a requirement for your use case, then you need to use a
>single
>> >> message anyway.
>> >
>> > No on the receiving site I think that I'll need to accept to copy
>the
>> data anyway.
>> > --
>> > Distributed Systems Research Group
>> > Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
>> > Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
>> > WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
>> > ___
>> > zeromq-dev mailing list
>> > zeromq-dev@lists.zeromq.org
>> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>

Regarding multicast, Linux binary packages built on OBS are built with PGM 
support, so you can just grab the ones for your distro:

https://build.opensuse.org/project/show/network:messaging:zeromq:git-stable


signature.asc
Description: PGP signature
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Rare feature combination possible?

[Peter Krey]

Have you tried to build zmq with the NORM protocol? Sounds like that will
give you almost what you need.

On Tue, Aug 29, 2017 at 11:36 AM, Patrik VV.  wrote:

> Just my 5 cents: AFAIK the RADIO/DISH sockets are the newer, thread-safe
> sockets, which don't support multi part messages (ZMQ_SNDMORE option is not
> accepted).
>
> On 29 Aug 2017, at 14:35, Stephan Opfer  wrote:
>
> >> Note that multi-part does _not_ imply contiguous memory location on the
> >> receive end (it might happen but there's no guarantee anywhere). So if
> >> that's a requirement for your use case, then you need to use a single
> >> message anyway.
> >
> > No on the receiving site I think that I'll need to accept to copy the
> data anyway.
> > --
> > Distributed Systems Research Group
> > Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
> > Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
> > WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Rare feature combination possible?

[Patrik VV.]

Just my 5 cents: AFAIK the RADIO/DISH sockets are the newer, thread-safe 
sockets, which don't support multi part messages (ZMQ_SNDMORE option is not 
accepted). 

On 29 Aug 2017, at 14:35, Stephan Opfer  wrote:

>> Note that multi-part does _not_ imply contiguous memory location on the
>> receive end (it might happen but there's no guarantee anywhere). So if
>> that's a requirement for your use case, then you need to use a single
>> message anyway.
> 
> No on the receiving site I think that I'll need to accept to copy the data 
> anyway.
> -- 
> Distributed Systems Research Group
> Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
> Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
> WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Rare feature combination possible?

[Stephan Opfer]

Note that multi-part does _not_ imply contiguous memory location on the
receive end (it might happen but there's no guarantee anywhere). So if
that's a requirement for your use case, then you need to use a single
message anyway.


No on the receiving site I think that I'll need to accept to copy the data 
anyway.
--
Distributed Systems Research Group
Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Rare feature combination possible?

[Luca Boccassi]

On Tue, 2017-08-29 at 12:38 +0200, Stephan Opfer wrote:
> Hi Luca,
> 
> the reason for me to use multi-part messages is, that I have the
> content of a message already in memory, but not in one continues
> memory sequence. So if I want to achieve zero-copy, I need to compose
> the message from several memory locations, that should not be copied.
> 
> When I understand you right, I just need to send several messages
> (one for each memory location)?
> 
> I think, that this would break my attempt to use Cap'n Proto
> (de)serialization. At least it could be tricky to recompose the Cap'n
> Proto message on the receiving side.
> 
> My current workflow contained the following step:
> 
> 1. Compose the Cap'n Proto message by let the message pointing to the
> different parts / memory locations.
> 
> 2. Send the different parts of the Cap'n Proto message as multi-part
> message via ZMQ + UDP + Multicast.
> 
> 3. Receive the Multi-Part message and reconstruct a Cap'n Proto
> message from it.
> 
> Instead step 3. now would need to receive several messages (incl.
> handling missing messages) in order to recreate the Cap'n Proto
> message.
> 
> My think that zero-copy isn't worth the hassle, in that case. So
> instead I tend to copy everything into one continuous memory sequence
> and send it as one message, which would just mean that I can compose
> the Cap'n Proto message without taking care of zero-copy.
> 
> Any thoughts?
> 
> Greetings,
>    Stephan

Note that multi-part does _not_ imply contiguous memory location on the
receive end (it might happen but there's no guarantee anywhere). So if
that's a requirement for your use case, then you need to use a single
message anyway.

-- 
Kind regards,
Luca Boccassi

signature.asc
Description: This is a digitally signed message part
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Rare feature combination possible?

[Stephan Opfer]

Hi Luca,

the reason for me to use multi-part messages is, that I have the content of a 
message already in memory, but not in one continues memory sequence. So if I 
want to achieve zero-copy, I need to compose the message from several memory 
locations, that should not be copied.

When I understand you right, I just need to send several messages (one for each 
memory location)?

I think, that this would break my attempt to use Cap'n Proto (de)serialization. 
At least it could be tricky to recompose the Cap'n Proto message on the 
receiving side.

My current workflow contained the following step:

1. Compose the Cap'n Proto message by let the message pointing to the different 
parts / memory locations.

2. Send the different parts of the Cap'n Proto message as multi-part message 
via ZMQ + UDP + Multicast.

3. Receive the Multi-Part message and reconstruct a Cap'n Proto message from it.

Instead step 3. now would need to receive several messages (incl. handling 
missing messages) in order to recreate the Cap'n Proto message.

My think that zero-copy isn't worth the hassle, in that case. So instead I tend 
to copy everything into one continuous memory sequence and send it as one 
message, which would just mean that I can compose the Cap'n Proto message 
without taking care of zero-copy.

Any thoughts?

Greetings,
  Stephan



--
Distributed Systems Research Group
Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Rare feature combination possible?

[Luca Boccassi]

On Mon, 2017-08-28 at 14:11 +0200, Stephan Opfer wrote:
> Hi all,
> 
> I would like to send Multi-Part Messages via a zmq_send_const(..)
> call over a RADIO socket through UDP at a Multicast Address. So is it
> possible to:
> 
> - Send Multi-Part Messages
> - Send Messages with Zero-Copy (e.g. via zmq_send_const)
> - Send over a RADIO socket and receive it at a DISH socket
> - Send over the raw UDP protocol
> - Send to a multicast address (e.g. 244.0.0.1)
> 
> If not, which feature do I need to drop?
> 
> UDP and Multicast are actually a must have for me.
> 
> Zero-Copy and Multi-Part Messages are somehow bind to each other, as
> I try to send data from different locations in memory in one multi-
> part message.
> 
> About the socket type I don't care, but I read often that UDP only
> works with RADIO/DISH.
> 
> What is the current status in ZMQ Version 4.2.3?
> 
> Greetings,
>    Stephan

Hi,

multi-part does not work with UDP.

Note that multi-part is needed for atomic delivery of messages - iow:
either all parts are delivered with guaranteed ordering or none is.

UDP means no guarantees on delivery and ordering.

If that's not a feature you require, than you could do without.

Also note that zmq_send_const, as the manpage and name suggest, it's
for constant memory only, eg: zmq_send_const (socket, "constant
string", ...

For zero-copy for non-constant buffers look at zmq_msg_init_data.

radio/dish over udp multicast (ipv4) can work with zmq_msg_init_data.

-- 
Kind regards,
Luca Boccassi

signature.asc
Description: This is a digitally signed message part
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev