Re: [zeromq-dev] Is shared ownership possible while sending with zero-copy?

[Thomas Rodgers]

If values are small (<32 bytes) then there are no allocations, as the
message type holds that much data directly.

On Thu, Aug 31, 2017 at 1:54 PM, Patrik VV.  wrote:

> I'm just curious, how large are those sensor values, how many do you keep
> around, and to how many other robots do you intend to send them?
>
> Could it be premature optimization? Just asking because maybe it's not
> worth the extra effort to make it zero-copy. Just copy and pass ownership
> to ZMQ.
>
> Regards, Patrik
>
> On 31 Aug 2017, at 20:06, Thomas Rodgers  wrote:
>
> Unfortunately that's not possible, libzmq exposes only a C API, and even
> though it is implemented in C++, it deliberately targets pre-C++11
> compilers.
>
> Further to the 'mark and sweep' idea, or more generally, deferred
> reclamation. You could have the callback place the message to be freed on a
> (possibly lock free, Boost has a handy one) queue and signal a 'reaper'
> thread (waiting on a condition_variable). The reaper thread wakes up,
> reclaims all queued message buffers then returns to waiting.
>
> On Thu, Aug 31, 2017 at 10:55 AM Stephan Opfer 
> wrote:
>
>> > Another, more complicated way, would be to implement a mark
>> > garbage collector of sorts: instead of freeing the buffer, the callback
>> > you register with zmq_msg_init_data would mark the buffer as done (in a
>> > thread safe way!). Then your application's garbage collector can sweep
>> > it.
>>
>> It would be nice, if I could pass over a copy  of (not reference or
>> pointer to) a shared_ptr that owns the buffer, but with the call back and
>> the "void * hint" this wasn't possible for me.
>>
>> --
>> Distributed Systems Research Group
>> Stephan Opfer  T. +49 561 804-6280 <+49%20561%208046280>  F. +49 561
>> 804-6277 <+49%20561%208046277>
>> Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
>> WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Is shared ownership possible while sending with zero-copy?

[Patrik VV.]

I'm just curious, how large are those sensor values, how many do you keep 
around, and to how many other robots do you intend to send them?

Could it be premature optimization? Just asking because maybe it's not worth 
the extra effort to make it zero-copy. Just copy and pass ownership to ZMQ.

Regards, Patrik

> On 31 Aug 2017, at 20:06, Thomas Rodgers  wrote:
> 
> Unfortunately that's not possible, libzmq exposes only a C API, and even 
> though it is implemented in C++, it deliberately targets pre-C++11 compilers.
> 
> Further to the 'mark and sweep' idea, or more generally, deferred 
> reclamation. You could have the callback place the message to be freed on a 
> (possibly lock free, Boost has a handy one) queue and signal a 'reaper' 
> thread (waiting on a condition_variable). The reaper thread wakes up, 
> reclaims all queued message buffers then returns to waiting.
> 
>> On Thu, Aug 31, 2017 at 10:55 AM Stephan Opfer  
>> wrote:
>> > Another, more complicated way, would be to implement a mark
>> > garbage collector of sorts: instead of freeing the buffer, the callback
>> > you register with zmq_msg_init_data would mark the buffer as done (in a
>> > thread safe way!). Then your application's garbage collector can sweep
>> > it.
>> 
>> It would be nice, if I could pass over a copy  of (not reference or pointer 
>> to) a shared_ptr that owns the buffer, but with the call back and the "void 
>> * hint" this wasn't possible for me.
>> 
>> --
>> Distributed Systems Research Group
>> Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
>> Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
>> WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Is shared ownership possible while sending with zero-copy?

[Thomas Rodgers]

Unfortunately that's not possible, libzmq exposes only a C API, and even
though it is implemented in C++, it deliberately targets pre-C++11
compilers.

Further to the 'mark and sweep' idea, or more generally, deferred
reclamation. You could have the callback place the message to be freed on a
(possibly lock free, Boost has a handy one) queue and signal a 'reaper'
thread (waiting on a condition_variable). The reaper thread wakes up,
reclaims all queued message buffers then returns to waiting.

On Thu, Aug 31, 2017 at 10:55 AM Stephan Opfer 
wrote:

> > Another, more complicated way, would be to implement a mark
> > garbage collector of sorts: instead of freeing the buffer, the callback
> > you register with zmq_msg_init_data would mark the buffer as done (in a
> > thread safe way!). Then your application's garbage collector can sweep
> > it.
>
> It would be nice, if I could pass over a copy  of (not reference or
> pointer to) a shared_ptr that owns the buffer, but with the call back and
> the "void * hint" this wasn't possible for me.
>
> --
> Distributed Systems Research Group
> Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
> Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
> WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Is shared ownership possible while sending with zero-copy?

[Stephan Opfer]

Another, more complicated way, would be to implement a mark
garbage collector of sorts: instead of freeing the buffer, the callback
you register with zmq_msg_init_data would mark the buffer as done (in a
thread safe way!). Then your application's garbage collector can sweep
it.


It would be nice, if I could pass over a copy  of (not reference or pointer to) a 
shared_ptr that owns the buffer, but with the call back and the "void * hint" 
this wasn't possible for me.

--
Distributed Systems Research Group
Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Is shared ownership possible while sending with zero-copy?

[Luca Boccassi]

On Thu, 2017-08-31 at 15:30 +0100, Luca Boccassi wrote:
> On Thu, 2017-08-31 at 16:16 +0200, Stephan Opfer wrote:
> > Hi all,
> > 
> > I am trying to use zmq_msg_init_data in order to have the zero-copy
> > advantage.
> > 
> > Now it is required, that I either pass a function, that can
> > deallocate the data after zmq sent it, or I pass NULL in order to
> > keep the ownership.
> > 
> > In the latter case: How do I know that zmq has done his work and I
> > can deallocate the data?
> 
> Locally, the only sure way is when the socket has been closed and the
> context has been terminated. But that of course is not ideal in a
> long-
> lived application.
> 
> If your protocol implements ACKS (request-reply pattern for example),
> then it's implicit that when receiving a response the original
> request
> was received.
> 
> > Another idea was to pass a lambda function (with a capture) to zmq,
> > in order to make zmq send a signal by calling the lambda function.
> > The problem is, that it is not allowed to pass lambda functions
> > with
> > captures as function pointers and without captures, I have no clue
> > how zmq could pass the signal.
> > 
> > Maybe somebody of you now already have an idea?
> > 
> > The general idea is:
> > 
> > My robotic application keeps some sensor values for a while and
> > sends
> > some of the values to other robots via zmq. Therefore I don't want
> > the data to be copied and I don't want zmq to deallocate it. I want
> > my own application to deallocate it, if it is not needed anymore
> > and
> > zmq did send everything.
> > 
> > Greetings,
> > 
> >    Stephan
> 
> A solution could be to make a shallow-copy of the message via
> zmq_msg_copy. This will not copy the data buffer, only the message
> metadata and references, which are a few bytes at most. The data
> buffer
> ownership is refcounted.
> 
> Then when your application is done with the data, you can call
> zmq_msg_close on the _copy_, which will decrease the refcount.
> 
> So the last one to be finished with the data, whether it's your
> application or the library, will deallocate it.

Another, more complicated way, would be to implement a mark
garbage collector of sorts: instead of freeing the buffer, the callback
you register with zmq_msg_init_data would mark the buffer as done (in a
thread safe way!). Then your application's garbage collector can sweep
it.

It's certainly more complicated than taking a shallow copy in C/C++ but
in some languages it might be easier.

-- 
Kind regards,
Luca Boccassi

signature.asc
Description: This is a digitally signed message part
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Is shared ownership possible while sending with zero-copy?

[Luca Boccassi]

On Thu, 2017-08-31 at 16:16 +0200, Stephan Opfer wrote:
> Hi all,
> 
> I am trying to use zmq_msg_init_data in order to have the zero-copy
> advantage.
> 
> Now it is required, that I either pass a function, that can
> deallocate the data after zmq sent it, or I pass NULL in order to
> keep the ownership.
> 
> In the latter case: How do I know that zmq has done his work and I
> can deallocate the data?

Locally, the only sure way is when the socket has been closed and the
context has been terminated. But that of course is not ideal in a long-
lived application.

If your protocol implements ACKS (request-reply pattern for example),
then it's implicit that when receiving a response the original request
was received.

> Another idea was to pass a lambda function (with a capture) to zmq,
> in order to make zmq send a signal by calling the lambda function.
> The problem is, that it is not allowed to pass lambda functions with
> captures as function pointers and without captures, I have no clue
> how zmq could pass the signal.
> 
> Maybe somebody of you now already have an idea?
> 
> The general idea is:
> 
> My robotic application keeps some sensor values for a while and sends
> some of the values to other robots via zmq. Therefore I don't want
> the data to be copied and I don't want zmq to deallocate it. I want
> my own application to deallocate it, if it is not needed anymore and
> zmq did send everything.
> 
> Greetings,
> 
>    Stephan

A solution could be to make a shallow-copy of the message via
zmq_msg_copy. This will not copy the data buffer, only the message
metadata and references, which are a few bytes at most. The data buffer
ownership is refcounted.

Then when your application is done with the data, you can call
zmq_msg_close on the _copy_, which will decrease the refcount.

So the last one to be finished with the data, whether it's your
application or the library, will deallocate it.

-- 
Kind regards,
Luca Boccassi

signature.asc
Description: This is a digitally signed message part
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] Is shared ownership possible while sending with zero-copy?

[Stephan Opfer]

Hi all,

I am trying to use zmq_msg_init_data in order to have the zero-copy advantage.

Now it is required, that I either pass a function, that can deallocate the data 
after zmq sent it, or I pass NULL in order to keep the ownership.

In the latter case: How do I know that zmq has done his work and I can 
deallocate the data?

Another idea was to pass a lambda function (with a capture) to zmq, in order to 
make zmq send a signal by calling the lambda function. The problem is, that it 
is not allowed to pass lambda functions with captures as function pointers and 
without captures, I have no clue how zmq could pass the signal.

Maybe somebody of you now already have an idea?

The general idea is:

My robotic application keeps some sensor values for a while and sends some of 
the values to other robots via zmq. Therefore I don't want the data to be 
copied and I don't want zmq to deallocate it. I want my own application to 
deallocate it, if it is not needed anymore and zmq did send everything.

Greetings,

  Stephan

--
Distributed Systems Research Group
Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zmq_msg_send returns "Resource temporarily unavailable"

[Stephan Opfer]

Hi Radek,

can you sketch some lines for this. I also thought about passing a shared_ptr 
as capture of a lambda function, but captures are not allowed in lambdas when 
passing it as function pointers. I did write another message to the mailing 
list, as the topic now changed a little bit.

Greetings,
  Stephan

--
Distributed Systems Research Group
Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zmq_msg_send returns "Resource temporarily unavailable"

[Radek Svoboda]

Hi Stephan,

I was solving the same problem some time ago. The trick is to call the
object "destructor" from the deallocator method. You can use e.g.
std::shared_ptr with reference counting to relay the actual destruction
when both your application and ZMQ are done with the "object". Note that
the deallocator method has additional "hint" parameter, which you can use
for this.

Best regards
Radek


On Thu, Aug 31, 2017 at 12:49 PM, Stephan Opfer 
wrote:

> Hi Luca,
>
> thank you for your hints. I am currently stuck with a little problem, that
> is probably related to your hin about cleaning up the messages to early. My
> current state is uploaded on gist: https://gist.github.com/Stepha
> nOpfer/98e32c13c822c33e06d56bc82956c7c2
>
> The thing is, that the messages are transfered correctly, when my process
> waits for some time, before it ends. So my question is, what do I don't
> understand correctly here?
>
> My current impression, or explanation of this phenomenon, is that the
> zmq_msg_send method is working asynchronously and the process ended, before
> zmq can send everything correctly.
>
> If that is the case I am not sure, what the best way will be to manage the
> data in my real application. How do I know, that zmq has finished sending
> my data, so that it would be ok to delete it. If I would let zmq decide to
> delete the data (by passing a deallocator method to zmq_msg_init_data), it
> could happen that it deletes the data to early for other parts of my
> application.
>
> Greetings,
>   Stephan
>
> --
> Distributed Systems Research Group
> Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
> Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
> WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zmq_msg_send returns "Resource temporarily unavailable"

[Luca Boccassi]

On Thu, 2017-08-31 at 12:49 +0200, Stephan Opfer wrote:
> Hi Luca,
> 
> thank you for your hints. I am currently stuck with a little problem,
> that is probably related to your hin about cleaning up the messages
> to early. My current state is uploaded on gist: https://gist.github.c
> om/StephanOpfer/98e32c13c822c33e06d56bc82956c7c2
> 
> The thing is, that the messages are transfered correctly, when my
> process waits for some time, before it ends. So my question is, what
> do I don't understand correctly here?
> 
> My current impression, or explanation of this phenomenon, is that the
> zmq_msg_send method is working asynchronously and the process ended,
> before zmq can send everything correctly.
> 
> If that is the case I am not sure, what the best way will be to
> manage the data in my real application. How do I know, that zmq has
> finished sending my data, so that it would be ok to delete it. If I
> would let zmq decide to delete the data (by passing a deallocator
> method to zmq_msg_init_data), it could happen that it deletes the
> data to early for other parts of my application.
> 
> Greetings,
>    Stephan

First of all, use the APIs to cleanly close the sockets and terminate
the context.

zmq_close (socket);
zmq_ctx_term (ctx);

When the socket is closed, there is grace periods for messages queued
up to the I/O thread but not yet sent - this is called the ZMQ_LINGER
timer. The default is 30 seconds. You can increase it (or set it to
unlimited with -1) via zmq_setsockopt.

Unlimited linger with TCP/IPC can lead to waiting forever if the other
end disappears, but with UDP there should not be this problem IIRC, as
it just sends the messages.

-- 
Kind regards,
Luca Boccassi

signature.asc
Description: This is a digitally signed message part
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zmq_msg_send returns "Resource temporarily unavailable"

[Stephan Opfer]

Hi Luca,

thank you for your hints. I am currently stuck with a little problem, that is 
probably related to your hin about cleaning up the messages to early. My 
current state is uploaded on gist: 
https://gist.github.com/StephanOpfer/98e32c13c822c33e06d56bc82956c7c2

The thing is, that the messages are transfered correctly, when my process waits 
for some time, before it ends. So my question is, what do I don't understand 
correctly here?

My current impression, or explanation of this phenomenon, is that the 
zmq_msg_send method is working asynchronously and the process ended, before zmq 
can send everything correctly.

If that is the case I am not sure, what the best way will be to manage the data 
in my real application. How do I know, that zmq has finished sending my data, 
so that it would be ok to delete it. If I would let zmq decide to delete the 
data (by passing a deallocator method to zmq_msg_init_data), it could happen 
that it deletes the data to early for other parts of my application.

Greetings,
  Stephan

--
Distributed Systems Research Group
Stephan Opfer  T. +49 561 804-6280  F. +49 561 804-6277
Univ. Kassel,  FB 16,  Wilhelmshöher Allee 73,  D-34121 Kassel
WWW: http://www.uni-kassel.de/go/vs_stephan-opfer/
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev