Re: [zeromq-dev] On hooking memory allocations
Using a memory pool (as in fixed-block allocator) is a somewhat different
requirement from using custom malloc/free implementation. I am not sure if
the library can make use of memory pools. The logical place to use memory
pools is for incoming message buffer management. This is for all practical
purposes the only place where a lot of allocations are happening and faster
allocation might make a difference. However the library interface promises
that message points to a contiguous buffer of message size, which makes it
difficult to integrate with memory pools.
On Mon, Nov 28, 2016 at 2:34 PM, Jens Auer
wrote:
> Yeah, I know that. I always find this more a hack than a real design. It
> is also not so nice to use with a C++-allocator object because I have to
> hook it with global functions. And being able to set this for ZeroMQ only
> e.g. gives me the freedom to use a memory pool there and standard
> allocators everywhere else. I welcome this flexibility.
>
>
>
> Cheers,
>
> Jens
>
>
>
> *Von:* zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] *Im
> Auftrag von *Max Kozlovsky
> *Gesendet:* Montag, 28. November 2016 20:48
>
> *An:* ZeroMQ development list
> *Betreff:* Re: [zeromq-dev] On hooking memory allocations
>
>
>
> Each program can be linked with a separate malloc implementation if the
> user so desires. Libraries don't need to be aware which implementation it
> is. Different malloc implementation can be even substituted at run time on
> platforms with dynamic linking support (LD_PRELOAD etc).
>
>
>
> On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer
> wrote:
>
> Hi,
>
>
>
> yes and no. If you overwrite it globally at compute-time every program on
> the system has to use your custom implementation. So if you deliver your
> ZeroMQ library with your program it will work, but what if my program wants
> a different custom allocator?
>
>
>
> Cheers,
>
> Jens
>
>
>
> *Von:* zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] *Im
> Auftrag von *Max Kozlovsky
> *Gesendet:* Montag, 28. November 2016 20:36
> *An:* ZeroMQ development list
> *Betreff:* Re: [zeromq-dev] On hooking memory allocations
>
>
>
> Hi,
>
>
>
> Would not globally overwriting malloc/free with the custom implementation
> achieve the desired behavior (instead of providing hooks for installing
> malloc overrides in each and every library)?
>
>
>
> Max
>
>
>
> On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens wrote:
>
> Hi,
>
> I don't see a big problem with the C API except that C doesn't support
> overloads. So if the function has a new name, e.g.
> zmq_ctx_new_with_allocator, everything stays plain C. The default instance
> would be a
>
> static void* malloc_(size_t n, void*) {return malloc(n);}
> static void free_(void* ptr, size_t n, void*) {free(ptr);}
>
> allocator_t alloc{
> NULL,
> malloc_,
> free_
> };
>
> context_t then stores the member and gets methods to forward memory
> allocations to the function pointers, passing the hint pointer as an
> additional argument.
>
> In my C++ code, I can then use an allocator
> static void* allocate(size_t n, void* obj) {return
> static_cast>(obj)->allocate(n); }
> static void free_(void* ptr, size_t n, void*obj) {
> static_cast>(obj)->deallocate(ptr, n); }
>
> std::allocator a;
> allocator_t zmqAlloc{
>&a,
>allocate,
>free_
> };
>
> void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
>
> I think this should work?
>
> Best wishes,
> Jens
>
> --
> Dr. Jens Auer | CGI | Software Engineer
> CGI Deutschland Ltd. & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> jens.a...@cgi.com
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter de.cgi.com/pflichtangaben.
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
> CGI Group Inc. and its affiliates may be contained in this message. If you
> are not a recipient indicated or intended in this message (or responsible
> for delivery of this message to such person), or you think for any reason
> that this message may have been addressed to you in error, you may not use
> or copy or deliver this message to anyone else. In such case, you should
> destroy this message and are asked to notify the sender by reply e-mail.
>
> > -Original Message-
> > From: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf
> Of
> > Luca Boccassi
> > Sent: 28 November 2016 12:30
> > To: ZeroMQ development list
> > Subject: Re: [zeromq-dev] On hooking memory allocations
> >
> > That would work for an internal API, but given we expose a C API
> unfortunately I
> > don't think that would work as a public API :-( And I think for this use
> case they
> > would require a public API.
> >
> > As an external API, a new zmq_ctx_set that takes a callback would have
> been ideal,
> > but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes
> a callback
> > pointer would be the next best.
> >
> > An alternative would be to have a system similar to what we use for th
Re: [zeromq-dev] On hooking memory allocations
Yeah, I know that. I always find this more a hack than a real design. It is
also not so nice to use with a C++-allocator object because I have to hook it
with global functions. And being able to set this for ZeroMQ only e.g. gives me
the freedom to use a memory pool there and standard allocators everywhere else.
I welcome this flexibility.
Cheers,
Jens
Von: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] Im Auftrag von Max
Kozlovsky
Gesendet: Montag, 28. November 2016 20:48
An: ZeroMQ development list
Betreff: Re: [zeromq-dev] On hooking memory allocations
Each program can be linked with a separate malloc implementation if the user so
desires. Libraries don't need to be aware which implementation it is. Different
malloc implementation can be even substituted at run time on platforms with
dynamic linking support (LD_PRELOAD etc).
On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer wrote:
Hi,
yes and no. If you overwrite it globally at compute-time every program on the
system has to use your custom implementation. So if you deliver your ZeroMQ
library with your program it will work, but what if my program wants a
different custom allocator?
Cheers,
Jens
Von: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] Im Auftrag von Max
Kozlovsky
Gesendet: Montag, 28. November 2016 20:36
An: ZeroMQ development list
Betreff: Re: [zeromq-dev] On hooking memory allocations
Hi,
Would not globally overwriting malloc/free with the custom implementation
achieve the desired behavior (instead of providing hooks for installing malloc
overrides in each and every library)?
Max
On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens wrote:
Hi,
I don't see a big problem with the C API except that C doesn't support
overloads. So if the function has a new name, e.g. zmq_ctx_new_with_allocator,
everything stays plain C. The default instance would be a
static void* malloc_(size_t n, void*) {return malloc(n);}
static void free_(void* ptr, size_t n, void*) {free(ptr);}
allocator_t alloc{
NULL,
malloc_,
free_
};
context_t then stores the member and gets methods to forward memory allocations
to the function pointers, passing the hint pointer as an additional argument.
In my C++ code, I can then use an allocator
static void* allocate(size_t n, void* obj) {return
static_cast>(obj)->allocate(n); }
static void free_(void* ptr, size_t n, void*obj) {
static_cast>(obj)->deallocate(ptr, n); }
std::allocator a;
allocator_t zmqAlloc{
&a,
allocate,
free_
};
void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
I think this should work?
Best wishes,
Jens
--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens.a...@cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
de.cgi.com/pflichtangaben.
CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
Group Inc. and its affiliates may be contained in this message. If you are not
a recipient indicated or intended in this message (or responsible for delivery
of this message to such person), or you think for any reason that this message
may have been addressed to you in error, you may not use or copy or deliver
this message to anyone else. In such case, you should destroy this message and
are asked to notify the sender by reply e-mail.
> -Original Message-
> From: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of
> Luca Boccassi
> Sent: 28 November 2016 12:30
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] On hooking memory allocations
>
> That would work for an internal API, but given we expose a C API
> unfortunately I
> don't think that would work as a public API :-( And I think for this use case
> they
> would require a public API.
>
> As an external API, a new zmq_ctx_set that takes a callback would have been
> ideal,
> but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes a
> callback
> pointer would be the next best.
>
> An alternative would be to have a system similar to what we use for the poll
> implementation (epoll kqueue select etc), but this would be a build-time
> option,
> and the implementation would have to be checked in, which I don't think is an
> option for this case, right?
>
> On Mon, 2016-11-28 at 10:51 +, Auer, Jens wrote:
> > Hi,
> >
> > I am just a user, but I would love to see this change. I have thinking
> > about this and I would like to be able to pass a C++ allocator object
> > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > define a struct in the interface
> >
> > struct allocator_t
> > {
> > void* hint;
> > void* (allocate)(size_t, void*);
> > void (deallocate)(void*, size_t, void*); };
> >
> > and store this in the context object. Since I don't think that this should
> > be
> changed during runtime, I would create a new zmq_ctx_new overload which take
Re: [zeromq-dev] On hooking memory allocations
Each program can be linked with a separate malloc implementation if the
user so desires. Libraries don't need to be aware which implementation it
is. Different malloc implementation can be even substituted at run time on
platforms with dynamic linking support (LD_PRELOAD etc).
On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer
wrote:
> Hi,
>
>
>
> yes and no. If you overwrite it globally at compute-time every program on
> the system has to use your custom implementation. So if you deliver your
> ZeroMQ library with your program it will work, but what if my program wants
> a different custom allocator?
>
>
>
> Cheers,
>
> Jens
>
>
>
> *Von:* zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] *Im
> Auftrag von *Max Kozlovsky
> *Gesendet:* Montag, 28. November 2016 20:36
> *An:* ZeroMQ development list
> *Betreff:* Re: [zeromq-dev] On hooking memory allocations
>
>
>
> Hi,
>
>
>
> Would not globally overwriting malloc/free with the custom implementation
> achieve the desired behavior (instead of providing hooks for installing
> malloc overrides in each and every library)?
>
>
>
> Max
>
>
>
> On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens wrote:
>
> Hi,
>
> I don't see a big problem with the C API except that C doesn't support
> overloads. So if the function has a new name, e.g.
> zmq_ctx_new_with_allocator, everything stays plain C. The default instance
> would be a
>
> static void* malloc_(size_t n, void*) {return malloc(n);}
> static void free_(void* ptr, size_t n, void*) {free(ptr);}
>
> allocator_t alloc{
> NULL,
> malloc_,
> free_
> };
>
> context_t then stores the member and gets methods to forward memory
> allocations to the function pointers, passing the hint pointer as an
> additional argument.
>
> In my C++ code, I can then use an allocator
> static void* allocate(size_t n, void* obj) {return
> static_cast>(obj)->allocate(n); }
> static void free_(void* ptr, size_t n, void*obj) {
> static_cast>(obj)->deallocate(ptr, n); }
>
> std::allocator a;
> allocator_t zmqAlloc{
>&a,
>allocate,
>free_
> };
>
> void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
>
> I think this should work?
>
> Best wishes,
> Jens
>
> --
> Dr. Jens Auer | CGI | Software Engineer
> CGI Deutschland Ltd. & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> jens.a...@cgi.com
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter de.cgi.com/pflichtangaben.
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
> CGI Group Inc. and its affiliates may be contained in this message. If you
> are not a recipient indicated or intended in this message (or responsible
> for delivery of this message to such person), or you think for any reason
> that this message may have been addressed to you in error, you may not use
> or copy or deliver this message to anyone else. In such case, you should
> destroy this message and are asked to notify the sender by reply e-mail.
>
> > -Original Message-
> > From: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf
> Of
> > Luca Boccassi
> > Sent: 28 November 2016 12:30
> > To: ZeroMQ development list
> > Subject: Re: [zeromq-dev] On hooking memory allocations
> >
> > That would work for an internal API, but given we expose a C API
> unfortunately I
> > don't think that would work as a public API :-( And I think for this use
> case they
> > would require a public API.
> >
> > As an external API, a new zmq_ctx_set that takes a callback would have
> been ideal,
> > but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes
> a callback
> > pointer would be the next best.
> >
> > An alternative would be to have a system similar to what we use for the
> poll
> > implementation (epoll kqueue select etc), but this would be a build-time
> option,
> > and the implementation would have to be checked in, which I don't think
> is an
> > option for this case, right?
> >
> > On Mon, 2016-11-28 at 10:51 +, Auer, Jens wrote:
> > > Hi,
> > >
> > > I am just a user, but I would love to see this change. I have thinking
> > > about this and I would like to be able to pass a C++ allocator object
> > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > define a struct in the interface
> > >
> > > struct allocator_t
> > > {
> > > void* hint;
> > > void* (allocate)(size_t, void*);
> > > void (deallocate)(void*, size_t, void*); };
> > >
> > > and store this in the context object. Since I don't think that this
> should be
> > changed during runtime, I would create a new zmq_ctx_new overload which
> takes a
> > parameter of type allocator_t. The default value would be to call
> malloc/free.
> > >
> > > Cheers,
> > > Jens
> > >
> > > --
> > > Jens Auer | CGI | Software-Engineer
> > > CGI (Germany) GmbH & Co. KG
> > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > T: +49 6151 36860 154
> > > jens.a...@cgi.com
> > > Unsere Pflichtangaben gemäß § 35a G
Re: [zeromq-dev] On hooking memory allocations
Hi,
yes and no. If you overwrite it globally at compute-time every program on the
system has to use your custom implementation. So if you deliver your ZeroMQ
library with your program it will work, but what if my program wants a
different custom allocator?
Cheers,
Jens
Von: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] Im Auftrag von Max
Kozlovsky
Gesendet: Montag, 28. November 2016 20:36
An: ZeroMQ development list
Betreff: Re: [zeromq-dev] On hooking memory allocations
Hi,
Would not globally overwriting malloc/free with the custom implementation
achieve the desired behavior (instead of providing hooks for installing malloc
overrides in each and every library)?
Max
On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens wrote:
Hi,
I don't see a big problem with the C API except that C doesn't support
overloads. So if the function has a new name, e.g. zmq_ctx_new_with_allocator,
everything stays plain C. The default instance would be a
static void* malloc_(size_t n, void*) {return malloc(n);}
static void free_(void* ptr, size_t n, void*) {free(ptr);}
allocator_t alloc{
NULL,
malloc_,
free_
};
context_t then stores the member and gets methods to forward memory allocations
to the function pointers, passing the hint pointer as an additional argument.
In my C++ code, I can then use an allocator
static void* allocate(size_t n, void* obj) {return
static_cast>(obj)->allocate(n); }
static void free_(void* ptr, size_t n, void*obj) {
static_cast>(obj)->deallocate(ptr, n); }
std::allocator a;
allocator_t zmqAlloc{
&a,
allocate,
free_
};
void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
I think this should work?
Best wishes,
Jens
--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens.a...@cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
de.cgi.com/pflichtangaben.
CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
Group Inc. and its affiliates may be contained in this message. If you are not
a recipient indicated or intended in this message (or responsible for delivery
of this message to such person), or you think for any reason that this message
may have been addressed to you in error, you may not use or copy or deliver
this message to anyone else. In such case, you should destroy this message and
are asked to notify the sender by reply e-mail.
> -Original Message-
> From: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of
> Luca Boccassi
> Sent: 28 November 2016 12:30
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] On hooking memory allocations
>
> That would work for an internal API, but given we expose a C API
> unfortunately I
> don't think that would work as a public API :-( And I think for this use case
> they
> would require a public API.
>
> As an external API, a new zmq_ctx_set that takes a callback would have been
> ideal,
> but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes a
> callback
> pointer would be the next best.
>
> An alternative would be to have a system similar to what we use for the poll
> implementation (epoll kqueue select etc), but this would be a build-time
> option,
> and the implementation would have to be checked in, which I don't think is an
> option for this case, right?
>
> On Mon, 2016-11-28 at 10:51 +, Auer, Jens wrote:
> > Hi,
> >
> > I am just a user, but I would love to see this change. I have thinking
> > about this and I would like to be able to pass a C++ allocator object
> > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > define a struct in the interface
> >
> > struct allocator_t
> > {
> > void* hint;
> > void* (allocate)(size_t, void*);
> > void (deallocate)(void*, size_t, void*); };
> >
> > and store this in the context object. Since I don't think that this should
> > be
> changed during runtime, I would create a new zmq_ctx_new overload which takes
> a
> parameter of type allocator_t. The default value would be to call malloc/free.
> >
> > Cheers,
> > Jens
> >
> > --
> > Jens Auer | CGI | Software-Engineer
> > CGI (Germany) GmbH & Co. KG
> > Rheinstraße 95 | 64295 Darmstadt | Germany
> > T: +49 6151 36860 154
> > jens.a...@cgi.com
> > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
> de.cgi.com/pflichtangaben.
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
> > CGI
> Group Inc. and its affiliates may be contained in this message. If you are
> not a
> recipient indicated or intended in this message (or responsible for delivery
> of this
> message to such person), or you think for any reason that this message may
> have
> been addressed to you in error, you may not use or copy or deliver this
> message to
> anyone else. In such ca
Re: [zeromq-dev] On hooking memory allocations
Hi,
Would not globally overwriting malloc/free with the custom implementation
achieve the desired behavior (instead of providing hooks for installing
malloc overrides in each and every library)?
Max
On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens wrote:
> Hi,
>
> I don't see a big problem with the C API except that C doesn't support
> overloads. So if the function has a new name, e.g.
> zmq_ctx_new_with_allocator, everything stays plain C. The default instance
> would be a
>
> static void* malloc_(size_t n, void*) {return malloc(n);}
> static void free_(void* ptr, size_t n, void*) {free(ptr);}
>
> allocator_t alloc{
> NULL,
> malloc_,
> free_
> };
>
> context_t then stores the member and gets methods to forward memory
> allocations to the function pointers, passing the hint pointer as an
> additional argument.
>
> In my C++ code, I can then use an allocator
> static void* allocate(size_t n, void* obj) {return
> static_cast>(obj)->allocate(n); }
> static void free_(void* ptr, size_t n, void*obj) {
> static_cast>(obj)->deallocate(ptr, n); }
>
> std::allocator a;
> allocator_t zmqAlloc{
>&a,
>allocate,
>free_
> };
>
> void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
>
> I think this should work?
>
> Best wishes,
> Jens
>
> --
> Dr. Jens Auer | CGI | Software Engineer
> CGI Deutschland Ltd. & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> jens.a...@cgi.com
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter de.cgi.com/pflichtangaben.
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
> CGI Group Inc. and its affiliates may be contained in this message. If you
> are not a recipient indicated or intended in this message (or responsible
> for delivery of this message to such person), or you think for any reason
> that this message may have been addressed to you in error, you may not use
> or copy or deliver this message to anyone else. In such case, you should
> destroy this message and are asked to notify the sender by reply e-mail.
>
>
> > -Original Message-
> > From: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf
> Of
> > Luca Boccassi
> > Sent: 28 November 2016 12:30
> > To: ZeroMQ development list
> > Subject: Re: [zeromq-dev] On hooking memory allocations
> >
> > That would work for an internal API, but given we expose a C API
> unfortunately I
> > don't think that would work as a public API :-( And I think for this use
> case they
> > would require a public API.
> >
> > As an external API, a new zmq_ctx_set that takes a callback would have
> been ideal,
> > but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes
> a callback
> > pointer would be the next best.
> >
> > An alternative would be to have a system similar to what we use for the
> poll
> > implementation (epoll kqueue select etc), but this would be a build-time
> option,
> > and the implementation would have to be checked in, which I don't think
> is an
> > option for this case, right?
> >
> > On Mon, 2016-11-28 at 10:51 +, Auer, Jens wrote:
> > > Hi,
> > >
> > > I am just a user, but I would love to see this change. I have thinking
> > > about this and I would like to be able to pass a C++ allocator object
> > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > define a struct in the interface
> > >
> > > struct allocator_t
> > > {
> > > void* hint;
> > > void* (allocate)(size_t, void*);
> > > void (deallocate)(void*, size_t, void*); };
> > >
> > > and store this in the context object. Since I don't think that this
> should be
> > changed during runtime, I would create a new zmq_ctx_new overload which
> takes a
> > parameter of type allocator_t. The default value would be to call
> malloc/free.
> > >
> > > Cheers,
> > > Jens
> > >
> > > --
> > > Jens Auer | CGI | Software-Engineer
> > > CGI (Germany) GmbH & Co. KG
> > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > T: +49 6151 36860 154
> > > jens.a...@cgi.com
> > > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter
> > de.cgi.com/pflichtangaben.
> > >
> > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> to CGI
> > Group Inc. and its affiliates may be contained in this message. If you
> are not a
> > recipient indicated or intended in this message (or responsible for
> delivery of this
> > message to such person), or you think for any reason that this message
> may have
> > been addressed to you in error, you may not use or copy or deliver this
> message to
> > anyone else. In such case, you should destroy this message and are asked
> to notify
> > the sender by reply e-mail.
> > >
> > > Von: zeromq-dev [zeromq-dev-boun...@lists.zeromq.org]" im Auftrag von
> > > "Petteri Salo [petteri.s...@gmail.com]
> > > Gesendet: Montag, 28. November 2016 09:40
> > > An: zeromq-dev@l
Re: [zeromq-dev] No UDP broadcast message received using CZMQ zbeacon on RaspberryPi3
A few more tests and information about the issue:
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.4.26-v7+ #915 SMP Thu Oct 20 17:08:44 BST 2016 armv7l
GNU/Linux
Now testing with zmq and czmq from master compiled on RPI with default
options:
*CZMQ version: 40002*
*ZMQ version: 40201*
Tried setting ZSYS_INTERFACE with "*", "wlan0", "eth0" and also not set at
all ("").
Using VERBOSE with zbeacon always indicate the correct interface being used:
- Without setting or "eth0":
I: 16-11-28 15:30:09 zbeacon: interface=eth0 address=192.168.25.214
broadcast=192.168.25.255
I: 16-11-28 15:30:09 zbeacon: configured, hostname=192.168.25.214
- Setting as "*"
I: 16-11-28 15:25:18 zbeacon: configured, hostname=*
Running sudo netstat -uac while executing tests had the following results:
- No setting ("") - *notice that Recv-Q builds up with time!*:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 9408 0 192.168.25.255:5670 *:*
udp0 0 *:bootpc*:*
udp0 0 raspberrypi-001:ntp *:*
udp0 0 raspberrypi:ntp *:*
udp0 0 localhost:ntp *:*
udp0 0 *:ntp *:*
udp0 0 *:mdns *:*
udp0 0 *:38304 *:*
udp6 0 0 [::]:ntp[::]:*
udp6 0 0 [::]:mdns [::]:*
udp6 0 0 [::]:43395 [::]:*
- ZSYS_INTERFACE = "*":
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp0 0 255.255.255.255:5670*:*
udp0 0 *:bootpc*:*
udp0 0 raspberrypi-001:ntp *:*
udp0 0 raspberrypi:ntp *:*
udp0 0 localhost:ntp *:*
udp0 0 *:ntp *:*
udp0 0 *:mdns *:*
udp0 0 *:38304 *:*
udp6 0 0 [::]:ntp[::]:*
udp6 0 0 [::]:mdns [::]:*
udp6 0 0 [::]:43395 [::]:*
The only difference from the netstat output for the QtUdpSocket version is
that the Recv-Q is always zero.
Any tips are welcome...
Thanks!
Rodrigo Madruga.
2016-11-28 13:19 GMT-02:00 Wes Young :
> but is that what interface your app is listening (err telling zbeacon to
> listen on) for beacons on?
>
> (in addition in making sure you’re using the latest ver), follow the
> ZSYS_INTERFACE var a bit (it may or may not be set and be associating the
> beacon listener with the correct interface…)
>
> > On Nov 28, 2016, at 9:42 AM, Rodrigo Madruga <
> rodrigo.madr...@nexxtorage.com> wrote:
> >
> > Broadcast messages are indeed reaching the pi, as shown by tcpdump:
> >
> > pi@raspberrypi:~ $ sudo tcpdump udp port 5670
> > tcpdump: verbose output suppressed, use -v or -vv for full protocol
> decode
> > listening on wlan0, link-type EN10MB (Ethernet), capture size 262144
> bytes
> > 13:33:56.203230 IP [REDACTED].5670 > 192.168.1.255.5670: UDP, length
> 22
> > 13:34:01.072476 IP [REDACTED].5670 > 192.168.1.255.5670: UDP, length
> 22
>
> --
> wes
> wesyoung.me
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] No UDP broadcast message received using CZMQ zbeacon on RaspberryPi3
but is that what interface your app is listening (err telling zbeacon to listen on) for beacons on? (in addition in making sure you’re using the latest ver), follow the ZSYS_INTERFACE var a bit (it may or may not be set and be associating the beacon listener with the correct interface…) > On Nov 28, 2016, at 9:42 AM, Rodrigo Madruga > wrote: > > Broadcast messages are indeed reaching the pi, as shown by tcpdump: > > pi@raspberrypi:~ $ sudo tcpdump udp port 5670 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on wlan0, link-type EN10MB (Ethernet), capture size 262144 bytes > 13:33:56.203230 IP [REDACTED].5670 > 192.168.1.255.5670: UDP, length 22 > 13:34:01.072476 IP [REDACTED].5670 > 192.168.1.255.5670: UDP, length 22 -- wes wesyoung.me signature.asc Description: Message signed with OpenPGP using GPGMail ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] No UDP broadcast message received using CZMQ zbeacon on RaspberryPi3
I successfully used it through zyre on 20 raspberry pi1.
Am 28.11.2016 3:50 nachm. schrieb "Luca Boccassi" :
> Hi,
>
> There is a CZMQ 4.0.x series now, loads of changes have gone in since
> 3.0.2. I would recommend trying it as the first thing.
>
> https://github.com/zeromq/czmq/releases
>
> On Mon, 2016-11-28 at 12:42 -0200, Rodrigo Madruga wrote:
> > Hello all,
> >
> > I am developing a system that uses CZMQ's zbeacon (broadcast on UDP) for
> > app discovery. Beacon sender is Windows box and receiver is RaspberryPi3.
> >
> > As the subject says, no message is reaching the zbeacon actor.
> >
> > Using czmq3.0.2 and zmq4.1.6 built directly on pi to rule out
> > cross-compiling issues.
> >
> > Calling just zbeacon_test() returns "OK" without assertions errors.
> >
> > Broadcast messages are indeed reaching the pi, as shown by tcpdump:
> >
> > pi@raspberrypi:~ $ sudo tcpdump udp port 5670
> > tcpdump: verbose output suppressed, use -v or -vv for full protocol
> > decode
> > listening on wlan0, link-type EN10MB (Ethernet), capture size 262144
> > bytes
> > 13:33:56.203230 IP [REDACTED].5670 > 192.168.1.255.5670: UDP, length
> 22
> > 13:34:01.072476 IP [REDACTED].5670 > 192.168.1.255.5670: UDP, length
> 22
> >
> > The code used to test is based on zbeacon_test function:
> >
> > zactor_t *listener = zactor_new (zbeacon, NULL);
> > assert (listener);
> > zsock_send (listener, "si", "CONFIGURE", 5670);
> > hostname = zstr_recv (listener);
> > assert (*hostname);
> > free (hostname);
> >
> > zsock_send (listener, "sb", "SUBSCRIBE", "", 0);
> >
> > zpoller_t *poller = zpoller_new (listener, NULL);
> > assert (poller);
> > int64_t stop_at = zclock_mono () + 1; // wait 10 seconds
> > while (zclock_mono () < stop_at) {
> > long timeout = (long) (stop_at - zclock_mono ());
> > if (timeout < 0)
> > timeout = 0;
> > void *which = zpoller_wait (poller, timeout * ZMQ_POLL_MSEC);
> > if (which) {
> > char *ipaddress, *received;
> > zstr_recvx (which, &ipaddress, &received, NULL);
> > printf("From address %s:%s\n", ipaddress, received);
> > zstr_free (&ipaddress);
> > zstr_free (&received);
> > }
> > }
> > zpoller_destroy (&poller);
> >
> > // Stop listening
> > zstr_sendx (listener, "UNSUBSCRIBE", NULL);
> >
> > // Destroy the test nodes
> > zactor_destroy (&listener);
> >
> > I cross-compiled a simple Qt app with a QUdpSocket (below) and the
> messages
> > were received without issues:
> >
> > #include
> >
> > TestUDP::TestUDP(QObject *parent) :
> > QObject(parent)
> > {
> > qDebug() << "Binding UDP Socket";
> > socket = new QUdpSocket(this);
> > socket->bind(QHostAddress::Any, 5670);
> > connect(socket, &QUdpSocket::readyRead,this,
> &TestUDP::readyRead);
> > qDebug() << "Ready Read signal connected0, waiting for
> broadcasts";
> > }
> >
> > void TestUDP::readyRead(){
> > QByteArray buffer;
> > buffer.resize(socket->pendingDatagramSize());
> > //var to store headers from udp
> > QHostAddress sender;
> > quint16 sender_port;
> > socket->readDatagram(buffer.data(),buffer.size(), &sender,
> > &sender_port);
> > qDebug() << "Message from " << sender << " port " << sender_port;
> > qDebug() << "Msg: " << buffer;
> > }
> >
> > Maybe the problem is at the way zbeacon is dealing with the UDP socket...
> >
> > Has anyone successfully used a zbeacon on RPI3?
> >
> > Thanks in advance!
> >
> > Rodrigo Madruga.
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] No UDP broadcast message received using CZMQ zbeacon on RaspberryPi3
Hi,
There is a CZMQ 4.0.x series now, loads of changes have gone in since
3.0.2. I would recommend trying it as the first thing.
https://github.com/zeromq/czmq/releases
On Mon, 2016-11-28 at 12:42 -0200, Rodrigo Madruga wrote:
> Hello all,
>
> I am developing a system that uses CZMQ's zbeacon (broadcast on UDP) for
> app discovery. Beacon sender is Windows box and receiver is RaspberryPi3.
>
> As the subject says, no message is reaching the zbeacon actor.
>
> Using czmq3.0.2 and zmq4.1.6 built directly on pi to rule out
> cross-compiling issues.
>
> Calling just zbeacon_test() returns "OK" without assertions errors.
>
> Broadcast messages are indeed reaching the pi, as shown by tcpdump:
>
> pi@raspberrypi:~ $ sudo tcpdump udp port 5670
> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> decode
> listening on wlan0, link-type EN10MB (Ethernet), capture size 262144
> bytes
> 13:33:56.203230 IP [REDACTED].5670 > 192.168.1.255.5670: UDP, length 22
> 13:34:01.072476 IP [REDACTED].5670 > 192.168.1.255.5670: UDP, length 22
>
> The code used to test is based on zbeacon_test function:
>
> zactor_t *listener = zactor_new (zbeacon, NULL);
> assert (listener);
> zsock_send (listener, "si", "CONFIGURE", 5670);
> hostname = zstr_recv (listener);
> assert (*hostname);
> free (hostname);
>
> zsock_send (listener, "sb", "SUBSCRIBE", "", 0);
>
> zpoller_t *poller = zpoller_new (listener, NULL);
> assert (poller);
> int64_t stop_at = zclock_mono () + 1; // wait 10 seconds
> while (zclock_mono () < stop_at) {
> long timeout = (long) (stop_at - zclock_mono ());
> if (timeout < 0)
> timeout = 0;
> void *which = zpoller_wait (poller, timeout * ZMQ_POLL_MSEC);
> if (which) {
> char *ipaddress, *received;
> zstr_recvx (which, &ipaddress, &received, NULL);
> printf("From address %s:%s\n", ipaddress, received);
> zstr_free (&ipaddress);
> zstr_free (&received);
> }
> }
> zpoller_destroy (&poller);
>
> // Stop listening
> zstr_sendx (listener, "UNSUBSCRIBE", NULL);
>
> // Destroy the test nodes
> zactor_destroy (&listener);
>
> I cross-compiled a simple Qt app with a QUdpSocket (below) and the messages
> were received without issues:
>
> #include
>
> TestUDP::TestUDP(QObject *parent) :
> QObject(parent)
> {
> qDebug() << "Binding UDP Socket";
> socket = new QUdpSocket(this);
> socket->bind(QHostAddress::Any, 5670);
> connect(socket, &QUdpSocket::readyRead,this, &TestUDP::readyRead);
> qDebug() << "Ready Read signal connected0, waiting for broadcasts";
> }
>
> void TestUDP::readyRead(){
> QByteArray buffer;
> buffer.resize(socket->pendingDatagramSize());
> //var to store headers from udp
> QHostAddress sender;
> quint16 sender_port;
> socket->readDatagram(buffer.data(),buffer.size(), &sender,
> &sender_port);
> qDebug() << "Message from " << sender << " port " << sender_port;
> qDebug() << "Msg: " << buffer;
> }
>
> Maybe the problem is at the way zbeacon is dealing with the UDP socket...
>
> Has anyone successfully used a zbeacon on RPI3?
>
> Thanks in advance!
>
> Rodrigo Madruga.
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
signature.asc
Description: This is a digitally signed message part
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] No UDP broadcast message received using CZMQ zbeacon on RaspberryPi3
Hello all,
I am developing a system that uses CZMQ's zbeacon (broadcast on UDP) for
app discovery. Beacon sender is Windows box and receiver is RaspberryPi3.
As the subject says, no message is reaching the zbeacon actor.
Using czmq3.0.2 and zmq4.1.6 built directly on pi to rule out
cross-compiling issues.
Calling just zbeacon_test() returns "OK" without assertions errors.
Broadcast messages are indeed reaching the pi, as shown by tcpdump:
pi@raspberrypi:~ $ sudo tcpdump udp port 5670
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on wlan0, link-type EN10MB (Ethernet), capture size 262144
bytes
13:33:56.203230 IP [REDACTED].5670 > 192.168.1.255.5670: UDP, length 22
13:34:01.072476 IP [REDACTED].5670 > 192.168.1.255.5670: UDP, length 22
The code used to test is based on zbeacon_test function:
zactor_t *listener = zactor_new (zbeacon, NULL);
assert (listener);
zsock_send (listener, "si", "CONFIGURE", 5670);
hostname = zstr_recv (listener);
assert (*hostname);
free (hostname);
zsock_send (listener, "sb", "SUBSCRIBE", "", 0);
zpoller_t *poller = zpoller_new (listener, NULL);
assert (poller);
int64_t stop_at = zclock_mono () + 1; // wait 10 seconds
while (zclock_mono () < stop_at) {
long timeout = (long) (stop_at - zclock_mono ());
if (timeout < 0)
timeout = 0;
void *which = zpoller_wait (poller, timeout * ZMQ_POLL_MSEC);
if (which) {
char *ipaddress, *received;
zstr_recvx (which, &ipaddress, &received, NULL);
printf("From address %s:%s\n", ipaddress, received);
zstr_free (&ipaddress);
zstr_free (&received);
}
}
zpoller_destroy (&poller);
// Stop listening
zstr_sendx (listener, "UNSUBSCRIBE", NULL);
// Destroy the test nodes
zactor_destroy (&listener);
I cross-compiled a simple Qt app with a QUdpSocket (below) and the messages
were received without issues:
#include
TestUDP::TestUDP(QObject *parent) :
QObject(parent)
{
qDebug() << "Binding UDP Socket";
socket = new QUdpSocket(this);
socket->bind(QHostAddress::Any, 5670);
connect(socket, &QUdpSocket::readyRead,this, &TestUDP::readyRead);
qDebug() << "Ready Read signal connected0, waiting for broadcasts";
}
void TestUDP::readyRead(){
QByteArray buffer;
buffer.resize(socket->pendingDatagramSize());
//var to store headers from udp
QHostAddress sender;
quint16 sender_port;
socket->readDatagram(buffer.data(),buffer.size(), &sender,
&sender_port);
qDebug() << "Message from " << sender << " port " << sender_port;
qDebug() << "Msg: " << buffer;
}
Maybe the problem is at the way zbeacon is dealing with the UDP socket...
Has anyone successfully used a zbeacon on RPI3?
Thanks in advance!
Rodrigo Madruga.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] On hooking memory allocations
Hi,
I don't see a big problem with the C API except that C doesn't support
overloads. So if the function has a new name, e.g. zmq_ctx_new_with_allocator,
everything stays plain C. The default instance would be a
static void* malloc_(size_t n, void*) {return malloc(n);}
static void free_(void* ptr, size_t n, void*) {free(ptr);}
allocator_t alloc{
NULL,
malloc_,
free_
};
context_t then stores the member and gets methods to forward memory allocations
to the function pointers, passing the hint pointer as an additional argument.
In my C++ code, I can then use an allocator
static void* allocate(size_t n, void* obj) {return
static_cast>(obj)->allocate(n); }
static void free_(void* ptr, size_t n, void*obj) {
static_cast>(obj)->deallocate(ptr, n); }
std::allocator a;
allocator_t zmqAlloc{
&a,
allocate,
free_
};
void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
I think this should work?
Best wishes,
Jens
--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens.a...@cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
de.cgi.com/pflichtangaben.
CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
Group Inc. and its affiliates may be contained in this message. If you are not
a recipient indicated or intended in this message (or responsible for delivery
of this message to such person), or you think for any reason that this message
may have been addressed to you in error, you may not use or copy or deliver
this message to anyone else. In such case, you should destroy this message and
are asked to notify the sender by reply e-mail.
> -Original Message-
> From: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of
> Luca Boccassi
> Sent: 28 November 2016 12:30
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] On hooking memory allocations
>
> That would work for an internal API, but given we expose a C API
> unfortunately I
> don't think that would work as a public API :-( And I think for this use case
> they
> would require a public API.
>
> As an external API, a new zmq_ctx_set that takes a callback would have been
> ideal,
> but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes a
> callback
> pointer would be the next best.
>
> An alternative would be to have a system similar to what we use for the poll
> implementation (epoll kqueue select etc), but this would be a build-time
> option,
> and the implementation would have to be checked in, which I don't think is an
> option for this case, right?
>
> On Mon, 2016-11-28 at 10:51 +, Auer, Jens wrote:
> > Hi,
> >
> > I am just a user, but I would love to see this change. I have thinking
> > about this and I would like to be able to pass a C++ allocator object
> > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > define a struct in the interface
> >
> > struct allocator_t
> > {
> > void* hint;
> > void* (allocate)(size_t, void*);
> > void (deallocate)(void*, size_t, void*); };
> >
> > and store this in the context object. Since I don't think that this should
> > be
> changed during runtime, I would create a new zmq_ctx_new overload which takes
> a
> parameter of type allocator_t. The default value would be to call malloc/free.
> >
> > Cheers,
> > Jens
> >
> > --
> > Jens Auer | CGI | Software-Engineer
> > CGI (Germany) GmbH & Co. KG
> > Rheinstraße 95 | 64295 Darmstadt | Germany
> > T: +49 6151 36860 154
> > jens.a...@cgi.com
> > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
> de.cgi.com/pflichtangaben.
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
> > CGI
> Group Inc. and its affiliates may be contained in this message. If you are
> not a
> recipient indicated or intended in this message (or responsible for delivery
> of this
> message to such person), or you think for any reason that this message may
> have
> been addressed to you in error, you may not use or copy or deliver this
> message to
> anyone else. In such case, you should destroy this message and are asked to
> notify
> the sender by reply e-mail.
> >
> > Von: zeromq-dev [zeromq-dev-boun...@lists.zeromq.org]" im Auftrag von
> > "Petteri Salo [petteri.s...@gmail.com]
> > Gesendet: Montag, 28. November 2016 09:40
> > An: zeromq-dev@lists.zeromq.org
> > Betreff: [zeromq-dev] On hooking memory allocations
> >
> > Hello,
> >
> > Let me first do a little introduction as I'm new to this list. I'm a
> > software engineer
> with 15+ years of experience working on games at a company called Remedy
> Entertainment Ltd. We've done games for PC, and various games consoles over
> the
> years. Most recently we did Quantum Break for Xbox One.
> >
> > I've now been tasked with evaluating ZeroMQ.
Re: [zeromq-dev] On hooking memory allocations
Hi,
I haven't done enough analysis (yet) about current allocations in libzmq to
say which would work best for us, compile time or a runtime hooking. But
given that libzmq comes with source code, I would probably look for a
compile time option at first, like a macro. So roughly:
#ifdef USE_DEFAULT_ALLOCS
#define ZMQ_MALLOC ( ... ) malloc( ... )
#define ZMQ_FREE ( ... ) free( ... )
#else
#define ZMQ_MALLOC ( ... ) my_custom_malloc( ... )
#define ZMQ_FREE ( ... ) my_custom_free( ... )
#endif
and then use placement new in those places where the current code uses a
(plain) new.
-Petteri
On Mon, Nov 28, 2016 at 1:29 PM, Luca Boccassi
wrote:
> That would work for an internal API, but given we expose a C API
> unfortunately I don't think that would work as a public API :-( And I
> think for this use case they would require a public API.
>
> As an external API, a new zmq_ctx_set that takes a callback would have
> been ideal, but it only takes int. So perhaps a new
> zmq_ctx_set_allocator that takes a callback pointer would be the next
> best.
>
> An alternative would be to have a system similar to what we use for the
> poll implementation (epoll kqueue select etc), but this would be a
> build-time option, and the implementation would have to be checked in,
> which I don't think is an option for this case, right?
>
> On Mon, 2016-11-28 at 10:51 +, Auer, Jens wrote:
> > Hi,
> >
> > I am just a user, but I would love to see this change. I have thinking
> about this and I would like to be able to pass a C++ allocator object to
> ZeroMQ, so a simple function hook is not enough. My idea is to define a
> struct in the interface
> >
> > struct allocator_t
> > {
> > void* hint;
> > void* (allocate)(size_t, void*);
> > void (deallocate)(void*, size_t, void*);
> > };
> >
> > and store this in the context object. Since I don't think that this
> should be changed during runtime, I would create a new zmq_ctx_new overload
> which takes a parameter of type allocator_t. The default value would be to
> call malloc/free.
> >
> > Cheers,
> > Jens
> >
> > --
> > Jens Auer | CGI | Software-Engineer
> > CGI (Germany) GmbH & Co. KG
> > Rheinstraße 95 | 64295 Darmstadt | Germany
> > T: +49 6151 36860 154
> > jens.a...@cgi.com
> > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter de.cgi.com/pflichtangaben.
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> to CGI Group Inc. and its affiliates may be contained in this message. If
> you are not a recipient indicated or intended in this message (or
> responsible for delivery of this message to such person), or you think for
> any reason that this message may have been addressed to you in error, you
> may not use or copy or deliver this message to anyone else. In such case,
> you should destroy this message and are asked to notify the sender by reply
> e-mail.
> >
> > Von: zeromq-dev [zeromq-dev-boun...@lists.zeromq.org]" im Auftrag von
> "Petteri Salo [petteri.s...@gmail.com]
> > Gesendet: Montag, 28. November 2016 09:40
> > An: zeromq-dev@lists.zeromq.org
> > Betreff: [zeromq-dev] On hooking memory allocations
> >
> > Hello,
> >
> > Let me first do a little introduction as I'm new to this list. I'm a
> software engineer with 15+ years of experience working on games at a
> company called Remedy Entertainment Ltd. We've done games for PC, and
> various games consoles over the years. Most recently we did Quantum Break
> for Xbox One.
> >
> > I've now been tasked with evaluating ZeroMQ. One important feature of
> any library we use in games is the ability to hook all memory allocations -
> this is to allow the use of custom memory allocators and/or for tracking
> when and where memory is allocated.
> >
> > I've searched the libzmq source code and there is about 150 uses of new,
> malloc, realloc , etc.
> >
> > If we were to adopt libzmq we'd like to put in allocation hooks and that
> work would then be something that we'd like to contribute back to the
> project. Having those hooks in the main repository would then make it
> easier for us to adopt future changes to the library.
> >
> > So, my question is would this kind of change be something that would be
> accepted? Of course assuming that coding conventions, proper way of
> submitting the patch etc. are followed. I do realize that one would want to
> see the actual code before accepting. I'm interested in the principle of
> accepting a change such as this, since it would introduce a new "rule" for
> working ión libzmq source code : "All allocations shall go through an
> allocation hook."
> >
> > Best Regards,
> >
> > Petteri Salo
> >
> >
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> ___
> zeromq-de
Re: [zeromq-dev] On hooking memory allocations
That would work for an internal API, but given we expose a C API
unfortunately I don't think that would work as a public API :-( And I
think for this use case they would require a public API.
As an external API, a new zmq_ctx_set that takes a callback would have
been ideal, but it only takes int. So perhaps a new
zmq_ctx_set_allocator that takes a callback pointer would be the next
best.
An alternative would be to have a system similar to what we use for the
poll implementation (epoll kqueue select etc), but this would be a
build-time option, and the implementation would have to be checked in,
which I don't think is an option for this case, right?
On Mon, 2016-11-28 at 10:51 +, Auer, Jens wrote:
> Hi,
>
> I am just a user, but I would love to see this change. I have thinking about
> this and I would like to be able to pass a C++ allocator object to ZeroMQ, so
> a simple function hook is not enough. My idea is to define a struct in the
> interface
>
> struct allocator_t
> {
> void* hint;
> void* (allocate)(size_t, void*);
> void (deallocate)(void*, size_t, void*);
> };
>
> and store this in the context object. Since I don't think that this should be
> changed during runtime, I would create a new zmq_ctx_new overload which takes
> a parameter of type allocator_t. The default value would be to call
> malloc/free.
>
> Cheers,
> Jens
>
> --
> Jens Auer | CGI | Software-Engineer
> CGI (Germany) GmbH & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> jens.a...@cgi.com
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
> de.cgi.com/pflichtangaben.
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
> Group Inc. and its affiliates may be contained in this message. If you are
> not a recipient indicated or intended in this message (or responsible for
> delivery of this message to such person), or you think for any reason that
> this message may have been addressed to you in error, you may not use or copy
> or deliver this message to anyone else. In such case, you should destroy this
> message and are asked to notify the sender by reply e-mail.
>
> Von: zeromq-dev [zeromq-dev-boun...@lists.zeromq.org]" im Auftrag von
> "Petteri Salo [petteri.s...@gmail.com]
> Gesendet: Montag, 28. November 2016 09:40
> An: zeromq-dev@lists.zeromq.org
> Betreff: [zeromq-dev] On hooking memory allocations
>
> Hello,
>
> Let me first do a little introduction as I'm new to this list. I'm a software
> engineer with 15+ years of experience working on games at a company called
> Remedy Entertainment Ltd. We've done games for PC, and various games consoles
> over the years. Most recently we did Quantum Break for Xbox One.
>
> I've now been tasked with evaluating ZeroMQ. One important feature of any
> library we use in games is the ability to hook all memory allocations - this
> is to allow the use of custom memory allocators and/or for tracking when and
> where memory is allocated.
>
> I've searched the libzmq source code and there is about 150 uses of new,
> malloc, realloc , etc.
>
> If we were to adopt libzmq we'd like to put in allocation hooks and that work
> would then be something that we'd like to contribute back to the project.
> Having those hooks in the main repository would then make it easier for us to
> adopt future changes to the library.
>
> So, my question is would this kind of change be something that would be
> accepted? Of course assuming that coding conventions, proper way of
> submitting the patch etc. are followed. I do realize that one would want to
> see the actual code before accepting. I'm interested in the principle of
> accepting a change such as this, since it would introduce a new "rule" for
> working ión libzmq source code : "All allocations shall go through an
> allocation hook."
>
> Best Regards,
>
> Petteri Salo
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
signature.asc
Description: This is a digitally signed message part
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] On hooking memory allocations
Hello, First of all, welcome to the community! Sounds like you have a very interesting use case :-) So there is precedent for using some sort of a custom allocator, although it does not go as far as your requirements: https://github.com/zeromq/czmq/blob/master/include/czmq_prelude.h#L528 So no objections in principle I think. Regarding what can get merged and what can't, we use the C4 process as defined here: https://rfc.zeromq.org/spec:42/C4/ Basically any correct patch that doesn't break (too much) stuff can be accepted :-) I would recommend, following C4, to start sending PRs as soon as you can, with small changes. We have a DRAFT api process, where new stuff is hidden behind a --enable-drafts configure call. I would recommend making as much as possible of this new feature hidden behind the flag, as to avoid disrupting existing users (we know many users build straight from the master branch head). For example, and it's up to you so it's just a suggestion of course, you could start by changing all the mallocs to a common function sort of like czmq does, which without --enable-drafts is just an inline static that calls malloc. And then implement the logic, public and private as draft (this way you can change the public interface as much as you want until it's table). Then we can iterate from that. Again it's entirely up to you. But we've used this model in the past year to successfully introduce big changes without major disruption (eg: new sockets, new poller, etc). On Mon, 2016-11-28 at 10:40 +0200, Petteri Salo wrote: > Hello, > > Let me first do a little introduction as I'm new to this list. I'm a > software engineer with 15+ years of experience working on games at a > company called Remedy Entertainment Ltd. We've done games for PC, and > various games consoles over the years. Most recently we did Quantum Break > for Xbox One. > > I've now been tasked with evaluating ZeroMQ. One important feature of any > library we use in games is the ability to hook all memory allocations - > this is to allow the use of custom memory allocators and/or for tracking > when and where memory is allocated. > > I've searched the libzmq source code and there is about 150 uses of new, > malloc, realloc , etc. > > If we were to adopt libzmq we'd like to put in allocation hooks and that > work would then be something that we'd like to contribute back to the > project. Having those hooks in the main repository would then make it > easier for us to adopt future changes to the library. > > So, my question is would this kind of change be something that would be > accepted? Of course assuming that coding conventions, proper way of > submitting the patch etc. are followed. I do realize that one would want to > see the actual code before accepting. I'm interested in the principle of > accepting a change such as this, since it would introduce a new "rule" for > working ión libzmq source code : "All allocations shall go through an > allocation hook." > > Best Regards, > > Petteri Salo > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev signature.asc Description: This is a digitally signed message part ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] On hooking memory allocations
Hi,
I am just a user, but I would love to see this change. I have thinking about
this and I would like to be able to pass a C++ allocator object to ZeroMQ, so a
simple function hook is not enough. My idea is to define a struct in the
interface
struct allocator_t
{
void* hint;
void* (allocate)(size_t, void*);
void (deallocate)(void*, size_t, void*);
};
and store this in the context object. Since I don't think that this should be
changed during runtime, I would create a new zmq_ctx_new overload which takes a
parameter of type allocator_t. The default value would be to call malloc/free.
Cheers,
Jens
--
Jens Auer | CGI | Software-Engineer
CGI (Germany) GmbH & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens.a...@cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
de.cgi.com/pflichtangaben.
CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
Group Inc. and its affiliates may be contained in this message. If you are not
a recipient indicated or intended in this message (or responsible for delivery
of this message to such person), or you think for any reason that this message
may have been addressed to you in error, you may not use or copy or deliver
this message to anyone else. In such case, you should destroy this message and
are asked to notify the sender by reply e-mail.
Von: zeromq-dev [zeromq-dev-boun...@lists.zeromq.org]" im Auftrag von "Petteri
Salo [petteri.s...@gmail.com]
Gesendet: Montag, 28. November 2016 09:40
An: zeromq-dev@lists.zeromq.org
Betreff: [zeromq-dev] On hooking memory allocations
Hello,
Let me first do a little introduction as I'm new to this list. I'm a software
engineer with 15+ years of experience working on games at a company called
Remedy Entertainment Ltd. We've done games for PC, and various games consoles
over the years. Most recently we did Quantum Break for Xbox One.
I've now been tasked with evaluating ZeroMQ. One important feature of any
library we use in games is the ability to hook all memory allocations - this is
to allow the use of custom memory allocators and/or for tracking when and where
memory is allocated.
I've searched the libzmq source code and there is about 150 uses of new,
malloc, realloc , etc.
If we were to adopt libzmq we'd like to put in allocation hooks and that work
would then be something that we'd like to contribute back to the project.
Having those hooks in the main repository would then make it easier for us to
adopt future changes to the library.
So, my question is would this kind of change be something that would be
accepted? Of course assuming that coding conventions, proper way of submitting
the patch etc. are followed. I do realize that one would want to see the actual
code before accepting. I'm interested in the principle of accepting a change
such as this, since it would introduce a new "rule" for working ión libzmq
source code : "All allocations shall go through an allocation hook."
Best Regards,
Petteri Salo
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] On hooking memory allocations
Hello, Let me first do a little introduction as I'm new to this list. I'm a software engineer with 15+ years of experience working on games at a company called Remedy Entertainment Ltd. We've done games for PC, and various games consoles over the years. Most recently we did Quantum Break for Xbox One. I've now been tasked with evaluating ZeroMQ. One important feature of any library we use in games is the ability to hook all memory allocations - this is to allow the use of custom memory allocators and/or for tracking when and where memory is allocated. I've searched the libzmq source code and there is about 150 uses of new, malloc, realloc , etc. If we were to adopt libzmq we'd like to put in allocation hooks and that work would then be something that we'd like to contribute back to the project. Having those hooks in the main repository would then make it easier for us to adopt future changes to the library. So, my question is would this kind of change be something that would be accepted? Of course assuming that coding conventions, proper way of submitting the patch etc. are followed. I do realize that one would want to see the actual code before accepting. I'm interested in the principle of accepting a change such as this, since it would introduce a new "rule" for working ión libzmq source code : "All allocations shall go through an allocation hook." Best Regards, Petteri Salo ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
