Hi, I'm not sure I understand your message.
The Cap'n Proto encoding is binary-compatible across all implementations (it wouldn't be a very good serialization format otherwise). The ABI issue we're discussing here is that of the libcapnp library -- that is, the C++ interfaces. pycapnp is implemented as a wrapper around libcapnp. Vitaly was discussing a case where there is a second Python extension loaded into the same program which *also* uses libcapnp and wishes to interact with pycapnp as well. Hence they would be passing C++ objects (not just serialized messages) back and forth, which requires C++ ABI compatibility (not just binary message encoding compatibility). -Kenton On Wed, May 3, 2017 at 5:00 PM, Hedge Hog <[email protected]> wrote: > Hi, > I'm contemplating working on the Ruby binding. It seems reasonable to > anticipate that I or others will strike this same issue. Some further > questions below... > > On Friday, 17 February 2017 09:29:23 UTC+11, Kenton Varda wrote: >> >> On Thu, Feb 16, 2017 at 12:51 PM, Scott Purdy <[email protected]> wrote: >> >>> Kenton, thanks for helping bring some clarity to this. It sounds like >>> our two options are: >>> >>> 1. Require pycapnp and our extensions to be compiled in the same >>> environment. We could potentially do this. We could make the install >>> process easy for end users by forking pycapnp and putting wheels up on PyPI >>> but we'd like to avoid that if possible. >>> >> >> I would argue that pycapnp should somehow export its version of libcapnp >> so that other Python extensions that also use libcapnp are able to reuse >> the same one. It makes sense for any Python extension that uses libcapnp.so >> to declare a dependency on pycapnp, I would think. >> > > I'm pretty sure I don't understand this correctly ;) > > Is it correct that issue only applies to CP's struct types (the case cited > in the OP)? > So when using all the other CP types we're good to go across different > environments? > I recall from the distant past some sensitivity issues around ABI > compatibility and `enum` types. > Now I'm not sure if the enum in CP's language maps that closely to the > compiler's `enum`, and if they too will expose the issue raised here. > > I know it is a lot to ask, but could the doc here [1] be updated to warn > users of these issues for each of CP's types? > > Is guidance to users as simple as 'use only the built in types in your > messages to minimise ABI compatibility risks/issues'? > i.e. are `List`, `Data` and `Text` subject to this same issue? > > [1]: https://capnproto.org/language.html#interfaces > > Best wishes > >> >> But I have no idea what this looks like logistically. >> >> 2. Pass the byte buffers, incurring a memory copy for anything that we >>> pass across the boundary. >>> >>> I'd like to explore #2 a bit more. Would this involve extracting the >>> segments from the pycapnp builder/reader, passing that to our extension, >>> and constructing a new builder/reader around the byte buffer? Or would we >>> have to construct a new message in the extension, pass the segments from >>> that back and find a way to copy that buffer into the pycapnp message >>> builder/reader? >>> >> >> There's no good way to share builders, since there would be no way for >> them to synchronize memory allocation. So, once a buffer has been passed, >> it needs to be read-only. >> >> If you are trying to build a message in Python code but have one branch >> of the message be built in C++ code, I think what you'll need to do is >> create a brand new MessageBuilder in C++, build just the C++ branch of the >> message there, and then pass this message to Python. In Python, you could >> read the message with a MessageReader and then copy the contents into the >> branch of the final message. This is where the copy is incurred -- when >> moving data from one message into another message. Presumably you can >> transmit individual messages between languages without any copies. >> >> -Kenton >> >> >>> We are also happy to put together a little demo project once we figure >>> this out so others that want to do something similar have a starting place. >>> >>> On Wednesday, February 15, 2017 at 2:59:56 PM UTC-8, Kenton Varda wrote: >>>> >>>> Hi Vitaly, >>>> >>>> For ABI compatibility, you'd need pycapnp built against exactly the >>>> same version of Cap'n Proto which you're using elsewhere in the process. >>>> Ideally both would link against the same libcapnp.so, although I *think* >>>> loading two copies of the library should not create problems as long as >>>> they are the same version. (This differs from libprotobuf, which definitely >>>> can't handle being loaded multiple times in the same process.) >>>> >>>> You may also need to make sure both copies are built with the same >>>> compiler. We're aware of at least one ABI incompatibility issue between >>>> Clang and GCC that affects Cap'n Proto. >>>> >>>> Of course, if you can't make anything work, you can always fall back to >>>> transferring byte buffers, at the expense of possibly needing to make a >>>> copy to merge the sub-messages into one overall message. >>>> >>>> -Kenton >>>> >>>> On Tue, Feb 14, 2017 at 4:46 PM, vitaly numenta < >>>> [email protected]> wrote: >>>> >>>>> I am experiencing binary compatibility issues trying to get pycapnp >>>>> serialization/deserialization working with C extensions. There appear to >>>>> be >>>>> ABI compatibility issues when passing C++ structs compiled in pycapnp into >>>>> our C extensions that are compiled in a different environment. >>>>> >>>>> When serializing an instance of a class that's implemented in NuPIC, >>>>> we create a message builder via pycapnp and pass it to the corresponding >>>>> instance's write method, which in turn invokes write methods of its >>>>> own contained members. This works fine for members whose classes are >>>>> implemented in python, but doesn't always work for those implemented in >>>>> the >>>>> nupic.bindings extension due to ABI issues. >>>>> >>>>> For example, when serializing the TemporalMemory class, we might >>>>> employ the following sequence: >>>>> >>>>> from nupic.proto import TemporalMemoryProto_capnp >>>>> >>>>> builder = TemporalMemoryProto_capnp.TemporalMemoryProto.new_message() >>>>> >>>>> temporal_memory.write(builder) >>>>> >>>>> Inside TemporalMemory.write(builder), we have something along these >>>>> lines: >>>>> >>>>> class TemporalMemory(object): >>>>> def write(self, builder): >>>>> builder.columnDimensions = list(self.columnDimensions) >>>>> self.connections.write(builder.connections) # pure python >>>>> *self._random.write(builder.random) # C++ Random class from >>>>> extension* >>>>> >>>>> >>>>> The Random class that's implemented inside the nupic.bindings >>>>> extension needs to rely on our own build of capnproto that's linked into >>>>> the extension, but this doesn't seem to be compatible with the object >>>>> constructed in pycapnp. >>>>> >>>>> We learned the hard way, after much trial and error, that we can't >>>>> simply pass the underlying message builders that were instantiated by >>>>> pycapnp's capnp.so module to our own build of capnproto contained in the >>>>> nupic.bindigns extension. This was particularly evident when working on >>>>> the >>>>> manylinux wheel for nupic.bindings, which needs to be compiled using the >>>>> toolchain and c/c++ runtimes from CentOS-6. This resulted in ABI >>>>> incompatibilities when the capnproto code compiled into the extension >>>>> attempts to operate on a message builder that was constructed by pycapnp's >>>>> build of capnp.so. The message builder instance created by pycapnp's >>>>> capnp.so appears corrupted when operated upon by the capnproto code linked >>>>> into the extension. >>>>> >>>>> >>>>> Is there any recommendation for handling this dual Python/C-extension >>>>> scenario that avoids the ABI compatibility problem with C++ objects? >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Cap'n Proto" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/capnproto. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Cap'n Proto" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> Visit this group at https://groups.google.com/group/capnproto. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > Visit this group at https://groups.google.com/group/capnproto. > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
