Thanks, you're right I had misunderstood where the issue was. Best wishes. Hedge
On Thu, May 4, 2017 at 3:21 PM, Kenton Varda <[email protected]> wrote: > Hi, > > I'm not sure I understand your message. > > The Cap'n Proto encoding is binary-compatible across all implementations (it > wouldn't be a very good serialization format otherwise). > > The ABI issue we're discussing here is that of the libcapnp library -- that > is, the C++ interfaces. pycapnp is implemented as a wrapper around libcapnp. > Vitaly was discussing a case where there is a second Python extension loaded > into the same program which *also* uses libcapnp and wishes to interact with > pycapnp as well. Hence they would be passing C++ objects (not just > serialized messages) back and forth, which requires C++ ABI compatibility > (not just binary message encoding compatibility). > > -Kenton > > On Wed, May 3, 2017 at 5:00 PM, Hedge Hog <[email protected]> wrote: >> >> Hi, >> I'm contemplating working on the Ruby binding. It seems reasonable to >> anticipate that I or others will strike this same issue. Some further >> questions below... >> >> On Friday, 17 February 2017 09:29:23 UTC+11, Kenton Varda wrote: >>> >>> On Thu, Feb 16, 2017 at 12:51 PM, Scott Purdy <[email protected]> wrote: >>>> >>>> Kenton, thanks for helping bring some clarity to this. It sounds like >>>> our two options are: >>>> >>>> 1. Require pycapnp and our extensions to be compiled in the same >>>> environment. We could potentially do this. We could make the install >>>> process >>>> easy for end users by forking pycapnp and putting wheels up on PyPI but >>>> we'd >>>> like to avoid that if possible. >>> >>> >>> I would argue that pycapnp should somehow export its version of libcapnp >>> so that other Python extensions that also use libcapnp are able to reuse the >>> same one. It makes sense for any Python extension that uses libcapnp.so to >>> declare a dependency on pycapnp, I would think. >> >> >> I'm pretty sure I don't understand this correctly ;) >> >> Is it correct that issue only applies to CP's struct types (the case cited >> in the OP)? >> So when using all the other CP types we're good to go across different >> environments? >> I recall from the distant past some sensitivity issues around ABI >> compatibility and `enum` types. >> Now I'm not sure if the enum in CP's language maps that closely to the >> compiler's `enum`, and if they too will expose the issue raised here. >> >> I know it is a lot to ask, but could the doc here [1] be updated to warn >> users of these issues for each of CP's types? >> >> Is guidance to users as simple as 'use only the built in types in your >> messages to minimise ABI compatibility risks/issues'? >> i.e. are `List`, `Data` and `Text` subject to this same issue? >> >> [1]: https://capnproto.org/language.html#interfaces >> >> Best wishes >>> >>> >>> But I have no idea what this looks like logistically. >>> >>>> 2. Pass the byte buffers, incurring a memory copy for anything that we >>>> pass across the boundary. >>>> >>>> I'd like to explore #2 a bit more. Would this involve extracting the >>>> segments from the pycapnp builder/reader, passing that to our extension, >>>> and >>>> constructing a new builder/reader around the byte buffer? Or would we have >>>> to construct a new message in the extension, pass the segments from that >>>> back and find a way to copy that buffer into the pycapnp message >>>> builder/reader? >>> >>> >>> There's no good way to share builders, since there would be no way for >>> them to synchronize memory allocation. So, once a buffer has been passed, it >>> needs to be read-only. >>> >>> If you are trying to build a message in Python code but have one branch >>> of the message be built in C++ code, I think what you'll need to do is >>> create a brand new MessageBuilder in C++, build just the C++ branch of the >>> message there, and then pass this message to Python. In Python, you could >>> read the message with a MessageReader and then copy the contents into the >>> branch of the final message. This is where the copy is incurred -- when >>> moving data from one message into another message. Presumably you can >>> transmit individual messages between languages without any copies. >>> >>> -Kenton >>> >>>> >>>> We are also happy to put together a little demo project once we figure >>>> this out so others that want to do something similar have a starting place. >>>> >>>> On Wednesday, February 15, 2017 at 2:59:56 PM UTC-8, Kenton Varda wrote: >>>>> >>>>> Hi Vitaly, >>>>> >>>>> For ABI compatibility, you'd need pycapnp built against exactly the >>>>> same version of Cap'n Proto which you're using elsewhere in the process. >>>>> Ideally both would link against the same libcapnp.so, although I *think* >>>>> loading two copies of the library should not create problems as long as >>>>> they >>>>> are the same version. (This differs from libprotobuf, which definitely >>>>> can't >>>>> handle being loaded multiple times in the same process.) >>>>> >>>>> You may also need to make sure both copies are built with the same >>>>> compiler. We're aware of at least one ABI incompatibility issue between >>>>> Clang and GCC that affects Cap'n Proto. >>>>> >>>>> Of course, if you can't make anything work, you can always fall back to >>>>> transferring byte buffers, at the expense of possibly needing to make a >>>>> copy >>>>> to merge the sub-messages into one overall message. >>>>> >>>>> -Kenton >>>>> >>>>> On Tue, Feb 14, 2017 at 4:46 PM, vitaly numenta >>>>> <[email protected]> wrote: >>>>>> >>>>>> I am experiencing binary compatibility issues trying to get pycapnp >>>>>> serialization/deserialization working with C extensions. There appear to >>>>>> be >>>>>> ABI compatibility issues when passing C++ structs compiled in pycapnp >>>>>> into >>>>>> our C extensions that are compiled in a different environment. >>>>>> >>>>>> When serializing an instance of a class that's implemented in NuPIC, >>>>>> we create a message builder via pycapnp and pass it to the corresponding >>>>>> instance's write method, which in turn invokes write methods of its own >>>>>> contained members. This works fine for members whose classes are >>>>>> implemented >>>>>> in python, but doesn't always work for those implemented in the >>>>>> nupic.bindings extension due to ABI issues. >>>>>> >>>>>> For example, when serializing the TemporalMemory class, we might >>>>>> employ the following sequence: >>>>>> >>>>>> from nupic.proto import TemporalMemoryProto_capnp >>>>>> >>>>>> builder = TemporalMemoryProto_capnp.TemporalMemoryProto.new_message() >>>>>> >>>>>> temporal_memory.write(builder) >>>>>> >>>>>> Inside TemporalMemory.write(builder), we have something along these >>>>>> lines: >>>>>> >>>>>> class TemporalMemory(object): >>>>>> def write(self, builder): >>>>>> builder.columnDimensions = list(self.columnDimensions) >>>>>> self.connections.write(builder.connections) # pure python >>>>>> self._random.write(builder.random) # C++ Random class from >>>>>> extension >>>>>> >>>>>> >>>>>> The Random class that's implemented inside the nupic.bindings >>>>>> extension needs to rely on our own build of capnproto that's linked into >>>>>> the >>>>>> extension, but this doesn't seem to be compatible with the object >>>>>> constructed in pycapnp. >>>>>> >>>>>> We learned the hard way, after much trial and error, that we can't >>>>>> simply pass the underlying message builders that were instantiated by >>>>>> pycapnp's capnp.so module to our own build of capnproto contained in the >>>>>> nupic.bindigns extension. This was particularly evident when working on >>>>>> the >>>>>> manylinux wheel for nupic.bindings, which needs to be compiled using the >>>>>> toolchain and c/c++ runtimes from CentOS-6. This resulted in ABI >>>>>> incompatibilities when the capnproto code compiled into the extension >>>>>> attempts to operate on a message builder that was constructed by >>>>>> pycapnp's >>>>>> build of capnp.so. The message builder instance created by pycapnp's >>>>>> capnp.so appears corrupted when operated upon by the capnproto code >>>>>> linked >>>>>> into the extension. >>>>>> >>>>>> >>>>>> Is there any recommendation for handling this dual Python/C-extension >>>>>> scenario that avoids the ABI compatibility problem with C++ objects? >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Cap'n Proto" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to [email protected]. >>>>>> Visit this group at https://groups.google.com/group/capnproto. >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Cap'n Proto" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> Visit this group at https://groups.google.com/group/capnproto. >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Cap'n Proto" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> Visit this group at https://groups.google.com/group/capnproto. > > -- πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα [The fox knows many things, but the hedgehog knows one big thing.] Archilochus, Greek poet (c. 680 BC – c. 645 BC) http://hedgehogshiatus.com -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
