I am experiencing binary compatibility issues trying to get pycapnp
serialization/deserialization working with C extensions. There appear to be
ABI compatibility issues when passing C++ structs compiled in pycapnp into
our C extensions that are compiled in a different environment.
When serializing an instance of a class that's implemented in NuPIC, we
create a message builder via pycapnp and pass it to the corresponding
instance's write method, which in turn invokes write methods of its own
contained members. This works fine for members whose classes are
implemented in python, but doesn't always work for those implemented in the
nupic.bindings extension due to ABI issues.
For example, when serializing the TemporalMemory class, we might employ the
following sequence:
from nupic.proto import TemporalMemoryProto_capnp
builder = TemporalMemoryProto_capnp.TemporalMemoryProto.new_message()
temporal_memory.write(builder)
Inside TemporalMemory.write(builder), we have something along these lines:
class TemporalMemory(object):
def write(self, builder):
builder.columnDimensions = list(self.columnDimensions)
self.connections.write(builder.connections) # pure python
*self._random.write(builder.random) # C++ Random class from extension*
The Random class that's implemented inside the nupic.bindings extension
needs to rely on our own build of capnproto that's linked into the
extension, but this doesn't seem to be compatible with the object
constructed in pycapnp.
We learned the hard way, after much trial and error, that we can't simply
pass the underlying message builders that were instantiated by pycapnp's
capnp.so module to our own build of capnproto contained in the
nupic.bindigns extension. This was particularly evident when working on the
manylinux wheel for nupic.bindings, which needs to be compiled using the
toolchain and c/c++ runtimes from CentOS-6. This resulted in ABI
incompatibilities when the capnproto code compiled into the extension
attempts to operate on a message builder that was constructed by pycapnp's
build of capnp.so. The message builder instance created by pycapnp's
capnp.so appears corrupted when operated upon by the capnproto code linked
into the extension.
Is there any recommendation for handling this dual Python/C-extension
scenario that avoids the ABI compatibility problem with C++ objects?
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
Visit this group at https://groups.google.com/group/capnproto.
