dhirschfeld commented on pull request #8255: URL: https://github.com/apache/arrow/pull/8255#issuecomment-704841013
> *what do you use serialize for currently?* I've just got a proof-of-concept arrow serialization framework which can serialize arbitrary Python objects (inheriting from a base class). Unfortunately, after implementing that I found it's not language-agnostic so it's languished as a bit of a curiosity. I need a language-agnostic serialization format which can serialise a Python `Dict[str, object]` where `object` is itself serializable to a `Dict[str, primitive_type]`. I'm performance sensitive and may need to serialize large arrays/DataFrames (*"primitive"* data-science types). Given that, I hoped `pyarrow` might fit the bill but it doesn't seem suited to serializing heterogeneous mappings? Currently wondering if `msgpack` might be better suited but interested to hear if this is something on the radar/roadmap for `arrow`? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
