[
https://issues.apache.org/jira/browse/ARROW-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16764444#comment-16764444
]
Wes McKinney commented on ARROW-585:
------------------------------------
Yes, in C++ the idea is to have a user-defined type registry. When defining a
new user type we need to provide the registry the following:
* A string (hopefully unique) identifier for the user-defined type.
* A deserialization callback, which, when provided with a serialized
representation of the user type, returns an instance of the type. So the
callback signature would be {{Status(const std::string& serialized,
std::shared_ptr<DataType>* type)}}
* A callback to "box" the physical data into a subclass of
{{std::shared_ptr<Array>}} so clients can then statically-cast to the
user-container
A user-defined type would look similar to this:
{code}
class MyUserType : public UserDefinedType {
public:
// Get serialized representation of type, for schema serialization
std::string Serialize() const override;
};
{code}
So in the IPC code paths
* Serialize is invoked, and this is stored in the {{custom_metadata}} field on
the wire. We need to decide what "key" is used something like "arrow_user_type"
* On read in the consumer, if "arrow_user_type" is populated, look in the type
registry, reconstruct the data type, then create the right "box" around the
data. We'll need to add a special case for user types in
https://github.com/apache/arrow/blob/master/cpp/src/arrow/array.cc#L884
All and all this doesn't seem like too much work to me once you decide how to
do it. In Java I'm not sure how this would have to be adapted.
> [C++] Define public API for user-defined data types
> ---------------------------------------------------
>
> Key: ARROW-585
> URL: https://issues.apache.org/jira/browse/ARROW-585
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Priority: Major
> Fix For: 0.13.0
>
>
> This will include:
> * Implementing a subclass of DataType
> * A "fallback" mechanism for receivers that do not understand our custom
> metadata
> * Implementing a serializer interface for custom metadata (to be send and
> received in an IPC setting)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
