[
https://issues.apache.org/jira/browse/KUDU-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17121069#comment-17121069
]
Grant Henke commented on KUDU-1276:
-----------------------------------
Kudu recently added support for a columnar, arrow compatible, format. There is
a WIP patch here to update the Python client to use it:
https://gerrit.cloudera.org/#/c/15661/
> Add a vectorized read/write interface for pandas DataFrame objects
> ------------------------------------------------------------------
>
> Key: KUDU-1276
> URL: https://issues.apache.org/jira/browse/KUDU-1276
> Project: Kudu
> Issue Type: New Feature
> Components: client, python
> Reporter: Wes McKinney
> Assignee: Jordan Birdsell
> Priority: Major
>
> A pandas read/write interface would make Kudu significantly easier to use for
> average Python data users.
> The layering is as follows:
> - Writer: "Vectorized" insert that accepts a C/C++ array of values plus an
> array (either bits or bytes) indicating nullness for nullable slots
> - Reader: Converts a row batch to NumPy arrays with missing data
> representation suitable for use in pandas. Ideally should not create more
> than one PyString object for each observed string value. Binary can be
> encoded as UTF8 string, while Timestamp will need to be converted to
> nanoseconds for pandas
> This would also give a very performant and relatively GIL-free data ingest
> path to the Kudu (and Kudu consumers like Impala) without a great deal of
> Python+Cython coding.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
