Andrew Wong has uploaded a new patch set (#4) to the change originally created by Todd Lipcon. ( http://gerrit.cloudera.org:8080/15661 )
Change subject: python: support for passing arrow data to python ...................................................................... python: support for passing arrow data to python This adds some support to pass arrow-formatted data to pyarrow. This takes the form of Arrow's C-structs, as well as helper classes to manage their memory. These structs are exposed in the Cython wrapper files. This also adds Python APIs that match the existing Python APIs. Namely, batch-level APIs that return one scan batch at a time, as well as a to_pandas() method that materializes the entire scan result as a DataFrame. As a sanity check for performance, I wrote a simple script (posted as a gist[1]) to write a bunch of rows and then serialize them to Pandas DataFrames: Materialized tuple DataFrame with 100000 rows in 0.7903690928 secs, averaged across 5 runs Materialized arrow DataFrame with 100000 rows in 0.05492771920000004 secs, averaged across 5 runs [1] https://gist.github.com/andrwng/bae4a4696eabb501b3b69f8db263a745 Change-Id: I5df4714bf5ac339d675f2a9169f8aeff06d30eed --- M python/kudu/__init__.py M python/kudu/client.pyx M python/kudu/libkudu_client.pxd M python/kudu/schema.pyx M python/kudu/tests/test_scanner.py M python/kudu/tests/util.py M src/kudu/client/CMakeLists.txt A src/kudu/client/arrow-internal.h A src/kudu/client/arrow.h M src/kudu/client/client-test.cc M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/columnar_scan_batch.cc M src/kudu/client/columnar_scan_batch.h M src/kudu/client/scanner-internal.cc M src/kudu/client/scanner-internal.h M src/kudu/client/schema.cc M src/kudu/client/schema.h M src/kudu/util/memory/arena.h 19 files changed, 539 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/61/15661/4 -- To view, visit http://gerrit.cloudera.org:8080/15661 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5df4714bf5ac339d675f2a9169f8aeff06d30eed Gerrit-Change-Number: 15661 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Grant Henke <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Wes McKinney <[email protected]>
