GitHub user BryanCutler opened a pull request:
https://github.com/apache/spark/pull/21939
[SPARK-23874][SQL][PYTHON] Upgrade Apache Arrow to 0.10.0
## What changes were proposed in this pull request?
Upgrade Apache Arrow to 0.10.0
Version 0.10.0 will allow for the following improvements and bug fixes:
* Allow for adding BinaryType support
* Bug fix related to array serialization ARROW-1973
* Python2 str will be made into an Arrow string instead of bytes ARROW-2101
* Python bytearrays are supported in as input to pyarrow ARROW-2141
* Java has common interface for reset to cleanup complex vectors in Spark
ArrowWriter ARROW-1962
* Cleanup pyarrow type equality checks ARROW-2423
* ArrowStreamWriter should not hold references to ArrowBlocks ARROW-2632,
ARROW-2645
* Improved low level handling of messages for RecordBatch ARROW-2704
## How was this patch tested?
existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BryanCutler/spark arrow-upgrade-010
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21939.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21939
----
commit 5e4061e16947d0c7d81b86cbccfa0e222b33d48b
Author: Bryan Cutler <cutlerb@...>
Date: 2018-04-09T19:26:06Z
made required code changes for upgrade
commit 8b63e6c39e2b88ba2f4d1e9c666e7dd456d265bf
Author: Bryan Cutler <cutlerb@...>
Date: 2018-04-09T19:26:54Z
remove unused import
commit 95a1bf9de43bcee8684adf47990dc64923a63c0c
Author: Bryan Cutler <cutlerb@...>
Date: 2018-04-20T16:47:05Z
added some TODOs
commit 739a12eca9dbd83256be388703098e4d1f9cab9e
Author: Bryan Cutler <cutlerb@...>
Date: 2018-05-18T20:36:21Z
change to using 0.10 snapshot
commit d1f7355e78367a1b8a4347887fa1780106c6bffd
Author: Bryan Cutler <cutlerb@...>
Date: 2018-08-01T00:40:39Z
replaced manifests
commit abedec3243cce5fa92ea3acf711f504bab6814f3
Author: Bryan Cutler <cutlerb@...>
Date: 2018-08-01T00:42:56Z
removed version snapshot
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
