GitHub user robbinspg opened a pull request:
https://github.com/apache/spark/pull/12501
[SPARK-13745][SQL]Support columnar in memory representation on Big Endian
platforms - implent by subclassing
## What changes were proposed in this pull request?
An alternative implementation of https://github.com/apache/spark/pull/12397
which uses subclasses to minimize any potential performance hits on Little
Endian
parquet datasource and ColumnarBatch tests fail on big-endian platforms
This patch adds support for the little-endian byte arrays being correctly
interpreted on a big-endian platform
## How was this patch tested?
Spark test builds ran on big endian z/Linux and regression build on little
endian amd64
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/robbinspg/spark-1 bigEndianViaSubclass
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12501.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12501
----
commit 1fc048385fb0fea93eef85f614586448a3ea7c2a
Author: Pete Robbins <[email protected]>
Date: 2016-04-14T13:50:34Z
Support columnar in memory representation on Big Endian platforms
commit 3eb481d8c30639c5b9a219e4891ccaccf73075b0
Author: Pete Robbins <[email protected]>
Date: 2016-04-14T19:24:48Z
Use ByteBuffer.wrap instead of allocate
commit 69fc667266c5efe97f796c6b4e8d14470168867d
Author: Pete Robbins <[email protected]>
Date: 2016-04-15T10:10:09Z
Fix offsets
commit a1f06106d321ca40bef6dcd7865484fd79976b08
Author: Pete Robbins <[email protected]>
Date: 2016-04-15T11:55:21Z
Wrap byte array once
commit a652865e9f59ca4cf4fc596141ae0511284462b4
Author: Pete Robbins <[email protected]>
Date: 2016-04-15T12:06:37Z
remove trailing spaces
commit 804740c9dbe3c4bbe8145cc119c997531634ebb1
Author: Pete Robbins <[email protected]>
Date: 2016-04-18T15:05:48Z
Merge branch 'master' of https://github.com/apache/spark.git into
apache-master
commit f109bda995a70be8787fd10f414a5be2125d97b2
Author: Pete Robbins <[email protected]>
Date: 2016-04-19T09:04:41Z
Merge branch 'master' of https://github.com/apache/spark.git into
apache-master
commit d7cbc84e1dfdae1036345956ea8f210c5c982b3d
Author: Pete Robbins <[email protected]>
Date: 2016-04-19T14:13:27Z
Big endian implementation using subclassing
commit 648b7ac9fa0f7466e05151d5f07a1e613a682bfb
Author: Pete Robbins <[email protected]>
Date: 2016-04-19T14:32:19Z
missing else clause
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
