GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/9477
[SPARK-7542] [SQL] Support off-heap index/sort buffer
This brings the support of off-heap memory for array inside BytesToBytesMap
and InMemorySorter, then we could allocate all the memory from off-heap for
execution.
This PR include #9383, see
https://github.com/apache/spark/commit/2b3277781c21d0efb20275bd5632a2d4f7f171c3
for the real changes.
Closes #8068
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark unsafe_timsort
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9477.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9477
----
commit 4e09081050633bc8baba4cebc5ffba6e9d900bae
Author: Davies Liu <[email protected]>
Date: 2015-10-30T19:08:29Z
Do hash-based aggregation for all records before switch to sort-based
commit 53dbdf2d4c8c547e6bd50a589bf0223e7ce95e84
Author: Davies Liu <[email protected]>
Date: 2015-10-30T19:24:08Z
merge the last map
commit 2e341f50b656d0effe36004b6abc68898a119f35
Author: Davies Liu <[email protected]>
Date: 2015-11-02T18:56:59Z
update tests
commit df44fc64ed1495a1c0f6f51a7014327b6a8750b7
Author: Davies Liu <[email protected]>
Date: 2015-11-02T20:57:18Z
fix bug
commit 6f3bb15b19cd326f677f15860cf215f57fd3671a
Author: Davies Liu <[email protected]>
Date: 2015-11-03T21:27:35Z
address comments, add regression test
commit fc5e052ff17560d02ef7cdeec91a4a30605c65f0
Author: Davies Liu <[email protected]>
Date: 2015-11-04T04:51:13Z
free the array after spilling
commit d89e03463b99e56fd25e5ada8f7d146b6749082f
Author: Davies Liu <[email protected]>
Date: 2015-11-04T06:11:58Z
refactor
commit 1c0c6c36a5a16c33ceb4cd43534ce02ec3c2b286
Author: Davies Liu <[email protected]>
Date: 2015-11-04T06:20:02Z
cleanup
commit d8422e15e70a1fab2535ca13ac01c0d3a7be19e9
Author: Davies Liu <[email protected]>
Date: 2015-11-04T06:27:11Z
throw better exception
commit cbeaedf1cc47365ea90db6478819ca02db5acaea
Author: Davies Liu <[email protected]>
Date: 2015-11-04T23:00:16Z
add more comments
commit fbce6fe74b8dccd0aefa98a1183ba1321b500a56
Author: Davies Liu <[email protected]>
Date: 2015-11-04T23:08:46Z
Merge branch 'master' of github.com:apache/spark into fix_switch
commit 10d71694ae07af68265bb36a957b4ff5320d8e72
Author: Davies Liu <[email protected]>
Date: 2015-11-04T23:09:16Z
fix conflict
commit 8a20e569fdb43e26804e1c71439fd2d02c5f5a69
Author: Davies Liu <[email protected]>
Date: 2015-11-04T23:21:08Z
Update UnsafeFixedWidthAggregationMap.java
commit f6a5f0629c0b462fa45c8da209f724c158fba078
Author: Davies Liu <[email protected]>
Date: 2015-11-04T23:46:46Z
fix build
commit 2b3277781c21d0efb20275bd5632a2d4f7f171c3
Author: Davies Liu <[email protected]>
Date: 2015-11-05T00:04:11Z
support off-heap index/sort buffer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
