GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/19472
Use MemoryBlock in UnsafeRow, UnsafeArrayData, and UnsafeMapData ## What changes were proposed in this pull request? Waiting for merging #19222 This PR tries to use `MemoryBlock` in `UnsafeRow`, `UnsafeArray`, and `UnsafeMap`. There are two advantages to use `MemoryBlock` 1. Increase flexibility of choosing memory type (e.g `byte[]`, `int[]`, `long[]`, or `DirectBuffer (offheap)`). 2. Improve runtime performance of memory access instead of using `Object`. ## How was this patch tested? Use existing testsuites You can merge this pull request into a Git repository by running: $ git pull https://github.com/kiszk/spark SPARK-22246 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19472.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19472 ---- commit 28e335d18da8b9c17b18d316943a90bfb290ee45 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-13T10:16:19Z introduce ByteArrayMemoryBlock, IntArrayMemoryBlock, LongArrayMemoryBlock, and OffheaMemoryBlock commit 47b05f3d5a1c2fc2f692cf42745eef9e738eb540 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-13T17:15:25Z OffHeapColumnVector uses UnsafeMemoryAllocator commit ae891eee8e214b63d204e794e2cb9f43f8a71e10 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-13T17:27:09Z UTF8String uses UnsafeMemoryAllocator commit 1f97cfe6db3e404c090df7ae9a8e1b5b88ca254b Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-13T17:36:57Z Platform.copymemory() in UsafeInMemorySorter uses new MemoryBlock commit 5472ab84daf37579ad9f6ca6d17beb652ddd5cfd Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-14T15:34:48Z address review comments commit fa3aa7eb3b87bfddf3923fe4bed1d4fa1081ae2e Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-14T17:53:33Z fix test failures (e.g. String in UnsafeArrayData) commit 186aa822459f314605e6b517da7148c1507b6401 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-18T16:13:13Z fix failures commit 15eab8fff1316cc0e1857e6cb0fdedfba2dc2d53 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-21T18:45:55Z minor update of UTF8String constructor commit dcc45804504c244c1dcd8898aa2004bdc9e0245f Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-22T11:00:12Z rename method name commit e43587efabbb6511ef1e2df3c86d50ac7ee75c4e Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-22T11:01:13Z remove unused code commit b4a9de1d7edf665e4092b68b55a72d0ad6355fd3 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-22T11:01:33Z reformat commit 0ea3cff303f5cd1d3bd9290ca3290fad977f5924 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-22T11:02:30Z update arrayEquals commit 92d9b3b91f654d9cbc7923132afe6118d90781e8 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-22T13:18:01Z rebase master commit 791be2e681d934f9d64faf984949b65228804d11 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-22T14:51:40Z make more methods final commit d1ffe0a67c6dc7888114c49cdeae61f125cf55df Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-22T15:39:57Z make fill method final in MemoryBlock commit 614933e042b9266118383daf254478bd3a9bf438 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-23T11:35:15Z fix test failures commit 81b454df13bbc48b7ae7638bb80b8fe7de820bbf Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-24T14:00:40Z add testsuite commit 81a0964de95ab38d3d06d8226caae37de0243d21 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-24T18:10:49Z pass concrete type to the first argument of Platform.get*/put* to get better performance commit c5f71f29e24306819824714e562aba2353777392 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-28T01:51:38Z rename methods related to hash commit a82220adc4b1f1ccc109c1922f889b2301d55a13 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-28T01:52:48Z added methods for MemoryBlock commit 5ad4c542f9371d12a8391139cee45388c5175764 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-28T03:44:30Z rebase with master commit 67d5a8b48cc851573bad4b825e3612a85cdbe432 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-09-28T04:37:25Z fix scala style error commit 03bc8f34df17de0d0cd927c2ea50673b60fca4af Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-10-11T02:14:34Z rebase with master commit e51fb6ae246a660ec2b0bdd1f6a53a0987d4a063 Author: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Date: 2017-10-11T10:10:21Z Use MemoryBlock in UnsafeRow, UnsafeArrayData, and UnsafeMapData ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org