[GitHub] spark pull request #20890: [WIP][SPARK-23779][SQL] TaskMemoryManager and Uns...

2018-04-06 Thread kiszk
Github user kiszk closed the pull request at:

https://github.com/apache/spark/pull/20890


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request #20890: [WIP][SPARK-23779][SQL] TaskMemoryManager and Uns...

2018-03-23 Thread kiszk
GitHub user kiszk opened a pull request:

https://github.com/apache/spark/pull/20890

[WIP][SPARK-23779][SQL] TaskMemoryManager and UnsafeSorter related classes 
use MemoryBlock

## What changes were proposed in this pull request?

Waiting for merging #19222

This PR tries to use `MemoryBlock` in `TaskMemoryManager` and 
`UnsafeSorter` related classes. There are two advantages to use `MemoryBlock`.

1. Has clean API calls rather than using a Java array or `PlatformMemory`
2. Improve runtime performance of memory access instead of using `Object` 
with `Platform.get/put...`.

## How was this patch tested?

Used existing UTs

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kiszk/spark SPARK-23779

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20890.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20890


commit f427ca2a0f30d70f42b4755d0d72318cfa0e4c77
Author: Kazuaki Ishizaki 
Date:   2017-09-13T10:16:19Z

introduce ByteArrayMemoryBlock, IntArrayMemoryBlock, LongArrayMemoryBlock, 
and OffheaMemoryBlock

commit 5d7ccdb0e845afcaa430bac3d21b519d35d1e6f4
Author: Kazuaki Ishizaki 
Date:   2017-09-13T17:15:25Z

OffHeapColumnVector uses UnsafeMemoryAllocator

commit 251fa09d4421085a6c97f56bb38108980a79b5c8
Author: Kazuaki Ishizaki 
Date:   2017-09-13T17:27:09Z

UTF8String uses UnsafeMemoryAllocator

commit 790bbe7f3ac52b2b1e4375684998737d6127a552
Author: Kazuaki Ishizaki 
Date:   2017-09-13T17:36:57Z

Platform.copymemory() in UsafeInMemorySorter uses new MemoryBlock

commit 93a792e7729f213fcef66f0ed9b33f45259d1ec3
Author: Kazuaki Ishizaki 
Date:   2017-09-14T15:34:48Z

address review comments

commit 0beab0308cdac57e85daf7794170a0ca899ab568
Author: Kazuaki Ishizaki 
Date:   2017-09-14T17:53:33Z

fix test failures (e.g. String in UnsafeArrayData)

commit fcf764c1aebdc847675f710c17ec8477d6022a40
Author: Kazuaki Ishizaki 
Date:   2017-09-18T16:13:13Z

fix failures

commit d2d2e50f8a2baf41d5b85127bf888da6f8bca343
Author: Kazuaki Ishizaki 
Date:   2017-09-21T18:45:55Z

minor update of UTF8String constructor

commit f5e10bb52c33856ddd3e1b1f8483b170e0167c53
Author: Kazuaki Ishizaki 
Date:   2017-09-22T11:00:12Z

rename method name

commit 1905e8ca4b3b8200fa56f5fc91899bb420a07628
Author: Kazuaki Ishizaki 
Date:   2017-09-22T11:01:13Z

remove unused code

commit 7778e586e94130749cec3f54a60b6fb24514647a
Author: Kazuaki Ishizaki 
Date:   2017-09-22T11:02:30Z

update arrayEquals

commit 4f96c82b151b78641bcfc92a65913048b055cfee
Author: Kazuaki Ishizaki 
Date:   2017-09-22T13:18:01Z

rebase master

commit d1d6ae90589c0fae6c64af0cb95696e270446228
Author: Kazuaki Ishizaki 
Date:   2017-09-22T14:51:40Z

make more methods final

commit 914dcd11d0d5ef014284868f2794cb4e5baa0958
Author: Kazuaki Ishizaki 
Date:   2017-09-22T15:39:57Z

make fill method final in MemoryBlock

commit 336e4b7bfd7edcb861edeac3ca115dead785b68a
Author: Kazuaki Ishizaki 
Date:   2017-09-23T11:35:15Z

fix test failures

commit 5be9ccb163832e1895b045730a06e79eb3b171cf
Author: Kazuaki Ishizaki 
Date:   2017-09-24T14:00:40Z

add testsuite

commit 43e6b572bd893bd42e58df930a34b7e31549a49a
Author: Kazuaki Ishizaki 
Date:   2017-09-24T18:10:49Z

pass concrete type to the first argument of Platform.get*/put* to get 
better performance

commit 05f024e566f828e9c3f836430c9c7b34da5e954b
Author: Kazuaki Ishizaki 
Date:   2017-09-28T01:51:38Z

rename methods related to hash

commit 9071cf6449123400f3d774664e1709337b05c555
Author: Kazuaki Ishizaki 
Date:   2017-09-28T01:52:48Z

added methods for MemoryBlock

commit 37ee9fa07a8f8faaeb097164422bb15958fa4b1c
Author: Kazuaki Ishizaki 
Date:   2017-09-28T03:44:30Z

rebase with master

commit d0b5d59bb31fe2845477ee243008992686e2f2a2
Author: Kazuaki Ishizaki 
Date:   2017-09-28T04:37:25Z

fix scala style error

commit 5cdad44717ccb510d4114d14cfff304bef9f5bb4
Author: Kazuaki Ishizaki 
Date:   2017-10-14T07:29:14Z

use MemoryBlock in Murmur3 for performance reason

commit 91028fa2ae34bb3ae667692112b8455d4394cbbd
Author: Kazuaki Ishizaki 
Date:   2017-10-14T07:29:30Z

fix typo in comment

commit 0210bd1e5f46f81617a35493d2cd0b737b4cf85d
Author: Kazuaki Ishizaki 
Date:   2017-10-29T12:32:38Z

address review comment

commit df6dad3762f4e918d503df75ae8fce052af8bf43
Author: Kazuaki Ishizaki 
Date:   2017-11-28T06:08:52Z

rebase with master

commit 1fa47a8c291ab54a8c1e386737769f467a76a672
Author: Kazuaki Ishizaki 
Date:   2018-02-20T12:17:26Z

fix failures

commit 01f9c8e8146ff8d21d18feba59c0c2ad83299e2a
Author: Kazuaki Ishizaki 
Date:   2018-02-20T17:30:11Z

fix failures in ArrowColumnVectorSuite and FeatureHasherSuite

commit 2ed8f82f13288e00e6352bdccbac0dd890