[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131713763
--- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala
---
@@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite {
assert(diskStore.getSize(blockId) === 0L)
}
+ test("blocks larger than 2gb") {
+val conf = new SparkConf()
+val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop =
true)
+val diskStore = new DiskStore(conf, diskBlockManager, new
SecurityManager(conf))
+
+val mb = 1024 * 1024
+val gb = 1024L * mb
+
+val blockId = BlockId("rdd_1_2")
+diskStore.put(blockId) { chan =>
+ val arr = new Array[Byte](mb)
+ for {
+_ <- 0 until 2048
+ } {
+val buf = ByteBuffer.wrap(arr)
+while (buf.hasRemaining()) {
+ chan.write(buf)
+}
+ }
+}
+
+val blockData = diskStore.getBytes(blockId)
--- End diff --
@kiszk, this is the test case I was referring to.
I actually introduced it prior to actually (hopefully) fixing the bug in
`DiskStore.getBytes`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131712834
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore(
}
+private class DiskBlockData(
+conf: SparkConf,
+file: File,
+blockSize: Long) extends BlockData {
+
+ private val minMemoryMapBytes =
conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
+
+ override def toInputStream(): InputStream = new FileInputStream(file)
+
+ /**
+ * Returns a Netty-friendly wrapper for the block's data.
+ *
+ * Please see `ManagedBuffer.convertToNetty()` for more details.
+ */
+ override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
+
+ override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer):
ChunkedByteBuffer = {
+Utils.tryWithResource(open()) { channel =>
+ var remaining = blockSize
+ val chunks = new ListBuffer[ByteBuffer]()
+ while (remaining > 0) {
+val chunkSize = math.min(remaining, Int.MaxValue)
+val chunk = allocator(chunkSize.toInt)
+remaining -= chunkSize
+JavaUtils.readFully(channel, chunk)
+chunk.flip()
+chunks += chunk
+ }
+ new ChunkedByteBuffer(chunks.toArray)
+}
+ }
+
+ override def toByteBuffer(): ByteBuffer = {
--- End diff --
indeed.
I chose to postpone the failure from `DiskStroe.getBytes` to this place as
I believe it introduces no regression while still allowing the more common
'streamin' like use-case.
further more, I think this plays well with the comment about future
deprecation of `org.apache.spark.network.buffer.ManagedBuffer#nioByteBuffer`
which seems to be the main reason for `BlockData` exposing the `toByteBuffer`
method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131709108
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore(
}
+private class DiskBlockData(
+conf: SparkConf,
+file: File,
+blockSize: Long) extends BlockData {
+
+ private val minMemoryMapBytes =
conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
+
+ override def toInputStream(): InputStream = new FileInputStream(file)
+
+ /**
+ * Returns a Netty-friendly wrapper for the block's data.
+ *
+ * Please see `ManagedBuffer.convertToNetty()` for more details.
+ */
+ override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
+
+ override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer):
ChunkedByteBuffer = {
+Utils.tryWithResource(open()) { channel =>
+ var remaining = blockSize
+ val chunks = new ListBuffer[ByteBuffer]()
+ while (remaining > 0) {
+val chunkSize = math.min(remaining, Int.MaxValue)
+val chunk = allocator(chunkSize.toInt)
+remaining -= chunkSize
+JavaUtils.readFully(channel, chunk)
+chunk.flip()
+chunks += chunk
+ }
+ new ChunkedByteBuffer(chunks.toArray)
+}
+ }
+
+ override def toByteBuffer(): ByteBuffer = {
--- End diff --
we will still hit the 2g limitation here, I'm wondering which end-to-end
use cases are affected by it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131549141
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore(
}
+private class DiskBlockData(
+conf: SparkConf,
+file: File,
+blockSize: Long) extends BlockData {
+
+ private val minMemoryMapBytes =
conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
+
+ override def toInputStream(): InputStream = new FileInputStream(file)
+
+ /**
+ * Returns a Netty-friendly wrapper for the block's data.
+ *
+ * Please see `ManagedBuffer.convertToNetty()` for more details.
+ */
+ override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
+
+ override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer):
ChunkedByteBuffer = {
+Utils.tryWithResource(open()) { channel =>
+ var remaining = blockSize
+ val chunks = new ListBuffer[ByteBuffer]()
+ while (remaining > 0) {
+val chunkSize = math.min(remaining, Int.MaxValue)
+val chunk = allocator(chunkSize.toInt)
+remaining -= chunkSize
+JavaUtils.readFully(channel, chunk)
+chunk.flip()
+chunks += chunk
+ }
+ new ChunkedByteBuffer(chunks.toArray)
+}
+ }
+
+ override def toByteBuffer(): ByteBuffer = {
+require( size < Int.MaxValue
--- End diff --
Sorry for confusing you. I mean the last line in your comment. I know the
`size` method has the same value as `blockSize` since I saw the `size` method.
Other places in `toByteBuffer` uses `blockSize` instead of `size`. For ease
of code reading, it would be good to use `blockSize` instead of `size` here.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131544351
--- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala
---
@@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite {
assert(diskStore.getSize(blockId) === 0L)
}
+ test("blocks larger than 2gb") {
+val conf = new SparkConf()
+val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop =
true)
+val diskStore = new DiskStore(conf, diskBlockManager, new
SecurityManager(conf))
+
+val mb = 1024*1024
--- End diff --
nit: `1024 * 1024`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131544332
--- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala
---
@@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite {
assert(diskStore.getSize(blockId) === 0L)
}
+ test("blocks larger than 2gb") {
+val conf = new SparkConf()
+val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop =
true)
+val diskStore = new DiskStore(conf, diskBlockManager, new
SecurityManager(conf))
+
+val mb = 1024*1024
+val gb = 1024L*mb
+
+val blockId = BlockId("rdd_1_2")
+diskStore.put(blockId) { chan =>
+ val arr = new Array[Byte](mb)
+ for{
+_ <- 0 until 2048
+ }{
--- End diff --
nit: `} {`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131544320
--- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala
---
@@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite {
assert(diskStore.getSize(blockId) === 0L)
}
+ test("blocks larger than 2gb") {
+val conf = new SparkConf()
+val diskBlockManager = new DiskBlockManager(conf, deleteFilesOnStop =
true)
+val diskStore = new DiskStore(conf, diskBlockManager, new
SecurityManager(conf))
+
+val mb = 1024*1024
+val gb = 1024L*mb
+
+val blockId = BlockId("rdd_1_2")
+diskStore.put(blockId) { chan =>
+ val arr = new Array[Byte](mb)
+ for{
--- End diff --
nit: `for (`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131543591
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore(
}
+private class DiskBlockData(
+conf: SparkConf,
+file: File,
+blockSize: Long) extends BlockData {
+
+ private val minMemoryMapBytes =
conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
+
+ override def toInputStream(): InputStream = new FileInputStream(file)
+
+ /**
+ * Returns a Netty-friendly wrapper for the block's data.
+ *
+ * Please see `ManagedBuffer.convertToNetty()` for more details.
+ */
+ override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
+
+ override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer):
ChunkedByteBuffer = {
+Utils.tryWithResource(open()) { channel =>
+ var remaining = blockSize
+ val chunks = new ListBuffer[ByteBuffer]()
+ while (remaining > 0) {
+val chunkSize = math.min(remaining, Int.MaxValue)
+val chunk = allocator(chunkSize.toInt)
+remaining -= chunkSize
+JavaUtils.readFully(channel, chunk)
+chunk.flip()
+chunks += chunk
+ }
+ new ChunkedByteBuffer(chunks.toArray)
+}
+ }
+
+ override def toByteBuffer(): ByteBuffer = {
+require( size < Int.MaxValue
--- End diff --
@kiszk , not sure I'm following your comment.
this requirement results with an explicit errors when one tries to obtain a
ByteBuffer larger than java.nio's limitations.
original code used to fail in line 115 when calling `ByteBuffer.allocate`
with block size larger than 2GB, newer code fails explicitly in this cae.
...or do you mean refer the `blockSize` val rather than the `size` method?
can't really see a difference in that case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131543602
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore(
}
+private class DiskBlockData(
+conf: SparkConf,
+file: File,
+blockSize: Long) extends BlockData {
+
+ private val minMemoryMapBytes =
conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
+
+ override def toInputStream(): InputStream = new FileInputStream(file)
+
+ /**
+ * Returns a Netty-friendly wrapper for the block's data.
+ *
+ * Please see `ManagedBuffer.convertToNetty()` for more details.
+ */
+ override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
+
+ override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer):
ChunkedByteBuffer = {
+Utils.tryWithResource(open()) { channel =>
+ var remaining = blockSize
+ val chunks = new ListBuffer[ByteBuffer]()
+ while (remaining > 0) {
+val chunkSize = math.min(remaining, Int.MaxValue)
+val chunk = allocator(chunkSize.toInt)
+remaining -= chunkSize
+JavaUtils.readFully(channel, chunk)
+chunk.flip()
+chunks += chunk
+ }
+ new ChunkedByteBuffer(chunks.toArray)
+}
+ }
+
+ override def toByteBuffer(): ByteBuffer = {
+require( size < Int.MaxValue
+ , s"can't create a byte buffer of size $blockSize"
++ s" since it exceeds Int.MaxValue ${Int.MaxValue}.")
+Utils.tryWithResource(open()) { channel =>
+ if (blockSize < minMemoryMapBytes) {
+// For small files, directly read rather than memory map.
+val buf = ByteBuffer.allocate(blockSize.toInt)
+JavaUtils.readFully(channel, buf)
+buf.flip()
+buf
+ } else {
+channel.map(MapMode.READ_ONLY, 0, file.length)
+ }
+}
+ }
+
+ override def size: Long = blockSize
+
+ override def dispose(): Unit = {}
+
+private def open() = new FileInputStream(file).getChannel
--- End diff --
will do :sunglasses:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131543345
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore(
}
+private class DiskBlockData(
+conf: SparkConf,
+file: File,
+blockSize: Long) extends BlockData {
+
+ private val minMemoryMapBytes =
conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
+
+ override def toInputStream(): InputStream = new FileInputStream(file)
+
+ /**
+ * Returns a Netty-friendly wrapper for the block's data.
+ *
+ * Please see `ManagedBuffer.convertToNetty()` for more details.
+ */
+ override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
+
+ override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer):
ChunkedByteBuffer = {
+Utils.tryWithResource(open()) { channel =>
+ var remaining = blockSize
+ val chunks = new ListBuffer[ByteBuffer]()
+ while (remaining > 0) {
+val chunkSize = math.min(remaining, Int.MaxValue)
+val chunk = allocator(chunkSize.toInt)
+remaining -= chunkSize
+JavaUtils.readFully(channel, chunk)
+chunk.flip()
+chunks += chunk
+ }
+ new ChunkedByteBuffer(chunks.toArray)
+}
+ }
+
+ override def toByteBuffer(): ByteBuffer = {
+require( size < Int.MaxValue
+ , s"can't create a byte buffer of size $blockSize"
++ s" since it exceeds Int.MaxValue ${Int.MaxValue}.")
+Utils.tryWithResource(open()) { channel =>
+ if (blockSize < minMemoryMapBytes) {
+// For small files, directly read rather than memory map.
+val buf = ByteBuffer.allocate(blockSize.toInt)
+JavaUtils.readFully(channel, buf)
+buf.flip()
+buf
+ } else {
+channel.map(MapMode.READ_ONLY, 0, file.length)
+ }
+}
+ }
+
+ override def size: Long = blockSize
+
+ override def dispose(): Unit = {}
+
+private def open() = new FileInputStream(file).getChannel
--- End diff --
nit: remove 2 spaces for better indent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [SPARK-3151][Block Manager] DiskStore.getBytes fa...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131543084
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore(
}
+private class DiskBlockData(
+conf: SparkConf,
+file: File,
+blockSize: Long) extends BlockData {
+
+ private val minMemoryMapBytes =
conf.getSizeAsBytes("spark.storage.memoryMapThreshold", "2m")
+
+ override def toInputStream(): InputStream = new FileInputStream(file)
+
+ /**
+ * Returns a Netty-friendly wrapper for the block's data.
+ *
+ * Please see `ManagedBuffer.convertToNetty()` for more details.
+ */
+ override def toNetty(): AnyRef = new DefaultFileRegion(file, 0, size)
+
+ override def toChunkedByteBuffer(allocator: (Int) => ByteBuffer):
ChunkedByteBuffer = {
+Utils.tryWithResource(open()) { channel =>
+ var remaining = blockSize
+ val chunks = new ListBuffer[ByteBuffer]()
+ while (remaining > 0) {
+val chunkSize = math.min(remaining, Int.MaxValue)
+val chunk = allocator(chunkSize.toInt)
+remaining -= chunkSize
+JavaUtils.readFully(channel, chunk)
+chunk.flip()
+chunks += chunk
+ }
+ new ChunkedByteBuffer(chunks.toArray)
+}
+ }
+
+ override def toByteBuffer(): ByteBuffer = {
+require( size < Int.MaxValue
--- End diff --
Is it better to check `blockSize`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18855: [Spark 3151][Block Manager] DiskStore.getBytes fa...
GitHub user eyalfa opened a pull request: https://github.com/apache/spark/pull/18855 [Spark 3151][Block Manager] DiskStore.getBytes fails for files larger than 2GB ## What changes were proposed in this pull request? introduced `DiskBlockData`, a new implementation of `BlockData` representing a whole file. this is somehow related to [SPARK-6236](https://issues.apache.org/jira/browse/SPARK-6236) as well This class follows the implementation of `EncryptedBlockData` just without the encryption. hence: * it uses FileOutputStream (todo: encrypted version actually uses `Channels.newInputStream`, not sure if it's the right choice for this) * `toNetty` is implemented in terms of `io.netty.channel.DefaultFileRegion#DefaultFileRegion` * `toByteBuffer` fails for files larger than 2GB (same behavior of the original code, just postponed a bit), it also respects the same configuration keys defined by the original code to choose between memory mapping and simple file read. (Please fill in changes proposed in this fix) ## How was this patch tested? added test to DiskStoreSuite and MemoryManagerSuite You can merge this pull request into a Git repository by running: $ git pull https://github.com/eyalfa/spark SPARK-3151 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18855.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18855 commit fc3f1d78e14a30dd2f71fc65ec59a2def5c1a0d4 Author: Eyal Farago Date: 2017-07-05T13:20:16Z SPARK-6235__take1: introduce a failing test. commit 84687380026a6a3bcded27be517094d3f690c3bb Author: Eyal Farago Date: 2017-07-30T20:06:05Z SPARK-6235__add_failing_tests: add failing tests for block manager suite. commit 15804497a477b8f97c08adfad5f0519504dc82f2 Author: Eyal Farago Date: 2017-08-01T17:34:26Z SPARK-6235__add_failing_tests: introduce a new BlockData implementation to represent a disk backed block data. commit c5028f50698c4fe48a06f5dd683dbee42f7e6b2b Author: Eyal Farago Date: 2017-08-05T19:57:41Z SPARK-6235__add_failing_tests: styling commit 908c7860688534d0bb77bcbebbd2e006a161fb74 Author: Eyal Farago Date: 2017-08-05T19:58:52Z SPARK-6235__add_failing_tests: adapt DiskStoreSuite to the modifications in the tested class. commit 67f4259ca16c3ca7c904c9ccc5de9acbc25d2271 Author: Eyal Farago Date: 2017-08-05T20:57:58Z SPARK-6235__add_failing_tests: try to reduce actual memory footprint of the >2gb tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
