Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2760#issuecomment-59848896
Reviewed again after the fact. LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19118588
--- Diff:
core/src/test/scala/org/apache/spark/broadcast/BroadcastSuite.scala ---
@@ -17,13 +17,18 @@
package org.apache.spark.broadcast
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19118825
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -227,6 +217,7 @@ private object TorrentBroadcast extends Logging
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19119141
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -104,29 +112,23 @@ private[spark] class TorrentBroadcast[T: ClassTag
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19119178
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -104,29 +112,23 @@ private[spark] class TorrentBroadcast[T: ClassTag
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19119195
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -62,6 +59,20 @@ private[spark] class TorrentBroadcast[T: ClassTag
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2866#issuecomment-59882025
Oh wow. Thanks for fixing this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2866#issuecomment-59882089
Actually instead of introducing a new one, what if we introduce a
compressed bitmap that tracks zero-sized blocks, and then use avg size to track
only non-zero blocks
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19130571
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -227,6 +217,7 @@ private object TorrentBroadcast extends Logging
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19130611
--- Diff:
core/src/test/scala/org/apache/spark/broadcast/BroadcastSuite.scala ---
@@ -84,6 +89,24 @@ class BroadcastSuite extends FunSuite with
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2866#issuecomment-59966852
Compressed bitmaps are in general just variants of run-length encoding that
is lossless. Which should be able to handle your case too.
---
If your project is set up for it
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1567#issuecomment-59967008
A short design doc would be nice. Just talk about the high level design and
how it is implemented. Thanks.
---
If your project is set up for it, you can reply to this
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2866#discussion_r19187905
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/MapStatusSuite.scala ---
@@ -46,6 +47,27 @@ class MapStatusSuite extends FunSuite
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2866#discussion_r19188111
--- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ---
@@ -112,35 +119,70 @@ private[spark] class CompressedMapStatus
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2866#discussion_r19188129
--- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ---
@@ -112,35 +119,70 @@ private[spark] class CompressedMapStatus
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2871#discussion_r19188778
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -63,12 +63,22 @@ private[spark] class TorrentBroadcast[T: ClassTag
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2871#discussion_r19197492
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -63,12 +63,22 @@ private[spark] class TorrentBroadcast[T: ClassTag
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2851#issuecomment-60048094
Can't we just have a general block reporting mechanism rather than special
casing for broadcast?
---
If your project is set up for it, you can reply to this emai
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2879#issuecomment-60048272
Merging in master. thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2866#issuecomment-60201281
@JoshRosen thanks for doing this. There is a chance that a normal hashset
is much slower than a bitmap. Can you test that? It might make a lot more sense
to use an
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2866#discussion_r19299903
--- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ---
@@ -112,35 +119,82 @@ private[spark] class CompressedMapStatus
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2911#issuecomment-60291186
LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19300117
--- Diff: core/src/main/scala/org/apache/spark/ui/UITables.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19300207
--- Diff: core/src/main/scala/org/apache/spark/ui/UITables.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19300234
--- Diff: core/src/main/scala/org/apache/spark/ui/UITables.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19300540
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala ---
@@ -21,12 +21,28 @@ import javax.servlet.http.HttpServletRequest
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19300512
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala ---
@@ -21,12 +21,28 @@ import javax.servlet.http.HttpServletRequest
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19300818
--- Diff: core/src/main/scala/org/apache/spark/ui/UITables.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19300885
--- Diff: core/src/main/scala/org/apache/spark/ui/storage/RDDPage.scala ---
@@ -19,16 +19,47 @@ package org.apache.spark.ui.storage
import
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19301013
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala ---
@@ -75,37 +89,45 @@ private[ui] class StreamingPage(parent
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2852#issuecomment-60294242
Looks pretty cool overall. I made some comments, mostly around clarity of
the code.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2851#issuecomment-60314638
Hi @CodingCat - This is a great & handy feature, but what I meant was that
we should NOT have custom code that tracks broadcast blocks. We can have
special UIs
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2913#discussion_r19310826
--- Diff:
core/src/main/scala/org/apache/spark/util/SparkUncaughtExceptionHandler.scala
---
@@ -15,17 +15,17 @@
* limitations under the License
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2913#discussion_r19310864
--- Diff:
core/src/main/scala/org/apache/spark/util/SparkUncaughtExceptionHandler.scala
---
@@ -15,17 +15,17 @@
* limitations under the License
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2866#issuecomment-60321093
LGTM!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2929#discussion_r19363491
--- Diff:
core/src/main/scala/org/apache/spark/network/nio/NioBlockTransferService.scala
---
@@ -95,16 +95,20 @@ final class NioBlockTransferService(conf
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2929#discussion_r19363531
--- Diff:
core/src/main/scala/org/apache/spark/network/nio/NioBlockTransferService.scala
---
@@ -95,16 +95,20 @@ final class NioBlockTransferService(conf
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19364501
--- Diff:
network/common/src/test/java/org/apache/spark/network/ProtocolSuite.java ---
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19364810
--- Diff:
network/common/src/main/java/org/apache/spark/network/util/SluiceConfig.java ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19364870
--- Diff:
network/common/src/main/java/org/apache/spark/network/util/DefaultConfigProvider.java
---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19383243
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockFetcher.scala ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19383250
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockFetcher.scala ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19383261
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockFetcher.scala ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19383297
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockFetcher.scala ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19383298
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockFetcher.scala ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19427924
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockFetcher.scala ---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19428104
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockRpcServer.scala ---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19437262
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/FileShuffleBlockManager.scala ---
@@ -24,14 +24,14 @@ import java.util.concurrent.atomic.AtomicInteger
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19437318
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -19,14 +19,13 @@ package org.apache.spark.storage
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19437966
--- Diff:
network/common/src/main/java/org/apache/spark/network/buffer/FileSegmentManagedBuffer.java
---
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19438048
--- Diff:
network/common/src/main/java/org/apache/spark/network/buffer/ManagedBuffer.java
---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19438149
--- Diff:
network/common/src/main/java/org/apache/spark/network/buffer/ManagedBuffer.java
---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19438223
--- Diff:
network/common/src/main/java/org/apache/spark/network/buffer/NettyManagedBuffer.java
---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19438500
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/ChunkFetchFailureException.java
---
@@ -15,18 +15,23 @@
* limitations under the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19438953
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClient.java
---
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19439010
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClient.java
---
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19439584
--- Diff:
network/common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19439894
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClient.java
---
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19439891
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClient.java
---
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19440017
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19440090
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19440233
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19440311
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19440361
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19440492
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportResponseHandler.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19440622
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportResponseHandler.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19440798
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportResponseHandler.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19440779
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportResponseHandler.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19440961
--- Diff:
network/common/src/main/java/org/apache/spark/network/protocol/Message.java ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19441087
--- Diff:
network/common/src/main/java/org/apache/spark/network/protocol/request/RequestMessage.java
---
@@ -15,11 +15,11 @@
* limitations under the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19441170
--- Diff:
network/common/src/main/java/org/apache/spark/network/protocol/response/ChunkFetchSuccess.java
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19441656
--- Diff:
network/common/src/main/java/org/apache/spark/network/protocol/Encodable.java
---
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19441669
--- Diff:
network/common/src/main/java/org/apache/spark/network/protocol/Encodable.java
---
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19441768
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/DefaultStreamManager.java
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19441890
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/DefaultStreamManager.java
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19441849
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/DefaultStreamManager.java
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19442016
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/DefaultStreamManager.java
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19442525
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/TransportClientHandler.java
---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19442563
--- Diff:
network/common/src/main/java/org/apache/spark/network/protocol/response/MessageDecoder.java
---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19452657
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java
---
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19452665
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java
---
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19452698
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/TransportServer.java
---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2753#discussion_r19452740
--- Diff:
network/common/src/main/java/org/apache/spark/network/util/SystemPropertyConfigProvider.java
---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2753#issuecomment-60708006
@aarondav I'm done with the pass. Let me know when you address the comment
and I will merge it (once Jenkins is happy).
---
If your project is set up for it, yo
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2969#discussion_r19454231
--- Diff: core/src/main/scala/org/apache/spark/network/ManagedBuffer.scala
---
@@ -106,7 +112,7 @@ final class FileSegmentManagedBuffer(val file: File,
val
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2969#discussion_r19454325
--- Diff: core/src/main/scala/org/apache/spark/storage/TachyonStore.scala
---
@@ -105,25 +106,17 @@ private[spark] class TachyonStore(
return None
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2953#issuecomment-60713308
Can you write a short high level design doc for this change and attach it
to JIRA?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2942#discussion_r19454416
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2942#discussion_r19454435
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454510
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454502
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454508
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454515
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454523
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454541
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454543
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454568
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454583
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454599
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala
---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2931#discussion_r19454635
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala
---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software
1 - 100 of 15085 matches
Mail list logo
