Repository: spark
Updated Branches:
refs/heads/branch-1.2 c4abb2eb4 -> a250ca369
[SPARK-4294][Streaming] UnionDStream stream should express the requirements in
the same way as TransformedDStream
In class TransformedDStream:
```scala
require(parents.length > 0, "List of DStreams to transform is empty")
require(parents.map(.ssc).distinct.size == 1, "Some of the DStreams have
different contexts")
require(parents.map(.slideDuration).distinct.size == 1,
"Some of the DStreams have different slide durations")
```
In class UnionDStream:
```scala
if (parents.length == 0)
{ throw new IllegalArgumentException("Empty array of parents") }
if (parents.map(.ssc).distinct.size > 1)
{ throw new IllegalArgumentException("Array of parents have different
StreamingContexts") }
if (parents.map(.slideDuration).distinct.size > 1)
{ throw new IllegalArgumentException("Array of parents have different slide
times") }
```
The function is the same, but the realization is not. I think they shoule be
the same.
Author: Yadong Qi <[email protected]>
Closes #3152 from watermen/bug-fix1 and squashes the following commits:
ed66db6 [Yadong Qi] Change transform to union
b6b3b8b [Yadong Qi] The same function should have the same realization.
(cherry picked from commit c3002c4a61c4fc5b966aa384c41c3cba33de0aa6)
Signed-off-by: Tathagata Das <[email protected]>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a250ca36
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a250ca36
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a250ca36
Branch: refs/heads/branch-1.2
Commit: a250ca369208b23503d7fff1cf9ee52e2e1ba3e2
Parents: c4abb2e
Author: Yadong Qi <[email protected]>
Authored: Wed Nov 19 15:53:06 2014 -0800
Committer: Tathagata Das <[email protected]>
Committed: Wed Nov 19 15:53:19 2014 -0800
----------------------------------------------------------------------
.../spark/streaming/dstream/UnionDStream.scala | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/a250ca36/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala
----------------------------------------------------------------------
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala
b/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala
index 57429a1..abbc40b 100644
---
a/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala
+++
b/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala
@@ -28,17 +28,10 @@ private[streaming]
class UnionDStream[T: ClassTag](parents: Array[DStream[T]])
extends DStream[T](parents.head.ssc) {
- if (parents.length == 0) {
- throw new IllegalArgumentException("Empty array of parents")
- }
-
- if (parents.map(_.ssc).distinct.size > 1) {
- throw new IllegalArgumentException("Array of parents have different
StreamingContexts")
- }
-
- if (parents.map(_.slideDuration).distinct.size > 1) {
- throw new IllegalArgumentException("Array of parents have different slide
times")
- }
+ require(parents.length > 0, "List of DStreams to union is empty")
+ require(parents.map(_.ssc).distinct.size == 1, "Some of the DStreams have
different contexts")
+ require(parents.map(_.slideDuration).distinct.size == 1,
+ "Some of the DStreams have different slide durations")
override def dependencies = parents.toList
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
