Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-147069563
[Test build #1871 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1871/consoleFull)
for PR 8881 at commit
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-147042776
I mean you need to trigger the Jenkins test again by typing "Jenkins,
retest this please." in Github comment box.
---
If your project is set up for it, you can reply
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-147042790
Or let the committer trigger the test for you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user jhu-chang commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-147042883
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-147071966
[Test build #1871 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1871/console)
for PR 8881 at commit
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-147072175
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/8881
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-146796751
[Test build #1865 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1865/console)
for PR 8881 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-146803222
[Test build #1866 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1866/consoleFull)
for PR 8881 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-146807452
[Test build #1866 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1866/console)
for PR 8881 at commit
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-147026433
It might be the problem of Jenkins, you'd better run the unit test again.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user jhu-chang commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-147026285
Hi, @srowen , @tdas @jerryshao
Do you know what's the reason of this failure?
---
If your project is set up for it, you can reply to this email and have your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-146794620
[Test build #1865 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1865/consoleFull)
for PR 8881 at commit
Github user jhu-chang commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-147038105
On my local environment, the unit test is ok.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user jhu-chang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r41492611
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -17,6 +17,7 @@
package
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r41481193
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -17,6 +17,7 @@
package
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-146439085
LGTM except some small comments :).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r41481438
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -38,6 +39,11 @@ class TransformedDStream[U:
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r41492868
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -17,6 +17,7 @@
package
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r41495059
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -17,6 +17,7 @@
package
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r41584948
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -38,6 +39,11 @@ class TransformedDStream[U: ClassTag] (
Github user jhu-chang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r41594725
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -38,6 +39,11 @@ class TransformedDStream[U:
Github user jhu-chang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r41594692
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -17,6 +17,7 @@
package
Github user jhu-chang commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-146418682
Hi, @tdas, I have checked in the fix for those issues, could you check
again?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r41189182
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -38,6 +39,11 @@ class TransformedDStream[U: ClassTag] (
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-145650573
@jhu-chang Could you fix the style issue and one minor issue that I pointed
out.
Style issues:
```
[error]
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-145064047
@jhu-chang can you fix up the style problem that fails the build? looks
like whitespace at the end of the lines
---
If your project is set up for it, you can reply to
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-144707086
[Test build #1833 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1833/consoleFull)
for PR 8881 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-144707377
[Test build #1833 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1833/console)
for PR 8881 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-144015558
[Test build #1824 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1824/console)
for PR 8881 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-144015277
[Test build #1824 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1824/consoleFull)
for PR 8881 at commit
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-143656104
Lgtm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r40509300
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -38,6 +39,12 @@ class TransformedDStream[U: ClassTag] (
Github user jhu-chang commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-143628908
```count``` with ```None``` will report NullPointerException since
```None``` will transform to ```null``` in [```transform``` function]
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-143269767
I cannot remember very clearly, `count()` may introduce incorrect result
when you use `None` instead of `RDD.empty` as I remembered.
In the early version some
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142897772
@jhu-chang I still don't see why you say a user can write logic to return a
null RDD, but can't write an if statement to handle it? a user who can use
`DStream` is
Github user jhu-chang commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142889379
@srowen Yes, user can write his own logical to deal with the case mentioned
last post. But user may still try to use the exist API like
```saveAsXXXFiles``` first
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142854071
@jhu-chang you can write application logic that selectively performs
additional operations or not on an RDD depending on whether it's empty. It's
just an if statement.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-143049207
Hi @jhu-chang , can you elaborate at what scenario emptyRDD will introduce
undefined behavior, I'm afraid I could not clearly catch what you mentioned. If
emptyRDD
Github user jhu-chang commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-143121026
@jerryshao This is an extreme sample, it is not a real case, just to
demonstrate the issue of RDD.empty, you can see the dependency increasing all
the time and final
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142656310
Looking at it again, @srowen is right. Returning `None` makes
`getOrCompute` think that no RDDs have been generated for a given time
(artifact of the fact that
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142709106
Absolutely agreed. This is something I wish I had done it right 3 years ago
:(
---
If your project is set up for it, you can reply to this email and have your
reply
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142689929
You should not be returning in null in the first place from a transform
function. May be we need to document this better. At least, in the code, we
should check for
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142694171
I think we'd better document the return value of `compute` function, I saw
lots of users in the community they're trying to return `None` instead of
`RDD.empty` if
GitHub user jhu-chang opened a pull request:
https://github.com/apache/spark/pull/8881
[SPARK-10772][Streaming][Scala]: NullPointerException when transform
function in DStream returns NULL
Currently, the ```TransformedDStream``` will using
```Some(transformFunc(parentRDDs,
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142517073
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142520996
LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/8881#discussion_r40177949
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala
---
@@ -210,6 +210,18 @@ class BasicOperationsSuite extends
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142526656
I'm not sure this is valid. Looking at how the result of `transform()` is
used, `None` means "try computing this again". Why would you want or need to
return a `null`
Github user jhu-chang commented on the pull request:
https://github.com/apache/spark/pull/8881#issuecomment-142795137
There exists a case that user needs to skip the operation after the
```transform```, especially, the output operation like saveAsXXXFiles (e.g.
snapshot the state of
50 matches
Mail list logo
