Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-63128807
Yeah, alright. Could you close this PR then?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user watermen closed the pull request at:
https://github.com/apache/spark/pull/2216
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user watermen opened a pull request:
https://github.com/apache/spark/pull/3237
[SPARK-3325] Add a parameter to the method print in class DStream.
I have deleted the repository with mistake, so I create a new PR. old
PR:https://github.com/apache/spark/pull/2216
def
Github user watermen commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-62833219
I have deleted the repository with mistake, so I create a new PR. new
PR:https://github.com/apache/spark/pull/3237
---
If your project is set up for it, you can reply
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-62534611
@watermen This PR hasnt been updated with the change @srowen suggested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r18627949
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStream.scala
---
@@ -94,6 +94,14 @@ class JavaDStream[T](val dstream:
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-58460160
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21515/consoleFull)
for PR 2216 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-58460408
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21515/consoleFull)
for PR 2216 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-58460410
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-58461058
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r18626981
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStream.scala
---
@@ -94,6 +94,14 @@ class JavaDStream[T](val dstream:
Github user watermen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r18627758
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStream.scala
---
@@ -94,6 +94,14 @@ class JavaDStream[T](val dstream:
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-57060715
Ah I see - I thought this was an abstract class instead of a trait being
modified in this patch.
This is not an error with the compatibility checker - it's a
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-57060863
Yeah so I looked into it a bit more and since `JavaDStream` extends
`JavaDStreamLike` this will break user code with custom DStream's. The issue is
that under the hood
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-57074623
I really wish that we could convert JavaDStreamLike / JavaRDDLike into
abstract base classes instead of traits, since there's no particular reason why
they should be
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-57008381
That should not be the reason. Logically speaking, that makes sense only if
we add a method that does not have a default implementation. Now whether Mima
tool is reasoning
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-56777443
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-56777484
LGTM too - this is a good idea!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-56777609
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20782/consoleFull)
for PR 2216 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-56782577
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-56782572
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20782/consoleFull)
for PR 2216 at commit
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-56787307
MIMA is complaining because a method is added to trait `JavaDStreamLike`. I
think it should just be suppressed, as there's no guarantee to callers that
this trait won't
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-56855642
@scrapcodes Can you take a look at this. Adding new methods to
JavaDStreamLike should not break binary compatibility.
---
If your project is set up for it, you can reply
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-56903783
@tdas Maybe we're seeing an error here because JavaDStreamLike compiles
down to a Java interface and adding new methods to a public interface can break
compatibility
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-56577585
Jenkins, test this again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-55364803
Binary compatibility is still the problem. I think adding optional
parameter to a scala function breaks binary compatibility. I suggest do the
same in Scala API as you have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-55375789
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20220/consoleFull)
for PR 2216 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-55381569
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20220/consoleFull)
for PR 2216 at commit
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-55354531
LGTM, if binary compatibility tests pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-55355509
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20198/consoleFull)
for PR 2216 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-55358417
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20198/consoleFull)
for PR 2216 at commit
Github user watermen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r17344850
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStreamLike.scala
---
@@ -50,8 +50,8 @@ trait JavaDStreamLike[T, This :
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r17348838
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStreamLike.scala
---
@@ -50,8 +50,8 @@ trait JavaDStreamLike[T, This :
Github user watermen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r1735
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStreamLike.scala
---
@@ -50,8 +50,8 @@ trait JavaDStreamLike[T, This :
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-54682610
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r17198878
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStreamLike.scala
---
@@ -50,8 +50,8 @@ trait JavaDStreamLike[T, This :
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-54683003
@watermen @srowen Should definitely add to both API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-54694366
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user watermen commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-54411613
@tdas When i use the function of updateStateByKey, and the number of key
are more than 10. I want to watch all values of keys to make sure the code is
right.
---
If
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-54414169
Ahh okay. As @ScrapCodes said, this can be done pretty easily with
foreachRDD, but I can see how this makes it a tad bit convenient. I am okay to
add this. However, please
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-54431716
@watermen @tdas PS do you think this can be added to the Java API too? it
needs to be a new additional method not a change to the existing one. Worth the
extra method,
Github user watermen commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-54577266
@srowen it doesn't need to add an existing method, see my [files changed].
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-54375823
Jenkins, this is ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-54376099
@watermen Can you explain the usecase under which this is a convenient
addition? I dont mind adding as long as i understand its usefulness.
---
If your project is set up
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r16942761
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -603,14 +603,14 @@ abstract class DStream[T: ClassTag] (
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-54028823
This change is okay to have, but to print N elements from Dstream you can
do something like dstream.foreachRDD(rdd = println(rdd.take(N).mkString)). I
will let
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r16946191
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -603,14 +603,14 @@ abstract class DStream[T: ClassTag] (
*
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r16946373
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -603,14 +603,14 @@ abstract class DStream[T: ClassTag] (
GitHub user watermen opened a pull request:
https://github.com/apache/spark/pull/2216
[SPARK-3325] Add a parameter to the method print in class DStream.
def print(num: Int = 10)
User can control the number of elements which to print.
You can merge this pull request into a Git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2216#issuecomment-53954443
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
50 matches
Mail list logo
