[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user lirui-intel commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58739690 Looks great! I think it's very useful to have these async APIs in java :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58740225 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21628/consoleFull) for PR 2760 at commit [`ff28e49`](https://github.com/apache/spark/commit/ff28e49d990577635fa148bd57461a387bd3466d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class JavaFutureActionWrapper[S, T](futureAction: FutureAction[S], converter: S = T)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58740227 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21628/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58740594 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21630/consoleFull) for PR 2760 at commit [`6f8f6ac`](https://github.com/apache/spark/commit/6f8f6ac668d74a3164bcf037f09c8353134b53f6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class JavaFutureActionWrapper[S, T](futureAction: FutureAction[S], converter: S = T)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58740597 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21630/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/2760 [SPARK-3902] Stabilize AsynRDDActions and add Java API This PR adds a Java API for AsyncRDDActions and promotes the API from `@Experimental` to stable. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JoshRosen/spark async-rdd-actions-in-java Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2760.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2760 commit 346e46ed8789ab72c709bec40c728568fd7294e5 Author: Josh Rosen joshro...@apache.org Date: 2014-10-11T02:16:49Z [SPARK-3902] Stabilize AsyncRDDActions; add Java API. commit ff28e49d990577635fa148bd57461a387bd3466d Author: Josh Rosen joshro...@apache.org Date: 2014-10-11T05:32:57Z Add MiMa excludes and fix a scalastyle error. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58738999 /cc's for review: - @rxin, who wrote the original AsyncRDDActions - @lirui-intel, who added an experimental Java API for `foreachAsync` in #2176, and - @vanzin, who added the `jobIds` method to expose job ids from FutureAction in #2337. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2760#discussion_r18739587
--- Diff:
core/src/main/java/org/apache/spark/api/java/JavaFutureAction.java ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.api.java;
+
+
+import java.util.List;
+import java.util.concurrent.Future;
+
+public interface JavaFutureActionT extends FutureT {
--- End diff --
I think that it makes sense to expose an extended version of the Java
`Future` API to users, since there may be a number of existing libraries for
consuming these standard future types.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2760#discussion_r18739588
--- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala ---
@@ -70,6 +70,11 @@ trait FutureAction[T] extends Future[T] {
override def isCompleted: Boolean
/**
+ * Returns whether the action has been cancelled.
+ */
+ def isCancelled: Boolean
--- End diff --
This method is new; I addd it to try to maintain feature parity between the
Java and Scala futures.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58739062 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21628/consoleFull) for PR 2760 at commit [`ff28e49`](https://github.com/apache/spark/commit/ff28e49d990577635fa148bd57461a387bd3466d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2760#discussion_r18739598
--- Diff: core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala
---
@@ -575,16 +575,49 @@ trait JavaRDDLike[T, This : JavaRDDLike[T, This]]
extends Serializable {
def name(): String = rdd.name
/**
- * :: Experimental ::
- * The asynchronous version of the foreach action.
- *
- * @param f the function to apply to all the elements of the RDD
- * @return a FutureAction for the action
+ * The asynchronous version of `count`, which returns a
+ * future for counting the number of elements in this RDD.
*/
- @Experimental
- def foreachAsync(f: VoidFunction[T]): FutureAction[Unit] = {
--- End diff --
Unfortunately, my PR breaks compatibility for this experimental Java API.
However, the previous version of this method hasn't been shipped in any Spark
releases yet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2760#discussion_r18739601
--- Diff: core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala
---
@@ -575,16 +575,49 @@ trait JavaRDDLike[T, This : JavaRDDLike[T, This]]
extends Serializable {
def name(): String = rdd.name
/**
- * :: Experimental ::
- * The asynchronous version of the foreach action.
- *
- * @param f the function to apply to all the elements of the RDD
- * @return a FutureAction for the action
+ * The asynchronous version of `count`, which returns a
+ * future for counting the number of elements in this RDD.
*/
- @Experimental
- def foreachAsync(f: VoidFunction[T]): FutureAction[Unit] = {
--- End diff --
yea i think this is fine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2760#discussion_r18739624 --- Diff: core/src/test/java/org/apache/spark/JavaAPISuite.java --- @@ -20,7 +20,9 @@ import java.io.*; import java.net.URI; import java.util.*; +import java.util.concurrent.*; +import org.apache.spark.api.java.*; --- End diff -- Whoops, IntelliJ messed up the import ordering :(. I'll fix this now so that it doesn't have to be addressed later once we add import-order style-checking. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58739341 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21630/consoleFull) for PR 2760 at commit [`6f8f6ac`](https://github.com/apache/spark/commit/6f8f6ac668d74a3164bcf037f09c8353134b53f6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
