[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11888 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-204083429 Merged into master thanks guys. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203772958 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203772960 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54588/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203772472 **[Test build #54588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54588/consoleFull)** for PR 11888 at commit [`39dd0ee`](https://github.com/apache/spark/commit/39dd0eed15d9c67604cb3430350c4f0f57c5f7e7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203744481 **[Test build #54588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54588/consoleFull)** for PR 11888 at commit [`39dd0ee`](https://github.com/apache/spark/commit/39dd0eed15d9c67604cb3430350c4f0f57c5f7e7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203744003 LGTM retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203739189 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54577/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203739188 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203739078 **[Test build #54577 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54577/consoleFull)** for PR 11888 at commit [`39dd0ee`](https://github.com/apache/spark/commit/39dd0eed15d9c67604cb3430350c4f0f57c5f7e7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203717852 **[Test build #54577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54577/consoleFull)** for PR 11888 at commit [`39dd0ee`](https://github.com/apache/spark/commit/39dd0eed15d9c67604cb3430350c4f0f57c5f7e7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203675470 OK, @rxin and I discussed this more offline. Our proposal is: ``` def getExecutorInfos: Seq[SparkExecutorInfo] private class SparkExecutorInfoImpl( val host: String, val port: Int, ... val cacheSize: Long, val numRunningTasks: Int) extends SparkExecutorInfo ``` Then we don't need to tie us down with the very specific `host:port` format, and we don't have to expose the executor IDs, which are just integers that don't mean much. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203670885 Yea that integer id is completely useless to users who want to figure out what to do with their clusters. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203665311 https://cloud.githubusercontent.com/assets/2133137/14159586/68f1d598-f68b-11e5-9e1d-0aa4b34a3910.png;> https://cloud.githubusercontent.com/assets/2133137/14159589/6ca0057a-f68b-11e5-86b9-3dbf92aa713f.png;> ``` scala> df.count() 16/03/30 15:21:58 INFO SparkContext: Starting job: count at :27 ... 16/03/30 15:21:58 INFO SparkDeploySchedulerBackend: Launching task 7 on executor id: 3 hostname: 192.168.0.209. 16/03/30 15:21:58 INFO SparkDeploySchedulerBackend: Launching task 8 on executor id: 0 hostname: 192.168.0.209. 16/03/30 15:21:58 INFO SparkDeploySchedulerBackend: Launching task 9 on executor id: 5 hostname: 192.168.0.209. 16/03/30 15:21:58 INFO SparkDeploySchedulerBackend: Launching task 10 on executor id: 1 hostname: 192.168.0.209. 16/03/30 15:21:58 INFO SparkDeploySchedulerBackend: Launching task 11 on executor id: 4 hostname: 192.168.0.209. 16/03/30 15:21:58 INFO SparkDeploySchedulerBackend: Launching task 12 on executor id: 2 hostname: 192.168.0.209. ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203658850 Can you paste me what an executor looks like? If you just tell me "executor id" as an end user, I have no clue what you are talking about. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203658485 What are executor ids? is that even an external concept? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203559116 My proposal: ``` def getExecutorIds(): Array[String] def getExecutorInfo(executorId: String): Option[SparkExecutorInfo] private class SparkExecutorInfoImpl( val id: String, val hostport: String, // host:port ... val cacheSize: Long, val numRunningTasks: Int) extends SparkExecutorInfo ``` This is more consistent with the existing status API, where we have things like ``` def getActiveJobIds(): Array[Int] def getJobInfo(jobId: Int): Option[SparkJobInfo] ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203555770 @rxin I don't understand the distinction between `getExecutors` and `getExecutorList`. Why not just be more explicit about what the strings represent? The other problem is the keys to the other maps are also expected to be `host:port` but I would normally expect them to be executor IDs. Also host can be confusing because it can be in one of many different formats (e.g. IP addr, all the hostname formats in EC2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/11888#discussion_r57933650
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusTracker.scala ---
@@ -104,4 +106,34 @@ class SparkStatusTracker private[spark] (sc:
SparkContext) {
}
}
}
+
+ /**
+ * Returns a list of all known executors, represented by string with
format: "host:port"
+ */
+ def getExecutors(): Array[String] = {
--- End diff --
actually let me move this to the main thread so it doesn't get collapsed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/11888#discussion_r57933508
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusTracker.scala ---
@@ -104,4 +106,34 @@ class SparkStatusTracker private[spark] (sc:
SparkContext) {
}
}
}
+
+ /**
+ * Returns a list of all known executors, represented by string with
format: "host:port"
+ */
+ def getExecutors(): Array[String] = {
--- End diff --
wait, how is `getExecutorList` different from `getExecutors`? Why not just
be more specific what the strings are?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203538706 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54526/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203538700 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203537841 **[Test build #54526 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54526/consoleFull)** for PR 11888 at commit [`0b9400e`](https://github.com/apache/spark/commit/0b9400e2a62f6a723be79264dfeddf947755f3d0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-203484294 **[Test build #54526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54526/consoleFull)** for PR 11888 at commit [`0b9400e`](https://github.com/apache/spark/commit/0b9400e2a62f6a723be79264dfeddf947755f3d0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11888#discussion_r57655794
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusTracker.scala ---
@@ -104,4 +106,34 @@ class SparkStatusTracker private[spark] (sc:
SparkContext) {
}
}
}
+
+ /**
+ * Returns a list of all known executors, represented by string with
format: "host:port"
+ */
+ def getExecutors(): Array[String] = {
--- End diff --
`getExecutorList` LGTM, I'll rename to it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11888#discussion_r57654956
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusTracker.scala ---
@@ -104,4 +106,34 @@ class SparkStatusTracker private[spark] (sc:
SparkContext) {
}
}
}
+
+ /**
+ * Returns a list of all known executors, represented by string with
format: "host:port"
+ */
+ def getExecutors(): Array[String] = {
--- End diff --
getExecutorList
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/11888#discussion_r57654799
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusTracker.scala ---
@@ -104,4 +106,34 @@ class SparkStatusTracker private[spark] (sc:
SparkContext) {
}
}
}
+
+ /**
+ * Returns a list of all known executors, represented by string with
format: "host:port"
+ */
+ def getExecutors(): Array[String] = {
--- End diff --
we can expose both, but I'd rather call it something more explicit like
`getExecutorHostPort` or something. Elsewhere in Spark I would think
`getExecutors: Array[String]` returns the executor IDs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11888#discussion_r57654386
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusTracker.scala ---
@@ -104,4 +106,34 @@ class SparkStatusTracker private[spark] (sc:
SparkContext) {
}
}
}
+
+ /**
+ * Returns a list of all known executors, represented by string with
format: "host:port"
+ */
+ def getExecutors(): Array[String] = {
--- End diff --
The other thing is I don't know if we want to query the scheduler every
time we want a list of executors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11888#discussion_r57654356
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusTracker.scala ---
@@ -104,4 +106,34 @@ class SparkStatusTracker private[spark] (sc:
SparkContext) {
}
}
}
+
+ /**
+ * Returns a list of all known executors, represented by string with
format: "host:port"
+ */
+ def getExecutors(): Array[String] = {
--- End diff --
If I just want a list of executors, why shouldn't I be able to get them? I
think it makes sense to have a more detailed version (maybe replace the
following 2), but having a simple one that returns just the list of executors
seem to make sense too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/11888#discussion_r57653397
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusTracker.scala ---
@@ -104,4 +106,34 @@ class SparkStatusTracker private[spark] (sc:
SparkContext) {
}
}
}
+
+ /**
+ * Returns a list of all known executors, represented by string with
format: "host:port"
+ */
+ def getExecutors(): Array[String] = {
--- End diff --
It seems kind of arbitrary that `getExecutors` returns `host:port` but not
IDs. I think it's better that we make a `SparkExecutorInfo` or something and
expose the `host:port` there, along with other things like cache size,
`numRunningTasks` etc. Then in the future we can add more things we want to
expose without tying ourselves with the `host:port` identifier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200324144 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53925/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200324143 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200323936 **[Test build #53925 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53925/consoleFull)** for PR 11888 at commit [`fe80390`](https://github.com/apache/spark/commit/fe8039013efd2e4504168dff264f725fb56665a9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200276711 **[Test build #53925 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53925/consoleFull)** for PR 11888 at commit [`fe80390`](https://github.com/apache/spark/commit/fe8039013efd2e4504168dff264f725fb56665a9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200276074 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200273101 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200273071 **[Test build #53921 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53921/consoleFull)** for PR 11888 at commit [`fe80390`](https://github.com/apache/spark/commit/fe8039013efd2e4504168dff264f725fb56665a9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200273102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53921/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200269875 **[Test build #53921 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53921/consoleFull)** for PR 11888 at commit [`fe80390`](https://github.com/apache/spark/commit/fe8039013efd2e4504168dff264f725fb56665a9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200268796 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200243417 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53904/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200243412 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200242940 **[Test build #53904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53904/consoleFull)** for PR 11888 at commit [`fe80390`](https://github.com/apache/spark/commit/fe8039013efd2e4504168dff264f725fb56665a9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200207584 **[Test build #53904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53904/consoleFull)** for PR 11888 at commit [`fe80390`](https://github.com/apache/spark/commit/fe8039013efd2e4504168dff264f725fb56665a9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200206025 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200196539 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53884/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200196538 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200196073 **[Test build #53884 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53884/consoleFull)** for PR 11888 at commit [`fe80390`](https://github.com/apache/spark/commit/fe8039013efd2e4504168dff264f725fb56665a9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200162987 It's probably ok. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200160851 I found it's difficulty to write tests for it. As it just collects the informations which are already exposed by the system, is it worth to test them again? cc @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14069][SQL] Improve SparkStatusTracker ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11888#issuecomment-200159464 **[Test build #53884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53884/consoleFull)** for PR 11888 at commit [`fe80390`](https://github.com/apache/spark/commit/fe8039013efd2e4504168dff264f725fb56665a9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
