[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 Thanx @vanzin! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @vanzin didn't mean to press or anything :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22381 You guys need to be a little patient... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @vanzin ready to merge. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97415/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #97415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97415/testReport)** for PR 22381 at commit [`a08836c`](https://github.com/apache/spark/commit/a08836cc9ca91164cf5dd95a63929d0613779cf2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4005/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #97415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97415/testReport)** for PR 22381 at commit [`a08836c`](https://github.com/apache/spark/commit/a08836cc9ca91164cf5dd95a63929d0613779cf2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @vanzin thanx for the review I think It looks ok now. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @vanzin could I get a merge pls? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97324/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #97324 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97324/testReport)** for PR 22381 at commit [`86bf7ec`](https://github.com/apache/spark/commit/86bf7ec4bbe1bca268f85464b73aa0cdcb0bf163). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3931/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #97324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97324/testReport)** for PR 22381 at commit [`86bf7ec`](https://github.com/apache/spark/commit/86bf7ec4bbe1bca268f85464b73aa0cdcb0bf163). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 Thanx a lot @aditanase ! I will update the PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user aditanase commented on the issue: https://github.com/apache/spark/pull/22381 @skonto @vanzin You can rest assured the metrics are not shipped anywhere. Collection is cheap/free and completely decoupled from reporting. The registry is only a data bag, shipping metrics is done via reporters that are configured to "pull" metrics from the registry at regular intervals. Also, you're mostly adding counters that are extremely simple. Histograms and timers are more involved but the cost associated is far outweighed by the benefits of having data points. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @vanzin AFAIK the operations just update metrics in the underlying dropwizard metrics library. I dont htink anything is shipped anywhere. How should I proceed? Is there anyone who is familiar with the internals? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22381 I don't really have an opinion about the metrics or names or things like that; but not being familiar with the internals of this metric system, I'd just like to make sure that updating these metrics is cheap and won't do things like actually try to send data to the sinks, which would be a no-no in this particular listener. Also, if the metrics system can be disabled (isn't it disabled by default?), I'd avoid even updating these metrics in the listener at all. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96215/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #96215 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96215/testReport)** for PR 22381 at commit [`3a2db16`](https://github.com/apache/spark/commit/3a2db16813a2bab3160f444fe0855855187a6178). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3211/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #96215 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96215/testReport)** for PR 22381 at commit [`3a2db16`](https://github.com/apache/spark/commit/3a2db16813a2bab3160f444fe0855855187a6178). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/22381 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/22381 UT fixed by #22452. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22381 Looks like different failures. I am not sure if these are known issues. I'd try again later today. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @srowen integration tests seem so flaky! Seen this before? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96159/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #96159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96159/testReport)** for PR 22381 at commit [`3a2db16`](https://github.com/apache/spark/commit/3a2db16813a2bab3160f444fe0855855187a6178). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #96159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96159/testReport)** for PR 22381 at commit [`3a2db16`](https://github.com/apache/spark/commit/3a2db16813a2bab3160f444fe0855855187a6178). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3173/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 jenkins, test this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 test failure is irrelevant. jenkins, test this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96153/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #96153 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96153/testReport)** for PR 22381 at commit [`3a2db16`](https://github.com/apache/spark/commit/3a2db16813a2bab3160f444fe0855855187a6178). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3169/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #96153 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96153/testReport)** for PR 22381 at commit [`3a2db16`](https://github.com/apache/spark/commit/3a2db16813a2bab3160f444fe0855855187a6178). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @erikerlandson I added an option to disable them, by default they are now disabled. Is it possible to move forward with this? @srowen ? Btw I removed any code triggered by this unless explicitly enabled. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #96151 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96151/testReport)** for PR 22381 at commit [`677a7c3`](https://github.com/apache/spark/commit/677a7c3da10ae7c241a1b8fc5cb9a9768109c6b1). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96151/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3167/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #96151 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96151/testReport)** for PR 22381 at commit [`677a7c3`](https://github.com/apache/spark/commit/677a7c3da10ae7c241a1b8fc5cb9a9768109c6b1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96076/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #96076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96076/testReport)** for PR 22381 at commit [`635f5aa`](https://github.com/apache/spark/commit/635f5aae4c91890c62b18edcf2b71400861cc463). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3115/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #96076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96076/testReport)** for PR 22381 at commit [`635f5aa`](https://github.com/apache/spark/commit/635f5aae4c91890c62b18edcf2b71400861cc463). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @erikerlandson I dont any tests for sources AFAIK, they exist for the metric system itself. From the commit history this look like an old feature. Regarding enabling or disabling them no at the moment. But it is fairly easy to do via spark conf as there is one flag for structured streaming already `spark.sql.streaming.metricsEnabled`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22381 These new metrics seem useful. Is there a way to provide unit or integration testing for it? Do these have enable/disable via metrics.properties files? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 ok will do. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22381 Yeah, this is bigger than adding one metric; raise it on dev@? I'm not sure who if anyone has an opinion on it. I'd rather hear more feedback on the history of these choices and what the right future state is. What's the use case, etc --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @srowen I was checking git history and I see that you merged this one: https://github.com/apache/spark/pull/22218 again related to metrics. Could you call someone who can do the merge? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @srowen thanks is there someone I could call? @vanzin ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22381 I don't think I know enough about monitoring to merge this. I am concerned about the duplication here, although I understand your argument why it's valuable. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 @srowen @xuanyuanking ready for another round. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95905/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #95905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95905/testReport)** for PR 22381 at commit [`a8fc89d`](https://github.com/apache/spark/commit/a8fc89d51971e16c37783cf336daa07d18e6d3c1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2991/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #95905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95905/testReport)** for PR 22381 at commit [`a8fc89d`](https://github.com/apache/spark/commit/a8fc89d51971e16c37783cf336daa07d18e6d3c1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95881/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95880/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #95881 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95881/testReport)** for PR 22381 at commit [`b43190d`](https://github.com/apache/spark/commit/b43190d5393217f751f38149839a060df88baf1f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #95880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95880/testReport)** for PR 22381 at commit [`b43190d`](https://github.com/apache/spark/commit/b43190d5393217f751f38149839a060df88baf1f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user dbrinegar commented on the issue: https://github.com/apache/spark/pull/22381 This is quite a nice improvement for monitoring. The rest api is great for UI and consolidated analytics, but monitoring through it is not as straightforward as when the data emits directly from the source like this. There is all kinds of nice context that we get when the data from this spark node is collected directly from the node itself, and not proxied through another collector / reporter. It is easier to build a monitoring data model across the cluster when node, jmx, pod, resource manifests, and spark data all align by virtue of coming from the same collector. Building a similar view of the cluster just from the rest api, as a comparison, is simply harder and quite challenging to do in general purpose terms. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 > that mirroring of metrics seems like a bigger question @srowen thanks for the comments, rest api is not always available or a standard way to export stuff. This is also the experience coming from [Opsclarity](https://www.opsclarity.com/product/features/) work. It it much more standard to use jmx for many systems out there AFAIK. This way using one component only like jmx exporter I can have all metrics in one place and no need to scrape the rest api. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2978/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22381 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2977/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #95880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95880/testReport)** for PR 22381 at commit [`b43190d`](https://github.com/apache/spark/commit/b43190d5393217f751f38149839a060df88baf1f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22381 **[Test build #95881 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95881/testReport)** for PR 22381 at commit [`b43190d`](https://github.com/apache/spark/commit/b43190d5393217f751f38149839a060df88baf1f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22381 please test this --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
