[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 Thank you for reviewing @holdenk ☺ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user holdenk commented on the issue: https://github.com/apache/spark/pull/20629 Merged to master :) Thank you :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93016/ Test PASSed. ---

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #93016 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93016/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #93016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93016/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/962/

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user holdenk commented on the issue: https://github.com/apache/spark/pull/20629 After I merged your other PR this has some minor conflicts so needs an update, but I'd be happy to try and get this in end of next week during my next review session :) ---

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 @holdenk @sethah any more comments on this? As 2.4 release is approaching I think it would be great to have this in, so we can remove the deprecated APIs in 3.0... Thanks. ---

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92477/ Test PASSed. ---

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #92477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92477/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92476/ Test FAILed. ---

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #92476 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92476/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #92477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92477/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/587/

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #92476 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92476/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #92475 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92475/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92475/ Test FAILed. ---

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/585/

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #92475 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92475/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 Thanks @holdenk @sethah, then I'll update this PR accordingly. Thanks. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user sethah commented on the issue: https://github.com/apache/spark/pull/20629 +1 for @mgaido91's plan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user holdenk commented on the issue: https://github.com/apache/spark/pull/20629 ping @sethah? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user holdenk commented on the issue: https://github.com/apache/spark/pull/20629 ping @sethah :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user holdenk commented on the issue: https://github.com/apache/spark/pull/20629 cc @sethah --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 kindly ping @holdenk @MLnick --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 kindly ping @holdenk @MLnick --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 @holdenk I am not sure about requiring or not cluster centers for this metric. On one side, since the `ClusteringEvaluator` should be a general interface for all clustering models and some of them

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89205/ Test PASSed. ---

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #89205 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89205/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89207/ Test PASSed. ---

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #89207 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89207/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution//

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #89207 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89207/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2220/

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #89205 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89205/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 @holdenk I am not sure I got 100% what you meant, so I'll try to answer but let me know if I missed something. The problem of doing 2 passes is related to cluster centers. The API of

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user holdenk commented on the issue: https://github.com/apache/spark/pull/20629 So when you say "second pass over the data" - from looking at this it seems like it would could do this with just a second map to look up the predictions in the already computed cluster centers,

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 kindly ping @MLnick @jkbradley @hhbyyh --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 kindly ping @MLnick @jkbradley @hhbyyh --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88307/ Test PASSed. ---

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #88307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88307/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1562/

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 @MLnick I checked and adding the `computeCosts` to `ClusteringEvaluator` has a small drawback: we have to compute the centers for each cluster and then we can compute the costs, which involved two

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #88307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88307/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 sorry @MLnick, what do you think about my previous comment? Any thoughts? Thanks. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 yes, I agree with you @MLnick. I'd like to ping also @jkbradley and @hhbyyh who drove the PR which introduce `computeCost`. In order to move the same metric to `ClusteringEvaluator`, I

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user MLnick commented on the issue: https://github.com/apache/spark/pull/20629 Sorry I mean putting the metric in evaluator and then also deprecating computCost On Sun, 18 Feb 2018 at 20:41, Nick Pentreath wrote: > Right - so while

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user MLnick commented on the issue: https://github.com/apache/spark/pull/20629 Right - so while it’s perhaps a lower quality metric it is different. So I wonder if deprecation is the right approach (vs say putting the within cluster sum squares into

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20629 thanks for taking a look at this @MLnick. No, it doesn't, in the sense that it returns a different result: this is the sum of the squared euclidean distance between a point and the centroid of the

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user MLnick commented on the issue: https://github.com/apache/spark/pull/20629 Just want to check - does `computeCost` do the same thing as the silhouette metric? --- - To unsubscribe, e-mail:

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87510/ Test PASSed. ---

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #87510 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87510/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20629 **[Test build #87510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87510/testReport)** for PR 20629 at commit

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/937/

[GitHub] spark issue #20629: [SPARK-23451][ML] Deprecate KMeans.computeCost

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20629 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional