[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4345#issuecomment-72781888 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26698/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4345#discussion_r24061664
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
---
@@ -132,14 +132,14 @@ class LogisticRegressionModel private[ml] (
override def transform(dataset: DataFrame, paramMap: ParamMap):
DataFrame = {
transformSchema(dataset.schema, paramMap, logging = true)
val map = this.paramMap ++ paramMap
-val scoreFunction: Vector = Double = (v) = {
+val scoreFunction = udf((v: Vector) = {
val margin = BLAS.dot(v, weights)
1.0 / (1.0 + math.exp(-margin))
-}
+} : Double)
--- End diff --
will do in a separate pr
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4345#discussion_r24061660
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -26,9 +27,8 @@ import
org.apache.spark.sql.catalyst.errors.TreeNodeException
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.types._
-/* Implicits */
import org.apache.spark.sql.TestData._
-import org.apache.spark.sql.test.TestSQLContext._
+import org.apache.spark.sql.test.TestSQLContext.{udf = _, _}
--- End diff --
We use this quite a bit in SQL tests though. Really, we should avoid
wildcard important. That's a future TODO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4345#issuecomment-72781877 [Test build #26698 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26698/consoleFull) for PR 4345 at commit [`639c0f8`](https://github.com/apache/spark/commit/639c0f8663c942b4f610e8256d6bb3bead20fbde). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4345 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4345#issuecomment-72769530 [Test build #26687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26687/consoleFull) for PR 4345 at commit [`b452b8d`](https://github.com/apache/spark/commit/b452b8d0c44488328367fed5b5f2eeba6f0c6c55). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4345#issuecomment-72769536 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26687/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4345#discussion_r24055018
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
---
@@ -132,14 +132,14 @@ class LogisticRegressionModel private[ml] (
override def transform(dataset: DataFrame, paramMap: ParamMap):
DataFrame = {
transformSchema(dataset.schema, paramMap, logging = true)
val map = this.paramMap ++ paramMap
-val scoreFunction: Vector = Double = (v) = {
+val scoreFunction = udf((v: Vector) = {
val margin = BLAS.dot(v, weights)
1.0 / (1.0 + math.exp(-margin))
-}
+} : Double)
--- End diff --
About the syntax, I like the following better
~~~
val margin = udf { v: Vector =
val margin = BLAS.dot(v, weights)
1.0 / (1.0 + math.exp(-margin))
}
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4345#issuecomment-72770311 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26688/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4345#issuecomment-72770304 [Test build #26688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26688/consoleFull) for PR 4345 at commit [`0a0b339`](https://github.com/apache/spark/commit/0a0b339f311573d775b1704ab2e0860e22746c3f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4345#discussion_r24055407
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -26,9 +27,8 @@ import
org.apache.spark.sql.catalyst.errors.TreeNodeException
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.types._
-/* Implicits */
import org.apache.spark.sql.TestData._
-import org.apache.spark.sql.test.TestSQLContext._
+import org.apache.spark.sql.test.TestSQLContext.{udf = _, _}
--- End diff --
Maybe it is useful to leave a comment here about what it does.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/4345#issuecomment-72771880 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4345#issuecomment-72772169 [Test build #26698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26698/consoleFull) for PR 4345 at commit [`639c0f8`](https://github.com/apache/spark/commit/639c0f8663c942b4f610e8256d6bb3bead20fbde). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4345#issuecomment-72772727 [Test build #26690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26690/consoleFull) for PR 4345 at commit [`639c0f8`](https://github.com/apache/spark/commit/639c0f8663c942b4f610e8256d6bb3bead20fbde). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5578][SQL][DataFrame] Provide a conveni...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4345#issuecomment-72772735 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26690/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
