[GitHub] spark issue #15434: [SPARK-17873][SQL] ALTER TABLE RENAME TO should allow us...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15434 cc @zhzhan Just FYI, the behavior of this DDL is different from Hive. If your team is migrating your Hive to Spark, you need to double check your scripts containing ALTER TABLE RENAME TO. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13065: [SPARK-15214][SQL] Code-generation for Generate
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13065 **[Test build #66802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66802/consoleFull)** for PR 13065 at commit [`8c14194`](https://github.com/apache/spark/commit/8c1419414aee2873497d6ce6564cc349f6f4f80e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13065: [SPARK-15214][SQL] Code-generation for Generate
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/13065 cc @davies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13065: [SPARK-15214][SQL] Code-generation for Generate
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13065 **[Test build #66803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66803/consoleFull)** for PR 13065 at commit [`459714c`](https://github.com/apache/spark/commit/459714c7f39046d3b7969e9133ef8aea7641d80d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12337: [SPARK-15566] Expose null checking function to Python la...
Github user kevincox commented on the issue: https://github.com/apache/spark/pull/12337 @holdenk The point is that this is inline. It doesn't require evaluating the whole dataframe and counting the nulls you find. Instead you use this on a column and it asserts that every value passing through is not null (or it raises an error). Then you continue using the dataframe like you would have and this check has almost no cost. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-le...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15360#discussion_r82947471
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala ---
@@ -358,53 +358,189 @@ class StatisticsSuite extends QueryTest with
TestHiveSingleton with SQLTestUtils
}
}
- test("generate column-level statistics and load them from hive
metastore") {
+ private def statsBeforeAfterUpdate(isAnalyzeTable: Boolean):
(Statistics, Statistics) = {
--- End diff --
`Analyze Table COMPUTE STATISTICS FOR COLUMNS` is also `Analyze Table`.
Thus, the input parm name is confusing. How about `isAnalyzeTable` ->
`isAnalyzeColumns`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-level sta...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15360 cc @cloud-fan I do not have any more comment. Could you check this please? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15435 **[Test build #66804 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66804/consoleFull)** for PR 15435 at commit [`805613c`](https://github.com/apache/spark/commit/805613ceb47ef873f9bea142791e6a55a5030471). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15435 **[Test build #66804 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66804/consoleFull)** for PR 15435 at commit [`805613c`](https://github.com/apache/spark/commit/805613ceb47ef873f9bea142791e6a55a5030471). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15435 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66804/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15435 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15433: [SPARK-17822] Use weak reference in JVMObjectTrac...
Github user techaddict closed the pull request at: https://github.com/apache/spark/pull/15433 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15433: [SPARK-17822] Use weak reference in JVMObjectTrac...
GitHub user techaddict reopened a pull request: https://github.com/apache/spark/pull/15433 [SPARK-17822] Use weak reference in JVMObjectTracker.objMap because it may leak JVM objects ## What changes were proposed in this pull request? Use weak reference in JVMObjectTracker.objMap because it may leak JVM objects ## How was this patch tested? existing tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/techaddict/spark SPARK-17822 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15433.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15433 commit 7023c40a99eaa81ee7bcd202a4b74df811d0cfc7 Author: Sandeep Singh Date: 2016-10-10T11:54:34Z [SPARK-17822] Use weak reference in JVMObjectTracker.objMap because it may leak JVM objects commit 69845947df62187eb40f3cc6468b52e38bdab897 Author: Sandeep Singh Date: 2016-10-10T13:23:56Z Merge branch 'master' into SPARK-17822 commit 995611d75351d24907ce2b22e7d33752cc803da3 Author: Sandeep Singh Date: 2016-10-11T13:13:09Z Merge branch 'master' into SPARK-17822 commit 8e763bef78fe147e84e1771f237a75ff42780705 Author: Sandeep Singh Date: 2016-10-12T06:33:26Z fix for failures commit 7d50d84f90fcda9e5dec79c9be834870c83443c4 Author: Sandeep Singh Date: 2016-10-12T06:34:23Z Merge branch 'master' into SPARK-17822 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15442: [SPARK-17853][STREAMING][KAFKA][DOC] make it clear that ...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15442 Merging in master/branch-2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15442: [SPARK-17853][STREAMING][KAFKA][DOC] make it clea...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15442 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15148: [SPARK-5992][ML] Locality Sensitive Hashing
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15148 **[Test build #66800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66800/consoleFull)** for PR 15148 at commit [`1b63173`](https://github.com/apache/spark/commit/1b6317396629b9f290a279dd735923c0fc8efd89). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class BitSampling(override val uid: String) extends LSH[BitSamplingModel]` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15148: [SPARK-5992][ML] Locality Sensitive Hashing
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15148 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66800/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15148: [SPARK-5992][ML] Locality Sensitive Hashing
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15148 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15307: [SPARK-17731][SQL][STREAMING] Metrics for structured str...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15307 **[Test build #66795 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66795/consoleFull)** for PR 15307 at commit [`4c08d56`](https://github.com/apache/spark/commit/4c08d569f7817e222550ef7578c6e01f90bc4ee0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15307: [SPARK-17731][SQL][STREAMING] Metrics for structured str...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15307 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15307: [SPARK-17731][SQL][STREAMING] Metrics for structured str...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15307 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66795/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15433: [SPARK-17822] Use weak reference in JVMObjectTracker.obj...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15433 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66797/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15433: [SPARK-17822] Use weak reference in JVMObjectTracker.obj...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15433 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15433: [SPARK-17822] Use weak reference in JVMObjectTracker.obj...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15433 **[Test build #66797 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66797/consoleFull)** for PR 15433 at commit [`7d50d84`](https://github.com/apache/spark/commit/7d50d84f90fcda9e5dec79c9be834870c83443c4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15435 **[Test build #66805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66805/consoleFull)** for PR 15435 at commit [`fdac2dc`](https://github.com/apache/spark/commit/fdac2dc7028b633ddda4d42d9f62ec9369b21b41). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-le...
Github user wzhfy commented on a diff in the pull request:
https://github.com/apache/spark/pull/15360#discussion_r82956403
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala ---
@@ -358,53 +358,189 @@ class StatisticsSuite extends QueryTest with
TestHiveSingleton with SQLTestUtils
}
}
- test("generate column-level statistics and load them from hive
metastore") {
+ private def statsBeforeAfterUpdate(isAnalyzeTable: Boolean):
(Statistics, Statistics) = {
--- End diff --
@gatorsmile OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12064: [SPARK-14272][ML] Evaluate GaussianMixtureModel with Log...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12064 **[Test build #66806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66806/consoleFull)** for PR 12064 at commit [`d5b9422`](https://github.com/apache/spark/commit/d5b94220956175befe1be4154e349ae79ebb9042). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-level sta...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15360 **[Test build #66807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66807/consoleFull)** for PR 15360 at commit [`d93d082`](https://github.com/apache/spark/commit/d93d082786b758be43689d79afcba45f67da1d49). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15433: [SPARK-17822] Use weak reference in JVMObjectTrac...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15433#discussion_r82960799
--- Diff: core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala
---
@@ -284,7 +286,7 @@ private[r] object JVMObjectTracker {
objId
}
- def remove(id: String): Option[Object] = {
+ def remove(id: String) {
--- End diff --
Although it's normal to return the object that was removed from map
methods, if this isn't desired, I'd declare this as `: Unit` to be clear
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15433: [SPARK-17822] Use weak reference in JVMObjectTrac...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15433#discussion_r82960952
--- Diff: core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala
---
@@ -263,18 +264,19 @@ private[r] object JVMObjectTracker {
// TODO: This map should be thread-safe if we want to support multiple
// connections at the same time
- private[this] val objMap = new HashMap[String, Object]
+ private[this] val objMap: ConcurrentMap[String, Object] =
--- End diff --
Not that it matters a lot, but do you need the reference type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15433: [SPARK-17822] Use weak reference in JVMObjectTrac...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15433#discussion_r82961129
--- Diff: core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala
---
@@ -263,18 +264,19 @@ private[r] object JVMObjectTracker {
// TODO: This map should be thread-safe if we want to support multiple
// connections at the same time
- private[this] val objMap = new HashMap[String, Object]
+ private[this] val objMap: ConcurrentMap[String, Object] =
+new MapMaker().weakValues().makeMap[String, Object]()
// TODO: We support only one connection now, so an integer is fine.
// Investigate using use atomic integer in the future.
private[this] var objCounter: Int = 0
def getObject(id: String): Object = {
-objMap(id)
+objMap.get(id)
}
def get(id: String): Option[Object] = {
-objMap.get(id)
+Option(objMap.get(id))
}
def put(obj: Object): String = {
--- End diff --
It kind of seems like this class is meant to be thread-safe, but this
isn't. You could use `AtomicLong` and `getAndIncrement`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15449: [SPARK-17884][SQL] To resolve Null pointer except...
GitHub user priyankagargnitk opened a pull request: https://github.com/apache/spark/pull/15449 [SPARK-17884][SQL] To resolve Null pointer exception when casting from empty string to interval type. ## What changes were proposed in this pull request? This change adds a check in castToInterval method of Cast expression , such that if converted value is null , then isNull variable should be set to true. Earlier, the expression Cast(Literal(), CalendarIntervalType) was throwing NullPointerException because of the above mentioned reason. ## How was this patch tested? Added test case in CastSuite.scala jira entry for detail: https://issues.apache.org/jira/browse/SPARK-17884 You can merge this pull request into a Git repository by running: $ git pull https://github.com/priyankagargnitk/spark SPARK-17884 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15449.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15449 commit 9dc99adecfadb60ef402f68e956a0f94734b1226 Author: prigarg Date: 2016-10-12T08:51:13Z [SPARK-17884][SQL] To resolve Null pointer exception when casting from empty string to interval type. ## What changes were proposed in this pull request? This change adds a check in castToInterval method of Cast expression , such that if converted value is null , then isNull variable should be set to true. Earlier, the expression Cast(Literal(), CalendarIntervalType) was throwing NullPointerException because of the above mentioned reason. ## How was this patch tested? Added test case in CastSuite.scala jira entry for detail: https://issues.apache.org/jira/browse/SPARK-17884 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15449: [SPARK-17884][SQL] To resolve Null pointer exception whe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15449 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13065: [SPARK-15214][SQL] Code-generation for Generate
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13065 **[Test build #66803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66803/consoleFull)** for PR 13065 at commit [`459714c`](https://github.com/apache/spark/commit/459714c7f39046d3b7969e9133ef8aea7641d80d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13065: [SPARK-15214][SQL] Code-generation for Generate
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13065 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13065: [SPARK-15214][SQL] Code-generation for Generate
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13065 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66803/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15342: [SPARK-11560] [MLLIB] Optimize KMeans implementation / r...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15342 Merged to master. I'm going to reopen a PR for just the duplicate centroids issue to re-table that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15342: [SPARK-11560] [MLLIB] Optimize KMeans implementat...
Github user srowen closed the pull request at: https://github.com/apache/spark/pull/15342 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to register Jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/9766 **[Test build #66801 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66801/consoleFull)** for PR 9766 at commit [`d481821`](https://github.com/apache/spark/commit/d4818217dc6e29a72a4e470dbe08cda197933162). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to register Jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/9766 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to register Jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/9766 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66801/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15386: [SPARK-17808][PYSPARK] Upgraded version of Pyrolite to 4...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15386 Merged to 2.0 too --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15230: [SPARK-17657] [SQL] Disallow Users to Change Table Type
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15230 **[Test build #66798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66798/consoleFull)** for PR 15230 at commit [`e19536c`](https://github.com/apache/spark/commit/e19536c3c645b70f6cf1df747a7798188acf2935). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15230: [SPARK-17657] [SQL] Disallow Users to Change Table Type
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15230 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66798/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15230: [SPARK-17657] [SQL] Disallow Users to Change Table Type
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15230 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15436: [SPARK-17875] [BUILD] Remove unneeded direct dependence ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15436 **[Test build #3326 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3326/consoleFull)** for PR 15436 at commit [`a5c5c31`](https://github.com/apache/spark/commit/a5c5c3146e702a5c6ac8a86648f58f44d13a95f2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15411: Updated master url
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15411 Ping @getintouchapp --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15411: Updated master url
Github user getintouchapp commented on the issue: https://github.com/apache/spark/pull/15411 Am I supposed to do anything here? I cleaned up the comments and I don't have permission to do anything else beyond that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13065: [SPARK-15214][SQL] Code-generation for Generate
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13065 **[Test build #66802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66802/consoleFull)** for PR 13065 at commit [`8c14194`](https://github.com/apache/spark/commit/8c1419414aee2873497d6ce6564cc349f6f4f80e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-level sta...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15360 **[Test build #66799 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66799/consoleFull)** for PR 15360 at commit [`1e64163`](https://github.com/apache/spark/commit/1e641633cbd38a4a990a1cebafeff7be276a0fec). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15429: [SPARK-17840] [DOCS] Add some pointers for wiki/CONTRIBU...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15429 **[Test build #66808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66808/consoleFull)** for PR 15429 at commit [`65f0bd3`](https://github.com/apache/spark/commit/65f0bd3a95087b50be8c44806a341183c58e1727). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13065: [SPARK-15214][SQL] Code-generation for Generate
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13065 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66802/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-level sta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15360 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13065: [SPARK-15214][SQL] Code-generation for Generate
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13065 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-level sta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15360 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66799/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15435 **[Test build #66805 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66805/consoleFull)** for PR 15435 at commit [`fdac2dc`](https://github.com/apache/spark/commit/fdac2dc7028b633ddda4d42d9f62ec9369b21b41). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66805/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15435 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15406: [Spark-17745][ml][PySpark] update NB python api - add we...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15406 **[Test build #66809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66809/consoleFull)** for PR 15406 at commit [`9f9591a`](https://github.com/apache/spark/commit/9f9591a0fbbd6e9ddc7b36ca7e7776543b98495a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15411: Updated master url
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15411 Yes thanks for updating the description. The title could be better. Yuhao also asked if this is something that looks like it needs to be done for more examples? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15427: [SPARK-17866][SPARK-17867][SQL] Fix Dataset.dropduplicat...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15427 My thoughts: 1. `Dataset.dropDuplicates()` should drop duplicates for all columns, the current implementation is wrong, this PR fixed it. 2. `Dataset.dropDuplicates(col: String)` should drop the first column matching the given name, or all matched columns? `Dataset.drop(col: String)` also drops all matched columns, the new behaviour seems reasonable. But we should be very careful, as this is a breaking change. cc @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to regis...
Github user zjffdu commented on a diff in the pull request:
https://github.com/apache/spark/pull/9766#discussion_r82969355
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala ---
@@ -412,6 +419,63 @@ class UDFRegistration private[sql] (functionRegistry:
FunctionRegistry) extends
//
/**
+ * Register a Java UDF class
+ * @param name
+ * @param className
+ * @param returnType
+ */
+ def registerJava(name: String, className: String, returnType: DataType):
Unit = {
+
+try {
+ // scalastyle:off classforname
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to regis...
Github user zjffdu commented on a diff in the pull request:
https://github.com/apache/spark/pull/9766#discussion_r82969338
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala ---
@@ -17,9 +17,15 @@
package org.apache.spark.sql
+
+import java.io.IOException
+import java.util.{List => JList, Map => JMap}
+
import scala.reflect.runtime.universe.TypeTag
import scala.util.Try
+import sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12064: [SPARK-14272][ML] Evaluate GaussianMixtureModel with Log...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12064 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12064: [SPARK-14272][ML] Evaluate GaussianMixtureModel with Log...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12064 **[Test build #66806 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66806/consoleFull)** for PR 12064 at commit [`d5b9422`](https://github.com/apache/spark/commit/d5b94220956175befe1be4154e349ae79ebb9042). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12064: [SPARK-14272][ML] Evaluate GaussianMixtureModel with Log...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12064 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66806/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to regis...
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/spark/pull/9766#discussion_r82969210 --- Diff: python/pyspark/sql/context.py --- @@ -202,6 +202,10 @@ def registerFunction(self, name, f, returnType=StringType()): """ self.sparkSession.catalog.registerFunction(name, f, returnType) +def registerJavaFunction(self, name, javaClassName, returnType): --- End diff -- Fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15411: Set master URL configuration in scala example
Github user getintouchapp commented on the issue: https://github.com/apache/spark/pull/15411 Fixed the title. Yes other examples are missing the master url config too. I will add them and create a pull request for the rest of the files once this is merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15411: Set master URL configuration in scala example
Github user getintouchapp commented on the issue: https://github.com/apache/spark/pull/15411 btw, for the rest of the files its not a major issue because they don't turn up in documentation. This one does at http://spark.apache.org/docs/latest/sql-programming-guide.html#starting-point-sparksession --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15406: [Spark-17745][ml][PySpark] update NB python api - add we...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15406 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66809/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15406: [Spark-17745][ml][PySpark] update NB python api - add we...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15406 **[Test build #66809 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66809/consoleFull)** for PR 15406 at commit [`9f9591a`](https://github.com/apache/spark/commit/9f9591a0fbbd6e9ddc7b36ca7e7776543b98495a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15411: Set master URL configuration in scala example
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15411 I think it's right to do this all at once, rather than in pieces. I must say I recall I opened a PR like this a long long time ago and Matei said the master was excluded on purpose because it was intended to be set by the environment running the example, which made some sense. I wonder if the logic is still the same or not? that is, if it's a runnable example, do we want to not override the master that the runner might set? or for doc-only example code, is it that we do need to show master being set programmatically? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15406: [Spark-17745][ml][PySpark] update NB python api - add we...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15406 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15411: Set master URL configuration in scala example
Github user getintouchapp commented on the issue: https://github.com/apache/spark/pull/15411 It does make sense to load master url dynamically thru environment or command line while running the example. However the documentation example should be a valid piece of code and not dependent on other hidden variables. I vote that we merge this pull request for the sake of clarity in documentation but for other examples we leave as is, since they could load the master URL dynamically --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15411: Set master URL configuration in scala example
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15411 OK are there other such examples that are of the same form? it makes sense to change any case like this in one go. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66810/consoleFull)** for PR 15382 at commit [`ef7a141`](https://github.com/apache/spark/commit/ef7a14102c1269fabf10667947adb8e314d775a0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-level sta...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15360 **[Test build #66807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66807/consoleFull)** for PR 15360 at commit [`d93d082`](https://github.com/apache/spark/commit/d93d082786b758be43689d79afcba45f67da1d49). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-level sta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15360 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66807/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15360: [SPARK-17073] [SQL] [FOLLOWUP] generate column-level sta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15360 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12933 **[Test build #66811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66811/consoleFull)** for PR 12933 at commit [`bbe908e`](https://github.com/apache/spark/commit/bbe908ef66898624166bcf06d1773008e7414f14). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15436: [SPARK-17875] [BUILD] Remove unneeded direct dependence ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15436 **[Test build #3326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3326/consoleFull)** for PR 15436 at commit [`a5c5c31`](https://github.com/apache/spark/commit/a5c5c3146e702a5c6ac8a86648f58f44d13a95f2). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12933 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66811/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12933 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12933: [Spark-15155][Mesos] Optionally ignore default role reso...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12933 **[Test build #66811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66811/consoleFull)** for PR 12933 at commit [`bbe908e`](https://github.com/apache/spark/commit/bbe908ef66898624166bcf06d1773008e7414f14). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15172: [SPARK-13331] AES support for over-the-wire encryption
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15172 **[Test build #66812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66812/consoleFull)** for PR 15172 at commit [`deb63a7`](https://github.com/apache/spark/commit/deb63a73f266a71784f86b382bea8b5d18bd5bf3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15436: [SPARK-17875] [BUILD] Remove unneeded direct dependence ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15436 **[Test build #66813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66813/consoleFull)** for PR 15436 at commit [`84bdf1b`](https://github.com/apache/spark/commit/84bdf1b9727efe8b5bd183a5ac6a9def85d9c851). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15429: [SPARK-17840] [DOCS] Add some pointers for wiki/CONTRIBU...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15429 **[Test build #66808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66808/consoleFull)** for PR 15429 at commit [`65f0bd3`](https://github.com/apache/spark/commit/65f0bd3a95087b50be8c44806a341183c58e1727). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15429: [SPARK-17840] [DOCS] Add some pointers for wiki/CONTRIBU...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15429 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66808/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15429: [SPARK-17840] [DOCS] Add some pointers for wiki/CONTRIBU...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15429 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15414: [SPARK-17848][ML] Move LabelCol datatype cast int...
Github user zhengruifeng commented on a diff in the pull request:
https://github.com/apache/spark/pull/15414#discussion_r82988194
--- Diff: mllib/src/test/scala/org/apache/spark/ml/PredictorSuite.scala ---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.linalg._
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.types._
+
+class PredictorSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
+
+ import testImplicits._
+
+ class MockPredictor(override val uid: String)
+extends Predictor[Vector, MockPredictor, MockPredictionModel] {
+
+override def train(dataset: Dataset[_]): MockPredictionModel = {
+ require(dataset.schema("label").dataType == DoubleType)
+ new MockPredictionModel(uid)
+}
+
+override def copy(extra: ParamMap): MockPredictor = defaultCopy(extra)
+ }
+
+ class MockPredictionModel(override val uid: String)
+extends PredictionModel[Vector, MockPredictionModel] {
+
+override def predict(features: Vector): Double = 1.0
+
+override def copy(extra: ParamMap): MockPredictionModel =
defaultCopy(extra)
+ }
+
+ test("should support all NumericType labels and not support other
types") {
+val predictor = new MockPredictor("mock")
+MLTestingUtils.checkNumericTypes[MockPredictionModel, MockPredictor](
--- End diff --
OK, I will update this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15414: [SPARK-17848][ML] Move LabelCol datatype cast into Predi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15414 **[Test build #66814 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66814/consoleFull)** for PR 15414 at commit [`6ef17b7`](https://github.com/apache/spark/commit/6ef17b7382f494cd7e5df55b64521172f51ff0cb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15230: [SPARK-17657] [SQL] Disallow Users to Change Tabl...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15230#discussion_r82990434
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -225,6 +225,11 @@ case class AlterTableSetPropertiesCommand(
val catalog = sparkSession.sessionState.catalog
val table = catalog.getTableMetadata(tableName)
DDLUtils.verifyAlterTableType(catalog, table, isView)
+// Not allowed to switch the table type.
+if (properties.contains("EXTERNAL")) {
--- End diff --
Then should we put this check in
`HiveExternalCatalog.verifyTableProperties`? I think it's a hive specific
limitation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15432: [SPARK-17854][SQL] rand/randn allows null/long as input ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15432 **[Test build #66815 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66815/consoleFull)** for PR 15432 at commit [`6f8f3f3`](https://github.com/apache/spark/commit/6f8f3f33f9b67d77285048bfd7d794990e072b8a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15450: [SPARK-3261] [MLLIB] KMeans clusterer can return ...
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/15450 [SPARK-3261] [MLLIB] KMeans clusterer can return duplicate cluster centers ## What changes were proposed in this pull request? Return potentially fewer than k cluster centers in cases where k distinct centroids aren't available or aren't selected. ## How was this patch tested? Existing tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/srowen/spark SPARK-3261 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15450.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15450 commit 42279b8e042aedaf54ae3900fc1b050d2a1dacef Author: Sean Owen Date: 2016-10-12T12:30:02Z Return potentially fewer than k cluster centers in cases where k distinct centroids aren't available or aren't selected --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15450: [SPARK-3261] [MLLIB] KMeans clusterer can return duplica...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15450 **[Test build #66816 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66816/consoleFull)** for PR 15450 at commit [`42279b8`](https://github.com/apache/spark/commit/42279b8e042aedaf54ae3900fc1b050d2a1dacef). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14653: [SPARK-10931][PYSPARK][ML] PySpark ML Models should cont...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14653 **[Test build #3328 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3328/consoleFull)** for PR 14653 at commit [`e706c7e`](https://github.com/apache/spark/commit/e706c7ee78aaebb5d3e625d651d37b9d088b6441). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15420: [SPARK-17855][CORE] Remove query string from jar url
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15420 **[Test build #3327 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3327/consoleFull)** for PR 15420 at commit [`d418568`](https://github.com/apache/spark/commit/d4185682e16c8813799a30db4d86ebdaf0b5361f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15448: [SPARK-17108][SQL]: Fix BIGINT and INT comparison failur...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15448 **[Test build #3330 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3330/consoleFull)** for PR 15448 at commit [`ec3d552`](https://github.com/apache/spark/commit/ec3d55296abc9f355a0f0db0f40e04abb4b58d94). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12337: [SPARK-15566] Expose null checking function to Python la...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12337 **[Test build #3331 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3331/consoleFull)** for PR 12337 at commit [`c671e4f`](https://github.com/apache/spark/commit/c671e4fe7f7a2dd08048e96c6c7c0a6485d063b9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15411: Set master URL configuration in scala example
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15411 **[Test build # has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder//consoleFull)** for PR 15411 at commit [`5324034`](https://github.com/apache/spark/commit/532403476678a8161d18a30ef12b21bffb4d5f92). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
