[GitHub] spark issue #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods f...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15913 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69108/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods f...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15913 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods f...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15913 **[Test build #69108 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69108/consoleFull)** for PR 15913 at commit [`0c85fd4`](https://github.com/apache/spark/commit/0c85fd4143c666c06a3db9c89ac50eb4981b2849). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15977 **[Test build #69118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69118/consoleFull)** for PR 15977 at commit [`2f31e72`](https://github.com/apache/spark/commit/2f31e7272a3e8eff3612216f0949f83c34a09023). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16001: [SPARK-18575][WEB] Keep same style: adjust the position ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16001 **[Test build #69117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69117/consoleFull)** for PR 16001 at commit [`a422955`](https://github.com/apache/spark/commit/a422955ea14a66f748c38102710bf89f3d23797e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16001: SPARK-18575: Keep same style: adjust the position...
GitHub user uncleGen opened a pull request: https://github.com/apache/spark/pull/16001 SPARK-18575: Keep same style: adjust the position of driver log links ## What changes were proposed in this pull request? NOT BUG, just adjust the position of driver log link to keep the same style with other executors log link.  ## How was this patch tested? no You can merge this pull request into a Git repository by running: $ git pull https://github.com/uncleGen/spark SPARK-18575 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16001.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16001 commit a422955ea14a66f748c38102710bf89f3d23797e Author: uncleGen Date: 2016-11-24T07:40:43Z SPARK-18575: Keep same style: adjust the position of driver log links --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14612: [SPARK-16803] [SQL] SaveAsTable does not work whe...
Github user gatorsmile closed the pull request at: https://github.com/apache/spark/pull/14612 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14995 **[Test build #69116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69116/consoleFull)** for PR 14995 at commit [`04172e0`](https://github.com/apache/spark/commit/04172e03c11165f961ec1c4dd8a66bc568d5620a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE and DESC ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15358 **[Test build #69115 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69115/consoleFull)** for PR 15358 at commit [`bc9a508`](https://github.com/apache/spark/commit/bc9a5082cf4576cfa7a6d4911db74bc0476c4360). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE and DESC ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15358 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching Using ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15975 **[Test build #69114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69114/consoleFull)** for PR 15975 at commit [`5c5b3ca`](https://github.com/apache/spark/commit/5c5b3cab4ec77484467e960319abbddb49313952). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15977 **[Test build #69113 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69113/consoleFull)** for PR 15977 at commit [`9bb1264`](https://github.com/apache/spark/commit/9bb1264f91af3bc855ae523a06526e830f6f85e0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16000: [SPARK-18537][Web UI]Add a REST api to spark streaming
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16000 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching Using ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15975 The test cases of the write path will be added after https://github.com/apache/spark/pull/15966 is merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16000: [SPARK-18537][Web UI]Add a REST api to spark streaming
Github user ChorPangChan commented on the issue: https://github.com/apache/spark/pull/16000 hi @vanzin new PR against master is ready. please have a look --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15975#discussion_r89445082
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -137,7 +137,8 @@ private[sql] case class JDBCRelation(
}
override def toString: String = {
+val partitioningInfo = if (parts.nonEmpty) s"
[numPartitions=${parts.length}]"
// credentials should not be included in the plan output, table
information is sufficient.
-s"JDBCRelation(${jdbcOptions.table})"
+s"JDBCRelation(${jdbcOptions.table})" + partitioningInfo
--- End diff --
Fixed. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15975#discussion_r89445068
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -159,7 +159,11 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends Logging {
* @since 1.4.0
*/
def jdbc(url: String, table: String, properties: Properties): DataFrame
= {
-jdbc(url, table, JDBCRelation.columnPartition(null), properties)
+// connectionProperties should override settings in extraOptions.
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15966 **[Test build #69112 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69112/consoleFull)** for PR 15966 at commit [`f9db374`](https://github.com/apache/spark/commit/f9db374d7c90f8d8d7875a13cc17644821779e88). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...
GitHub user ChorPangChan opened a pull request: https://github.com/apache/spark/pull/16000 [SPARK-18537][Web UI]Add a REST api to spark streaming a new PR for https://github.com/apache/spark/pull/15974 against master ## What changes were proposed in this pull request? 1. implement a package(org.apache.spark.streaming.status.api.v1) that serve the same purpose as org.apache.spark.status.api.v1 1. register the api path through StreamingPage 1. retrive the streaming informateion through StreamingJobProgressListener this api should cover exceptly the same amount of information as you can get from the web interface the implementation is base on the current REST implementation of spark-core and will be available for running applications only https://issues.apache.org/jira/browse/SPARK-18537 You can merge this pull request into a Git repository by running: $ git pull https://github.com/ChorPangChan/spark stream-api-dev-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16000.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16000 commit 680a59aab2f8b3624f9f39fbc78bb5cbd7ec3bac Author: Chan Chor Pang Date: 2016-10-26T05:36:44Z compile ok, try to test commit 04f9e9c914b58c14c98845b5529472333b348590 Author: Chan Chor Pang Date: 2016-10-26T07:39:40Z add path /streamingapi commit da20ce37b88770e2cb997ed48976ba2542305e6b Author: Chan Chor Pang Date: 2016-10-28T05:42:40Z need attach to some where commit 3468d40336091201108169b168bd612c0f5fcf77 Author: Chan Chor Pang Date: 2016-11-02T02:10:03Z no writer yet commit 525ae5fe25a47ed3884c2bf15143dcb932581f48 Author: Chan Chor Pang Date: 2016-11-02T05:13:48Z not work, may be the data need to be in Iterator form commit f854767cb5b3bb4a303d3418b426b73d02599c25 Author: Chan Chor Pang Date: 2016-11-08T06:05:24Z remove unuse file commit 34c8b3b90a9b045e3d6b6ac86d270edff9ed24a3 Author: Chan Chor Pang Date: 2016-11-02T06:18:17Z package name didnt change in the copy process commit 170b18904f041dfeb271f54fdb408ad2f575a2ee Author: Chan Chor Pang Date: 2016-11-07T04:35:43Z try to get the real info commit 2f51c59a37f994c6bee2dd65d1517b32e7d9776d Author: saturday_s Date: 2016-11-14T09:51:02Z Refactor to fit scalastyle. commit 76324b7c6f8849bef7d45363d925fd95efbbedcf Author: saturday_s Date: 2016-11-16T04:40:24Z Try to get startTime. commit 68d734f07b43b44127ae5f698db39d671aaa59c1 Author: saturday_s Date: 2016-11-16T04:53:41Z Change api path prefix. commit ccfe0f5f28db73bb300d43c32d40d6e0e596c77c Author: saturday_s Date: 2016-11-16T09:13:59Z Implement statistics api. commit 2d1e88440902c5212f43746f5c0b7f282b7a6243 Author: saturday_s Date: 2016-11-17T02:59:58Z Implement receivers api. commit 0d9f6b9667ef774f5a8c868a453e3d68b66a6702 Author: saturday_s Date: 2016-11-17T04:46:13Z Fix last-error-info format. commit 8088fa5bad4c0e15bb14abfc0ee7475ba4ad138b Author: saturday_s Date: 2016-11-17T05:08:30Z Implement one-receiver api. commit f1da6b1f2856b761696ae9d767836af6417e4f43 Author: saturday_s Date: 2016-11-17T05:21:39Z Fix access level issue of `ErrorWrapper`. commit 4d8138191f1529137e4c1e858998bd78477ca739 Author: saturday_s Date: 2016-11-18T01:30:30Z Synchronize to listener when getting info from it. commit 17cb832cedb4b2cfeff5e501a9f71378b3402cee Author: saturday_s Date: 2016-11-18T05:30:15Z Implement batch(es) api. commit 137e8fb7de34b39b218939b371062e225adc958e Author: saturday_s Date: 2016-11-18T06:55:42Z Remove details of outputOpsInfo from batchInfo. commit 08f33522251ff20b14af15952ac918cbcfada551 Author: saturday_s Date: 2016-11-18T08:35:55Z Implement outputOpsInfo api. commit 477e71de47bbde642a9222729c73b7dd52318529 Author: saturday_s Date: 2016-11-18T09:37:04Z Try another approach to get outputOpsInfo. commit 7ddac2929343ad60f733166d25ed485fa3976cc0 Author: saturday_s Date: 2016-11-21T02:03:55Z Try another more approach to get outputOpsInfo. commit e0fe970fa64fc87de277a7f63f39423608cfef52 Author: saturday_s Date: 2016-11-21T02:41:25Z Continue trying to get outputOpsInfo(jobIds). commit 35963312dcf722b98cd3b0dabff97d398ccd020c Author: saturday_s Date: 2016-11-21T04:14:38Z Fix outputOpsInfo and jobIds issue. commit 9760492cb826c7552c453e3c55a1098455eaa0bc Author: saturday_s Date: 2016-11-21T04:35:55Z Fix syntax error. commit 65b39078d54408d8ac1ee608a21e49a978e7415d Author: saturday_s Date: 2016-11-21T05:23:31Z Consolidate the param check logic into one place. commit 0080bccf75862d9643ead1c98bf600287227f40d Author: saturday_s Date: 2016-11-21T05:28:37Z Use the same regex-pattern with `ApiRootResource`
[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15966 Retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15966 The only one failure is irrelevant to this PR. ``` [info] KafkaSourceStressForDontFailOnDataLossSuite: [info] - stress test for failOnDataLoss=false *** FAILED *** (1 minute, 58 seconds) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/15994 cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15994 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69107/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15994 **[Test build #69107 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69107/consoleFull)** for PR 15994 at commit [`dbf23a1`](https://github.com/apache/spark/commit/dbf23a1df9ab650794a3e4938deb4cdf7ad680fc). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated me...
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/15913#discussion_r89443000
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala ---
@@ -216,13 +216,6 @@ final class ChiSqSelectorModel private[ml] (
@Since("1.6.0")
def setOutputCol(value: String): this.type = set(outputCol, value)
- /**
- * @group setParam
- */
- @Since("1.6.0")
- @deprecated("labelCol is not used by ChiSqSelectorModel.", "2.0.0")
- def setLabelCol(value: String): this.type = set(labelCol, value)
--- End diff --
All python ML models don't support setter/getter params currently, so no
need to remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated me...
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/15913#discussion_r89442919
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala ---
@@ -183,6 +183,11 @@ class GBTRegressionModel private[ml](
@Since("1.4.0")
override def trees: Array[DecisionTreeRegressionModel] = _trees
+ /**
+ * Number of trees in ensemble
+ */
+ val getNumTrees: Int = trees.length
--- End diff --
Added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated me...
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/15913#discussion_r89442550
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
---
@@ -176,8 +176,12 @@ private[classification] trait LogisticRegressionParams
extends ProbabilisticClas
}
}
- override def validateParams(): Unit = {
+ override protected def validateAndTransformSchema(
--- End diff --
I think it's a bug that ```validateParams ``` was never used. It should
validate params interaction before fitting(if necessary), this is why we
deprecate ```validateParams ``` and move what it does to ```transformSchema```.
We do not have corresponding test cases before, so no test was broken when we
deprecated ```validateParams```. I added test cases in this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods f...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15913 **[Test build #69111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69111/consoleFull)** for PR 15913 at commit [`1d8cef5`](https://github.com/apache/spark/commit/1d8cef5f251614f0aef37fdd19426f5bf298beb4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14136 **[Test build #69110 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69110/consoleFull)** for PR 14136 at commit [`4ace3bc`](https://github.com/apache/spark/commit/4ace3bc5402866761c6f1b61600c4d4e71321598). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15999: [WIP][SPARK-3359][BUILD][DOCS] More changes to resolve j...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15999 **[Test build #69109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69109/consoleFull)** for PR 15999 at commit [`cfce0e8`](https://github.com/apache/spark/commit/cfce0e89a52bb96d38b7132f7d0bd45331d933e3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15999: [WIP][SPARK-3359][BUILD][DOCS] More changes to resolve j...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15999 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15999: [WIP][SPARK-3359][BUILD][DOCS] More changes to resolve j...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15999 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69105/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15999: [WIP][SPARK-3359][BUILD][DOCS] More changes to resolve j...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15999 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15999: [WIP][SPARK-3359][BUILD][DOCS] More changes to resolve j...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15999 **[Test build #69105 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69105/consoleFull)** for PR 15999 at commit [`cfce0e8`](https://github.com/apache/spark/commit/cfce0e89a52bb96d38b7132f7d0bd45331d933e3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated me...
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/15913#discussion_r89439866
--- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala ---
@@ -317,8 +317,28 @@ private[ml] trait TreeEnsembleParams extends
DecisionTreeParams {
}
}
-/** Used for [[RandomForestParams]] */
-private[ml] trait HasFeatureSubsetStrategy extends Params {
+/**
+ * Parameters for Random Forest algorithms.
+ */
+private[ml] trait RandomForestParams extends TreeEnsembleParams {
+
+ /**
+ * Number of trees to train (>= 1).
+ * If 1, then no bootstrapping is used. If > 1, then bootstrapping is
done.
+ * TODO: Change to always do bootstrapping (simpler). SPARK-7130
+ * (default = 20)
+ * @group param
+ */
+ final val numTrees: IntParam = new IntParam(this, "numTrees", "Number of
trees to train (>= 1)",
--- End diff --
Good suggestion, added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated me...
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/15913#discussion_r89439490
--- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala ---
@@ -443,12 +425,11 @@ private[ml] trait GBTParams extends
TreeEnsembleParams with HasMaxIter with HasS
* (default = 0.1)
* @group setParam
*/
- def setStepSize(value: Double): this.type = set(stepSize, value)
-
- override def validateParams(): Unit = {
+ def setStepSize(value: Double): this.type = {
require(ParamValidators.inRange(0, 1, lowerInclusive = false,
upperInclusive = true)(
--- End diff --
The original ```validateParams()``` was only used to check for interactions
between parameters. Parameter value checks which do not depend on other
parameters are handled by Param.validate()``` at the definition, i.e.
```
final val k = new IntParam(this, "k", "The number of clusters to create. " +
"Must be > 1.", ParamValidators.gt(1))
```
However, ```stepSize``` was a trait and inherited by lots of sub-classes
which have different constraints for this parameter, so I add the check at
setter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods f...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15913 **[Test build #69108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69108/consoleFull)** for PR 15913 at commit [`0c85fd4`](https://github.com/apache/spark/commit/0c85fd4143c666c06a3db9c89ac50eb4981b2849). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated me...
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/15913#discussion_r89438884
--- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala ---
@@ -95,11 +95,6 @@ private[ml] trait TreeEnsembleModel[M <:
DecisionTreeModel] {
/** Trees in this ensemble. Warning: These have null parent Estimators.
*/
def trees: Array[M]
- /**
- * Number of trees in ensemble
- */
- val getNumTrees: Int = trees.length
--- End diff --
@sethah You explained this problem very clearly and totally right. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15966 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69104/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15966 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15966 **[Test build #69104 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69104/consoleFull)** for PR 15966 at commit [`f9db374`](https://github.com/apache/spark/commit/f9db374d7c90f8d8d7875a13cc17644821779e88). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15993: [SPARK-18050][SQL] do not create default database...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15993#discussion_r89437989
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ---
@@ -92,8 +92,12 @@ private[sql] class SharedState(val sparkContext:
SparkContext) extends Logging {
{
val defaultDbDefinition = CatalogDatabase(
SessionCatalog.DEFAULT_DATABASE, "default database", warehousePath,
Map())
-// Initialize default database if it doesn't already exist
-externalCatalog.createDatabase(defaultDbDefinition, ignoreIfExists =
true)
+// Initialize default database if it doesn't exist
+if (!externalCatalog.databaseExists(SessionCatalog.DEFAULT_DATABASE)) {
--- End diff --
Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15949 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15949 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69103/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15949 **[Test build #69103 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69103/consoleFull)** for PR 15949 at commit [`4df5675`](https://github.com/apache/spark/commit/4df56754b45247d73ec2e560d012f6b47a1d4853). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15994 **[Test build #69107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69107/consoleFull)** for PR 15994 at commit [`dbf23a1`](https://github.com/apache/spark/commit/dbf23a1df9ab650794a3e4938deb4cdf7ad680fc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15949 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69102/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15949 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15949 **[Test build #69102 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69102/consoleFull)** for PR 15949 at commit [`3b403bd`](https://github.com/apache/spark/commit/3b403bd4377388c74fd1ff62282d57200eb094e8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15994 **[Test build #69106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69106/consoleFull)** for PR 15994 at commit [`7c23f6f`](https://github.com/apache/spark/commit/7c23f6ff03a8a1f5117f9846c8941268ae46e4ed). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69106/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15994 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15994 **[Test build #69106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69106/consoleFull)** for PR 15994 at commit [`7c23f6f`](https://github.com/apache/spark/commit/7c23f6ff03a8a1f5117f9846c8941268ae46e4ed). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/15994 @srowen I exclude the fill function from mima, is it ok? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15904: [SPARK-18470][STREAMING][WIP] Provide Spark Strea...
Github user uncleGen closed the pull request at: https://github.com/apache/spark/pull/15904 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15974: [SPARK-18537] [Web UI]Add a REST api to spark str...
Github user ChorPangChan closed the pull request at: https://github.com/apache/spark/pull/15974 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15974: [SPARK-18537] [Web UI]Add a REST api to spark streaming
Github user ChorPangChan commented on the issue: https://github.com/apache/spark/pull/15974 @uncleGen Thankyou @vanzin OK --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15971: [SPARK-18535][UI][YARN] Redact sensitive information fro...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15971 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15971: [SPARK-18535][UI][YARN] Redact sensitive information fro...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15971 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69100/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15971: [SPARK-18535][UI][YARN] Redact sensitive information fro...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15971 **[Test build #69100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69100/consoleFull)** for PR 15971 at commit [`49015ac`](https://github.com/apache/spark/commit/49015acaf35a2e6b2491c57cd1c71d12ce5a8a64). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15999: [WIP][SPARK-3359][BUILD][DOCS] More changes to resolve j...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15999 **[Test build #69105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69105/consoleFull)** for PR 15999 at commit [`cfce0e8`](https://github.com/apache/spark/commit/cfce0e89a52bb96d38b7132f7d0bd45331d933e3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15999: [WIP][SPARK-3359][BUILD][DOCS] More changes to resolve j...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15999 I should double check the built javadoc. Will leave some images in the changes. BTW, this still does not fully resolve the problem (cc @srowen). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15999: [WIP][SPARK-3359][BUILD][DOCS] More changes to re...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/15999
[WIP][SPARK-3359][BUILD][DOCS] More changes to resolve javadoc 8 errors
that will help unidoc/genjavadoc compatibility
## What changes were proposed in this pull request?
This PR only tries to fix things that looks pretty straightforward and were
fixed in previous PRs before.
This PR roughly fixes several things as below:
- Fix unrecognisable class and method links in javadoc by changing it from
`[[..]]` to `` `...` ``
```
[error]
.../spark/sql/core/target/java/org/apache/spark/sql/streaming/DataStreamReader.java:226:
error: reference not found
[error]* Loads text files and returns a {@link DataFrame} whose
schema starts with a string column named
[error]^
```
- Fix an exception annotation and remove code backticks in `@throws`
annotation
Currently, sbt unidoc with Java 8 complains as below:
```
[error] .../java/org/apache/spark/sql/streaming/StreamingQuery.java:72:
error: unexpected text
[error]* @throws StreamingQueryException, if this query
has terminated with an exception.
```
`@throws` should specify the correct class name from
`StreamingQueryException,` to `StreamingQueryException` without backticks. (see
[JDK-8007644](https://bugs.openjdk.java.net/browse/JDK-8007644)).
- Fix `[[http..]]` to ``.
```diff
- *
[[https://blogs.oracle.com/java-platform-group/entry/diagnosing_tls_ssl_and_https
Oracle
- * blog page]].
+ * https://blogs.oracle.com/java-platform-group/entry/diagnosing_tls_ssl_and_https";>
+ * Oracle blog page.
```
`[[http...]]` link markdown in scaladoc is unrecognisable in javadoc.
- It seems class can't have `@return` annotation. So, two cases of this
were removed.
```
[error]
.../java/org/apache/spark/mllib/regression/IsotonicRegression.java:27: error:
invalid use of @return
[error]* @return New instance of IsotonicRegression.
```
- Fix < to `<` > to `>` according to HTML rules.
- Fix `` complaint
- Exclude unrecognisable in javadoc, `@constructor`, `@todo` and
`@groupname`.
## How was this patch tested?
Manually tested by `jekyll build` with Java 7 and 8
```
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
```
```
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
```
Note: this does not yet make sbt unidoc suceed with Java 8 yet but it
reduces the number of errors with Java 8.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-3359-errors
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15999.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15999
commit 405ed5827966beec4e3d13bb57ddc062695b387e
Author: hyukjinkwon
Date: 2016-11-21T07:14:16Z
Fix errors first round
commit 937de1915c98128e905ca471d66d99b97af3d3ae
Author: hyukjinkwon
Date: 2016-11-23T05:25:54Z
Add throw annotations
commit 2c74b056e1b7e4746a9735a790e68fa6e286733f
Author: hyukjinkwon
Date: 2016-11-23T08:20:02Z
Second fix
commit 00bdc0ed62488bbcb2b83dd3b96701bf72fd2330
Author: hyukjinkwon
Date: 2016-11-23T09:45:00Z
Third fix
commit 4b4cb9b27d6e296a28e233e3e44923b246dd9358
Author: hyukjinkwon
Date: 2016-11-23T09:59:21Z
Remove empty p element
commit 69f661575078c24f29e2b708228b41f0b15505b8
Author: hyukjinkwon
Date: 2016-11-23T13:04:35Z
Fix wrongly formatted http URLs
commit c30ddfa4bafbeccf1eee0d87cffd1d6362a35f4a
Author: hyukjinkwon
Date: 2016-11-23T15:16:57Z
Fix hyperlinks
commit 831e8e4656720aab0a8662ff70af8c9bc83dea1c
Author: hyukjinkwon
Date: 2016-11-24T02:03:36Z
Fix typos and missed ones
commit cfce0e89a52bb96d38b7132f7d0bd45331d933e3
Author: hyukjinkwon
Date: 2016-11-24T02:45:50Z
Fix wrongly printed code block
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69101/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15994 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15994 **[Test build #69101 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69101/consoleFull)** for PR 15994 at commit [`6cb667e`](https://github.com/apache/spark/commit/6cb667ee99a9d09fdbb27e3caf94926bc79e8474). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15990: [SPARK-18559] [SQL] Fix HLL++ with small relative error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15990 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15990: [SPARK-18559] [SQL] Fix HLL++ with small relative error
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15990 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69099/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15990: [SPARK-18559] [SQL] Fix HLL++ with small relative error
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15990 **[Test build #69099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69099/consoleFull)** for PR 15990 at commit [`39420f9`](https://github.com/apache/spark/commit/39420f9c895f6c4c09f6db10c55ecd76c3f7eb35). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15966 **[Test build #69104 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69104/consoleFull)** for PR 15966 at commit [`f9db374`](https://github.com/apache/spark/commit/f9db374d7c90f8d8d7875a13cc17644821779e88). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69098/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #69098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69098/consoleFull)** for PR 15915 at commit [`5210ce3`](https://github.com/apache/spark/commit/5210ce3303af77cd626400c49867bfc908d813db). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15974: [SPARK-18537] [Web UI]Add a REST api to spark streaming
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/15974 @ChorPangChan we will not merge new features into a maintenance branch. You need to submit a new PR against master. You can't change this one to be against master, you have to open a new one. Please close this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/15915 Haven't forgotten about this; it's on my list to review post-Thanksgiving. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15974: [SPARK-18537] [Web UI]Add a REST api to spark streaming
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15974 OK --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15949 **[Test build #69103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69103/consoleFull)** for PR 15949 at commit [`4df5675`](https://github.com/apache/spark/commit/4df56754b45247d73ec2e560d012f6b47a1d4853). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15986 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69095/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15986 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15986: [SPARK-18553][CORE][branch-2.0] Fix leak of TaskSetManag...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15986 **[Test build #69095 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69095/consoleFull)** for PR 15986 at commit [`26ca80d`](https://github.com/apache/spark/commit/26ca80d22f873c0d37cbb3892fc9c4de196f1d12). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15949 **[Test build #69102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69102/consoleFull)** for PR 15949 at commit [`3b403bd`](https://github.com/apache/spark/commit/3b403bd4377388c74fd1ff62282d57200eb094e8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15949 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69097/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15949 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15949 **[Test build #69097 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69097/consoleFull)** for PR 15949 at commit [`13d1367`](https://github.com/apache/spark/commit/13d1367fd6ac553d7692298b62823e94553a32f3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15949 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69096/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15949 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15949 **[Test build #69096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69096/consoleFull)** for PR 15949 at commit [`1547b63`](https://github.com/apache/spark/commit/1547b63b56199a3b0c7a221dc57b3a88577e3eb0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15993: [SPARK-18050][SQL] do not create default database...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15993#discussion_r89429181
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ---
@@ -92,8 +92,12 @@ private[sql] class SharedState(val sparkContext:
SparkContext) extends Logging {
{
val defaultDbDefinition = CatalogDatabase(
SessionCatalog.DEFAULT_DATABASE, "default database", warehousePath,
Map())
-// Initialize default database if it doesn't already exist
-externalCatalog.createDatabase(defaultDbDefinition, ignoreIfExists =
true)
+// Initialize default database if it doesn't exist
+if (!externalCatalog.databaseExists(SessionCatalog.DEFAULT_DATABASE)) {
--- End diff --
if they create database with `IF NOT EXISTS`, yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15993: [SPARK-18050][SQL] do not create default database...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15993#discussion_r89428849
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ---
@@ -92,8 +92,12 @@ private[sql] class SharedState(val sparkContext:
SparkContext) extends Logging {
{
val defaultDbDefinition = CatalogDatabase(
SessionCatalog.DEFAULT_DATABASE, "default database", warehousePath,
Map())
-// Initialize default database if it doesn't already exist
-externalCatalog.createDatabase(defaultDbDefinition, ignoreIfExists =
true)
+// Initialize default database if it doesn't exist
+if (!externalCatalog.databaseExists(SessionCatalog.DEFAULT_DATABASE)) {
--- End diff --
The users will still see the above error message, if the database already
exists, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15949 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69094/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15974: [SPARK-18537] [Web UI]Add a REST api to spark streaming
Github user ChorPangChan commented on the issue: https://github.com/apache/spark/pull/15974 hey @uncleGen I think we need to settle this down in order to make progress. I had a quick look in your code, as i said above, modifying spark-core is no a good idea to implement the streaming REST api because user can use spark without streaming at least as a spark user, I don't want something not related to spark-core inside it. if you already come up a even better design, then it will be another story, but for now, I think my implementation is the way to go how do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15949 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15949: [SPARK-18339] [SPARK-18513] [SQL] Don't push down curren...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15949 **[Test build #69094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69094/consoleFull)** for PR 15949 at commit [`baafe4a`](https://github.com/apache/spark/commit/baafe4ad9d49d78f8f96eccf07642f456fe21635). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15994: [SPARK-18555][SQL][WIP]DataFrameNaFunctions.fill miss up...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15994 **[Test build #69101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69101/consoleFull)** for PR 15994 at commit [`6cb667e`](https://github.com/apache/spark/commit/6cb667ee99a9d09fdbb27e3caf94926bc79e8474). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15620: [SPARK-18091] [SQL] Deep if expressions cause Generated ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15620 can you try sbt? The jenkins run tests using sbt instead of maven --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15993: [SPARK-18050][SQL] do not create default database...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15993#discussion_r89426408
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ---
@@ -92,8 +92,12 @@ private[sql] class SharedState(val sparkContext:
SparkContext) extends Logging {
{
val defaultDbDefinition = CatalogDatabase(
SessionCatalog.DEFAULT_DATABASE, "default database", warehousePath,
Map())
-// Initialize default database if it doesn't already exist
-externalCatalog.createDatabase(defaultDbDefinition, ignoreIfExists =
true)
+// Initialize default database if it doesn't exist
+if (!externalCatalog.databaseExists(SessionCatalog.DEFAULT_DATABASE)) {
--- End diff --
checking existence and then creating database is not atomic. It's ok here
because we can't drop default database, but it may not work for general
`createDatabase`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15544: [SPARK-17997] [SQL] Add an aggregation function for coun...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/15544 @rxin @hvanhovell @cloud-fan Do you think this is a useful aggregate function? It used to compute ndv for bins in a histogram, but we are not using histogram for now. It still can be used to compute ndv for a sequence of intervals. If you think it's useful, I'll update it, otherwise I'll close it. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15971: [SPARK-18535][UI][YARN] Redact sensitive information fro...
Github user markgrover commented on the issue: https://github.com/apache/spark/pull/15971 Ok, looks like all is good now! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15971: [SPARK-18535][UI][YARN] Redact sensitive information fro...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15971 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
