[GitHub] spark pull request #16068: [SPARK-18637][SQL]Stateful UDF should be consider...

2016-12-05 Thread zhzhan
Github user zhzhan commented on a diff in the pull request: https://github.com/apache/spark/pull/16068#discussion_r91026585 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala --- @@ -487,6 +488,52 @@ class HiveUDFSuite extends QueryTest with

[GitHub] spark pull request #16068: [SPARK-18637][SQL]Stateful UDF should be consider...

2016-12-05 Thread zhzhan
Github user zhzhan commented on a diff in the pull request: https://github.com/apache/spark/pull/16068#discussion_r91026433 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala --- @@ -487,6 +488,52 @@ class HiveUDFSuite extends QueryTest with

[GitHub] spark issue #16161: [SPARK-18717][SQL] Make code generation for Scala Map wo...

2016-12-05 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16161 cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #16103: [SPARK-18374][ML]Incorrect words in StopWords/english.tx...

2016-12-05 Thread hhbyyh
Github user hhbyyh commented on the issue: https://github.com/apache/spark/pull/16103 Thanks for the review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16167: [DO NOT MERGE]Remove workaround for Netty memory leak

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16167 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69711/ Test PASSed. ---

[GitHub] spark issue #16167: [DO NOT MERGE]Remove workaround for Netty memory leak

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16167 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16167: [DO NOT MERGE]Remove workaround for Netty memory leak

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16167 **[Test build #69711 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69711/consoleFull)** for PR 16167 at commit

[GitHub] spark issue #15994: [SPARK-18555][SQL]DataFrameNaFunctions.fill miss up orig...

2016-12-05 Thread windpiger
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/15994 ok ,thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark pull request #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial...

2016-12-05 Thread sethah
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/16149#discussion_r91021502 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -479,7 +479,12 @@ object GeneralizedLinearRegression

[GitHub] spark pull request #16068: [SPARK-18637][SQL]Stateful UDF should be consider...

2016-12-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16068#discussion_r91020060 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala --- @@ -487,6 +488,52 @@ class HiveUDFSuite extends QueryTest

[GitHub] spark issue #16166: [SPARK-18734][SS] Represent timestamp in StreamingQueryP...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16166 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69710/ Test FAILed. ---

[GitHub] spark issue #16166: [SPARK-18734][SS] Represent timestamp in StreamingQueryP...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16166 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16166: [SPARK-18734][SS] Represent timestamp in StreamingQueryP...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16166 **[Test build #69710 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69710/consoleFull)** for PR 16166 at commit

[GitHub] spark issue #16068: [SPARK-18637][SQL]Stateful UDF should be considered as n...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16068 **[Test build #69716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69716/consoleFull)** for PR 16068 at commit

[GitHub] spark issue #16068: [SPARK-18637][SQL]Stateful UDF should be considered as n...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16068 **[Test build #69715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69715/consoleFull)** for PR 16068 at commit

[GitHub] spark issue #16128: [SPARK-18671][SS][TEST] Added tests to ensure stability ...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16128 **[Test build #69714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69714/consoleFull)** for PR 16128 at commit

[GitHub] spark issue #16129: [SPARK-18678][ML] Skewed feature subsampling in Random f...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16129 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16129: [SPARK-18678][ML] Skewed feature subsampling in Random f...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16129 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69709/ Test PASSed. ---

[GitHub] spark issue #16131: [SPARK-18701][ML] Fix Poisson GLM failure due to wrong i...

2016-12-05 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16131 @srowen Done. Thanks for the suggestion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #16137: [SPARK-18708][CORE] Improvement/improve docs in spark co...

2016-12-05 Thread Mironor
Github user Mironor commented on the issue: https://github.com/apache/spark/pull/16137 @srowen I reversed obvious comments as well as some minor changes (such as capitalizing). I only left javadoc for some of the non-trivial public api. I can reverse changes for comments where the

[GitHub] spark issue #16129: [SPARK-18678][ML] Skewed feature subsampling in Random f...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16129 **[Test build #69709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69709/consoleFull)** for PR 16129 at commit

[GitHub] spark issue #16128: [SPARK-18671][SS][TEST] Added tests to ensure stability ...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16128 **[Test build #3468 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3468/consoleFull)** for PR 16128 at commit

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread Mironor
Github user Mironor commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91015596 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1144,13 +1218,19 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91015460 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1144,13 +1218,19 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16138 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69713/ Test FAILed. ---

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16138 **[Test build #69713 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69713/consoleFull)** for PR 16138 at commit

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16138 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread ChorPangChan
Github user ChorPangChan commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r91014959 --- Diff: streaming/src/main/java/org/apache/spark/streaming/status/api/v1/BatchStatus.java --- @@ -0,0 +1,30 @@ +/* + * Licensed to the

[GitHub] spark issue #16163: [SPARK-18730] Post Jenkins test report page instead of t...

2016-12-05 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/16163 @srowen Thanks. I sent this one because the `consoleFull` page frequently freezes my browser these days, not mentioning viewing Jenkins build results via mobile phone... --- If your project is

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16138 **[Test build #69713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69713/consoleFull)** for PR 16138 at commit

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69712/consoleFull)** for PR 16014 at commit

[GitHub] spark issue #16165: [SPARK-18733] [WEBUI] HistoryServer: Add config option t...

2016-12-05 Thread seyfe
Github user seyfe commented on the issue: https://github.com/apache/spark/pull/16165 Hi @srowen, Thanks for the quick feedback. Let me get rid of the on/off knob for inprogress files. Would you like me to remove the maxAge setting for inprogress files as well? I

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread Mironor
Github user Mironor commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91013560 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1144,13 +1218,19 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark issue #16167: [DO NOT MERGE]Remove workaround for Netty memory leak

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16167 **[Test build #69711 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69711/consoleFull)** for PR 16167 at commit

[GitHub] spark issue #16128: [SPARK-18671][SS][TEST] Added tests to ensure stability ...

2016-12-05 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16128 LGTM pending tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark pull request #16167: [DO NOT MERGE]Remove workaround for Netty memory ...

2016-12-05 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/16167 [DO NOT MERGE]Remove workaround for Netty memory leak ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch

[GitHub] spark issue #16165: [SPARK-18733] [WEBUI] HistoryServer: Add config option t...

2016-12-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16165 I don't think it makes sense to expose yet another set of settings for this. I think the risk of course is that this accidentally cleans up another instance's work in progress. However if

[GitHub] spark issue #16166: [SPARK-18734][SS] Represent timestamp in StreamingQueryP...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16166 **[Test build #69710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69710/consoleFull)** for PR 16166 at commit

[GitHub] spark pull request #16166: [SPARK-18734][SS] Represent timestamp in Streamin...

2016-12-05 Thread tdas
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/16166 [SPARK-18734][SS] Represent timestamp in StreamingQueryProgress as formatted string instead of millis ## What changes were proposed in this pull request? Easier to read while debugging as a

[GitHub] spark pull request #16165: [SPARK-18733] [WEBUI] HistoryServer: Add config o...

2016-12-05 Thread seyfe
GitHub user seyfe opened a pull request: https://github.com/apache/spark/pull/16165 [SPARK-18733] [WEBUI] HistoryServer: Add config option to cleanup in-progress files ## What changes were proposed in this pull request? Add 2 new config parameters 1)

[GitHub] spark issue #16165: [SPARK-18733] [WEBUI] HistoryServer: Add config option t...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16165 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #16148: [SPARK-18325][SparkR][ML] SparkR ML wrappers exam...

2016-12-05 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16148#discussion_r91011837 --- Diff: examples/src/main/r/ml/lda.R --- @@ -0,0 +1,46 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +#

[GitHub] spark issue #16148: [SPARK-18325][SparkR][ML] SparkR ML wrappers example cod...

2016-12-05 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16148 this is great, thanks! btw, how are these examples getting run? is there a way to know if the examples are broken because of API changes? --- If your project is set up for it, you can reply to

[GitHub] spark pull request #16148: [SPARK-18325][SparkR][ML] SparkR ML wrappers exam...

2016-12-05 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16148#discussion_r91011772 --- Diff: examples/src/main/r/ml/randomForest.R --- @@ -0,0 +1,63 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] spark pull request #16148: [SPARK-18325][SparkR][ML] SparkR ML wrappers exam...

2016-12-05 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16148#discussion_r91011734 --- Diff: examples/src/main/r/ml/randomForest.R --- @@ -0,0 +1,63 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] spark pull request #16148: [SPARK-18325][SparkR][ML] SparkR ML wrappers exam...

2016-12-05 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16148#discussion_r91011578 --- Diff: docs/sparkr.md --- @@ -512,39 +512,33 @@ head(teenagers) # Machine Learning -SparkR supports the following machine

[GitHub] spark pull request #16148: [SPARK-18325][SparkR][ML] SparkR ML wrappers exam...

2016-12-05 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16148#discussion_r91011559 --- Diff: docs/sparkr.md --- @@ -512,39 +512,33 @@ head(teenagers) # Machine Learning -SparkR supports the following machine

[GitHub] spark issue #16150: [SPARK-18349][SparkR]:Update R API documentation on ml m...

2016-12-05 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16150 there is also this form `\code{apriori} (the label distribution)` and this form `\item{\code{docConcentration}}{concentration parameter commonly named \code{alpha} ` --- If

[GitHub] spark issue #16150: [SPARK-18349][SparkR]:Update R API documentation on ml m...

2016-12-05 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16150 thanks, there is also the issue with `\code{numOfInputs}` vs `number of iterations IRLS takes` - should it be a "variable" (and thus wrapped with `\code{something}` - or should it be a

[GitHub] spark pull request #16160: [SPARK-18721][SS]Fix ForeachSink with watermark +...

2016-12-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16160 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #16164: [SPARK-18732][WEB-UI] The Y axis ranges of "schedulingDe...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16164 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16164: [SPARK-18732][WEB-UI] The Y axis ranges of "schedulingDe...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16164 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69708/ Test PASSed. ---

[GitHub] spark issue #16164: [SPARK-18732][WEB-UI] The Y axis ranges of "schedulingDe...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16164 **[Test build #69708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69708/consoleFull)** for PR 16164 at commit

[GitHub] spark issue #16159: [SPARK-18697][BUILD] Upgrade sbt plugins

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16159 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16159: [SPARK-18697][BUILD] Upgrade sbt plugins

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16159 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69704/ Test PASSed. ---

[GitHub] spark issue #16159: [SPARK-18697][BUILD] Upgrade sbt plugins

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16159 **[Test build #69704 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69704/consoleFull)** for PR 16159 at commit

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91010366 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1417,27 +1551,31 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91010077 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1417,27 +1551,31 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91010001 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1401,8 +1532,11 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #16098: [SPARK-18672][CORE] Close recordwriter in SparkHa...

2016-12-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16098 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #16160: [SPARK-18721][SS]Fix ForeachSink with watermark + append

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16160 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16160: [SPARK-18721][SS]Fix ForeachSink with watermark + append

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16160 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69706/ Test PASSed. ---

[GitHub] spark issue #16160: [SPARK-18721][SS]Fix ForeachSink with watermark + append

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16160 **[Test build #69706 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69706/consoleFull)** for PR 16160 at commit

[GitHub] spark issue #16163: [SPARK-18730] Post Jenkins test report page instead of t...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16163 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16098: [SPARK-18672][CORE] Close recordwriter in SparkHadoopMap...

2016-12-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16098 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #16163: [SPARK-18730] Post Jenkins test report page instead of t...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16163 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69703/ Test PASSed. ---

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91009699 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1620,8 +1766,9 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread Mironor
Github user Mironor commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91009631 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1417,27 +1551,31 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark issue #16163: [SPARK-18730] Post Jenkins test report page instead of t...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16163 **[Test build #69703 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69703/testReport)** for PR 16163 at commit

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread Mironor
Github user Mironor commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91009155 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1401,8 +1532,11 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark issue #16139: [SPARK-18705][ML][DOC] Update user guide to reflect one ...

2016-12-05 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/16139 ping @yanboliang --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91008727 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -119,22 +119,22 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91008621 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -119,22 +119,22 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91008657 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -119,22 +119,22 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91008467 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -923,15 +971,13 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #16137: [SPARK-18708][CORE] Improvement/improve docs in s...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16137#discussion_r91008183 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1417,27 +1551,31 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark issue #16128: [SPARK-18671][SS][TEST] Added tests to ensure stability ...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16128 **[Test build #3468 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3468/consoleFull)** for PR 16128 at commit

[GitHub] spark issue #16142: [SPARK-18716][CORE] Restrict the disk usage of spark eve...

2016-12-05 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/16142 @srowen If I have understand what you mean correctly, the **"log rotation"** is different with **"job event log clean up"**. The "job event log" is used to reply to build spark history ui.

[GitHub] spark issue #16128: [SPARK-18671][SS][TEST] Added tests to ensure stability ...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16128 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16128: [SPARK-18671][SS][TEST] Added tests to ensure stability ...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16128 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69705/ Test FAILed. ---

[GitHub] spark issue #16128: [SPARK-18671][SS][TEST] Added tests to ensure stability ...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16128 **[Test build #69705 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69705/consoleFull)** for PR 16128 at commit

[GitHub] spark pull request #16159: [SPARK-18697][BUILD] Upgrade sbt plugins

2016-12-05 Thread weiqingy
Github user weiqingy commented on a diff in the pull request: https://github.com/apache/spark/pull/16159#discussion_r91006592 --- Diff: project/SparkBuild.scala --- @@ -596,19 +596,17 @@ object Hive { } object Assembly { - import sbtassembly.AssemblyUtils._

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

2016-12-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15998 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

2016-12-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15998 thanks, merging to master/2.1! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

2016-12-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15998#discussion_r91006319 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala --- @@ -346,6 +346,31 @@ abstract class

[GitHub] spark issue #16164: [SPARK-18732][WEB-UI] The Y axis ranges of "schedulingDe...

2016-12-05 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/16164 @srowen Indeed, it is not a normal case. And I found this problem when the streaming job went wrong. As you said > one can compare the graphs visually. It still may mislead users

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

2016-12-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15998#discussion_r91006034 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala --- @@ -346,6 +346,31 @@ abstract class

[GitHub] spark pull request #16037: [SPARK-18471][MLLIB] In LBFGS, avoid sending huge...

2016-12-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16037#discussion_r91005791 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala --- @@ -241,16 +241,27 @@ object LBFGS extends Logging { val bcW =

[GitHub] spark issue #16142: [SPARK-18716][CORE] Restrict the disk usage of spark eve...

2016-12-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16142 Yes, but the alternative is reimplementing an ad-hoc log rotation system here, which isn't great either. Are you saying the history server already manages logs? pardon, I don't know it at all. ---

[GitHub] spark issue #16142: [SPARK-18716][CORE] Restrict the disk usage of spark eve...

2016-12-05 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/16142 @srowen Spark History Server may do the clean-up work. The precondition is we start it and it keeps running. Besides, if there are abundant applications constantly, the event log may still take up

[GitHub] spark issue #16146: [SPARK-18091] [SQL] [BACKPORT-1.6] Deep if expressions c...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16146 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69707/ Test FAILed. ---

[GitHub] spark issue #16146: [SPARK-18091] [SQL] [BACKPORT-1.6] Deep if expressions c...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16146 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16146: [SPARK-18091] [SQL] [BACKPORT-1.6] Deep if expressions c...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16146 **[Test build #69707 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69707/consoleFull)** for PR 16146 at commit

[GitHub] spark issue #16129: [SPARK-18678][ML] Skewed feature subsampling in Random f...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16129 **[Test build #69709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69709/consoleFull)** for PR 16129 at commit

[GitHub] spark issue #16164: [SPARK-18732][WEB-UI] The Y axis ranges of "schedulingDe...

2016-12-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16164 CC @zsxwing because it works this way on purpose, so that one can compare the graphs visually. Usually these values aren't too different in scale; it's a problem here because scheduling delay is

[GitHub] spark issue #16147: [SPARK-18718][TESTS] Skip some test failures due to path...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16147 Just FYI, I ran some more tests for each package for myself and grepped `local-cluster` before submitting this PR and It seems there are not many same instances. If I face the same problems

[GitHub] spark issue #16164: [SPARK-18732][WEB-UI] The Y axis ranges of "schedulingDe...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16164 **[Test build #69708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69708/consoleFull)** for PR 16164 at commit

[GitHub] spark issue #16142: [SPARK-18716][CORE] Restrict the disk usage of spark eve...

2016-12-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16142 Hm, does Spark generally manage log rotation? I confess ignorance. It's something you often leave to a cron job or something to archive and clean up. --- If your project is set up for it, you can

[GitHub] spark pull request #16128: [SPARK-18671][SS][TEST] Added tests to ensure sta...

2016-12-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/16128#discussion_r91003428 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala --- @@ -1022,6 +1021,33 @@ class FileStreamSourceSuite extends

[GitHub] spark pull request #16128: [SPARK-18671][SS][TEST] Added tests to ensure sta...

2016-12-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/16128#discussion_r91003243 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala --- @@ -81,7 +81,14 @@ private object JsonUtils { */

  1   2   3   4   5   6   >