[GitHub] spark issue #16131: [SPARK-18701][ML] Fix Poisson GLM failure due to wrong i...

2016-12-05 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16131 @sethah Thanks for the review. I have updated according to your suggestion. @yanboliang @srowen Please take another look. Thanks. --- If your project is set up for it, you

[GitHub] spark issue #16140: [SPARK-18714][SQL] Add a simple time function to SparkSe...

2016-12-05 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16140 The cost to maintain this seems very small though, and I'd definitely use it all the time in the repl. In Databricks this is not an issue since the environment always appends the time, but I really

[GitHub] spark pull request #16131: [SPARK-18701][ML] Fix Poisson GLM failure due to ...

2016-12-05 Thread sethah
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/16131#discussion_r90986964 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -505,7 +505,8 @@ object GeneralizedLinearRegression

[GitHub] spark pull request #16131: [SPARK-18701][ML] Fix Poisson GLM failure due to ...

2016-12-05 Thread sethah
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/16131#discussion_r90986789 --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala --- @@ -497,6 +500,7 @@ class

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16156 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16156 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69688/ Test PASSed. ---

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16156 **[Test build #69688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69688/consoleFull)** for PR 16156 at commit

[GitHub] spark issue #15722: [SPARK-18208] [Shuffle] Executor OOM due to a growing Lo...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15722 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15722: [SPARK-18208] [Shuffle] Executor OOM due to a growing Lo...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15722 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69689/ Test FAILed. ---

[GitHub] spark issue #15722: [SPARK-18208] [Shuffle] Executor OOM due to a growing Lo...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15722 **[Test build #69689 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69689/consoleFull)** for PR 15722 at commit

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-05 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/16030 I'm checking whether the original behavior is consistent (do we always respect data schema column order when partition columns are included in data schema?). If not, I call it a bug and we just

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-05 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/16030 yeah, that's what worries me. but does that merit keeping bad inconsistent behavior forever? Maybe a dev list question? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #16155: [SPARK-18722][SS]Move no data rate limit from StreamExec...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16155 **[Test build #69700 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69700/consoleFull)** for PR 16155 at commit

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-05 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/16030 @brkyvz I agree that always moving all partitioned columns to the end of the schema is more consistent and intuitive. However, users may have ordinal-dependent code like this: ```scala

[GitHub] spark issue #16155: [SPARK-18722][SS]Move no data rate limit from StreamExec...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16155 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69686/ Test PASSed. ---

[GitHub] spark issue #16155: [SPARK-18722][SS]Move no data rate limit from StreamExec...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16155 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16155: [SPARK-18722][SS]Move no data rate limit from StreamExec...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16155 **[Test build #69686 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69686/consoleFull)** for PR 16155 at commit

[GitHub] spark issue #16162: [SPARK-18729][SS]Move DataFrame.collect out of synchroni...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16162 **[Test build #69699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69699/consoleFull)** for PR 16162 at commit

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-05 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/16030 Even if it is a bug or not, I think we should have a consistent story for all of these. It's weird that the behavior is different for CatalogTables, for Vectorized and non-vectorized parquet reader,

[GitHub] spark pull request #16162: [SPARK-18729][SS]Move DataFrame.collect out of sy...

2016-12-05 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/16162 [SPARK-18729][SS]Move DataFrame.collect out of synchronized block in MemorySink ## What changes were proposed in this pull request? Move DataFrame.collect out of synchronized block so

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16156 @liancheng Ah, thank you. I should have tested this first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16156 Would there be another way to avoid try-catch? I think it is a normal reading path logic and it seems it might not be safe to rely on exception handling. --- If your project is set up for

[GitHub] spark issue #16160: [SPARK-18721][SS]Fix ForeachSink with watermark + append

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16160 **[Test build #69698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69698/consoleFull)** for PR 16160 at commit

[GitHub] spark issue #16158: [SPARK-18724][ML] Add TuningSummary for TrainValidationS...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16158 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69690/ Test PASSed. ---

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90982394 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java --- @@ -107,7 +107,16 @@

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/16156 Hey @xwu0226 @gatorsmile, did some investigation, and I don't think this is a bug now. Please refer to [my JIRA comment][1] for more details. [1]:

[GitHub] spark issue #16158: [SPARK-18724][ML] Add TuningSummary for TrainValidationS...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16158 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16158: [SPARK-18724][ML] Add TuningSummary for TrainValidationS...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16158 **[Test build #69690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69690/consoleFull)** for PR 16158 at commit

[GitHub] spark pull request #16160: [SPARK-18721][SS]Fix ForeachSink with watermark +...

2016-12-05 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/16160#discussion_r90982137 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ForeachSinkSuite.scala --- @@ -204,6 +204,55 @@ class ForeachSinkSuite extends

[GitHub] spark issue #16154: [WIP] [SPARK-17822] Make JVMObjectTracker a member varia...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16154 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69684/ Test PASSed. ---

[GitHub] spark issue #16154: [WIP] [SPARK-17822] Make JVMObjectTracker a member varia...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16154 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16154: [WIP] [SPARK-17822] Make JVMObjectTracker a member varia...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16154 **[Test build #69684 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69684/consoleFull)** for PR 16154 at commit

[GitHub] spark issue #16068: [SPARK-18637][SQL]Stateful UDF should be considered as n...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16068 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16068: [SPARK-18637][SQL]Stateful UDF should be considered as n...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16068 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69687/ Test PASSed. ---

[GitHub] spark issue #16068: [SPARK-18637][SQL]Stateful UDF should be considered as n...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16068 **[Test build #69687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69687/consoleFull)** for PR 16068 at commit

[GitHub] spark pull request #16160: [SPARK-18721][SS]Fix ForeachSink with watermark +...

2016-12-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/16160#discussion_r90980149 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ForeachSinkSuite.scala --- @@ -204,6 +204,55 @@ class ForeachSinkSuite extends

[GitHub] spark pull request #16160: [SPARK-18721][SS]Fix ForeachSink with watermark +...

2016-12-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/16160#discussion_r90979980 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ForeachSinkSuite.scala --- @@ -204,6 +204,55 @@ class ForeachSinkSuite extends

[GitHub] spark issue #14789: [SPARK-17209][YARN] Add the ability to manually update c...

2016-12-05 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/14789 bq. but I think the api makes sense to be very similar, or at least in the same sort of class I think it will be hard to have the same API serve both use cases. You could have one API to

[GitHub] spark issue #16155: [SPARK-18722][SS]Move no data rate limit from StreamExec...

2016-12-05 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/16155 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark pull request #16160: [SPARK-18721][SS]Fix ForeachSink with watermark +...

2016-12-05 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/16160#discussion_r90979548 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ForeachSink.scala --- @@ -32,46 +31,26 @@ import

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread xwu0226
Github user xwu0226 commented on the issue: https://github.com/apache/spark/pull/16156 For normal parquet reader case, we have the following code ```Scala } else { logDebug(s"Falling back to parquet-mr") // ParquetRecordReader returns UnsafeRow

[GitHub] spark issue #16160: [SPARK-18721][SS]Fix ForeachSink with watermark + append

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16160 **[Test build #69697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69697/consoleFull)** for PR 16160 at commit

[GitHub] spark issue #16161: [SPARK-18717][SQL] Make code generation for Scala Map wo...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16161 **[Test build #69696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69696/consoleFull)** for PR 16161 at commit

[GitHub] spark pull request #16161: [SPARK-18717][SQL] Make code generation for Scala...

2016-12-05 Thread aray
GitHub user aray opened a pull request: https://github.com/apache/spark/pull/16161 [SPARK-18717][SQL] Make code generation for Scala Map work with immutable.Map also ## What changes were proposed in this pull request? Fixes compile errors in generated code when user has

[GitHub] spark pull request #16160: [SPARK-18721][SS]Fix ForeachSink with watermark +...

2016-12-05 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/16160 [SPARK-18721][SS]Fix ForeachSink with watermark + append ## What changes were proposed in this pull request? Right now ForeachSink creates a new physical plan, so StreamExecution cannot

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread xwu0226
Github user xwu0226 commented on the issue: https://github.com/apache/spark/pull/16156 @liancheng I see. In normal parquet reader, ParquetFileFormat is using hadoop's `ParquetRecordReader`, which we can not add such toleration code. --- If your project is set up for it, you can

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread ChorPangChan
Github user ChorPangChan commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90976630 --- Diff: streaming/src/main/java/org/apache/spark/streaming/status/api/v1/BatchStatus.java --- @@ -0,0 +1,30 @@ +/* + * Licensed to the

[GitHub] spark issue #16155: [SPARK-18722][SS]Move no data rate limit from StreamExec...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16155 **[Test build #69694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69694/consoleFull)** for PR 16155 at commit

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90977399 --- Diff: streaming/src/main/java/org/apache/spark/streaming/status/api/v1/BatchStatus.java --- @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #16113: [SPARK-18657][SPARK-18668] Make StreamingQuery.id persis...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16113 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69693/ Test FAILed. ---

[GitHub] spark issue #16113: [SPARK-18657][SPARK-18668] Make StreamingQuery.id persis...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16113 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16113: [SPARK-18657][SPARK-18668] Make StreamingQuery.id persis...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16113 **[Test build #69695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69695/consoleFull)** for PR 16113 at commit

[GitHub] spark issue #11692: [SPARK-13852][YARN]handle the InterruptedException cause...

2016-12-05 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/11692 Wow this is old. @WangTaoTheTonic I'm not sure this is the right fix. The code in YarnClientSchedulerBackend is already catching InterruptedException: ``` try { val

[GitHub] spark pull request #16153: [SPARK-18694][SS] Add StreamingQuery.explain and ...

2016-12-05 Thread zsxwing
Github user zsxwing closed the pull request at: https://github.com/apache/spark/pull/16153 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #16153: [SPARK-18694][SS] Add StreamingQuery.explain and excepti...

2016-12-05 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16153 Merging to 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #16150: [SPARK-18349][SparkR]:Update R API documentation on ml m...

2016-12-05 Thread wangmiao1981
Github user wangmiao1981 commented on the issue: https://github.com/apache/spark/pull/16150 cc @felixcheung --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16113: [SPARK-18657][SPARK-18668] Make StreamingQuery.id persis...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16113 **[Test build #69693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69693/consoleFull)** for PR 16113 at commit

[GitHub] spark pull request #16157: [SPARK-18723][DOC] Expanded programming guide inf...

2016-12-05 Thread michalsenkyr
Github user michalsenkyr commented on a diff in the pull request: https://github.com/apache/spark/pull/16157#discussion_r90975179 --- Diff: docs/programming-guide.md --- @@ -347,7 +347,7 @@ Some notes on reading files with Spark: Apart from text files, Spark's Scala API

[GitHub] spark issue #16153: [SPARK-18694][SS] Add StreamingQuery.explain and excepti...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16153 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16153: [SPARK-18694][SS] Add StreamingQuery.explain and excepti...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16153 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69682/ Test PASSed. ---

[GitHub] spark issue #16153: [SPARK-18694][SS] Add StreamingQuery.explain and excepti...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16153 **[Test build #69682 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69682/consoleFull)** for PR 16153 at commit

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/16156 @xwu0226 Just tested that this issue also affects the normal Parquet reader (by setting `spark.sql.parquet.enableVectorizedReader` to `false`). That's also why #9940 couldn't take a similar

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90974262 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -578,4 +578,66 @@ class

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90974235 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java --- @@ -107,7 +107,16 @@

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90970052 --- Diff: streaming/src/main/java/org/apache/spark/streaming/status/api/v1/BatchStatus.java --- @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90972417 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/status/api/v1/SecurityFilter.scala --- @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90973570 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingJobProgressListener.scala --- @@ -39,6 +39,8 @@ private[streaming] class

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90971231 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/status/api/v1/AllReceiversResource.scala --- @@ -0,0 +1,80 @@ +/* + * Licensed to the

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90970807 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/scheduler/StreamingListener.scala --- @@ -66,6 +69,9 @@ case class

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90970255 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala --- @@ -45,7 +45,7 @@ import org.apache.spark.storage.StorageLevel

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90971580 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/status/api/v1/AllReceiversResource.scala --- @@ -0,0 +1,80 @@ +/* + * Licensed to the

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90971838 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/status/api/v1/JacksonMessageWriter.scala --- @@ -0,0 +1,94 @@ +/* + * Licensed to the

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

2016-12-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16000#discussion_r90973292 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/status/api/v1/StreamingApiRootResource.scala --- @@ -0,0 +1,133 @@ +/* + * Licensed to

[GitHub] spark issue #16151: [SPARK-18719] Add spark.ui.showConsoleProgress to config...

2016-12-05 Thread nchammas
Github user nchammas commented on the issue: https://github.com/apache/spark/pull/16151 @davies - Should this also be cherry-picked into 2.0 and 2.1? I think this config has been there for a while, just without documentation. 😊 --- If your project is set up for it, you

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90973620 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java --- @@ -107,7 +107,16 @@ public

[GitHub] spark issue #16131: [SPARK-18701][ML] Fix Poisson GLM failure due to wrong i...

2016-12-05 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16131 @srowen @yanboliang I have updated the code and further cleaned up the test. Please review and let me know if there is any question. Thanks. --- If your project is set up for it, you

[GitHub] spark pull request #16151: [SPARK-18719] Add spark.ui.showConsoleProgress to...

2016-12-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16151 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90972771 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java --- @@ -107,7 +107,16 @@

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90972713 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -578,4 +578,66 @@ class

[GitHub] spark issue #16151: [SPARK-18719] Add spark.ui.showConsoleProgress to config...

2016-12-05 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/16151 lgtm, merging into master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90972508 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java --- @@ -107,7 +107,16 @@

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90972121 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java --- @@ -107,7 +107,16 @@ public

[GitHub] spark issue #16147: [SPARK-18718][TESTS] Skip some test failures due to path...

2016-12-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16147 It seems some tests are being failed in some cases when running executor as process (e.g. `local-cluster`). In this case, it fails because simply the classpath is too long

[GitHub] spark issue #16159: [SPARK-18697][BUILD] Upgrade sbt plugins

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16159 **[Test build #69692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69692/consoleFull)** for PR 16159 at commit

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90971885 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java --- @@ -107,7 +107,16 @@

[GitHub] spark issue #16159: [SPARK-18697][BUILD] Upgrade sbt plugins

2016-12-05 Thread weiqingy
Github user weiqingy commented on the issue: https://github.com/apache/spark/pull/16159 The references to upgrade sbt-assembly as following: https://github.com/sbt/sbt-assembly/blob/master/Migration.md

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-05 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/16030 @brkyvz I also worry about the behavior change. Let me check whether the original behavior is by design or by accident. If it is a bug from the very beginning, then we should just fix it in this

[GitHub] spark pull request #16159: [SPARK-18697][BUILD] Upgrade sbt plugins

2016-12-05 Thread weiqingy
GitHub user weiqingy opened a pull request: https://github.com/apache/spark/pull/16159 [SPARK-18697][BUILD] Upgrade sbt plugins ## What changes were proposed in this pull request? This PR is to upgrade sbt plugins. The following sbt plugins will be upgraded: ```

[GitHub] spark pull request #16158: [SPARK-18724][ML] Add TuningSummary for TrainVali...

2016-12-05 Thread hhbyyh
Github user hhbyyh commented on a diff in the pull request: https://github.com/apache/spark/pull/16158#discussion_r90970832 --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala --- @@ -226,6 +230,29 @@ class TrainValidationSplitModel private[ml] (

[GitHub] spark pull request #16157: [SPARK-18723][DOC] Expanded programming guide inf...

2016-12-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16157#discussion_r90970609 --- Diff: docs/programming-guide.md --- @@ -347,7 +347,7 @@ Some notes on reading files with Spark: Apart from text files, Spark's Scala API also

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90970607 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -578,4 +578,66 @@ class

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/16156 BTW, I think this PR is a cleaner fix than #9940, which introduces a temporary metadata while merging two `StructType`s and erased it in a later phase. We may want to remove the hack done in

[GitHub] spark issue #16158: [SPARK-18724][ML] Add TuningSummary for TrainValidationS...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16158 **[Test build #69690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69690/consoleFull)** for PR 16158 at commit

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

2016-12-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15998 **[Test build #69691 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69691/consoleFull)** for PR 15998 at commit

[GitHub] spark issue #16156: [SPARK-18539][SQL]: tolerate pushed-down filter on non-e...

2016-12-05 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/16156 Actually, PR #9940 should have already fixed this issue. I'm checking why it doesn't work under 2.0.1 for 2.0.2. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #16158: [SPARK-18724][ML] Add TuningSummary for TrainVali...

2016-12-05 Thread hhbyyh
GitHub user hhbyyh opened a pull request: https://github.com/apache/spark/pull/16158 [SPARK-18724][ML] Add TuningSummary for TrainValidationSplit ## What changes were proposed in this pull request? jira: https://issues.apache.org/jira/browse/SPARK-18724 Currently

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

2016-12-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15998 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90969603 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -578,4 +578,66 @@ class

[GitHub] spark pull request #16156: [SPARK-18539][SQL]: tolerate pushed-down filter o...

2016-12-05 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16156#discussion_r90969322 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -578,4 +578,66 @@ class

[GitHub] spark issue #16157: [SPARK-18723][DOC] Expanded programming guide informatio...

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16157 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

<    1   2   3   4   5   6   >