[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100648519 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala --- @@ -0,0 +1,213 @@ +/* + * Licensed to

[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100648210 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -394,36 +447,32 @@ class JacksonParser( }

[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100648139 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -48,69 +47,102 @@ class JacksonParser( /

[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100647494 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -48,69 +47,110 @@ class JacksonParser( /

[GitHub] spark pull request #16878: [SPARK-19539][SQL] Block duplicate temp table dur...

2017-02-10 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16878#discussion_r100647313 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -425,7 +425,9 @@ class SparkSqlAstBuilder(conf: SQLConf) extend

[GitHub] spark issue #16826: [WIP][SPARK-19540][SQL] Add ability to clone SparkSessio...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16826 **[Test build #72727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72727/testReport)** for PR 16826 at commit [`6da6bda`](https://github.com/apache/spark/commit/6d

[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100647047 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala --- @@ -31,10 +31,17 @@ import org.apache.spark.sql.catalyst.

[GitHub] spark pull request #16878: [SPARK-19539][SQL] Block duplicate temp table dur...

2017-02-10 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16878#discussion_r100646859 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -425,7 +425,9 @@ class SparkSqlAstBuilder(conf: SQLConf) extend

[GitHub] spark issue #15604: [SPARK-18066] [CORE] [TESTS] Add Pool usage policies tes...

2017-02-10 Thread erenavsarogullari
Github user erenavsarogullari commented on the issue: https://github.com/apache/spark/pull/15604 Hi @kayousterhout and @squito, Firstly, many thanks for the review. All comments are addressed. This is ready to re-review ;) --- If your project is set up for it, you can reply to th

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72726/testReport)** for PR 16386 at commit [`0f4686d`](https://github.com/apache/spark/commit/0f

[GitHub] spark pull request #16878: [SPARK-19539][SQL] Block duplicate temp table dur...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16878#discussion_r100646662 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -425,7 +425,9 @@ class SparkSqlAstBuilder(conf: SQLConf) exte

[GitHub] spark pull request #15604: [SPARK-18066] [CORE] [TESTS] Add Pool usage polic...

2017-02-10 Thread erenavsarogullari
Github user erenavsarogullari commented on a diff in the pull request: https://github.com/apache/spark/pull/15604#discussion_r100646612 --- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala --- @@ -178,4 +180,97 @@ class PoolSuite extends SparkFunSuite with Local

[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-10 Thread NathanHowell
Github user NathanHowell commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100646497 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -48,69 +47,98 @@ class JacksonParser(

[GitHub] spark pull request #16878: [SPARK-19539][SQL] Block duplicate temp table dur...

2017-02-10 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16878#discussion_r100646267 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -425,7 +425,9 @@ class SparkSqlAstBuilder(conf: SQLConf) extend

[GitHub] spark pull request #16871: [SPARK-19550][BUILD][CORE][WIP] Remove Java 7 sup...

2017-02-10 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16871#discussion_r100646239 --- Diff: project/SparkBuild.scala --- @@ -765,16 +750,6 @@ object CopyDependencies { } -object Java8TestSettings { - import BuildCo

[GitHub] spark issue #15604: [SPARK-18066] [CORE] [TESTS] Add Pool usage policies tes...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15604 **[Test build #72725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72725/testReport)** for PR 15604 at commit [`f84abe7`](https://github.com/apache/spark/commit/f8

[GitHub] spark pull request #16871: [SPARK-19550][BUILD][CORE][WIP] Remove Java 7 sup...

2017-02-10 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16871#discussion_r100645947 --- Diff: project/SparkBuild.scala --- @@ -765,16 +750,6 @@ object CopyDependencies { } -object Java8TestSettings { - import BuildCo

[GitHub] spark issue #15604: [SPARK-18066] [CORE] [TESTS] Add Pool usage policies tes...

2017-02-10 Thread erenavsarogullari
Github user erenavsarogullari commented on the issue: https://github.com/apache/spark/pull/15604 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

[GitHub] spark issue #16871: [SPARK-19550][BUILD][CORE][WIP] Remove Java 7 support

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16871 **[Test build #72724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72724/testReport)** for PR 16871 at commit [`7f8a2cb`](https://github.com/apache/spark/commit/7f

[GitHub] spark pull request #16871: [SPARK-19550][BUILD][CORE][WIP] Remove Java 7 sup...

2017-02-10 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16871#discussion_r100645304 --- Diff: project/SparkBuild.scala --- @@ -765,16 +750,6 @@ object CopyDependencies { } -object Java8TestSettings { - import BuildCo

[GitHub] spark pull request #16871: [SPARK-19550][BUILD][CORE][WIP] Remove Java 7 sup...

2017-02-10 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16871#discussion_r100644859 --- Diff: project/SparkBuild.scala --- @@ -765,16 +750,6 @@ object CopyDependencies { } -object Java8TestSettings { - import BuildCo

[GitHub] spark pull request #16878: [SPARK-19539][SQL] Block duplicate temp table dur...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16878#discussion_r100644845 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -425,7 +425,9 @@ class SparkSqlAstBuilder(conf: SQLConf) exte

[GitHub] spark issue #15604: [SPARK-18066] [CORE] [TESTS] Add Pool usage policies tes...

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15604 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15604: [SPARK-18066] [CORE] [TESTS] Add Pool usage policies tes...

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15604 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72722/ Test FAILed. ---

[GitHub] spark issue #15604: [SPARK-18066] [CORE] [TESTS] Add Pool usage policies tes...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15604 **[Test build #72722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72722/testReport)** for PR 15604 at commit [`84513b1`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #15009: [SPARK-17443][SPARK-11035] Stop Spark Application if lau...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15009 **[Test build #72723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72723/testReport)** for PR 15009 at commit [`cc2c0be`](https://github.com/apache/spark/commit/cc

[GitHub] spark issue #15604: [SPARK-18066] [CORE] [TESTS] Add Pool usage policies tes...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15604 **[Test build #72722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72722/testReport)** for PR 15604 at commit [`84513b1`](https://github.com/apache/spark/commit/84

[GitHub] spark pull request #16871: [SPARK-19550][BUILD][CORE][WIP] Remove Java 7 sup...

2017-02-10 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16871#discussion_r100644266 --- Diff: project/SparkBuild.scala --- @@ -765,16 +750,6 @@ object CopyDependencies { } -object Java8TestSettings { - import BuildCo

[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100644275 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -48,69 +47,98 @@ class JacksonParser( //

[GitHub] spark issue #14864: [SPARK-15453] [SQL] FileSourceScanExec to extract `outpu...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14864 This is an optimization and we usually don't backport optimizations. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #16871: [SPARK-19550][BUILD][CORE][WIP] Remove Java 7 sup...

2017-02-10 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16871#discussion_r100643906 --- Diff: streaming/src/test/java/org/apache/spark/streaming/Java8APISuite.java --- @@ -15,7 +15,7 @@ * limitations under the License. */

[GitHub] spark issue #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint Inference...

2017-02-10 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/16067 @gatorsmile can we also add it in branch-2.0? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #16871: [SPARK-19550][BUILD][CORE][WIP] Remove Java 7 sup...

2017-02-10 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16871#discussion_r100643005 --- Diff: project/SparkBuild.scala --- @@ -765,16 +750,6 @@ object CopyDependencies { } -object Java8TestSettings { - import BuildCo

[GitHub] spark pull request #16871: [SPARK-19550][BUILD][CORE][WIP] Remove Java 7 sup...

2017-02-10 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16871#discussion_r100643153 --- Diff: streaming/src/test/java/org/apache/spark/streaming/Java8APISuite.java --- @@ -15,7 +15,7 @@ * limitations under the License. */

[GitHub] spark issue #14864: [SPARK-15453] [SQL] FileSourceScanExec to extract `outpu...

2017-02-10 Thread ankravch
Github user ankravch commented on the issue: https://github.com/apache/spark/pull/14864 Was this patch merged in spark2.0.1 release? If so, how can it be enforced/turned on? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #16878: [SPARK-19539][SQL] Block duplicate temp table dur...

2017-02-10 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16878#discussion_r100642343 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -425,7 +425,9 @@ class SparkSqlAstBuilder(conf: SQLConf) extend

[GitHub] spark issue #15125: [SPARK-5484][GraphX] Periodically do checkpoint in Prege...

2017-02-10 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/15125 @viirya @dding3 I'm going to rerun our big connected components computation with the changes I've suggested to validate that it still performs and completes as expected. Given the time required to c

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100641170 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/util/PeriodicGraphCheckpointer.scala --- @@ -87,10 +88,7 @@ private[mllib] class PeriodicGraphCheck

[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-10 Thread NathanHowell
Github user NathanHowell commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100640620 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -48,69 +47,98 @@ class JacksonParser(

[GitHub] spark pull request #16886: [SPARK-19548][SQL] Support Hive UDFs which return...

2017-02-10 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16886 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16886: [SPARK-19548][SQL] Support Hive UDFs which return typed ...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16886 thanks, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wi

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100640256 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -123,16 +127,25 @@ object Pregel extends Logging { s" but got ${maxIte

[GitHub] spark issue #16886: [SPARK-19548][SQL] Support Hive UDFs which return typed ...

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16886 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16886: [SPARK-19548][SQL] Support Hive UDFs which return typed ...

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16886 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72719/ Test PASSed. ---

[GitHub] spark issue #16886: [SPARK-19548][SQL] Support Hive UDFs which return typed ...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16886 **[Test build #72719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72719/testReport)** for PR 16886 at commit [`8cf25b9`](https://github.com/apache/spark/commit/8

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100638292 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -123,16 +127,25 @@ object Pregel extends Logging { s" but got ${maxIte

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100638130 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/util/PeriodicGraphCheckpointer.scala --- @@ -76,7 +77,7 @@ import org.apache.spark.storage.StorageL

[GitHub] spark issue #16889: [SPARK-17668][SQL] Use Expressions for conversions to/fr...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16889 **[Test build #72720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72720/testReport)** for PR 16889 at commit [`ac09ad5`](https://github.com/apache/spark/commit/ac

[GitHub] spark issue #16880: [SPARK-19542][SS]Delete the temp checkpoint if a query i...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16880 **[Test build #72721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72721/testReport)** for PR 16880 at commit [`cae981f`](https://github.com/apache/spark/commit/ca

[GitHub] spark pull request #16889: [SPARK-17668][SQL] Use Expressions for conversion...

2017-02-10 Thread koertkuipers
GitHub user koertkuipers opened a pull request: https://github.com/apache/spark/pull/16889 [SPARK-17668][SQL] Use Expressions for conversions to/from user types in UDFs ## What changes were proposed in this pull request? do not merge this is a first attempt at trying to

[GitHub] spark pull request #16880: [SPARK-19542][SS]Delete the temp checkpoint if a ...

2017-02-10 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/16880#discussion_r100636660 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala --- @@ -323,6 +329,20 @@ class StreamExecution(

[GitHub] spark issue #16161: [SPARK-18717][SQL] Make code generation for Scala Map wo...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16161 I'm fine to backport this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wis

[GitHub] spark pull request #15945: [SPARK-12978][SQL] Merge unnecessary partial aggr...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15945#discussion_r100633797 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/MergePartialAggregate.scala --- @@ -0,0 +1,96 @@ +/* + * Licensed to t

[GitHub] spark issue #16699: [SPARK-18710][ML] Add offset in GLM

2017-02-10 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16699 @sethah Thanks much for your review. Regarding prediction, both R and my implementation here allow prediction with offsets. If the users want to get the predicted rates (instead of cou

[GitHub] spark pull request #15945: [SPARK-12978][SQL] Merge unnecessary partial aggr...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15945#discussion_r100632491 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateExec.scala --- @@ -0,0 +1,68 @@ +/* + * Licensed to the Apach

[GitHub] spark pull request #15945: [SPARK-12978][SQL] Merge unnecessary partial aggr...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15945#discussion_r100632348 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala --- @@ -82,32 +81,14 @@ class QueryExecution(val sparkSession: Spark

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100632148 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/impl/PeriodicRDDCheckpointerSuite.scala --- @@ -23,7 +23,7 @@ import org.apache.spark.{SparkContext,

[GitHub] spark issue #16395: [SPARK-17075][SQL] implemented filter estimation

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16395 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72718/ Test PASSed. ---

[GitHub] spark issue #16395: [SPARK-17075][SQL] implemented filter estimation

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16395 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16395: [SPARK-17075][SQL] implemented filter estimation

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16395 **[Test build #72718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72718/testReport)** for PR 16395 at commit [`96b14de`](https://github.com/apache/spark/commit/9

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100631975 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointerSuite.scala --- @@ -21,6 +21,7 @@ import org.apache.hadoop.fs.Path

[GitHub] spark issue #16161: [SPARK-18717][SQL] Make code generation for Scala Map wo...

2017-02-10 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/16161 Shall we backport this to branch-2.1? I'd consider this as a bug because of the following snippet fail in Spark 2.1: ```scala case class Wrapper1(value: Option[Map[String, String]])

[GitHub] spark pull request #15945: [SPARK-12978][SQL] Merge unnecessary partial aggr...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15945#discussion_r100629464 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/PhysicalOptimizer.scala --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Soft

[GitHub] spark issue #16664: [SPARK-18120 ][SQL] Call QueryExecutionListener callback...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16664 @salilsurendran yes, and we can send another PR to fix the InsertXXX command problem --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #16664: [SPARK-18120 ][SQL] Call QueryExecutionListener callback...

2017-02-10 Thread salilsurendran
Github user salilsurendran commented on the issue: https://github.com/apache/spark/pull/16664 @cloud-fan From what I understand we need to modify InsertXXX command to carry all the write options instead of the change suggested in this PR. Right now the QueryExecution object doesn't ca

[GitHub] spark issue #16664: [SPARK-18120 ][SQL] Call QueryExecutionListener callback...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16664 @vanzin yes, InsertXXX command will carry all the write options. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark issue #16664: [SPARK-18120 ][SQL] Call QueryExecutionListener callback...

2017-02-10 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16664 Does that mean the information would show up in the plan? That would be great. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #16864: [SPARK-19527][Core] Approximate Size of Intersect...

2017-02-10 Thread Bcpoole
Github user Bcpoole commented on a diff in the pull request: https://github.com/apache/spark/pull/16864#discussion_r100624882 --- Diff: common/sketch/src/main/java/org/apache/spark/util/sketch/BloomFilterImpl.java --- @@ -221,6 +221,49 @@ public BloomFilter mergeInPlace(BloomFilte

[GitHub] spark pull request #16750: [SPARK-18937][SQL] Timezone support in CSV/JSON p...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16750#discussion_r100624733 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala --- @@ -859,6 +859,48 @@ class CSVSuite extends QueryTest

[GitHub] spark pull request #16750: [SPARK-18937][SQL] Timezone support in CSV/JSON p...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16750#discussion_r100623330 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala --- @@ -58,13 +59,15 @@ private[sql] class JSONOptions(

[GitHub] spark issue #16699: [SPARK-18710][ML] Add offset in GLM

2017-02-10 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/16699 @actuaryzhang This is looking pretty good overall. Regarding the prediction logic, R glm does not allow you to predict with offsets, correct? I notice that statsmodels in Python _does_ allow it. For,

[GitHub] spark issue #16886: [SPARK-19548][SQL] Support Hive UDFs which return typed ...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16886 **[Test build #72719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72719/testReport)** for PR 16886 at commit [`8cf25b9`](https://github.com/apache/spark/commit/8c

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16887 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16887 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16887 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72713/ Test FAILed. ---

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16887 **[Test build #72713 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72713/testReport)** for PR 16887 at commit [`81b4243`](https://github.com/apache/spark/commit/8

[GitHub] spark pull request #16887: [SPARK-19549] Allow providing reason for stage/jo...

2017-02-10 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16887 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16887 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72717/ Test PASSed. ---

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16887 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16887 Merging in master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #16888: [SPARK-19552] [BUILD] Upgrade Netty version to 4.1.8 fin...

2017-02-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16888 BTW for Netty we shouldn't just bump to the highest version. We should use the maintenance branches. --- If your project is set up for it, you can reply to this email and have your reply appear on Gi

[GitHub] spark issue #16888: [SPARK-19552] [BUILD] Upgrade Netty version to 4.1.8 fin...

2017-02-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16888 Shouldn't we use netty-4.0.44.Final rather than 4.1.x? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16887 **[Test build #72717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72717/testReport)** for PR 16887 at commit [`50c54b1`](https://github.com/apache/spark/commit/5

[GitHub] spark pull request #16854: [SPARK-15463][SQL] Add an API to load DataFrame f...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16854#discussion_r100613684 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -361,6 +362,41 @@ class DataFrameReader private[sql](sparkSession: Sp

[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100612936 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -48,69 +47,98 @@ class JacksonParser( //

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100612840 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -123,16 +127,25 @@ object Pregel extends Logging { s" but got ${maxIte

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r100611930 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -0,0 +1,623 @@ +/*

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r100611664 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -0,0 +1,623 @@ +/*

[GitHub] spark pull request #16860: [SPARK-18613][ML] make spark.mllib LDA dependenci...

2017-02-10 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16860 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r100611368 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -0,0 +1,623 @@ +/*

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r100611222 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -0,0 +1,623 @@ +/*

[GitHub] spark issue #16860: [SPARK-18613][ML] make spark.mllib LDA dependencies in s...

2017-02-10 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16860 LGTM Merging with master Thanks @sueann ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-10 Thread NathanHowell
Github user NathanHowell commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100610662 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -48,69 +47,98 @@ class JacksonParser(

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100609529 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -123,16 +127,25 @@ object Pregel extends Logging { s" but got ${maxIte

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r100608877 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -0,0 +1,623 @@ +/*

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100608839 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -123,16 +127,25 @@ object Pregel extends Logging { s" but got ${maxIte

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16887 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72716/ Test PASSed. ---

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-02-10 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r100608545 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -0,0 +1,623 @@ +/*

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16887 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16887 **[Test build #72716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72716/testReport)** for PR 16887 at commit [`a5f4946`](https://github.com/apache/spark/commit/a

<    1   2   3   4   5   >