[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-10-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r143288711 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -266,6 +267,10 @@ final class DataFrameWriter[T] private[sql](ds: Datase

[GitHub] spark pull request #19429: [SPARK-20055] [Docs] Added documentation for load...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19429#discussion_r143288308 --- Diff: docs/sql-programming-guide.md --- @@ -479,6 +481,47 @@ source type can be converted into other types using this syntax. +

[GitHub] spark pull request #18460: [SPARK-21247][SQL] Type comparison should respect...

2017-10-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/18460#discussion_r143288219 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -100,6 +101,17 @@ object TypeCoercion {

[GitHub] spark pull request #19429: [SPARK-20055] [Docs] Added documentation for load...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19429#discussion_r143288202 --- Diff: examples/src/main/java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java --- @@ -115,7 +115,20 @@ private static void runBasicDataS

[GitHub] spark pull request #19429: [SPARK-20055] [Docs] Added documentation for load...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19429#discussion_r143287943 --- Diff: examples/src/main/resources/people.csv --- @@ -0,0 +1,3 @@ +name;age;job +Jorge;30;Developer +Bob;32;Developer --- End diff --

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread ajbozarth
Github user ajbozarth commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143287894 --- Diff: core/src/main/resources/org/apache/spark/ui/static/utils.js --- @@ -46,3 +46,64 @@ function formatBytes(bytes, type) { var i = Math.floo

[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 Ok Yes, I just noticed that comment. Will get back to you on this one ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spar

[GitHub] spark pull request #19429: [SPARK-20055] [Docs] Added documentation for load...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19429#discussion_r143287807 --- Diff: examples/src/main/python/sql/datasource.py --- @@ -53,6 +53,11 @@ def basic_datasource_example(spark): df.select("name", "age").write.s

[GitHub] spark pull request #19429: [SPARK-20055] [Docs] Added documentation for load...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19429#discussion_r143287505 --- Diff: examples/src/main/java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java --- @@ -115,7 +115,20 @@ private static void runBasicDataS

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143287411 --- Diff: core/src/main/resources/org/apache/spark/ui/static/utils.js --- @@ -46,3 +46,64 @@ function formatBytes(bytes, type) { var i = Math.flo

[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...

2017-10-06 Thread ajbozarth
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 Thanks, I must've missed that in the description. You've take care of all but my last comment, I'm not seeing the accumulators table, have you checked it shows up when theres accumulators? ---

[GitHub] spark issue #19394: [SPARK-22170][SQL] Reduce memory consumption in broadcas...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19394 **[Test build #82525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82525/testReport)** for PR 19394 at commit [`5b92fe2`](https://github.com/apache/spark/commit/5b

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread ajbozarth
Github user ajbozarth commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143286538 --- Diff: core/src/main/resources/org/apache/spark/ui/static/utils.js --- @@ -46,3 +46,64 @@ function formatBytes(bytes, type) { var i = Math.floo

[GitHub] spark pull request #19394: [SPARK-22170][SQL] Reduce memory consumption in b...

2017-10-06 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/19394#discussion_r143286130 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala --- @@ -73,25 +73,37 @@ case class BroadcastExchangeEx

[GitHub] spark pull request #19394: [SPARK-22170][SQL] Reduce memory consumption in b...

2017-10-06 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/19394#discussion_r143286104 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala --- @@ -73,25 +73,37 @@ case class BroadcastExchangeEx

[GitHub] spark pull request #19394: [SPARK-22170][SQL] Reduce memory consumption in b...

2017-10-06 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/19394#discussion_r143285737 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/ConfigBehaviorSuite.scala --- @@ -58,7 +58,7 @@ class ConfigBehaviorSuite extends QueryTest with Shar

[GitHub] spark issue #19394: [SPARK-22170][SQL] Reduce memory consumption in broadcas...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19394 **[Test build #82524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82524/testReport)** for PR 19394 at commit [`f6613c2`](https://github.com/apache/spark/commit/f6

[GitHub] spark pull request #18460: [SPARK-21247][SQL] Type comparison should respect...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18460#discussion_r143285500 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -100,6 +101,17 @@ object TypeCoercion {

[GitHub] spark pull request #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19451#discussion_r143284824 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1242,6 +1243,53 @@ object ReplaceIntersectWithSemi

[GitHub] spark pull request #19336: [SPARK-21947][SS] Check and report error when mon...

2017-10-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19336 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19336: [SPARK-21947][SS] Check and report error when monotonica...

2017-10-06 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19336 LGTM. Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revi

[GitHub] spark issue #18460: [SPARK-21247][SQL] Type comparison should respect case-s...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18460 **[Test build #82523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82523/testReport)** for PR 18460 at commit [`c72aa18`](https://github.com/apache/spark/commit/c7

[GitHub] spark issue #18460: [SPARK-21247][SQL] Type comparison should respect case-s...

2017-10-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18460 The test cases are added. Thank you, @gatorsmile ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For a

[GitHub] spark pull request #19444: [SPARK-22214][SQL] Refactor the list hive partiti...

2017-10-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19444 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19444: [SPARK-22214][SQL] Refactor the list hive partitions cod...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19444 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 @ajbozarth Regarding your earlier comment on the missing functionality of show additional metrics, I have mentioned that in the description of the PR. I restate it below: Because of the

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143280105 --- Diff: core/src/main/resources/org/apache/spark/ui/static/utils.js --- @@ -46,3 +46,64 @@ function formatBytes(bytes, type) { var i = Math.flo

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143279779 --- Diff: core/src/main/resources/org/apache/spark/ui/static/taskspages.js --- @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143278211 --- Diff: core/src/main/resources/org/apache/spark/ui/static/taskspages.js --- @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143277618 --- Diff: core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala --- @@ -346,7 +346,7 @@ class UISeleniumSuite extends SparkFunSuite with WebBrows

[GitHub] spark issue #19444: [SPARK-22214][SQL] Refactor the list hive partitions cod...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19444 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19444: [SPARK-22214][SQL] Refactor the list hive partitions cod...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19444 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82519/ Test PASSed. ---

[GitHub] spark issue #19444: [SPARK-22214][SQL] Refactor the list hive partitions cod...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19444 **[Test build #82519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82519/testReport)** for PR 19444 at commit [`1e119ea`](https://github.com/apache/spark/commit/1

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143274906 --- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala --- @@ -170,6 +170,17 @@ class ExecutorsListener(storageStatusListener: Storag

[GitHub] spark issue #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19451 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143274132 --- Diff: core/src/main/resources/org/apache/spark/ui/static/taskspages.js --- @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143274000 --- Diff: core/src/main/resources/org/apache/spark/ui/static/taskspages.js --- @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143274034 --- Diff: core/src/main/resources/org/apache/spark/ui/static/taskspages.js --- @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143274083 --- Diff: core/src/main/resources/org/apache/spark/ui/static/taskspages.js --- @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143273927 --- Diff: core/src/main/resources/org/apache/spark/ui/static/taskspages.js --- @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request #19270: [SPARK-21809] : Change Stage Page to use datatabl...

2017-10-06 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/19270#discussion_r143273867 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/AllStagesResource.scala --- @@ -138,21 +155,61 @@ private[v1] object AllStagesResource {

[GitHub] spark pull request #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-06 Thread sathiyapk
GitHub user sathiyapk opened a pull request: https://github.com/apache/spark/pull/19451 SPARK-22181 Adds ReplaceExceptWithNotFilter rule ## What changes were proposed in this pull request? Adds a new optimisation rule 'ReplaceExceptWithNotFilter' that replaces Except logica

[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82522/testReport)** for PR 19270 at commit [`25b5215`](https://github.com/apache/spark/commit/25

[GitHub] spark pull request #18460: [SPARK-21247][SQL] Type comparison should respect...

2017-10-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/18460#discussion_r143273532 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -100,6 +101,17 @@ object TypeCoercion {

[GitHub] spark pull request #18460: [SPARK-21247][SQL] Type comparison should respect...

2017-10-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/18460#discussion_r143273000 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -100,6 +101,17 @@ object TypeCoercion {

[GitHub] spark issue #18966: [SPARK-21751][SQL] CodeGeneraor.splitExpressions counts ...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18966 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #18966: [SPARK-21751][SQL] CodeGeneraor.splitExpressions counts ...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18966 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82518/ Test PASSed. ---

[GitHub] spark issue #18966: [SPARK-21751][SQL] CodeGeneraor.splitExpressions counts ...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18966 **[Test build #82518 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82518/testReport)** for PR 18966 at commit [`b04c09c`](https://github.com/apache/spark/commit/b

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-10-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r143272791 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1213,6 +1213,71 @@ case class ToUTCTimesta

[GitHub] spark issue #19450: [SPARK-22218] spark shuffle services fails to update sec...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19450 **[Test build #82521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82521/testReport)** for PR 19450 at commit [`5a9ef13`](https://github.com/apache/spark/commit/5a

[GitHub] spark issue #19450: [SPARK-22218] spark shuffle services fails to update sec...

2017-10-06 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19450 ping @vanzin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@

[GitHub] spark pull request #19450: [SPARK-22218] spark shuffle services fails to upd...

2017-10-06 Thread tgravescs
GitHub user tgravescs opened a pull request: https://github.com/apache/spark/pull/19450 [SPARK-22218] spark shuffle services fails to update secret on app re-attempts This patch fixes application re-attempts when running spark on yarn using the external shuffle service with securit

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19448 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82517/ Test PASSed. ---

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19448 **[Test build #82517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82517/testReport)** for PR 19448 at commit [`e6fdbdc`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19448 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18664 I am okay with proceeding separately for dealing with timezone, and matching the behaviour with Arrow to the existing behaviour without Arrow here with respect to timezone. Less sure ye

[GitHub] spark pull request #18732: [SPARK-20396][SQL][PySpark] groupby().apply() wit...

2017-10-06 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/18732#discussion_r143263694 --- Diff: python/pyspark/sql/group.py --- @@ -192,7 +193,69 @@ def pivot(self, pivot_col, values=None): jgd = self._jgd.pivot(pivot_col)

[GitHub] spark issue #19372: [SPARK-22156][MLLIB] Fix update equation of learning rat...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19372 **[Test build #3943 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3943/testReport)** for PR 19372 at commit [`2ea3f18`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-06 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/18664 If we all agree on the necessity of a design doc first, I can create a Jira and we can make progress there. What do you all think? @BryanCutler @gatorsmile @HyukjinKwon --- -

[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...

2017-10-06 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18229 gentle ping @mkesselaers --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-06 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/18664 I agree. I think some high level document describing these differences so we can discuss it. I think we should be more careful about Arrow-version behavior before releasing support for timestamp

[GitHub] spark issue #19077: [SPARK-21860][core]Improve memory reuse for heap memory ...

2017-10-06 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19077 gentle ping @jerryshao for review --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mai

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18664 Yup, I admit there could be some exceptions (there have been actually) but that should still be the baseline we should basically pursue. Probably, we could treat this Arrow optimisation as an ev

[GitHub] spark issue #19449: [SPARK-22219][SQL] Refactor code to get a value for "spa...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19449 **[Test build #82520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82520/testReport)** for PR 19449 at commit [`be4220d`](https://github.com/apache/spark/commit/be

[GitHub] spark pull request #19449: [SPARK-22219][SQL] Refactor code to get a value f...

2017-10-06 Thread kiszk
GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/19449 [SPARK-22219][SQL] Refactor code to get a value for "spark.sql.codegen.comments" ## What changes were proposed in this pull request? This PR refactors code to get a value for "spark.sql.code

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-10-06 Thread zivanfi
Github user zivanfi commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r143257840 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1213,6 +1213,71 @@ case class ToUTCTimest

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-06 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/18664 > The baseline should be (as said above): Internal optimisation should not introduce any behaviour change, and we are discouraged to change the previous behaviour unless it has bugs in general.

[GitHub] spark issue #18732: [SPARK-20396][SQL][PySpark] groupby().apply() with panda...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18732 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82515/ Test PASSed. ---

[GitHub] spark issue #18732: [SPARK-20396][SQL][PySpark] groupby().apply() with panda...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18732 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #18732: [SPARK-20396][SQL][PySpark] groupby().apply() with panda...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18732 **[Test build #82515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82515/testReport)** for PR 18732 at commit [`284ba00`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #19424: [SPARK-22197][SQL] push down operators to data source be...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19424 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82514/ Test PASSed. ---

[GitHub] spark issue #19424: [SPARK-22197][SQL] push down operators to data source be...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19424 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19424: [SPARK-22197][SQL] push down operators to data source be...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19424 **[Test build #82514 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82514/testReport)** for PR 19424 at commit [`24f1a75`](https://github.com/apache/spark/commit/2

[GitHub] spark pull request #19294: [SPARK-21549][CORE] Respect OutputFormats with no...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19294#discussion_r143252186 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -57,6 +60,15 @@ class HadoopMapReduceCommitProtoco

[GitHub] spark pull request #19294: [SPARK-21549][CORE] Respect OutputFormats with no...

2017-10-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19294#discussion_r143250260 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -57,6 +60,15 @@ class HadoopMapReduceCommitProtocol(j

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-10-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r143246203 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TimestampTableTimeZone.scala --- @@ -0,0 +1,213 @@ +/* + * Licensed to

[GitHub] spark issue #18931: [SPARK-21717][SQL] Decouple consume functions of physica...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18931 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-10-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r143245229 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -230,6 +230,13 @@ case class AlterTableSetPropertiesCommand(

[GitHub] spark issue #18931: [SPARK-21717][SQL] Decouple consume functions of physica...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18931 **[Test build #82516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82516/testReport)** for PR 18931 at commit [`edb73d6`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #19447: [SPARK-22215][SQL] Add configuration to set the threshol...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19447 We still need to add a test case. We also should capture the exception and issue a better one; otherwise, users will not know what they should do when they hit such a confusing error. cc

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-10-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r143243769 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1213,6 +1213,71 @@ case class ToUTCTimesta

[GitHub] spark issue #18931: [SPARK-21717][SQL] Decouple consume functions of physica...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18931 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82516/ Test FAILed. ---

[GitHub] spark issue #19424: [SPARK-22197][SQL] push down operators to data source be...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19424 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-10-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r143244400 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -266,6 +267,10 @@ final class DataFrameWriter[T] private[sql](ds: Datase

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-10-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r143243338 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1213,6 +1213,71 @@ case class ToUTCTimestamp

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-10-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r143243228 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -1015,6 +1020,10 @@ object DateTimeUtils { gues

[GitHub] spark issue #19424: [SPARK-22197][SQL] push down operators to data source be...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19424 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82513/ Test PASSed. ---

[GitHub] spark issue #19424: [SPARK-22197][SQL] push down operators to data source be...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19424 **[Test build #82513 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82513/testReport)** for PR 19424 at commit [`e880328`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-10-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18664 The baseline should be (as said above): Internal optimisation should not introduce any behaviour change, and we are discouraged to change the previous behaviour unless it has bugs in general.

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-10-06 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r143242256 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1213,6 +1213,71 @@ case class ToUTCTimesta

[GitHub] spark issue #19443: [SPARK-22212][SQL][PySpark] Some SQL functions in Python...

2017-10-06 Thread jsnowacki
Github user jsnowacki commented on the issue: https://github.com/apache/spark/pull/19443 The only other reason for Python I can think of, if the above are not compelling enough, is that the issue with function not having call-by-column-name option is that we'll get the error only at t

[GitHub] spark issue #19444: [SPARK-22214][SQL] Refactor the list hive partitions cod...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19444 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.ap

[GitHub] spark pull request #18460: [SPARK-21247][SQL] Type comparision should respec...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18460#discussion_r143238804 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -100,6 +101,17 @@ object TypeCoercion {

[GitHub] spark issue #18460: [SPARK-21247][SQL] Type comparision should respect case-...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18460 I see. There is a typo in PR title: `comparision`. Will review the fix soon. --- - To unsubscribe, e-mail: reviews-un

[GitHub] spark issue #19438: [SPARK-22208] [SQL] Improve percentile_approx by not rou...

2017-10-06 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/19438 Maybe we can run some of the major test suites locally and update all the results. --- - To unsubscribe, e-mail: reviews-uns

[GitHub] spark issue #19444: [SPARK-22214][SQL] Refactor the list hive partitions cod...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19444 **[Test build #82519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82519/testReport)** for PR 19444 at commit [`1e119ea`](https://github.com/apache/spark/commit/1e

[GitHub] spark issue #19438: [SPARK-22208] [SQL] Improve percentile_approx by not rou...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19438 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19438: [SPARK-22208] [SQL] Improve percentile_approx by not rou...

2017-10-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19438 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82512/ Test FAILed. ---

[GitHub] spark issue #19438: [SPARK-22208] [SQL] Improve percentile_approx by not rou...

2017-10-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19438 **[Test build #82512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82512/testReport)** for PR 19438 at commit [`aae2769`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #19444: [SPARK-22214][SQL] Refactor the list hive partitions cod...

2017-10-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19444 LGTM except a minor comment. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mai

<    1   2   3   4   >