[GitHub] spark issue #21190: [SPARK-22938][SQL][followup] Assert that SQLConf.get is ...

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21190 **[Test build #90191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90191/testReport)** for PR 21190 at commit

[GitHub] spark issue #21118: SPARK-23325: Use InternalRow when reading with DataSourc...

2018-05-04 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21118 @cloud-fan, let me clarify what I'm getting at here. It appears that Spark makes at least one copy of data to unsafe when reading any Parquet row. If the projection includes partition

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21040 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21040 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90202/ Test FAILed. ---

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21040 **[Test build #90202 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90202/testReport)** for PR 21040 at commit

[GitHub] spark pull request #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to b...

2018-05-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21122#discussion_r186132760 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -1354,7 +1354,8 @@ class HiveDDLSuite val

[GitHub] spark pull request #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21040#discussion_r186133103 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -378,6 +378,138 @@ case class

[GitHub] spark issue #21118: SPARK-23325: Use InternalRow when reading with DataSourc...

2018-05-04 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21118 I just did a performance test based on our 2.1.1 and a real table. I tested a full scan of an hour of data with a single data filter. The scan had 13,083 tasks and read 1084.8 GB. I used

[GitHub] spark issue #21173: [SPARK-23856][SQL] Add an option `queryTimeout` in JDBCO...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21173 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90197/ Test PASSed. ---

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-05-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21122 +1, LGTM, too. Thank you for choosing this path, @gatorsmile ! --- - To unsubscribe, e-mail:

[GitHub] spark issue #18801: SPARK-10878 Fix race condition when multiple clients res...

2018-05-04 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18801 I can take over this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20787: [MINOR][DOCS] Documenting months_between direction

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20787 **[Test build #90205 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90205/testReport)** for PR 20787 at commit

[GitHub] spark issue #21235: [SPARK-24181][SQL] Better error message for writing sort...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21235 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2920/

[GitHub] spark issue #21235: [SPARK-24181][SQL] Better error message for writing sort...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21235 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-04 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186182632 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -413,6 +413,25 @@ class DataFrameFunctionsSuite extends

[GitHub] spark pull request #21236: [SPARK-23935][SQL] Adding map_entries function

2018-05-04 Thread mn-mikke
GitHub user mn-mikke opened a pull request: https://github.com/apache/spark/pull/21236 [SPARK-23935][SQL] Adding map_entries function ## What changes were proposed in this pull request? This PR adds `map_entries` function that returns an unordered array of all entries in

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21040 **[Test build #90196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90196/testReport)** for PR 21040 at commit

[GitHub] spark pull request #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to b...

2018-05-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/21122#discussion_r186142381 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala --- @@ -199,8 +199,8 @@ class InMemoryCatalog(

[GitHub] spark issue #21173: [SPARK-23856][SQL] Add an option `queryTimeout` in JDBCO...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21173 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-04 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186151824 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -413,6 +413,25 @@ class DataFrameFunctionsSuite extends

[GitHub] spark issue #18447: [SPARK-21232][SQL][SparkR][PYSPARK] New built-in SQL fun...

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18447 **[Test build #90199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90199/testReport)** for PR 18447 at commit

[GitHub] spark pull request #21145: [SPARK-24073][SQL]: Rename DataReaderFactory to R...

2018-05-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21145#discussion_r186163175 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/ReadTask.java --- @@ -22,20 +22,20 @@ import

[GitHub] spark issue #21049: [SPARK-23957][SQL] Remove redundant sort operators from ...

2018-05-04 Thread henryr
Github user henryr commented on the issue: https://github.com/apache/spark/pull/21049 I might be a bit of a hardliner on this, but I think it's correct to eliminate the {{ORDER BY}} from common table expressions (e.g. MSSQL agrees with me, see [this

[GitHub] spark issue #21235: [SPARK-24181][SQL] Better error message for writing sort...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21235 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2919/

[GitHub] spark issue #21200: [SPARK-24039][SS] Do continuous processing writes with m...

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21200 **[Test build #90201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90201/testReport)** for PR 21200 at commit

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-04 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186126547 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -413,6 +413,25 @@ class DataFrameFunctionsSuite extends QueryTest

[GitHub] spark pull request #21155: [SPARK-23927][SQL] Add "sequence" expression

2018-05-04 Thread wajda
Github user wajda commented on a diff in the pull request: https://github.com/apache/spark/pull/21155#discussion_r186134394 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1059,3 +1063,316 @@ case class

[GitHub] spark issue #16411: [SPARK-17984][YARN][Mesos][Deploy][WIP] add executor lau...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16411 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18447: [SPARK-21232][SQL][SparkR][PYSPARK] New built-in SQL fun...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18447 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21235: [SPARK-24181][SQL] Better error message for writing sort...

2018-05-04 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21235 cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21234: [SPARK-24124] Spark history server should create spark.h...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21234 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90200/ Test PASSed. ---

[GitHub] spark issue #21234: [SPARK-24124] Spark history server should create spark.h...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21234 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21235: [SPARK-24181][SQL] Better error message for writing sort...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21235 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21236: [SPARK-23935][SQL] Adding map_entries function

2018-05-04 Thread mn-mikke
Github user mn-mikke commented on the issue: https://github.com/apache/spark/pull/21236 cc @ueshin @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21236: [SPARK-23935][SQL] Adding map_entries function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21236 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21145: [SPARK-24073][SQL]: Rename DataReaderFactory to ReadTask...

2018-05-04 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21145 @gengliangwang, we can follow up with a rename for the streaming classes that already use this API. But there is no need to do that right now and make this commit larger. I think I've

[GitHub] spark issue #21145: [SPARK-24073][SQL]: Rename DataReaderFactory to ReadTask...

2018-05-04 Thread jose-torres
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/21145 I don't see the problem with the name ReadTask. In RDDs, we call the serializable representation of a partition for distribution to executors just Partition, and I've always found this pretty

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-05-04 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21070 @rdblue [I made the input test date sorted

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21040 **[Test build #90202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90202/testReport)** for PR 21040 at commit

[GitHub] spark issue #21217: [SPARK-24151][SQL] Fix CURRENT_DATE, CURRENT_TIMESTAMP t...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21217 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90194/ Test PASSed. ---

[GitHub] spark issue #21217: [SPARK-24151][SQL] Fix CURRENT_DATE, CURRENT_TIMESTAMP t...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21217 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21040 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90196/ Test PASSed. ---

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21040 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21040#discussion_r186139938 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -378,6 +378,138 @@ case class

[GitHub] spark issue #21216: [SPARK-24149][YARN] Retrieve all federated namespaces to...

2018-05-04 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21216 > I disagree that it is federation. It's just declaring multiple HDFS services in the same config file. I am using the terminology which is used in the Hadoop website. The

[GitHub] spark issue #21021: [SPARK-23921][SQL] Add array_sort function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21021 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90198/ Test PASSed. ---

[GitHub] spark issue #21021: [SPARK-23921][SQL] Add array_sort function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21021 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21028 **[Test build #90209 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90209/testReport)** for PR 21028 at commit

[GitHub] spark issue #21190: [SPARK-22938][SQL][followup] Assert that SQLConf.get is ...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21190 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90191/ Test PASSed. ---

[GitHub] spark issue #21190: [SPARK-22938][SQL][followup] Assert that SQLConf.get is ...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21190 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21028 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21070: [SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.

2018-05-04 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21070 @maropu, I'd recommend looking at the Parquet files using [`parquet-cli`](http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22parquet-cli%22) to see if you're getting reasonable min/max stats for your

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21040 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21040 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2917/

[GitHub] spark issue #21217: [SPARK-24151][SQL] Fix CURRENT_DATE, CURRENT_TIMESTAMP t...

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21217 **[Test build #90194 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90194/testReport)** for PR 21217 at commit

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21040 **[Test build #90203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90203/testReport)** for PR 21040 at commit

[GitHub] spark issue #21216: [SPARK-24149][YARN] Retrieve all federated namespaces to...

2018-05-04 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21216 > This is a valid configuration and it is the easiest configuration for HDFS federation. I disagree that it is federation. It's just declaring multiple HDFS services in the same config

[GitHub] spark issue #21235: [SPARK-24181][SQL] Better error message for writing sort...

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21235 **[Test build #90208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90208/testReport)** for PR 21235 at commit

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21028 **[Test build #90192 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90192/testReport)** for PR 21028 at commit

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21040 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21145: [SPARK-24073][SQL]: Rename DataReaderFactory to R...

2018-05-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21145#discussion_r186140632 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/ReadTask.java --- @@ -22,20 +22,20 @@ import

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21040 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2918/

[GitHub] spark issue #21158: [SPARK-23850][sql] Add separate config for SQL options r...

2018-05-04 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21158 @gatorsmile seriously, why are you focusing on the fact that I'm changing that default value when I wrote a bunch of code to actually maintain the URL redaction on the SQL plans, which was the

[GitHub] spark pull request #21145: [SPARK-24073][SQL]: Rename DataReaderFactory to R...

2018-05-04 Thread henryr
Github user henryr commented on a diff in the pull request: https://github.com/apache/spark/pull/21145#discussion_r186143060 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/ReadTask.java --- @@ -22,20 +22,20 @@ import

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-04 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186149012 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -413,6 +413,25 @@ class DataFrameFunctionsSuite extends QueryTest

[GitHub] spark issue #21021: [SPARK-23921][SQL] Add array_sort function

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21021 **[Test build #90198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90198/testReport)** for PR 21021 at commit

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-04 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186153677 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -413,6 +413,25 @@ class DataFrameFunctionsSuite extends QueryTest

[GitHub] spark issue #21145: [SPARK-24073][SQL]: Rename DataReaderFactory to ReadTask...

2018-05-04 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21145 @gatorsmile, the Spark UI has used the term "task" for years to refer to the same thing. I don't think it is unreasonable to use the same term.

[GitHub] spark issue #18447: [SPARK-21232][SQL][SparkR][PYSPARK] New built-in SQL fun...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18447 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90199/ Test PASSed. ---

[GitHub] spark pull request #18801: SPARK-10878 Fix race condition when multiple clie...

2018-05-04 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/18801#discussion_r186160092 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitUtilsSuite.scala --- @@ -255,4 +256,20 @@ class SparkSubmitUtilsSuite extends

[GitHub] spark issue #21235: [SPARK-24181][SQL] Better error message for writing sort...

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21235 **[Test build #90206 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90206/testReport)** for PR 21235 at commit

[GitHub] spark issue #21155: [SPARK-23927][SQL] Add "sequence" expression

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21155 **[Test build #90207 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90207/testReport)** for PR 21155 at commit

[GitHub] spark issue #21158: [SPARK-23850][sql] Add separate config for SQL options r...

2018-05-04 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21158 > usernames could have potentially personal identifiable information User names are not sensitive information. There's a ton of places where you can see them outside of the Spark UI (have

[GitHub] spark pull request #21235: [SPARK-24181][SQL] Better error message for writi...

2018-05-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21235#discussion_r186179922 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -339,9 +339,16 @@ final class DataFrameWriter[T] private[sql](ds:

[GitHub] spark issue #21236: [SPARK-23935][SQL] Adding map_entries function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21236 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to b...

2018-05-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21122#discussion_r186119360 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -1354,7 +1354,8 @@ class HiveDDLSuite val

[GitHub] spark pull request #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to b...

2018-05-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21122#discussion_r186138866 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -1354,7 +1354,8 @@ class HiveDDLSuite val

[GitHub] spark pull request #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to b...

2018-05-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21122#discussion_r186143143 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -1354,7 +1354,8 @@ class HiveDDLSuite val

[GitHub] spark pull request #21217: [SPARK-24151][SQL] Fix CURRENT_DATE, CURRENT_TIME...

2018-05-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/21217#discussion_r186150689 --- Diff: docs/sql-programming-guide.md --- @@ -1812,6 +1812,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see -

[GitHub] spark pull request #21200: [SPARK-24039][SS] Do continuous processing writes...

2018-05-04 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21200#discussion_r186123363 --- Diff: sql/core/pom.xml --- @@ -146,6 +146,11 @@ parquet-avro test + + org.mockito +

[GitHub] spark issue #20787: [MINOR][DOCS] Documenting months_between direction

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20787 **[Test build #90204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90204/testReport)** for PR 20787 at commit

[GitHub] spark issue #20787: [MINOR][DOCS] Documenting months_between direction

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20787 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90204/ Test FAILed. ---

[GitHub] spark issue #20787: [MINOR][DOCS] Documenting months_between direction

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20787 **[Test build #90204 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90204/testReport)** for PR 20787 at commit

[GitHub] spark issue #20787: [MINOR][DOCS] Documenting months_between direction

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20787 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21216: [SPARK-24149][YARN] Retrieve all federated namespaces to...

2018-05-04 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21216 > So, what about adding a check if viewfs is enabled: if so we skip the code added here That sounds ok. --- - To

[GitHub] spark pull request #21235: [SPARK-24181][SQL] Better error message for writi...

2018-05-04 Thread dbtsai
GitHub user dbtsai opened a pull request: https://github.com/apache/spark/pull/21235 [SPARK-24181][SQL] Better error message for writing sorted data ## What changes were proposed in this pull request? The exception message should clearly distinguish sorting and bucketing in

[GitHub] spark issue #21234: [SPARK-24124] Spark history server should create spark.h...

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21234 **[Test build #90200 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90200/testReport)** for PR 21234 at commit

[GitHub] spark issue #21049: [SPARK-23957][SQL] Remove redundant sort operators from ...

2018-05-04 Thread dilipbiswal
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/21049 @henryr I might be a bit of a hardliner on this, but I think it's correct to eliminate the {{ORDER BY}} from common table expressions (e.g. MSSQL agrees with me, see this link). DB>>

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90192/ Test PASSed. ---

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-04 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r186140513 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -413,6 +413,25 @@ class DataFrameFunctionsSuite extends

[GitHub] spark pull request #21233: [PYTHON] Upgrade to Py4J 0.10.7

2018-05-04 Thread HyukjinKwon
Github user HyukjinKwon closed the pull request at: https://github.com/apache/spark/pull/21233 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21173: [SPARK-23856][SQL] Add an option `queryTimeout` in JDBCO...

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21173 **[Test build #90197 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90197/testReport)** for PR 21173 at commit

[GitHub] spark pull request #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to b...

2018-05-04 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21122#discussion_r186152832 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -1354,7 +1354,8 @@ class HiveDDLSuite val

[GitHub] spark issue #21235: [SPARK-24181][SQL] Better error message for writing sort...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21235 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21235: [SPARK-24181][SQL] Better error message for writing sort...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21235 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90206/ Test FAILed. ---

[GitHub] spark issue #21235: [SPARK-24181][SQL] Better error message for writing sort...

2018-05-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21235 **[Test build #90206 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90206/testReport)** for PR 21235 at commit

[GitHub] spark issue #21242: [SPARK-23657][SQL] Document and expose the internal data...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21242 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21242: [SPARK-23657][SQL] Document and expose the internal data...

2018-05-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21242 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2932/

[GitHub] spark pull request #21092: [SPARK-23984][K8S] Initial Python Bindings for Py...

2018-05-04 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r186239751 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala --- @@ -101,17 +112,29 @@ private[spark]

[GitHub] spark pull request #21092: [SPARK-23984][K8S] Initial Python Bindings for Py...

2018-05-04 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r186240772 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/bindings/PythonDriverFeatureStep.scala --- @@ -0,0 +1,59

  1   2   3   4   5   6   >