[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21319 **[Test build #90573 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90573/testReport)** for PR 21319 at commit [`ca6ccb2`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21319 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21319 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90573/ Test PASSed. ---

[GitHub] spark pull request #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwr...

2018-05-14 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21257#discussion_r187935923 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -163,6 +169,12 @@ class HadoopMapReduceCommitProtoc

[GitHub] spark issue #21266: [SPARK-24206][SQL] Improve DataSource read benchmark cod...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21266 **[Test build #90579 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90579/testReport)** for PR 21266 at commit [`fc96adb`](https://github.com/apache/spark/commit/fc

[GitHub] spark issue #21311: [SPARK-24257][SQL]LongToUnsafeRowMap calculate the new s...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21311 **[Test build #90574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90574/testReport)** for PR 21311 at commit [`22a2767`](https://github.com/apache/spark/commit/22

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21291 **[Test build #90583 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90583/testReport)** for PR 21291 at commit [`3a14bd6`](https://github.com/apache/spark/commit/3a

[GitHub] spark issue #19293: [SPARK-22079][SQL] Serializer in HiveOutputWriter miss l...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19293 **[Test build #90580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90580/testReport)** for PR 19293 at commit [`45477fb`](https://github.com/apache/spark/commit/45

[GitHub] spark issue #21165: [Spark-20087][CORE] Attach accumulators / metrics to 'Ta...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21165 **[Test build #90585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90585/testReport)** for PR 21165 at commit [`05d1d9c`](https://github.com/apache/spark/commit/05

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21317 **[Test build #90576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90576/testReport)** for PR 21317 at commit [`aa0ccc0`](https://github.com/apache/spark/commit/aa

[GitHub] spark issue #21311: [SPARK-24257][SQL]LongToUnsafeRowMap calculate the new s...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21311 **[Test build #90575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90575/testReport)** for PR 21311 at commit [`d9d8e62`](https://github.com/apache/spark/commit/d9

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21317 **[Test build #90578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90578/testReport)** for PR 21317 at commit [`aa0ccc0`](https://github.com/apache/spark/commit/aa

[GitHub] spark issue #20933: [SPARK-23817][SQL]Migrate ORC file format read path to d...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20933 **[Test build #90584 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90584/testReport)** for PR 20933 at commit [`67b1748`](https://github.com/apache/spark/commit/67

[GitHub] spark issue #21114: [SPARK-22371][CORE] Return None instead of throwing an e...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21114 **[Test build #90577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90577/testReport)** for PR 21114 at commit [`8b30733`](https://github.com/apache/spark/commit/8b

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21320 **[Test build #90582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90582/testReport)** for PR 21320 at commit [`9e301b3`](https://github.com/apache/spark/commit/9e

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20894 **[Test build #90581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90581/testReport)** for PR 20894 at commit [`2bd2713`](https://github.com/apache/spark/commit/2b

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20894 **[Test build #90581 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90581/testReport)** for PR 20894 at commit [`2bd2713`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20894 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90581/ Test FAILed. ---

[GitHub] spark pull request #21321: [SPARK-24268][SQL] Use datatype.simpleString in e...

2018-05-14 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/21321 [SPARK-24268][SQL] Use datatype.simpleString in error messages ## What changes were proposed in this pull request? SPARK-22893 tried to unify error messages about dataTypes. Unfortunately,

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20894 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21321 **[Test build #90586 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90586/testReport)** for PR 21321 at commit [`ada7667`](https://github.com/apache/spark/commit/ad

[GitHub] spark issue #21165: [Spark-20087][CORE] Attach accumulators / metrics to 'Ta...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21165 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21165: [Spark-20087][CORE] Attach accumulators / metrics to 'Ta...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21165 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90585/ Test FAILed. ---

[GitHub] spark issue #21165: [Spark-20087][CORE] Attach accumulators / metrics to 'Ta...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21165 **[Test build #90585 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90585/testReport)** for PR 21165 at commit [`05d1d9c`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #21266: [SPARK-24206][SQL] Improve DataSource read benchmark cod...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21266 **[Test build #90579 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90579/testReport)** for PR 21266 at commit [`fc96adb`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #21266: [SPARK-24206][SQL] Improve DataSource read benchmark cod...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21266 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90579/ Test FAILed. ---

[GitHub] spark issue #21266: [SPARK-24206][SQL] Improve DataSource read benchmark cod...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21266 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3194/ Tes

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21317 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3099/ --- --

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21317 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21317 **[Test build #90578 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90578/testReport)** for PR 21317 at commit [`aa0ccc0`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21317 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21317 **[Test build #90576 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90576/testReport)** for PR 21317 at commit [`aa0ccc0`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90576/ Test PASSed. ---

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90578/ Test PASSed. ---

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21317 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21246: [SPARK-23901][SQL] Add masking functions

2018-05-14 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21246#discussion_r187940989 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/MaskExpressionsUtils.java --- @@ -0,0 +1,80 @@ +/* + * Licensed to

[GitHub] spark issue #21266: [SPARK-24206][SQL] Improve DataSource read benchmark cod...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21266 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21266: [SPARK-24206][SQL] Improve DataSource read benchmark cod...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21266 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3195/ Tes

[GitHub] spark pull request #21246: [SPARK-23901][SQL] Add masking functions

2018-05-14 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21246#discussion_r187941913 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala --- @@ -0,0 +1,569 @@ +/* + * Licensed to th

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21317 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3099/ ---

[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...

2018-05-14 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21299 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21320 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21320 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3196/ Tes

[GitHub] spark pull request #21290: [SPARK-24241][Submit]Do not fail fast when dynami...

2018-05-14 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21290#discussion_r187943795 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -76,6 +75,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[

[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21299 **[Test build #90587 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90587/testReport)** for PR 21299 at commit [`a100dea`](https://github.com/apache/spark/commit/a1

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21291 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3197/ Tes

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21291 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21246: [SPARK-23901][SQL] Add masking functions

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21246 **[Test build #90588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90588/testReport)** for PR 21246 at commit [`06b8b6c`](https://github.com/apache/spark/commit/06

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20894 **[Test build #90589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90589/testReport)** for PR 20894 at commit [`21f8b10`](https://github.com/apache/spark/commit/21

[GitHub] spark issue #20933: [SPARK-23817][SQL]Migrate ORC file format read path to d...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20933 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20933: [SPARK-23817][SQL]Migrate ORC file format read path to d...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20933 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3198/ Tes

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-14 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r187946945 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala --- @@ -136,6 +136,59 @@ class Collection

[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21299 **[Test build #90590 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90590/testReport)** for PR 21299 at commit [`01e288a`](https://github.com/apache/spark/commit/01

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21321 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21321 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3199/ Tes

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-14 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21291 The test failure is caused by a bug. I proposed a fix at #21312. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.or

[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21299 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3200/ Tes

[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21299 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-14 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r187950642 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -529,6 +567,239 @@ case class ArrayConta

[GitHub] spark pull request #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-14 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21028#discussion_r187950523 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -529,6 +567,239 @@ case class ArrayConta

[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21299 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...

2018-05-14 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21312 @viirya Thanks for catching this! I think we have many tests that excise the array types. I am curious why this is not caught by existing tests, e.g: https://github.com/apache/spark/

[GitHub] spark issue #21246: [SPARK-23901][SQL] Add masking functions

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21246 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21299 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3202/ Tes

[GitHub] spark issue #21246: [SPARK-23901][SQL] Add masking functions

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21246 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3201/ Tes

[GitHub] spark pull request #21291: [SPARK-24242][SQL] RangeExec should have correct ...

2018-05-14 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21291#discussion_r187951662 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala --- @@ -621,6 +621,25 @@ class PlannerSuite extends SharedSQLContext {

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21291 **[Test build #90591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90591/testReport)** for PR 21291 at commit [`59499ad`](https://github.com/apache/spark/commit/59

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21028 **[Test build #90592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90592/testReport)** for PR 21028 at commit [`49d9372`](https://github.com/apache/spark/commit/49

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-05-14 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21082 Gentle ping @hvanhovell @ueshin Is there anything else I can do to move forward with this PR? --- - To unsubscribe,

[GitHub] spark issue #21286: [SPARK-24238][SQL] HadoopFsRelation can't append the sam...

2018-05-14 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/21286 > cc @steveloughran who I believe is the expert in this area. I suppose "Stepped through the FileOutputCommit operations with a debugger and a pen and paper" counts, given the complexi

[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...

2018-05-14 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21312 @icexelloss It only happens when there are more than one batch in each partition. Existing tests do not hit this condition. That is why the added test here is doing a `repartition`: `df = se

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3203/ Tes

[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21028 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwr...

2018-05-14 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/21257#discussion_r187953870 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -163,6 +169,12 @@ class HadoopMapReduceCommitPr

[GitHub] spark pull request #21106: [SPARK-23711][SQL][WIP] Add fallback logic for Un...

2018-05-14 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21106#discussion_r187954345 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedUnsafeProjection.scala --- @@ -87,12 +87,11 @@ class InterpretedU

[GitHub] spark pull request #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwr...

2018-05-14 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/21257#discussion_r187954805 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -235,4 +247,23 @@ class HadoopMapReduceCommitPr

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21291 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21291 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3204/ Tes

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21257 **[Test build #90593 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90593/testReport)** for PR 21257 at commit [`273b1af`](https://github.com/apache/spark/commit/27

[GitHub] spark issue #21106: [SPARK-23711][SQL] Add fallback generator for UnsafeProj...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21106 **[Test build #90594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90594/testReport)** for PR 21106 at commit [`de88f88`](https://github.com/apache/spark/commit/de

[GitHub] spark pull request #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwr...

2018-05-14 Thread zheh12
Github user zheh12 commented on a diff in the pull request: https://github.com/apache/spark/pull/21257#discussion_r187959698 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -163,6 +169,12 @@ class HadoopMapReduceCommitProtocol(

[GitHub] spark issue #21193: [SPARK-24121][SQL] Add API for handling expression code ...

2018-05-14 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21193 ping @hvanhovell @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: r

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

2018-05-14 Thread ssimeonov
Github user ssimeonov commented on the issue: https://github.com/apache/spark/pull/20894 @HyukjinKwon we are one of the Spark users experiencing this problem in the real world: dealing with dirty data produced by a variety of third party systems. Documentation doesn't solve anything h

[GitHub] spark issue #21106: [SPARK-23711][SQL] Add fallback generator for UnsafeProj...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21106 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21106: [SPARK-23711][SQL] Add fallback generator for UnsafeProj...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21106 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3205/ Tes

[GitHub] spark issue #21153: [SPARK-24058][ML][PySpark] Default Params in ML should b...

2018-05-14 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21153 Not sure why AppVeyor build failed. How to re-trigger AppVeyor build? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apac

[GitHub] spark pull request #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwr...

2018-05-14 Thread zheh12
Github user zheh12 commented on a diff in the pull request: https://github.com/apache/spark/pull/21257#discussion_r187960677 --- Diff: core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala --- @@ -120,7 +120,8 @@ abstract class FileCommitProtocol { * Spe

[GitHub] spark issue #21286: [SPARK-24238][SQL] HadoopFsRelation can't append the sam...

2018-05-14 Thread zheh12
Github user zheh12 commented on the issue: https://github.com/apache/spark/pull/21286 Thanks @cloud-fan @steveloughran for your reply, I will look more detail on this problem. --- - To unsubscribe, e-mail: reviews-u

[GitHub] spark pull request #21114: [SPARK-22371][CORE] Return None instead of throwi...

2018-05-14 Thread artemrd
Github user artemrd commented on a diff in the pull request: https://github.com/apache/spark/pull/21114#discussion_r187970775 --- Diff: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala --- @@ -237,6 +236,65 @@ class AccumulatorSuite extends SparkFunSuite with Matchers wi

[GitHub] spark issue #21114: [SPARK-22371][CORE] Return None instead of throwing an e...

2018-05-14 Thread artemrd
Github user artemrd commented on the issue: https://github.com/apache/spark/pull/21114 This issue is more like a race condition, so the test needs to generate a specific sequence of events to reproduce the issue. I agree it's probably too specific. What is Spark approach to reproduce

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-14 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21321 `jsonExpressions.scala` also has the same issue? https://github.com/apache/spark/blob/b6c50d7820aafab172835633fb0b35899e93146b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions

[GitHub] spark pull request #21193: [SPARK-24121][SQL] Add API for handling expressio...

2018-05-14 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21193#discussion_r187978104 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala --- @@ -623,8 +624,14 @@ case class Cast(child: Expression, dataT

[GitHub] spark issue #21193: [SPARK-24121][SQL] Add API for handling expression code ...

2018-05-14 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21193 I have no comment on this pr and LGTM (I'm looking forward to your follow-up about the manual reference issue.). --- - To unsubsc

[GitHub] spark issue #21286: [SPARK-24238][SQL] HadoopFsRelation can't append the sam...

2018-05-14 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21286 Thanks @steveloughran for your deep explanation! Spark does have a unique job id, but it's only unique within a SparkContext, we may have 2 different spark applications writing to the same

[GitHub] spark pull request #21165: [Spark-20087][CORE] Attach accumulators / metrics...

2018-05-14 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21165#discussion_r187981280 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -287,6 +287,28 @@ private[spark] class Executor( notifyAll()

[GitHub] spark issue #21165: [Spark-20087][CORE] Attach accumulators / metrics to 'Ta...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21165 **[Test build #90595 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90595/testReport)** for PR 21165 at commit [`945c1d5`](https://github.com/apache/spark/commit/94

[GitHub] spark pull request #21193: [SPARK-24121][SQL] Add API for handling expressio...

2018-05-14 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21193#discussion_r187982910 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala --- @@ -623,8 +624,14 @@ case class Cast(child: Expression, dat

[GitHub] spark issue #21193: [SPARK-24121][SQL] Add API for handling expression code ...

2018-05-14 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21193 @viirya actually I had just one last comment: https://github.com/apache/spark/pull/21193#discussion_r187612068 --- - To unsubsc

[GitHub] spark issue #21165: [Spark-20087][CORE] Attach accumulators / metrics to 'Ta...

2018-05-14 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21165 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apa

<    1   2   3   4   5   6   >