[GitHub] spark issue #21175: [SPARK-24107][CORE] ChunkedByteBuffer.writeFully method ...

2018-05-14 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/21175 No, I mean that the code here can simply follow the write call as straight through code. We don't need to guard against exceptions here because the duplicate of the buffer is used only by a

[GitHub] spark pull request #21175: [SPARK-24107][CORE] ChunkedByteBuffer.writeFully ...

2018-05-14 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21175#discussion_r188160813 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -63,10 +63,15 @@ private[spark] class ChunkedByteBuffer(var chunks:

[GitHub] spark pull request #21175: [SPARK-24107][CORE] ChunkedByteBuffer.writeFully ...

2018-05-14 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21175#discussion_r188161280 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -63,10 +63,15 @@ private[spark] class ChunkedByteBuffer(var chunks:

[GitHub] spark issue #21327: [SPARK-24107][CORE][followup] ChunkedByteBuffer.writeFul...

2018-05-14 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21327 cc @manbuyun @JoshRosen @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional command

[GitHub] spark pull request #21327: [SPARK-24107][CORE][followup] ChunkedByteBuffer.w...

2018-05-14 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/21327 [SPARK-24107][CORE][followup] ChunkedByteBuffer.writeFully method has not reset the limit value ## What changes were proposed in this pull request? According to the discussion in https:/

[GitHub] spark issue #21327: [SPARK-24107][CORE][followup] ChunkedByteBuffer.writeFul...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21327 **[Test build #90622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90622/testReport)** for PR 21327 at commit [`cd2f0e3`](https://github.com/apache/spark/commit/cd

[GitHub] spark issue #21327: [SPARK-24107][CORE][followup] ChunkedByteBuffer.writeFul...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21327 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21327: [SPARK-24107][CORE][followup] ChunkedByteBuffer.writeFul...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21327 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3219/ Tes

[GitHub] spark pull request #21114: [SPARK-22371][CORE] Return None instead of throwi...

2018-05-14 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21114#discussion_r188164036 --- Diff: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala --- @@ -237,6 +236,65 @@ class AccumulatorSuite extends SparkFunSuite with Matchers

[GitHub] spark issue #21221: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21221 **[Test build #90613 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90613/testReport)** for PR 21221 at commit [`10ed328`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #21221: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21221 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90613/ Test PASSed. ---

[GitHub] spark issue #21221: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21221 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-05-14 Thread liyinan926
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/21260 @er0sin PVCs can be mounted similarly to the example below: ``` spark.kubernetes.driver.volumes.persistentVolumeClaim.pv1.mount.path=/mnt spark.kubernetes.driver.volumes.persisten

[GitHub] spark issue #20800: [SPARK-23627][SQL] Provide isEmpty in Dataset

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20800 **[Test build #90615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90615/testReport)** for PR 20800 at commit [`f30d3ec`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #20800: [SPARK-23627][SQL] Provide isEmpty in Dataset

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20800 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90615/ Test PASSed. ---

[GitHub] spark issue #20800: [SPARK-23627][SQL] Provide isEmpty in Dataset

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20800 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21114: [SPARK-22371][CORE] Return None instead of throwi...

2018-05-14 Thread artemrd
Github user artemrd commented on a diff in the pull request: https://github.com/apache/spark/pull/21114#discussion_r188170023 --- Diff: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala --- @@ -237,6 +236,65 @@ class AccumulatorSuite extends SparkFunSuite with Matchers wi

[GitHub] spark issue #21310: [SPARK-24256][SQL] SPARK-24256: ExpressionEncoder should...

2018-05-14 Thread fangshil
Github user fangshil commented on the issue: https://github.com/apache/spark/pull/21310 @viirya @cloud-fan before I add test, could you guys take a look and advise if the approach taken in this patch is acceptable? --- ---

[GitHub] spark issue #17335: [SPARK-19995][YARN] Register tokens to current UGI to av...

2018-05-14 Thread rajeshcode
Github user rajeshcode commented on the issue: https://github.com/apache/spark/pull/17335 Is this patch will work for spark-sql --master local mode as well. In our environment localmode is not supporting proxy user where as yarn mode looks ok. Do we have a solution for proxy

[GitHub] spark issue #21310: [SPARK-24256][SQL] SPARK-24256: ExpressionEncoder should...

2018-05-14 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21310 I will take a look later today. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21114: [SPARK-22371][CORE] Return None instead of throwing an e...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21114 **[Test build #90623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90623/testReport)** for PR 21114 at commit [`1b1b1fa`](https://github.com/apache/spark/commit/1b

[GitHub] spark issue #20894: [SPARK-23786][SQL] Checking column names of csv headers

2018-05-14 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/20894 @gengliangwang @gatorsmile May I ask you to look at this PR one more time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spar

[GitHub] spark issue #20800: [SPARK-23627][SQL] Provide isEmpty in Dataset

2018-05-14 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20800 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #20800: [SPARK-23627][SQL] Provide isEmpty in Dataset

2018-05-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20800 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21295: [SPARK-24230][SQL] Fix SpecificParquetRecordReaderBase w...

2018-05-14 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21295 So far, I guess so, @vanzin . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR

2018-05-14 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21313 Yea, it's not necessarily to add all but adding them in a batch sounds making sense at least once you got used to it. --- -

[GitHub] spark issue #20800: [SPARK-23627][SQL] Provide isEmpty in Dataset

2018-05-14 Thread goungoun
Github user goungoun commented on the issue: https://github.com/apache/spark/pull/20800 Thanks!! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.

[GitHub] spark pull request #21328: Ci

2018-05-14 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/21328 Ci ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was teste

[GitHub] spark pull request #21328: Ci

2018-05-14 Thread wangyum
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/21328 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21324: [SPARK-24035][SQL] SQL syntax for Pivot - fix ant...

2018-05-14 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21324#discussion_r188177303 --- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 --- @@ -398,7 +398,7 @@ hintStatement ; fromCl

[GitHub] spark issue #21324: [SPARK-24035][SQL] SQL syntax for Pivot - fix antlr warn...

2018-05-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21324 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21312 **[Test build #90618 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90618/testReport)** for PR 21312 at commit [`a2d1444`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21312 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90618/ Test PASSed. ---

[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21312 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21329: [SPARK-24277][SQL] Code clean up in SQL module: H...

2018-05-14 Thread gengliangwang
GitHub user gengliangwang opened a pull request: https://github.com/apache/spark/pull/21329 [SPARK-24277][SQL] Code clean up in SQL module: HadoopMapReduceCommitProtocol ## What changes were proposed in this pull request? In HadoopMapReduceCommitProtocol and FileFormatWrite

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21329 **[Test build #90624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90624/testReport)** for PR 21329 at commit [`353606c`](https://github.com/apache/spark/commit/35

[GitHub] spark pull request #21324: [SPARK-24035][SQL] SQL syntax for Pivot - fix ant...

2018-05-14 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21324 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21208: [SPARK-23925][SQL] Add array_repeat collection function

2018-05-14 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21208 LGTM. Btw, seems like commits in this PR are not connected with your account yet. Can we use the email in the commits or something else when we merge this pr? --- --

[GitHub] spark issue #21318: [minor] Update docs for functions.scala to make it clear...

2018-05-14 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21318 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #21318: [minor] Update docs for functions.scala to make it clear...

2018-05-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21318 It's still going to fail because I haven't updated it yet. Will do tomorrow. --- - To unsubscribe, e-mail: reviews-unsubscr...@spar

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21329 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3220/ Tes

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21329 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21246: [SPARK-23901][SQL] Add masking functions

2018-05-14 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21246#discussion_r188179821 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/MaskExpressionsUtils.java --- @@ -0,0 +1,80 @@ +/* + * Licensed to th

[GitHub] spark issue #21326: [SPARK-24275][SQL] Revise doc comments in InputPartition

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21326 **[Test build #90620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90620/testReport)** for PR 21326 at commit [`76a116a`](https://github.com/apache/spark/commit/7

[GitHub] spark pull request #21266: [SPARK-24206][SQL] Improve DataSource read benchm...

2018-05-14 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21266#discussion_r188179709 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataSourceReadBenchmark.scala --- @@ -0,0 +1,826 @@ +/* + * Licensed to the Apache Software Fou

[GitHub] spark issue #21318: [minor] Update docs for functions.scala to make it clear...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21318 **[Test build #90625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90625/testReport)** for PR 21318 at commit [`83c191f`](https://github.com/apache/spark/commit/83

[GitHub] spark issue #21326: [SPARK-24275][SQL] Revise doc comments in InputPartition

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21326 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90620/ Test PASSed. ---

[GitHub] spark issue #21326: [SPARK-24275][SQL] Revise doc comments in InputPartition

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21326 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21318: [minor] Update docs for functions.scala to make it clear...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21318 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21318: [minor] Update docs for functions.scala to make it clear...

2018-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21318 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3221/ Tes

[GitHub] spark pull request #21246: [SPARK-23901][SQL] Add masking functions

2018-05-14 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21246#discussion_r188181176 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala --- @@ -0,0 +1,569 @@ +/* + * Licensed to the

[GitHub] spark pull request #21246: [SPARK-23901][SQL] Add masking functions

2018-05-14 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21246#discussion_r188181573 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala --- @@ -0,0 +1,569 @@ +/* + * Licensed to the

[GitHub] spark pull request #21246: [SPARK-23901][SQL] Add masking functions

2018-05-14 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21246#discussion_r188181134 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala --- @@ -0,0 +1,569 @@ +/* + * Licensed to the

[GitHub] spark pull request #21246: [SPARK-23901][SQL] Add masking functions

2018-05-14 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21246#discussion_r188181885 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MaskExpressionsSuite.scala --- @@ -0,0 +1,236 @@ +/* + * Licensed to

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21257 **[Test build #90619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90619/testReport)** for PR 21257 at commit [`6821795`](https://github.com/apache/spark/commit/6

<    1   2   3   4   5   6