[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21416 LGTM (I didn't look that carefully though) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional command

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3656/ Tes

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21447: [SPARK-24339][SQL]Add project for transform/map/reduce s...

2018-05-28 Thread xdcjie
Github user xdcjie commented on the issue: https://github.com/apache/spark/pull/21447 @maropu I updated the commet. In summary, with this pr can reduce the time of scan and assemble data. In our scenario, the relation(table) have 700 columns. --- --

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21416 @rxin I simplified the test cases as you suggested. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91242 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91242/testReport)** for PR 21416 at commit [`fed2846`](https://github.com/apache/spark/commit/fe

[GitHub] spark pull request #21416: [SPARK-24371] [SQL] Added isInCollection in DataF...

2018-05-28 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/21416#discussion_r191317978 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala --- @@ -392,9 +396,97 @@ class ColumnExpressionSuite extends QueryTest wit

[GitHub] spark pull request #21416: [SPARK-24371] [SQL] Added isInCollection in DataF...

2018-05-28 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/21416#discussion_r191317980 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala --- @@ -392,9 +396,97 @@ class ColumnExpressionSuite extends QueryTest wit

[GitHub] spark pull request #21442: [SPARK-24402] [SQL] Optimize `In` expression when...

2018-05-28 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21442#discussion_r191314972 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -219,7 +219,14 @@ object ReorderAssociativeOperat

[GitHub] spark pull request #21442: [SPARK-24402] [SQL] Optimize `In` expression when...

2018-05-28 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21442#discussion_r191314675 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -219,7 +219,14 @@ object ReorderAssociativeOperat

[GitHub] spark pull request #21436: [SPARK-24250][SQL][Follow-up] support accessing S...

2018-05-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21436 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21436: [SPARK-24250][SQL][Follow-up] support accessing S...

2018-05-28 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21436#discussion_r191313872 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala --- @@ -1021,21 +1021,33 @@ object SparkSession extends Logging { /**

[GitHub] spark issue #21436: [SPARK-24250][SQL][Follow-up] support accessing SQLConf ...

2018-05-28 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21436 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21443: [SPARK-24369][SQL] Correct handling for multiple distinc...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21443 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3655/ Tes

[GitHub] spark issue #21443: [SPARK-24369][SQL] Correct handling for multiple distinc...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21443 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21447: [SPARK-24339][SQL]Add project for transform/map/reduce s...

2018-05-28 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21447 Could you add explain result differences with/without this pr in the description? --- - To unsubscribe, e-mail: reviews-unsubscr.

[GitHub] spark pull request #21447: [SPARK-24339][SQL]Add project for transform/map/r...

2018-05-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21447#discussion_r191312085 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -338,6 +338,17 @@ class AstBuilder(conf: SQLConf) extends

[GitHub] spark issue #21447: [SPARK-24339][SQL]Add project for transform/map/reduce s...

2018-05-28 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21447 @gatorsmile Can you trigger this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-ma

[GitHub] spark issue #21443: [SPARK-24369][SQL] Correct handling for multiple distinc...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21443 **[Test build #91241 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91241/testReport)** for PR 21443 at commit [`29e6485`](https://github.com/apache/spark/commit/29

[GitHub] spark issue #21447: [SPARK-24339][SQL]Add project for transform/map/reduce s...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21447 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21447: [SPARK-24339][SQL]Add project for transform/map/reduce s...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21447 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21447: [SPARK-24339][SQL]Add project for transform/map/r...

2018-05-28 Thread xdcjie
GitHub user xdcjie opened a pull request: https://github.com/apache/spark/pull/21447 [SPARK-24339][SQL]Add project for transform/map/reduce sql to prune column ## What changes were proposed in this pull request? Transform query do not have Project Node, so that it will scan

[GitHub] spark pull request #21378: [SPARK-24326][Mesos] add support for local:// sch...

2018-05-28 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21378#discussion_r191310691 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -418,17 +417,33 @@ private

[GitHub] spark issue #21409: [SPARK-24365][SQL] Add Data Source write benchmark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21409 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21409: [SPARK-24365][SQL] Add Data Source write benchmark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21409 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91237/ Test PASSed. ---

[GitHub] spark issue #21409: [SPARK-24365][SQL] Add Data Source write benchmark

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21409 **[Test build #91237 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91237/testReport)** for PR 21409 at commit [`8ffba61`](https://github.com/apache/spark/commit/8

[GitHub] spark pull request #21416: [SPARK-24371] [SQL] Added isInCollection in DataF...

2018-05-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/21416#discussion_r191306678 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala --- @@ -392,9 +396,97 @@ class ColumnExpressionSuite extends QueryTest with

[GitHub] spark pull request #21416: [SPARK-24371] [SQL] Added isInCollection in DataF...

2018-05-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/21416#discussion_r191306654 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala --- @@ -392,9 +396,97 @@ class ColumnExpressionSuite extends QueryTest with

[GitHub] spark pull request #21443: [SPARK-24369][SQL] Correct handling for multiple ...

2018-05-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21443#discussion_r191306683 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala --- @@ -151,7 +152,7 @@ object RewriteDistinct

[GitHub] spark pull request #21443: [SPARK-24369][SQL] Correct handling for multiple ...

2018-05-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21443#discussion_r191306423 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala --- @@ -687,4 +687,12 @@ class DataFrameAggregateSuite extends QueryTest

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-28 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21437 cc @ueshin @HyukjinKwon @BryanCutler --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comman

[GitHub] spark issue #18717: [SPARK-21510] [SQL] Add isMaterialized() and eager persi...

2018-05-28 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18717 The target of this ticket is 2.4? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-ma

[GitHub] spark issue #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener.aggrega...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21438 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener.aggrega...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21438 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91234/ Test PASSed. ---

[GitHub] spark issue #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener.aggrega...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21438 **[Test build #91234 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91234/testReport)** for PR 21438 at commit [`eb87d2d`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #21443: [SPARK-24369][SQL] Correct handling for multiple distinc...

2018-05-28 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21443 cc @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark pull request #21443: [SPARK-24369][SQL] Correct handling for multiple ...

2018-05-28 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21443#discussion_r191302814 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala --- @@ -151,7 +152,7 @@ object RewriteDist

[GitHub] spark issue #21445: [SPARK-24404][SS] Increase currentEpoch when meet a Epoc...

2018-05-28 Thread LiangchangZ
Github user LiangchangZ commented on the issue: https://github.com/apache/spark/pull/21445 > Looks like the patch is needed only with #21353 #21332 #21293 as of now, right? If then please > state the condition in JIRA issue description as well as PR's description so that we don't

[GitHub] spark pull request #21409: [SPARK-24365][SQL] Add Data Source write benchmar...

2018-05-28 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21409#discussion_r191302688 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceWriteBenchmark.scala --- @@ -0,0 +1,145 @@ +/* + * Licen

[GitHub] spark issue #21446: [SPARK-19613][SS][TEST] Random.nextString is not safe fo...

2018-05-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21446 Thank you for review and merging, @HyukjinKwon . Thank you all! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apa

[GitHub] spark pull request #21443: [SPARK-24369][SQL] Correct handling for multiple ...

2018-05-28 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21443#discussion_r191302155 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala --- @@ -687,4 +687,12 @@ class DataFrameAggregateSuite extends Query

[GitHub] spark pull request #21409: [SPARK-24365][SQL] Add Data Source write benchmar...

2018-05-28 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21409#discussion_r191302141 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceWriteBenchmark.scala --- @@ -0,0 +1,145 @@ +/* + * Licen

[GitHub] spark pull request #21439: [SPARK-24391][SQL] Support arrays of any types by...

2018-05-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21439#discussion_r191299100 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -523,6 +523,8 @@ case class JsonToStructs(

[GitHub] spark pull request #21439: [SPARK-24391][SQL] Support arrays of any types by...

2018-05-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21439#discussion_r191298921 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -523,6 +523,8 @@ case class JsonToStructs(

[GitHub] spark pull request #21439: [SPARK-24391][SQL] Support arrays of any types by...

2018-05-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21439#discussion_r191298844 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -523,6 +523,8 @@ case class JsonToStructs(

[GitHub] spark issue #21439: [SPARK-24391][SQL] Support arrays of any types by from_j...

2018-05-28 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21439 Can we also accept primitive arrays in `to_json`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additiona

[GitHub] spark pull request #21409: [SPARK-24365][SQL] Add Data Source write benchmar...

2018-05-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21409#discussion_r191297180 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceWriteBenchmark.scala --- @@ -0,0 +1,145 @@ +/* + * Licensed to

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-28 Thread ssuchter
ing MBP:~/src/ssuchter-spark% echo $REVISION 1d8a265d13 MBP:~/src/ssuchter-spark% echo $DATE 20180528 MBP:~/src/ssuchter-spark% ./dev/make-distribution.sh --name ${DATE}-${REVISION} --tgz -DzincPort=${ZINC_PORT} -Phadoop-2.7 -Pkubernetes -Pkinesis-asl -Phive -Phive-th

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

2018-05-28 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/21370 @viirya @gatorsmile @ueshin @felixcheung @HyukjinKwon The refactor about generating html code out of `Dataset.scala` was done in 94f3414. Please help to check whether it is appropriate when

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21426 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3654/ Tes

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21426 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3653/ Tes

[GitHub] spark issue #21445: [SPARK-24404][SS] Increase currentEpoch when meet a Epoc...

2018-05-28 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/21445 ``` Looks like the patch is needed only with #21353 #21332 #21293 as of now, right? ``` @HeartSaVioR Yes, sorry for the late explanation. The background is we are running POC based

[GitHub] spark issue #21426: [SPARK-24384][PYTHON][SPARK SUBMIT] Add .py files correc...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21426 **[Test build #91240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91240/testReport)** for PR 21426 at commit [`f015e0d`](https://github.com/apache/spark/commit/f0

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #91239 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91239/testReport)** for PR 13599 at commit [`44500fc`](https://github.com/apache/spark/commit/44

[GitHub] spark pull request #21420: [SPARK-24377][Spark Submit] make --py-files work ...

2018-05-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21420 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21420: [SPARK-24377][Spark Submit] make --py-files work in non ...

2018-05-28 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21420 Thanks @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark issue #21420: [SPARK-24377][Spark Submit] make --py-files work in non ...

2018-05-28 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21420 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/13599 (Oops, the test failure was legitimate.) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional co

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/13599 Actually lets also loop in @ifilonenko who's been thinking about similar things but with more of a K8s bent. --- - To unsubscrib

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21437 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91233/ Test FAILed. ---

[GitHub] spark issue #21437: [SPARK-24397][PYSPARK] Added TaskContext.getLocalPropert...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21437 **[Test build #91233 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91233/testReport)** for PR 21437 at commit [`b9d8dd3`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/13599 It's certainly closer, I haven't had a chance to take a look super recently (been focused on the PySpark K8s integration). I'm still hesitant about this merged as-is from a skim through, but maybe a

[GitHub] spark pull request #21446: [SPARK-19613][SS][TEST] Random.nextString is not ...

2018-05-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21446 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21444: Branch 2.3

2018-05-28 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21444 @mozammal mind closing this please? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional command

[GitHub] spark issue #21446: [SPARK-19613][SS][TEST] Random.nextString is not safe fo...

2018-05-28 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21446 Merged to master and branch-2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #91238 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91238/testReport)** for PR 13599 at commit [`d9a5f00`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91238/ Test FAILed. ---

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3652/ Tes

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #91238 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91238/testReport)** for PR 13599 at commit [`d9a5f00`](https://github.com/apache/spark/commit/d9

[GitHub] spark issue #21446: [SPARK-19613][SS][TEST] Random.nextString is not safe fo...

2018-05-28 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21446 Yea, I was facing this problem too. Thanks for fixing this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.or

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/13599 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark issue #21409: [SPARK-24365][SQL] Add Data Source write benchmark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21409 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3651/ Tes

[GitHub] spark issue #21409: [SPARK-24365][SQL] Add Data Source write benchmark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21409 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3522/ ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3522/ --- --

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3650/ Tes

[GitHub] spark issue #21409: [SPARK-24365][SQL] Add Parquet write benchmark

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21409 **[Test build #91237 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91237/testReport)** for PR 21409 at commit [`8ffba61`](https://github.com/apache/spark/commit/8f

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20697 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91236/ Test FAILed. ---

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91236 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91236/testReport)** for PR 20697 at commit [`1d8a265`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91235/ Test FAILed. ---

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3649/ Tes

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #91235 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91235/testReport)** for PR 13599 at commit [`d9a5f00`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91236 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91236/testReport)** for PR 20697 at commit [`1d8a265`](https://github.com/apache/spark/commit/1d

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-28 Thread ssuchter
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/20697 Fixed the bug. @mccheah I'd appreciate your eyes on commit 1d8a265, for both correctness and style. (I haven't used Scala before this project, so I'm very not confidence in the best way to do thing

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #91235 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91235/testReport)** for PR 13599 at commit [`d9a5f00`](https://github.com/apache/spark/commit/d9

[GitHub] spark issue #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener.aggrega...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21438 **[Test build #91234 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91234/testReport)** for PR 21438 at commit [`eb87d2d`](https://github.com/apache/spark/commit/eb

[GitHub] spark pull request #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener....

2018-05-28 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21438#discussion_r191285757 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala --- @@ -159,7 +159,7 @@ class SQLAppStatusListener(

[GitHub] spark issue #21438: [SPARK-24398] [SQL] Improve SQLAppStatusListener.aggrega...

2018-05-28 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21438 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@sp

[GitHub] spark issue #21446: [SPARK-19613][SS][TEST] Random.nextString is not safe fo...

2018-05-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21446 Thank you for reviewing, @felixcheung and @HeartSaVioR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #21288: [SPARK-24206][SQL] Improve FilterPushdownBenchmar...

2018-05-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21288#discussion_r191283013 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala --- @@ -131,211 +132,214 @@ object FilterPushdownBe

[GitHub] spark pull request #21378: [SPARK-24326][Mesos] add support for local:// sch...

2018-05-28 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21378#discussion_r191271894 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -418,17 +417,34 @@ private

  1   2   3   4   >