[GitHub] spark issue #19920: [SPARK-21672][CORE] Remove SHS-specific application / at...

2017-12-07 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19920 @shaneknapp any idea why the PR builder is not testing this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19920: [SPARK-21672][CORE] Remove SHS-specific application / at...

2017-12-07 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19920 err, ignore that --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19920: [SPARK-21672][CORE] Remove SHS-specific applicati...

2017-12-07 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19920#discussion_r155653359 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala --- @@ -30,7 +31,8 @@ private[history] class HistoryPage(parent:

[GitHub] spark pull request #19920: [SPARK-21672][CORE] Remove SHS-specific applicati...

2017-12-07 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19920#discussion_r155640885 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -17,16 +17,15 @@ package org.apache.spark.status

[GitHub] spark pull request #19924: [SPARK-22187][SS][REVERT] Revert change in state ...

2017-12-07 Thread tdas
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/19924 [SPARK-22187][SS][REVERT] Revert change in state row format for mapGroupsWithState ## What changes were proposed in this pull request? #19416 changed the format in which rows were encoded in

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19717 **[Test build #84613 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84613/testReport)** for PR 19717 at commit

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19717 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19923: [SPARK-22721] BytesToBytesMap peak memory not updated.

2017-12-07 Thread juliuszsompolski
Github user juliuszsompolski commented on the issue: https://github.com/apache/spark/pull/19923 jenkins retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19923: [SPARK-22721] BytesToBytesMap peak memory not updated.

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19923 **[Test build #84627 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84627/testReport)** for PR 19923 at commit

[GitHub] spark pull request #19922: [SPARK-22688][SQL][HOTFIX] Upgrade Janino version...

2017-12-07 Thread srowen
Github user srowen closed the pull request at: https://github.com/apache/spark/pull/19922 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84630/testReport)** for PR 19872 at commit

[GitHub] spark pull request #19813: [SPARK-22600][SQL] Fix 64kb limit for deeply nest...

2017-12-07 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19813#discussion_r155684442 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExpressionCodegen.scala --- @@ -0,0 +1,237 @@ +/* + *

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84632 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84632/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84632/ Test FAILed. ---

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-07 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/19717 I understand the concern here. I think we can guard against accidental use in client mode to post a nicer error message. @liyinan926, how about testing if `kubernetes.default.svc` is accessible?

[GitHub] spark issue #19923: [SPARK-22721] BytesToBytesMap peak memory not updated.

2017-12-07 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19923 yay! LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19920: [SPARK-21672][CORE] Remove SHS-specific application / at...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19920 **[Test build #84620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84620/testReport)** for PR 19920 at commit

[GitHub] spark issue #19683: [SPARK-21657][SQL] optimize explode quadratic memory con...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19683 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84614/ Test PASSed. ---

[GitHub] spark issue #19683: [SPARK-21657][SQL] optimize explode quadratic memory con...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19683 **[Test build #84614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84614/testReport)** for PR 19683 at commit

[GitHub] spark issue #19922: [SPARK-22688][SQL][HOTFIX] Upgrade Janino version to 3.0...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19922 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19922: [SPARK-22688][SQL][HOTFIX] Upgrade Janino version to 3.0...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19922 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84615/ Test PASSed. ---

[GitHub] spark pull request #19813: [SPARK-22600][SQL] Fix 64kb limit for deeply nest...

2017-12-07 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19813#discussion_r155673475 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExpressionCodegen.scala --- @@ -0,0 +1,237 @@ +/* + *

[GitHub] spark pull request #19813: [SPARK-22600][SQL] Fix 64kb limit for deeply nest...

2017-12-07 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19813#discussion_r155673528 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExpressionCodegen.scala --- @@ -0,0 +1,237 @@ +/* + *

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-07 Thread liyinan926
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/19717 There are two cases to consider here. 1) If the client that creates the `SparkContext` is running in a Kubernetes pod, it should work the same way as the driver pod of an application submitted

[GitHub] spark issue #19852: [SPARK-22655][PySpark] Throw exception rather than exit ...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19852 **[Test build #84633 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84633/testReport)** for PR 19852 at commit

[GitHub] spark issue #19920: [SPARK-21672][CORE] Remove SHS-specific application / at...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19920 **[Test build #84620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84620/testReport)** for PR 19920 at commit

[GitHub] spark pull request #18113: [SPARK-20890][SQL] Added min and max typed aggreg...

2017-12-07 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18113#discussion_r155688313 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala --- @@ -263,6 +262,25 @@ class DatasetAggregatorSuite extends

[GitHub] spark issue #19926: [SPARK-22733] Split StreamExecution into MicroBatchExecu...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19926 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84625/ Test FAILed. ---

[GitHub] spark issue #19926: [SPARK-22733] Split StreamExecution into MicroBatchExecu...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19926 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #18995: [SPARK-21787][SQL] Support for pushing down filte...

2017-12-07 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18995 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-12-07 Thread henryr
Github user henryr commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r155690933 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -294,11 +304,22 @@ private void

[GitHub] spark issue #19925: [SPARK-22732] Add Structured Streaming APIs to DataSourc...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19925 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84624/ Test PASSed. ---

[GitHub] spark issue #19925: [SPARK-22732] Add Structured Streaming APIs to DataSourc...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19925 **[Test build #84624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84624/testReport)** for PR 19925 at commit

[GitHub] spark issue #19925: [SPARK-22732] Add Structured Streaming APIs to DataSourc...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19925 **[Test build #84622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84622/testReport)** for PR 19925 at commit

[GitHub] spark issue #19925: [SPARK-22732] Add Structured Streaming APIs to DataSourc...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19925 **[Test build #84624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84624/testReport)** for PR 19925 at commit

[GitHub] spark issue #19721: [SPARK-22496][SQL]thrift server adds operation logs

2017-12-07 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19721 cc @liufengdb --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19920: [SPARK-21672][CORE] Remove SHS-specific application / at...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19920 **[Test build #84618 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84618/testReport)** for PR 19920 at commit

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

2017-12-07 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/19527 > For example with 5 categories, we don't know [0.0, 0.0, 0.0, 0.0, 0.0] means last category or invalid value. For the semantics I described ("OPTION 1"), I think it is clear: * Last

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84630 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84630/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84630/ Test FAILed. ---

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-07 Thread liyinan926
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/19717 Actually given that in our implementations the executors talk to the driver through a Kubernetes headless service that is only created by the submission client. I don't think any of the two

[GitHub] spark pull request #19783: [SPARK-21322][SQL] support histogram in filter ca...

2017-12-07 Thread ron8hu
Github user ron8hu commented on a diff in the pull request: https://github.com/apache/spark/pull/19783#discussion_r155683828 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/EstimationUtils.scala --- @@ -114,4 +114,144 @@ object

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84631 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84631/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84631/ Test FAILed. ---

[GitHub] spark issue #19926: [SPARK-22733] Split StreamExecution into MicroBatchExecu...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19926 **[Test build #84625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84625/testReport)** for PR 19926 at commit

[GitHub] spark issue #18995: [SPARK-21787][SQL] Support for pushing down filters for ...

2017-12-07 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18995 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19920: [SPARK-21672][CORE] Remove SHS-specific applicati...

2017-12-07 Thread smurakozi
Github user smurakozi commented on a diff in the pull request: https://github.com/apache/spark/pull/19920#discussion_r155657306 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala --- @@ -88,4 +90,9 @@ private[history] class HistoryPage(parent:

[GitHub] spark pull request #19920: [SPARK-21672][CORE] Remove SHS-specific applicati...

2017-12-07 Thread smurakozi
Github user smurakozi commented on a diff in the pull request: https://github.com/apache/spark/pull/19920#discussion_r155657274 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala --- @@ -30,7 +31,8 @@ private[history] class HistoryPage(parent:

[GitHub] spark pull request #19920: [SPARK-21672][CORE] Remove SHS-specific applicati...

2017-12-07 Thread smurakozi
Github user smurakozi commented on a diff in the pull request: https://github.com/apache/spark/pull/19920#discussion_r155657473 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -17,16 +17,15 @@ package org.apache.spark.status

[GitHub] spark issue #19925: [SPARK-22732] Add Structured Streaming APIs to DataSourc...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19925 **[Test build #84623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84623/testReport)** for PR 19925 at commit

[GitHub] spark issue #19922: [SPARK-22688][SQL][HOTFIX] Upgrade Janino version to 3.0...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19922 **[Test build #84615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84615/testReport)** for PR 19922 at commit

[GitHub] spark issue #19923: [SPARK-22721] BytesToBytesMap peak memory not updated.

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19923 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19852: [SPARK-22655][PySpark] Throw exception rather tha...

2017-12-07 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19852#discussion_r155686222 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala --- @@ -319,7 +319,7 @@ private[spark] abstract class BasePythonRunner[IN,

[GitHub] spark pull request #19813: [SPARK-22600][SQL] Fix 64kb limit for deeply nest...

2017-12-07 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19813#discussion_r155688125 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExpressionCodegen.scala --- @@ -0,0 +1,237 @@ +/* + *

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize exist...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19864 **[Test build #84612 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84612/testReport)** for PR 19864 at commit

[GitHub] spark issue #19683: [SPARK-21657][SQL] optimize explode quadratic memory con...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19683 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19924: [SPARK-22187][SS][REVERT] Revert change in state row for...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19924 **[Test build #84621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84621/testReport)** for PR 19924 at commit

[GitHub] spark issue #19925: [SPARK-22732] Add Structured Streaming APIs to DataSourc...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19925 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84622/ Test FAILed. ---

[GitHub] spark issue #19925: [SPARK-22732] Add Structured Streaming APIs to DataSourc...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19925 **[Test build #84622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84622/testReport)** for PR 19925 at commit

[GitHub] spark issue #19925: [SPARK-22732] Add Structured Streaming APIs to DataSourc...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19925 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19843: [SPARK-22644][ML][TEST][WIP] Make ML testsuite support S...

2017-12-07 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/19843 I'll make a call: Given that the SQL tests do not use clearActive, let's not bother with it. If we see flakiness, then we can try adding clearActive as a fix. ---

[GitHub] spark issue #19921: [SPARK-22452][SQL] Add getDouble to DataSourceV2Options

2017-12-07 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19921 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84617 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84617/testReport)** for PR 19893 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/19872 I end up removing `UDFColumn` class and using the existing `Aggregate` logical plan for pandas group_agg UDF. This reuses a lot of code being to existing `Aggregate` and minimize the code

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84632/testReport)** for PR 19872 at commit

[GitHub] spark issue #19924: [SPARK-22187][SS][REVERT] Revert change in state row for...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19924 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19924: [SPARK-22187][SS][REVERT] Revert change in state row for...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19924 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84621/ Test PASSed. ---

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-07 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19717 All existing clusters work with client mode, so you're on your own here... maybe setting an environment variable when starting the driver and checking that it's set in your code. Or something. ---

[GitHub] spark issue #19852: [SPARK-22655][PySpark] Throw exception rather than exit ...

2017-12-07 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19852 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19783: [SPARK-21322][SQL] support histogram in filter ca...

2017-12-07 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19783#discussion_r155690722 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -332,8 +332,41 @@ case class

[GitHub] spark issue #19923: [SPARK-22721] BytesToBytesMap peak memory not updated.

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19923 **[Test build #84619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84619/testReport)** for PR 19923 at commit

[GitHub] spark pull request #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to ...

2017-12-07 Thread wesm
Github user wesm commented on a diff in the pull request: https://github.com/apache/spark/pull/19884#discussion_r155647982 --- Diff: python/pyspark/sql/types.py --- @@ -1658,13 +1657,13 @@ def from_arrow_type(at): spark_type = FloatType() elif at ==

[GitHub] spark pull request #19499: [SPARK-22279][SQL] Turn on spark.sql.hive.convert...

2017-12-07 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19499 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2017-12-07 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19893 Yeah, I don't think this can be added before figuring out SQL tests. ``` $ grep 'POSSIBLE THREAD LEAK' unit-tests.log | wc -l 158 ``` ---

[GitHub] spark issue #19591: [SPARK-11035][core] Add in-process Spark app launcher.

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19591 **[Test build #84629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84629/testReport)** for PR 19591 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/19872 And to @holdenk 's question. Pandas group_agg udf fundamentally uses different physical plan than the existing java/scala udf and therefore it's hard to combine them together. I don't know a

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84631/testReport)** for PR 19872 at commit

[GitHub] spark issue #19921: [SPARK-22452][SQL] Add getDouble to DataSourceV2Options

2017-12-07 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19921 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize exist...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19864 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84612/ Test PASSed. ---

[GitHub] spark issue #19864: [SPARK-22673][SQL] InMemoryRelation should utilize exist...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19864 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19884: [WIP][SPARK-22324][SQL][PYTHON] Upgrade Arrow to ...

2017-12-07 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19884#discussion_r155668850 --- Diff: python/pyspark/sql/types.py --- @@ -1658,13 +1657,13 @@ def from_arrow_type(at): spark_type = FloatType() elif at ==

[GitHub] spark issue #19499: [SPARK-22279][SQL] Turn on spark.sql.hive.convertMetasto...

2017-12-07 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19499 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19923: [SPARK-22721] BytesToBytesMap peak memory not updated.

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19923 **[Test build #84619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84619/testReport)** for PR 19923 at commit

[GitHub] spark issue #19923: [SPARK-22721] BytesToBytesMap peak memory not updated.

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19923 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84619/ Test FAILed. ---

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-07 Thread liyinan926
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/19717 @vanzin Sorry I didn't see your question. We don't yet support client mode and no one has ever tried a true client mode, i.e., the driver runs on the client machine outside a Kubernetes cluster.

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84617/ Test PASSed. ---

[GitHub] spark issue #19920: [SPARK-21672][CORE] Remove SHS-specific application / at...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19920 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19920: [SPARK-21672][CORE] Remove SHS-specific application / at...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19920 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84618/ Test PASSed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84628/testReport)** for PR 19872 at commit

[GitHub] spark issue #19926: [SPARK-22733] Split StreamExecution into MicroBatchExecu...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19926 **[Test build #84625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84625/testReport)** for PR 19926 at commit

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-07 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19717 I don't think anyone answered my previous question: > One question though: what happens if someone does new SparkContext(myConf) in their code with a > k8s URL? That's a valid way of

[GitHub] spark issue #19921: [SPARK-22452][SQL] Add getDouble to DataSourceV2Options

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19921 **[Test build #84626 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84626/testReport)** for PR 19921 at commit

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-07 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19717 It's more than a docs issue though, it should be enforced in the code, and as far as I've seen the code path I mentioned does not generate an error (or at least not an obvious one). ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84628/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

2017-12-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84628 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84628/testReport)** for PR 19872 at commit

  1   2   3   4   5   >