[GitHub] spark issue #19768: [SPARK-22535][PYSPARK] Sleep before killing the python w...

2017-11-16 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19768 Thanks! Merging to 2.2 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19720: [SPARK-22494][SQL] Fix 64KB limit exception with Coalesc...

2017-11-16 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19720 No, a query with a `coalesce` with many/complex parameters will hit this problem. A query with a lot of small `coalesce` will not have the problem. For `AtLeastNNonNulls ` the fix would be safe

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151557718 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -298,7 +304,10 @@ private void

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151556907 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala --- @@ -355,9 +361,31 @@ class

[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...

2017-11-16 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19760 (And even `SparkSubmit` now creates `SparkConf`.) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...

2017-11-16 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19760 There's quite a lot of applications that use just `SparkContext`. And `SparkConf` is generally created before anything else (e.g. in yarn-cluster mode, the Spark code will instantiate `SparkConf`

[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19760 Since now `SparkSession` is the main entry, we just need `spark.conf.get(xxx)` to print deprecation message(still keep the old deprecated configs in `SparkConf`). I think then it's possible to

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151550352 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -298,7 +304,10 @@ private void

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19769 **[Test build #83950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83950/testReport)** for PR 19769 at commit

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151549968 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala --- @@ -355,9 +361,31 @@ class

[GitHub] spark issue #19768: [SPARK-22535][PYSPARK] Sleep before killing the python w...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19768 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19768: [SPARK-22535][PYSPARK] Sleep before killing the python w...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19768 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83945/ Test PASSed. ---

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151547788 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -284,6 +284,13 @@ object SQLConf { .booleanConf

[GitHub] spark issue #19768: [SPARK-22535][PYSPARK] Sleep before killing the python w...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19768 **[Test build #83945 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83945/testReport)** for PR 19768 at commit

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151547738 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -284,6 +284,13 @@ object SQLConf { .booleanConf

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19769 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83948/ Test FAILed. ---

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19769 **[Test build #83948 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83948/testReport)** for PR 19769 at commit

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19769 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19770: [SPARK-21571][WEB UI] Spark history server leaves incomp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19770 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19770: [SPARK-21571][WEB UI] Spark history server leaves incomp...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19770 **[Test build #83949 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83949/testReport)** for PR 19770 at commit

[GitHub] spark issue #19770: [SPARK-21571][WEB UI] Spark history server leaves incomp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19770 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83949/ Test FAILed. ---

[GitHub] spark issue #19770: [SPARK-21571][WEB UI] Spark history server leaves incomp...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19770 **[Test build #83949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83949/testReport)** for PR 19770 at commit

[GitHub] spark issue #19770: [SPARK-21571][WEB UI] Spark history server leaves incomp...

2017-11-16 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19770 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19770: [SPARK-21571][WEB UI] Spark history server leaves incomp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19770 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18791: [SPARK-21571][Scheduler] Spark history server leaves inc...

2017-11-16 Thread ericvandenbergfb
Github user ericvandenbergfb commented on the issue: https://github.com/apache/spark/pull/18791 See continuation of pull request at https://github.com/apache/spark/pull/19770 --- - To unsubscribe, e-mail:

[GitHub] spark pull request #18791: [SPARK-21571][Scheduler] Spark history server lea...

2017-11-16 Thread ericvandenbergfb
Github user ericvandenbergfb closed the pull request at: https://github.com/apache/spark/pull/18791 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19770: [SPARK-21571][WEB UI] Spark history server leaves incomp...

2017-11-16 Thread ericvandenbergfb
Github user ericvandenbergfb commented on the issue: https://github.com/apache/spark/pull/19770 This is a continuation of https://github.com/apache/spark/pull/18791 - the underlying code changed so had to reimplement. ---

[GitHub] spark pull request #19770: [SPARK-21571][WEB UI] Spark history server leaves...

2017-11-16 Thread ericvandenbergfb
GitHub user ericvandenbergfb opened a pull request: https://github.com/apache/spark/pull/19770 [SPARK-21571][WEB UI] Spark history server leaves incomplete or unreadable logs around forever ## What changes were proposed in this pull request? ** Updated pull request based

[GitHub] spark pull request #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC di...

2017-11-16 Thread devaraj-kavali
Github user devaraj-kavali commented on a diff in the pull request: https://github.com/apache/spark/pull/19741#discussion_r151541900 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -268,8 +268,13 @@

[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...

2017-11-16 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19760 It would be possible for configs declared in core; because then we can force SparkConf to initialize those classes. (e.g., referencing any config constant causes the `o.a.s.internal.config` package

[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19760 Generally having the deprecation message is good, but I hope that can be done within the config entry, instead of a central place in `SparkConf`. Is this possible? ---

[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...

2017-11-16 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19760 > the name withAlternatives says it's an alternative, That's the other thing. Having alternative names is just confusing; there should be a single name for a config, with others deprecated.

[GitHub] spark pull request #19760: [SPARK-22533][core] Handle deprecated names in Co...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19760#discussion_r151538808 --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala --- @@ -663,8 +663,10 @@ private[spark] object SparkConf extends Logging {

[GitHub] spark pull request #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC di...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19741#discussion_r151538353 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -268,8 +268,13 @@ private[spark]

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19769 **[Test build #83948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83948/testReport)** for PR 19769 at commit

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/19769 [SPARK-12297][SQL] Adjust timezone for int96 data from impala ## What changes were proposed in this pull request? Int96 data written by impala vs data written by hive & spark is stored

[GitHub] spark issue #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC disassoci...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19741 **[Test build #83947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83947/testReport)** for PR 19741 at commit

[GitHub] spark issue #18918: [SPARK-21707][SQL]Improvement a special case for non-det...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18918 what exactly are you proposing? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC di...

2017-11-16 Thread devaraj-kavali
Github user devaraj-kavali commented on a diff in the pull request: https://github.com/apache/spark/pull/19741#discussion_r151533418 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -268,8 +268,13 @@

[GitHub] spark pull request #18918: [SPARK-21707][SQL]Improvement a special case for ...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18918#discussion_r151533419 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -2029,4 +2029,13 @@ class DataFrameSuite extends QueryTest with

[GitHub] spark issue #19728: [SPARK-22498][SQL] Fix 64KB JVM bytecode limit problem w...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19728 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19728: [SPARK-22498][SQL] Fix 64KB JVM bytecode limit problem w...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19728 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83944/ Test PASSed. ---

[GitHub] spark issue #19728: [SPARK-22498][SQL] Fix 64KB JVM bytecode limit problem w...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19728 **[Test build #83944 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83944/testReport)** for PR 19728 at commit

[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19767 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83943/ Test PASSed. ---

[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19767 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19767 **[Test build #83943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83943/testReport)** for PR 19767 at commit

[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16976 Guys, please use the mailing list or JIRA ... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19630: [SPARK-22409] Introduce function type argument in pandas...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19630 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19630: [SPARK-22409] Introduce function type argument in pandas...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19630 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83942/ Test PASSed. ---

[GitHub] spark issue #19630: [SPARK-22409] Introduce function type argument in pandas...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19630 **[Test build #83942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83942/testReport)** for PR 19630 at commit

[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16976 It was renamed `multiLine` before the release. Could we try out it instead? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19720: [SPARK-22494][SQL] Fix 64KB limit exception with Coalesc...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19720 If there is a query with a lot of coalesce function, wouldn't it hit the 64kb issue? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19760 I don't exactly follow the purpose here, the name `withAlternatives` says it's an alternative, not deprecated, so it should not print deprecation message. Besides, I think `withAlternatives` is

[GitHub] spark pull request #19760: [SPARK-22533][core] Handle deprecated names in Co...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19760#discussion_r151524853 --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala --- @@ -663,8 +663,10 @@ private[spark] object SparkConf extends Logging {

[GitHub] spark pull request #19630: [SPARK-22409] Introduce function type argument in...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19630#discussion_r151524338 --- Diff: python/pyspark/sql/tests.py --- @@ -3166,6 +3166,92 @@ def test_filtered_frame(self): self.assertTrue(pdf.empty)

[GitHub] spark pull request #19630: [SPARK-22409] Introduce function type argument in...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19630#discussion_r151524279 --- Diff: python/pyspark/sql/tests.py --- @@ -3166,6 +3166,92 @@ def test_filtered_frame(self): self.assertTrue(pdf.empty)

[GitHub] spark pull request #19630: [SPARK-22409] Introduce function type argument in...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19630#discussion_r151523879 --- Diff: python/pyspark/sql/tests.py --- @@ -3166,6 +3166,92 @@ def test_filtered_frame(self): self.assertTrue(pdf.empty)

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r151521406 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -451,15 +468,22 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r151520818 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -175,17 +176,39 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r151521750 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -451,15 +468,22 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r151521101 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -228,24 +254,15 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r151520336 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -349,13 +349,22

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r151521265 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -427,10 +444,10 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r151521542 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -451,15 +468,22 @@ trait

[GitHub] spark issue #19631: [SPARK-22372][core, yarn] Make cluster submission use Sp...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19631 **[Test build #83946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83946/testReport)** for PR 19631 at commit

[GitHub] spark issue #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "spark.m...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17436 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83941/ Test PASSed. ---

[GitHub] spark issue #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "spark.m...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17436 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "spark.m...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17436 **[Test build #83941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83941/testReport)** for PR 17436 at commit

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r151510043 --- Diff: core/src/main/scala/org/apache/spark/status/ElementTrackingStore.scala --- @@ -0,0 +1,168 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19730: [SPARK-22500][SQL] Fix 64KB JVM bytecode limit pr...

2017-11-16 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19730#discussion_r151509423 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala --- @@ -1039,13 +1039,19 @@ case class Cast(child: Expression,

[GitHub] spark pull request #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC di...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19741#discussion_r151509249 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -268,8 +268,13 @@ private[spark]

[GitHub] spark issue #19768: [SPARK-22535][PYSPARK] Sleep before killing the python w...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19768 **[Test build #83945 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83945/testReport)** for PR 19768 at commit

[GitHub] spark issue #19768: [SPARK-22535][PYSPARK] Sleep before killing the python w...

2017-11-16 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19768 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19768: [SPARK-22535][PYSPARK] Sleep before killing the p...

2017-11-16 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/19768 [SPARK-22535][PYSPARK] Sleep before killing the python worker in PythRunner.MonitorThread (branch-2.2) ## What changes were proposed in this pull request? Backport #19762 to 2.2

[GitHub] spark issue #9428: [SPARK-8582][Core]Optimize checkpointing to avoid computi...

2017-11-16 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/9428 > I simply cant recompute things as results will be different. A task may run multiple times due to failure. Why is this not a problem for you? ---

[GitHub] spark pull request #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC di...

2017-11-16 Thread devaraj-kavali
Github user devaraj-kavali commented on a diff in the pull request: https://github.com/apache/spark/pull/19741#discussion_r151497034 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -268,8 +268,13 @@

[GitHub] spark pull request #19630: [SPARK-22409] Introduce function type argument in...

2017-11-16 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19630#discussion_r151493167 --- Diff: python/pyspark/sql/tests.py --- @@ -3166,6 +3166,92 @@ def test_filtered_frame(self): self.assertTrue(pdf.empty)

[GitHub] spark pull request #19760: [SPARK-22533][core] Handle deprecated names in Co...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19760#discussion_r151492713 --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala --- @@ -663,8 +663,10 @@ private[spark] object SparkConf extends Logging {

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-11-16 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19601 I see. Let us revisit this design later. I would appreciate it if you would review this columnar cache reader with simple primitive-type (non-nested) array. ---

[GitHub] spark issue #19720: [SPARK-22494][SQL] Fix 64KB limit exception with Coalesc...

2017-11-16 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19720 It's not about running slower. This PR solves the problem which makes the user facing an exception if there are a lot of arguments in `coalesce` (or `AtLestNNonNulls`), but what I am doing in the

[GitHub] spark issue #19630: [SPARK-22409] Introduce function type argument in pandas...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19630 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19631: [SPARK-22372][core, yarn] Make cluster submission...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19631#discussion_r151490038 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -216,7 +216,9 @@ private[spark] object

[GitHub] spark pull request #19630: [SPARK-22409] Introduce function type argument in...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19630#discussion_r151489959 --- Diff: python/pyspark/sql/tests.py --- @@ -3166,6 +3166,92 @@ def test_filtered_frame(self): self.assertTrue(pdf.empty)

[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files

2017-11-16 Thread vishnusram
Github user vishnusram commented on the issue: https://github.com/apache/spark/pull/16976 The wholeFile option doesn't seem to be working. **Test file content:** "num_col1","txt_col","num_col2" 10001,"regular string",20001 10002,"string with newline",20002

[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19767 **[Test build #83943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83943/testReport)** for PR 19767 at commit

[GitHub] spark issue #19728: [SPARK-22498][SQL] Fix 64KB JVM bytecode limit problem w...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19728 **[Test build #83944 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83944/testReport)** for PR 19728 at commit

[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19767 cc @kiszk @rednaxelafx @maropu @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...

2017-11-16 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/19767 [WIP][SPARK-22543][SQL] fix java 64kb compile error for deeply nested expressions ## What changes were proposed in this pull request? A frequently reported issue of Spark is the Java

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-11-16 Thread DaimonPl
Github user DaimonPl commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r151485646 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -961,6 +961,15 @@ object SQLConf { .booleanConf

[GitHub] spark issue #19765: [SPARK-22540][SQL] Fix incorrect avgSize in HighlyCompre...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19765 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83940/ Test PASSed. ---

[GitHub] spark issue #19765: [SPARK-22540][SQL] Fix incorrect avgSize in HighlyCompre...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19765 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #9428: [SPARK-8582][Core]Optimize checkpointing to avoid computi...

2017-11-16 Thread ferdonline
Github user ferdonline commented on the issue: https://github.com/apache/spark/pull/9428 Hello. I find this feature to be really important and I would be happy to contribute here. Even though we would potentially not support every use case, it would already be great if in the

[GitHub] spark pull request #19733: [SPARK-22501][SQL] Fix 64KB JVM bytecode limit pr...

2017-11-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19733 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19765: [SPARK-22540][SQL] Fix incorrect avgSize in HighlyCompre...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19765 **[Test build #83940 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83940/testReport)** for PR 19765 at commit

[GitHub] spark issue #19720: [SPARK-22494][SQL] Fix 64KB limit exception with Coalesc...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19720 hmm, isn't running slower better than can't run? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #19728: [SPARK-22498][SQL] Fix 64KB JVM bytecode limit pr...

2017-11-16 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19728#discussion_r151482932 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -63,15 +63,32 @@ case class Concat(children:

[GitHub] spark issue #19733: [SPARK-22501][SQL] Fix 64KB JVM bytecode limit problem w...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19733 thanks , merging to master/2.2! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19720: [SPARK-22494][SQL] Fix 64KB limit exception with Coalesc...

2017-11-16 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19720 @cloud-fan please do not backport this to 2.2. In 2.2 we don't have SPARK-18016 and this is adding new variables in the case of coalesce. Thus it can generate an higher pressure on the constant

[GitHub] spark pull request #19720: [SPARK-22494][SQL] Fix 64KB limit exception with ...

2017-11-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19720 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19720: [SPARK-22494][SQL] Fix 64KB limit exception with Coalesc...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19720 thanks, merging to master/2.2! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

<    1   2   3   4   >