[GitHub] spark issue #19630: [SPARK-22409] Introduce function type argument in pandas...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19630 To be honest, I know a super hacky funny workaround I use sometimes but .. want to know if there is any easy way or cleaner way .. ---

[GitHub] spark issue #11674: [SPARK-13838][SQL] Clear variable code to prevent it to ...

2017-11-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/11674 Oh. I see. After looking at the source file at that time:

[GitHub] spark issue #11674: [SPARK-13838][SQL] Clear variable code to prevent it to ...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/11674 Thanks! Since it's a small change, we can do it in future whole-stage-codegen-related PRs. --- - To unsubscribe, e-mail:

[GitHub] spark issue #11674: [SPARK-13838][SQL] Clear variable code to prevent it to ...

2017-11-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/11674 Ok. I think it should be safe to revert this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #11674: [SPARK-13838][SQL] Clear variable code to prevent it to ...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/11674 It's different from `evaluateRequiredVariables`, `evaluateRequiredVariables` pulls out the code to be evaluated and put it in the beginning of the generated code. However here we just clear the

[GitHub] spark pull request #19756: [SPARK-22527][SQL] Reuse coordinated exchanges if...

2017-11-16 Thread cenyuhai
Github user cenyuhai commented on a diff in the pull request: https://github.com/apache/spark/pull/19756#discussion_r151615572 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala --- @@ -109,3 +109,67 @@ case class ReuseExchange(conf: SQLConf)

[GitHub] spark pull request #19756: [SPARK-22527][SQL] Reuse coordinated exchanges if...

2017-11-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19756#discussion_r151614576 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala --- @@ -109,3 +109,67 @@ case class ReuseExchange(conf: SQLConf)

[GitHub] spark pull request #19756: [SPARK-22527][SQL] Reuse coordinated exchanges if...

2017-11-16 Thread cenyuhai
Github user cenyuhai commented on a diff in the pull request: https://github.com/apache/spark/pull/19756#discussion_r151612678 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala --- @@ -109,3 +109,67 @@ case class ReuseExchange(conf: SQLConf)

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19769 **[Test build #83958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83958/testReport)** for PR 19769 at commit

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19769 thanks for the reviews @cloud-fan @HyukjinKwon @ueshin . I've updated the PR to address all the comments, I believe --- - To

[GitHub] spark pull request #19756: [SPARK-22527][SQL] Reuse coordinated exchanges if...

2017-11-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19756#discussion_r151612155 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala --- @@ -109,3 +109,67 @@ case class ReuseExchange(conf: SQLConf)

[GitHub] spark pull request #19756: [SPARK-22527][SQL] Reuse coordinated exchanges if...

2017-11-16 Thread cenyuhai
Github user cenyuhai commented on a diff in the pull request: https://github.com/apache/spark/pull/19756#discussion_r151611386 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala --- @@ -109,3 +109,67 @@ case class ReuseExchange(conf: SQLConf)

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151610021 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -93,13 +94,18 @@ private

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151609609 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +96,113 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151609532 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala --- @@ -355,9 +361,31 @@ class

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-11-16 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r151607724 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -727,8 +769,8 @@ private[spark] class AppStatusListener(

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-11-16 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r151607855 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -40,8 +41,8 @@ private[spark] abstract class LiveEntity {

[GitHub] spark issue #19772: [SPARK-22538][ML] SQLTransformer should not unpersist po...

2017-11-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19772 Although touching ML codes, this is related to SQL. cc @cloud-fan --- - To unsubscribe, e-mail:

[GitHub] spark issue #19772: [SPARK-22538][ML] SQLTransformer should not unpersist po...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19772 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83955/ Test PASSed. ---

[GitHub] spark pull request #19630: [SPARK-22409] Introduce function type argument in...

2017-11-16 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19630#discussion_r151605217 --- Diff: python/pyspark/sql/tests.py --- @@ -3166,6 +3166,92 @@ def test_filtered_frame(self): self.assertTrue(pdf.empty)

[GitHub] spark issue #19772: [SPARK-22538][ML] SQLTransformer should not unpersist po...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19772 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19772: [SPARK-22538][ML] SQLTransformer should not unpersist po...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19772 **[Test build #83955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83955/testReport)** for PR 19772 at commit

[GitHub] spark pull request #19630: [SPARK-22409] Introduce function type argument in...

2017-11-16 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19630#discussion_r151605052 --- Diff: python/pyspark/sql/tests.py --- @@ -3348,18 +3385,6 @@ def test_vectorized_udf_null_string(self): res =

[GitHub] spark issue #19751: [SPARK-20653][core] Add cleaning of old elements from th...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19751 **[Test build #83957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83957/testReport)** for PR 19751 at commit

[GitHub] spark pull request #19630: [SPARK-22409] Introduce function type argument in...

2017-11-16 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19630#discussion_r151605081 --- Diff: python/pyspark/sql/tests.py --- @@ -3166,6 +3166,92 @@ def test_filtered_frame(self): self.assertTrue(pdf.empty)

[GitHub] spark issue #19630: [SPARK-22409] Introduce function type argument in pandas...

2017-11-16 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/19630 As a side question, is there an easy way to run coverage reports with pyspark tests? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19630: [SPARK-22409] Introduce function type argument in pandas...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19630 **[Test build #83956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83956/testReport)** for PR 19630 at commit

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-11-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r151603481 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala --- @@ -93,9 +93,9 @@ class JobProgressListener(conf: SparkConf) extends

[GitHub] spark issue #19772: [SPARK-22538][SQL] SQLTransformer should not unpersist p...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19772 **[Test build #83955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83955/testReport)** for PR 19772 at commit

[GitHub] spark pull request #19772: [SPARK-22538][SQL] SQLTransformer should not unpe...

2017-11-16 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/19772 [SPARK-22538][SQL] SQLTransformer should not unpersist possibly cached input dataset ## What changes were proposed in this pull request? `SQLTransformer.transform` unpersists input dataset

[GitHub] spark issue #19755: [SPARK-22524][SQL] Subquery shows reused on UI SQL tab e...

2017-11-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19755 Also update the UI images in the PR description? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-11-16 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r151324714 --- Diff: core/src/main/scala/org/apache/spark/status/ElementTrackingStore.scala --- @@ -0,0 +1,168 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-11-16 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r151445965 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala --- @@ -93,9 +93,9 @@ class JobProgressListener(conf: SparkConf)

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151596892 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -93,13 +94,18 @@ private

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151598171 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +96,113 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151597632 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala --- @@ -337,6 +341,8 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151597335 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java --- @@ -105,13 +108,23 @@ */

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151597861 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala --- @@ -30,6 +30,7 @@ import

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-11-16 Thread WeichenXu123
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/19627 My local test passed. This test failure looks like test system issue. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19627 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19627 **[Test build #83954 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83954/testReport)** for PR 19627 at commit

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19627 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83954/ Test FAILed. ---

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19627 **[Test build #83954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83954/testReport)** for PR 19627 at commit

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19627 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19627 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83953/ Test FAILed. ---

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19627 **[Test build #83953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83953/testReport)** for PR 19627 at commit

[GitHub] spark issue #19763: [SPARK-22537][core] Aggregation of map output statistics...

2017-11-16 Thread gczsjdy
Github user gczsjdy commented on the issue: https://github.com/apache/spark/pull/19763 Actually, the time gap is O(number of mappers * shuffle partitions). In this case, number of mappers is not very large, while users are more likely to get slowed down when they run on a big data

[GitHub] spark issue #19627: [SPARK-21088][ML][WIP] CrossValidator, TrainValidationSp...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19627 **[Test build #83953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83953/testReport)** for PR 19627 at commit

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-11-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r151597063 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -961,6 +961,15 @@ object SQLConf { .booleanConf

[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...

2017-11-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19767#discussion_r151595494 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala --- @@ -105,6 +105,41 @@ abstract class Expression extends

[GitHub] spark pull request #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile err...

2017-11-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19767#discussion_r151593662 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala --- @@ -105,6 +105,41 @@ abstract class Expression extends

[GitHub] spark issue #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP PARTITIO...

2017-11-16 Thread DazhuangSu
Github user DazhuangSu commented on the issue: https://github.com/apache/spark/pull/19691 @gatorsmile @dongjoon-hyun Could you give me some advice please? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...

2017-11-16 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/19767 Probably, I feel we better track the changes of actual bytecode size statistics (e.g, `maxCodeSize`) in tpcds, so I'll check later. ---

[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...

2017-11-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19767 Also from the numbers provided by @maropu, looks no significant regression. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...

2017-11-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19767 Seems a good approach that saves us much effort to add similar codes for many expressions. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19257: [SPARK-22042] [SQL] ReorderJoinPredicates can break when...

2017-11-16 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19257 how is this coming? it will be good to fix this in 2.2? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19631: [SPARK-22372][core, yarn] Make cluster submission use Sp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19631 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83951/ Test PASSed. ---

[GitHub] spark issue #19631: [SPARK-22372][core, yarn] Make cluster submission use Sp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19631 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19631: [SPARK-22372][core, yarn] Make cluster submission use Sp...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19631 **[Test build #83951 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83951/testReport)** for PR 19631 at commit

[GitHub] spark issue #11674: [SPARK-13838][SQL] Clear variable code to prevent it to ...

2017-11-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/11674 It is correct that we evaluate the variables before generating codes for bound attributes. The variables' codes are clear and won't be evaluated twice. Here this is a safety guard that

[GitHub] spark pull request #19766: [SPARK-22542][SQL] remove unused features in Colu...

2017-11-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19766 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19761: [SPARK-22479][SQL][BRANCH-2.2] Exclude credentials from ...

2017-11-16 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19761 @onursatici Could you please close it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19767: [WIP][SPARK-22543][SQL] fix java 64kb compile error for ...

2017-11-16 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/19767 I like this approach. Does this pr solve the same issue in #18641? or, orthogonal? Anyway, I checked the TPCDS perf with this current pr:

[GitHub] spark issue #19766: [SPARK-22542][SQL] remove unused features in ColumnarBat...

2017-11-16 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19766 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19771: [SPARK-22544][SS]FileStreamSource should use its own had...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19771 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19771: [SPARK-22544][SS]FileStreamSource should use its own had...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19771 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83952/ Test PASSed. ---

[GitHub] spark issue #19771: [SPARK-22544][SS]FileStreamSource should use its own had...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19771 **[Test build #83952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83952/testReport)** for PR 19771 at commit

[GitHub] spark pull request #19643: [SPARK-11421][CORE][PYTHON][R] Added ability for ...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19643#discussion_r151585367 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1838,12 +1852,21 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151575949 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java --- @@ -287,8 +300,8 @@ private

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151576417 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +96,113 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151578253 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +96,113 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151578179 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +96,113 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151580998 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +96,113 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151576716 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +96,113 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151576869 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +96,113 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151575898 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java --- @@ -105,13 +108,23 @@

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151578019 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +96,113 @@ class

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19769 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19769 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83950/ Test PASSed. ---

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19769 **[Test build #83950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83950/testReport)** for PR 19769 at commit

[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files

2017-11-16 Thread vishnusram
Github user vishnusram commented on the issue: https://github.com/apache/spark/pull/16976 Thanks Kwon. It worked --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC disassoci...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19741 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC disassoci...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19741 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83947/ Test PASSed. ---

[GitHub] spark issue #19741: [SPARK-14228][CORE][YARN] Lost executor of RPC disassoci...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19741 **[Test build #83947 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83947/testReport)** for PR 19741 at commit

[GitHub] spark issue #11674: [SPARK-13838][SQL] Clear variable code to prevent it to ...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/11674 for example, `spark.range(10).select('id + 1 as 'i).filter('i + 'i < 4)`. When the filter opetator consumes input, it already pre-evalute `i`, and `'id + 1` is only evaluated once, IIUC. ---

[GitHub] spark issue #19764: [SPARK-22539][SQL] Add second order for rangepartitioner...

2017-11-16 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19764 Doesn't this break the `Partitioner` contract? Rows for the same key will now map to multiple partitions. --- - To

[GitHub] spark pull request #19702: [SPARK-10365][SQL] Support Parquet logical type T...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19702#discussion_r151570606 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala --- @@ -372,23 +381,18 @@

[GitHub] spark pull request #19702: [SPARK-10365][SQL] Support Parquet logical type T...

2017-11-16 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19702#discussion_r151569727 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala --- @@ -372,23 +381,18 @@

[GitHub] spark issue #19771: [SPARK-22544][SS]FileStreamSource should use its own had...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19771 **[Test build #83952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83952/testReport)** for PR 19771 at commit

[GitHub] spark issue #19771: [SPARK-22544][SS]FileStreamSource should use its own had...

2017-11-16 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19771 cc @marmbrus --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19771: [SPARK-22544][SS]FileStreamSource should use its ...

2017-11-16 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/19771 [SPARK-22544][SS]FileStreamSource should use its own hadoop conf to call globPathIfNecessary ## What changes were proposed in this pull request? Pass the FileSystem created using the

[GitHub] spark issue #11674: [SPARK-13838][SQL] Clear variable code to prevent it to ...

2017-11-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/11674 If one variable is used as input to many expressions? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19756: [SPARK-22527][SQL] Reuse coordinated exchanges if possib...

2017-11-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19756 cc @cloud-fan for review. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19630: [SPARK-22409] Introduce function type argument in pandas...

2017-11-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19630 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151563276 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala --- @@ -355,9 +361,31 @@ class

[GitHub] spark issue #19631: [SPARK-22372][core, yarn] Make cluster submission use Sp...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19631 **[Test build #83951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83951/testReport)** for PR 19631 at commit

[GitHub] spark issue #19631: [SPARK-22372][core, yarn] Make cluster submission use Sp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19631 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19631: [SPARK-22372][core, yarn] Make cluster submission use Sp...

2017-11-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19631 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83946/ Test FAILed. ---

[GitHub] spark issue #19631: [SPARK-22372][core, yarn] Make cluster submission use Sp...

2017-11-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19631 **[Test build #83946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83946/testReport)** for PR 19631 at commit

[GitHub] spark pull request #19768: [SPARK-22535][PYSPARK] Sleep before killing the p...

2017-11-16 Thread zsxwing
Github user zsxwing closed the pull request at: https://github.com/apache/spark/pull/19768 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

  1   2   3   4   >