[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188289317 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownOperatorsToDataSource.scala --- @@ -17,48 +17,81 @@

[GitHub] spark issue #21289: [SPARK-24240] Add a config to control whether InMemoryFi...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21289 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21257 **[Test build #90636 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90636/testReport)** for PR 21257 at commit

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21321 cc @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21291 **[Test build #90646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90646/testReport)** for PR 21291 at commit

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21291 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3235/

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21291 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21332 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21332 **[Test build #90645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90645/testReport)** for PR 21332 at commit

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21332 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90645/ Test FAILed. ---

[GitHub] spark issue #21333: [SPARK-23778][CORE] Avoid unneeded shuffle when union ge...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21333 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21333: [SPARK-23778][CORE] Avoid unneeded shuffle when union ge...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21333 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3236/

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188283550 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,110 @@ case class

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21321 **[Test build #90638 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90638/testReport)** for PR 21321 at commit

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21319 **[Test build #90644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90644/testReport)** for PR 21319 at commit

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21329 **[Test build #90639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90639/testReport)** for PR 21329 at commit

[GitHub] spark issue #21268: [SPARK-24209][SHS] Automatic retrieve proxyBase from Kno...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21268 @vanzin do you happen to have any time for checking this please? Thanks. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21216: [SPARK-24149][YARN] Retrieve all federated namespaces to...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21216 @jerryshao @vanzin any more comments? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21208: [SPARK-23925][SQL] Add array_repeat collection function

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21208 **[Test build #90633 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90633/testReport)** for PR 21208 at commit

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188254469 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188254449 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark issue #21327: [SPARK-24107][CORE][followup] ChunkedByteBuffer.writeFul...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21327 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21330: [SPARK-22234] Support distinct window functions

2018-05-15 Thread jinxing64
GitHub user jinxing64 opened a pull request: https://github.com/apache/spark/pull/21330 [SPARK-22234] Support distinct window functions ## What changes were proposed in this pull request? This pr proposes to support distinct window functions. After this change, query like below

[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188254452 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala --- @@ -22,78 +22,57 @@ import

[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188255611 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownOperatorsToDataSource.scala --- @@ -17,48 +17,81 @@

[GitHub] spark issue #21246: [SPARK-23901][SQL] Add masking functions

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21246 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21246: [SPARK-23901][SQL] Add masking functions

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21246 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90634/ Test PASSed. ---

[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188255505 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownOperatorsToDataSource.scala --- @@ -17,48 +17,81 @@

[GitHub] spark issue #21286: [SPARK-24238][SQL] HadoopFsRelation can't append the sam...

2018-05-15 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/21286 > After the job is committed, `skip_dir/tab1/_temporary` will be deleted. Then when other jobs attempt to commit, an error will be reported. I see. Yes, that's

[GitHub] spark issue #18903: [SPARK-21590][SS]Window start time should support negati...

2018-05-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18903 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19691: [SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP PARTITIO...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19691 @DazhuangSu are you still working on this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188242730 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188241998 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188240371 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188241708 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188245774 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -434,18 +511,139 @@ case class SortMergeJoinExec(

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188246302 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -840,6 +1049,241 @@ private[joins] class

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188240094 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188230817 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -97,13 +100,13 @@ object PhysicalOperation extends

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21329 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3230/

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21329 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188252110 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188251861 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -97,13 +100,13 @@ object PhysicalOperation extends

[GitHub] spark pull request #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwr...

2018-05-15 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/21257#discussion_r188262174 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -163,6 +170,15 @@ class

[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewExecu...

2018-05-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21316 Is it better to update title and the PR description, too? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188269083 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -434,18 +511,139 @@ case class SortMergeJoinExec(

[GitHub] spark issue #21331: [SPARK-24276][SQL] Order of literals in IN should not af...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21331 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21293: [SPARK-24237][SS] Continuous shuffle dependency a...

2018-05-15 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21293#discussion_r188269208 --- Diff: core/src/main/scala/org/apache/spark/Dependency.scala --- @@ -65,15 +65,17 @@ abstract class NarrowDependency[T](_rdd: RDD[T]) extends

[GitHub] spark pull request #21293: [SPARK-24237][SS] Continuous shuffle dependency a...

2018-05-15 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21293#discussion_r188277683 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ContinuousShuffleMapTask.scala --- @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-05-15 Thread zecevicp
Github user zecevicp commented on the issue: https://github.com/apache/spark/pull/21109 @mgaido91 Regarding the amount of code, maybe you can suggest how to reduce it? Because I don't see a way... I think the code is well contained (mostly in separate new classes) and is not

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 Thank you so much for the suggestions! I tried to use IntelliJ a few times but at the end I always return to sbt/terminal/vim after some frustration (mainly due to not being able to configure

[GitHub] spark issue #21045: [WIP][SPARK-23931][SQL] Adds zip function to sparksql

2018-05-15 Thread mn-mikke
Github user mn-mikke commented on the issue: https://github.com/apache/spark/pull/21045 @DylanGuedes What about `eval.value`? Example: ``` val evals = children.map(_.genCode(ctx)) val args = ctx.freshName("args") val inputs = evals.zipWithIndex.map { case

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21321 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90638/ Test PASSed. ---

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-05-15 Thread zecevicp
Github user zecevicp commented on the issue: https://github.com/apache/spark/pull/21109 Well, that is the essence of the contribution: to have a moving window over the data, instead of a fixed block (per equi-join match). To implement a moving window you need something like a queue.

[GitHub] spark pull request #21322: [SPARK-24225][CORE] Support closing AutoClosable ...

2018-05-15 Thread JeetKunDoug
Github user JeetKunDoug commented on a diff in the pull request: https://github.com/apache/spark/pull/21322#discussion_r188295537 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -384,15 +385,36 @@ private[spark] class MemoryStore( }

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21321 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-05-15 Thread zecevicp
Github user zecevicp commented on the issue: https://github.com/apache/spark/pull/21109 Btw, thank you @mgaido91 and @kiszk for the comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21332 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3234/

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21332 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21332 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21332 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3237/

[GitHub] spark issue #21322: [SPARK-24225][CORE] Support closing AutoClosable objects...

2018-05-15 Thread JeetKunDoug
Github user JeetKunDoug commented on the issue: https://github.com/apache/spark/pull/21322 @cloud-fan So your suggestion makes sense - it seems like the best path forward is to check the `isBroadcast` flag on the BlockId passed in to `MemoryStore.remove` and release resources only if

[GitHub] spark issue #21289: [SPARK-24240] Add a config to control whether InMemoryFi...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21289 **[Test build #90637 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90637/testReport)** for PR 21289 at commit

[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188299230 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala --- @@ -22,78 +22,57 @@ import

[GitHub] spark pull request #21332: [SPARK-24236][SS] Continuous replacement for Shuf...

2018-05-15 Thread xuanyuanking
GitHub user xuanyuanking opened a pull request: https://github.com/apache/spark/pull/21332 [SPARK-24236][SS] Continuous replacement for ShuffleExchangeExec ## What changes were proposed in this pull request? 1. New RDD named ContinuousShuffleRowRDD 2. New case class

[GitHub] spark pull request #21333: [SPARK-23778][CORE] Avoid unneeded shuffle when u...

2018-05-15 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/21333 [SPARK-23778][CORE] Avoid unneeded shuffle when union gets an empty RDD ## What changes were proposed in this pull request? When a `union` is invoked on several RDDs of which one is an

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21329 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90639/ Test FAILed. ---

[GitHub] spark issue #21329: [SPARK-24277][SQL] Code clean up in SQL module: HadoopMa...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21329 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188283661 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,110 @@ case class

[GitHub] spark issue #21289: [SPARK-24240] Add a config to control whether InMemoryFi...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21289 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90637/ Test PASSed. ---

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21257 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90636/ Test PASSed. ---

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21257 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces...

2018-05-15 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21312 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21322: [SPARK-24225][CORE] Support closing AutoClosable ...

2018-05-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21322#discussion_r188306362 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -384,15 +385,36 @@ private[spark] class MemoryStore( }

[GitHub] spark issue #21291: [SPARK-24242][SQL] RangeExec should have correct outputO...

2018-05-15 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21291 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21332 **[Test build #90645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90645/testReport)** for PR 21332 at commit

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/21332 cc @jose-torres As we discussion in #21293, the main difference between us is whether we can reuse current implementation of scheduler and shuffle, but in this part about the

[GitHub] spark issue #21322: [SPARK-24225][CORE] Support closing AutoClosable objects...

2018-05-15 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21322 @JeetKunDoug The same issue we discuss above. I think if there's a deserialized version of the variable, it can be not in `MemoryStore` but only serialized bytes in disk store. The reason

[GitHub] spark issue #21333: [SPARK-23778][CORE] Avoid unneeded shuffle when union ge...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21333 **[Test build #90647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90647/testReport)** for PR 21333 at commit

[GitHub] spark issue #21165: [Spark-20087][CORE] Attach accumulators / metrics to 'Ta...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21165 **[Test build #90643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90643/testReport)** for PR 21165 at commit

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21109 @zecevicp for instance do we really need `InMemoryUnsafeRowQueue`? why `ExternalAppendOnlyUnsafeRowArray` is not ok? --- - To

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188286751 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,88 @@ case class

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188287355 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,88 @@ case class

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21319 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3233/

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21319 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21286: [SPARK-24238][SQL] HadoopFsRelation can't append the sam...

2018-05-15 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/21286 Does Spark have a jobID in writing path? Below path is an example in my debugging log: ```

[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...

2018-05-15 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21312 thanks, merging to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21312: [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong ...

2018-05-15 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21312 Thanks @HyukjinKwon @BryanCutler @cloud-fan @icexelloss --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21322: [SPARK-24225][CORE] Support closing AutoClosable objects...

2018-05-15 Thread JeetKunDoug
Github user JeetKunDoug commented on the issue: https://github.com/apache/spark/pull/21322 @viirya it seems from my admittedly cursory look at where we use the `cachedValues` reference map that we should be OK in this case - if there's a deserialized version of the variable (the only

[GitHub] spark pull request #21322: [SPARK-24225][CORE] Support closing AutoClosable ...

2018-05-15 Thread JeetKunDoug
Github user JeetKunDoug commented on a diff in the pull request: https://github.com/apache/spark/pull/21322#discussion_r188314886 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -384,15 +385,36 @@ private[spark] class MemoryStore( }

[GitHub] spark pull request #21322: [SPARK-24225][CORE] Support closing AutoClosable ...

2018-05-15 Thread JeetKunDoug
Github user JeetKunDoug commented on a diff in the pull request: https://github.com/apache/spark/pull/21322#discussion_r188325608 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -384,15 +385,36 @@ private[spark] class MemoryStore( }

[GitHub] spark issue #21332: [SPARK-24236][SS] Continuous replacement for ShuffleExch...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21332 **[Test build #90648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90648/testReport)** for PR 21332 at commit

[GitHub] spark issue #21257: [SPARK-24194] [SQL]HadoopFsRelation cannot overwrite a p...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21257 **[Test build #90636 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90636/testReport)** for PR 21257 at commit

[GitHub] spark issue #21289: [SPARK-24240] Add a config to control whether InMemoryFi...

2018-05-15 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/21289 @cloud-fan @adrian-ionescu I added a test, please check when you have time. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21289: [SPARK-24240] Add a config to control whether InMemoryFi...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21289 **[Test build #90637 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90637/testReport)** for PR 21289 at commit

[GitHub] spark issue #21321: [SPARK-24268][SQL] Use datatype.simpleString in error me...

2018-05-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21321 **[Test build #90638 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90638/testReport)** for PR 21321 at commit

[GitHub] spark pull request #21319: [SPARK-24267][SQL] explicitly keep DataSourceRead...

2018-05-15 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21319#discussion_r188232881 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownOperatorsToDataSource.scala --- @@ -17,48 +17,81 @@

[GitHub] spark pull request #21109: [SPARK-24020][SQL] Sort-merge join inner range op...

2018-05-15 Thread zecevicp
Github user zecevicp commented on a diff in the pull request: https://github.com/apache/spark/pull/21109#discussion_r188258635 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -131,13 +134,101 @@ object ExtractEquiJoinKeys extends

[GitHub] spark issue #21330: [SPARK-22234] Support distinct window functions

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21330 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3231/

[GitHub] spark issue #21330: [SPARK-22234] Support distinct window functions

2018-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21330 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

<    1   2   3   4   5   >