[GitHub] spark issue #19804: [WIP][SPARK-22573][SQL] Shouldn't inferFilters if it con...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19804 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #20977: [SPARK-23867][Scheduler] use droppedCount in logW...

2018-04-05 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20977#discussion_r179665313 --- Diff: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala --- @@ -166,7 +166,7 @@ private class AsyncEventQueue(val name: String,

[GitHub] spark issue #19804: [WIP][SPARK-22573][SQL] Shouldn't inferFilters if it con...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19804 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2021/

[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19802 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20962: [SPARK-23847][PYTHON][SQL]Add asc_nulls_first, asc_nulls...

2018-04-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20962 LGTM otherwise. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20962: [SPARK-23847][PYTHON][SQL]Add asc_nulls_first, as...

2018-04-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20962#discussion_r179664682 --- Diff: python/pyspark/sql/column.py --- @@ -450,21 +450,69 @@ def isin(self, *cols): Returns a sort expression based on the ascending order

[GitHub] spark pull request #20928: [MINOR][DOC] Fix some typos and grammar issues

2018-04-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20928 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20948: corrected filename for spark config

2018-04-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20948 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20962: [SPARK-23847][PYTHON][SQL]Add asc_nulls_first, as...

2018-04-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20962#discussion_r179664441 --- Diff: python/pyspark/sql/column.py --- @@ -450,21 +450,69 @@ def isin(self, *cols): Returns a sort expression based on the ascending order

[GitHub] spark issue #20928: [MINOR][DOC] Fix some typos and grammar issues

2018-04-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20928 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20871: [SPARK-23762][SQL] UTF8StringBuffer uses MemoryBlock

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20871 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2020/

[GitHub] spark issue #20871: [SPARK-23762][SQL] UTF8StringBuffer uses MemoryBlock

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20871 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20871: [SPARK-23762][SQL] UTF8StringBuffer uses MemoryBlock

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20871 **[Test build #88964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88964/testReport)** for PR 20871 at commit

[GitHub] spark issue #20871: [SPARK-23762][SQL] UTF8StringBuffer uses MemoryBlock

2018-04-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20871 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19881: [SPARK-22683][CORE] Add a fullExecutorAllocationDivisor ...

2018-04-05 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19881 Maybe instead of "divisor", we just have a "rate" or "factor" that can be floating point value, and use multiplication rather than division? This way people can also make it even more aggressive.

[GitHub] spark issue #20990: Fix typo in Python docstring kinesis example

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20990 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20990: Fix typo in Python docstring kinesis example

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20990 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20990: Fix typo in docstring

2018-04-05 Thread emirot
GitHub user emirot opened a pull request: https://github.com/apache/spark/pull/20990 Fix typo in docstring ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how

[GitHub] spark issue #20987: [SPARK-23816][CORE] Killed tasks should ignore FetchFail...

2018-04-05 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/20987 Thanks for fixing this @squito, LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #20987: [SPARK-23816][CORE] Killed tasks should ignore Fe...

2018-04-05 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/20987#discussion_r179654822 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -494,19 +507,6 @@ private[spark] class Executor(

[GitHub] spark issue #20871: [SPARK-23762][SQL] UTF8StringBuffer uses MemoryBlock

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20871 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20871: [SPARK-23762][SQL] UTF8StringBuffer uses MemoryBlock

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20871 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2019/

[GitHub] spark pull request #20886: [SPARK-19724][SQL]create a managed table with an ...

2018-04-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20886 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20886: [SPARK-19724][SQL]create a managed table with an existed...

2018-04-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20886 Thanks! Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20975: [SPARK-23863] [SQL] Wholetext mode should not add line b...

2018-04-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20975 Thanks for the details and confirmation, @barrenlake. I helped me understand the intention. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20871: [SPARK-23762][SQL] UTF8StringBuffer uses MemoryBlock

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20871 **[Test build #88963 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88963/testReport)** for PR 20871 at commit

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20989 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1981/ ---

[GitHub] spark issue #20953: [SPARK-23822][SQL] Improve error message for Parquet sch...

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20953 **[Test build #88962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88962/testReport)** for PR 20953 at commit

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20971 **[Test build #88961 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88961/testReport)** for PR 20971 at commit

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

2018-04-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20971 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

2018-04-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20971 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20988: [SPARK-23877][SQL]: Use filter predicates to prune parti...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20988 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88959/ Test PASSed. ---

[GitHub] spark pull request #20931: [SPARK-23815][Core]Spark writer dynamic partition...

2018-04-05 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20931#discussion_r179651241 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -186,7 +186,9 @@ class

[GitHub] spark issue #20953: [SPARK-23822][SQL] Improve error message for Parquet sch...

2018-04-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20953 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20961: [SPARK-23823][SQL] Keep origin in transformExpres...

2018-04-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20961 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20988: [SPARK-23877][SQL]: Use filter predicates to prune parti...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20988 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20989 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1981/ ---

[GitHub] spark pull request #20931: [SPARK-23815][Core]Spark writer dynamic partition...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20931#discussion_r179651085 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -186,7 +186,9 @@ class

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20989 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2018/

[GitHub] spark pull request #20931: [SPARK-23815][Core]Spark writer dynamic partition...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20931#discussion_r179651049 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -186,7 +186,9 @@ class

[GitHub] spark pull request #20931: [SPARK-23815][Core]Spark writer dynamic partition...

2018-04-05 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20931#discussion_r179651037 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -186,7 +186,9 @@ class

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20989 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20988: [SPARK-23877][SQL]: Use filter predicates to prune parti...

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20988 **[Test build #88959 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88959/testReport)** for PR 20988 at commit

[GitHub] spark issue #20961: [SPARK-23823][SQL] Keep origin in transformExpression

2018-04-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20961 Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #20970: [SPARK-23870][ML] Forward RFormula handleInvalid ...

2018-04-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20970 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20931: [SPARK-23815][Core]Spark writer dynamic partition...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20931#discussion_r179650482 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -186,7 +186,9 @@ class

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20989 **[Test build #88960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88960/testReport)** for PR 20989 at commit

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20989 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20989 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88960/ Test FAILed. ---

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20989 **[Test build #88960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88960/testReport)** for PR 20989 at commit

[GitHub] spark issue #20970: [SPARK-23870][ML] Forward RFormula handleInvalid Param t...

2018-04-05 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/20970 LGTM Merging with master Thanks @yogeshg ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20989 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20987: [SPARK-23816][CORE] Killed tasks should ignore FetchFail...

2018-04-05 Thread Ngone51
Github user Ngone51 commented on the issue: https://github.com/apache/spark/pull/20987 Things I'm concerned about is that does there exists a situation like 'a task gets killed after it gets a FetchFailure, but re-run (not by resubmit) later and gets a FetchFailure too without

[GitHub] spark pull request #20931: [SPARK-23815][Core]Spark writer dynamic partition...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20931#discussion_r179647871 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -186,7 +186,9 @@ class

[GitHub] spark pull request #20987: [SPARK-23816][CORE] Killed tasks should ignore Fe...

2018-04-05 Thread Ngone51
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/20987#discussion_r179647613 --- Diff: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala --- @@ -173,8 +173,26 @@ class ExecutorSuite extends SparkFunSuite with

[GitHub] spark pull request #20987: [SPARK-23816][CORE] Killed tasks should ignore Fe...

2018-04-05 Thread Ngone51
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/20987#discussion_r179647240 --- Diff: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala --- @@ -330,6 +362,15 @@ class FetchFailureHidingRDD( case t:

[GitHub] spark issue #19193: [WIP][SPARK-21896][SQL] Fix Stack Overflow when window f...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19193 I feel it's much simpler to just prohibit this case, as we don't allow nested aggregate function. Users can always use subquery to work around it. Do other databases support window function

[GitHub] spark issue #20987: [SPARK-23816][CORE] Killed tasks should ignore FetchFail...

2018-04-05 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20987 pinging some potential reviewers: @tgravescs @kayousterhout @zsxwing @mridulm --- - To unsubscribe, e-mail:

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19222 oh I missed one thing. @kiszk please create a new JIRA for this PR, as the original JIRA tries to introduce an infrastructure to enable memory accessing from c++, which is not what this PR did.

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2018-04-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19222 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19222 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20986: [SPARK-23864][SQL] Add unsafe object writing to U...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20986#discussion_r179645207 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala --- @@ -176,6 +141,23 @@ object

[GitHub] spark pull request #20986: [SPARK-23864][SQL] Add unsafe object writing to U...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20986#discussion_r179644816 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala --- @@ -176,6 +141,23 @@ object

[GitHub] spark pull request #20986: [SPARK-23864][SQL] Add unsafe object writing to U...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20986#discussion_r179644768 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala --- @@ -176,6 +141,23 @@ object

[GitHub] spark pull request #20986: [SPARK-23864][SQL] Add unsafe object writing to U...

2018-04-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20986#discussion_r179644530 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala --- @@ -286,23 +239,32 @@ object

[GitHub] spark issue #20975: [SPARK-23863] [SQL] Wholetext mode should not add line b...

2018-04-05 Thread barrenlake
Github user barrenlake commented on the issue: https://github.com/apache/spark/pull/20975 @HyukjinKwon Oh yes, you are right. This problem was encountered when I added the small file merge function to the Hive module. When I used select count(1) to count the merge result file, I

[GitHub] spark pull request #20975: [SPARK-23863] [SQL] Wholetext mode should not add...

2018-04-05 Thread barrenlake
Github user barrenlake closed the pull request at: https://github.com/apache/spark/pull/20975 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20987: [SPARK-23816][CORE] Killed tasks should ignore FetchFail...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20987 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20987: [SPARK-23816][CORE] Killed tasks should ignore FetchFail...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20987 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88958/ Test PASSed. ---

[GitHub] spark issue #20987: [SPARK-23816][CORE] Killed tasks should ignore FetchFail...

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20987 **[Test build #88958 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88958/testReport)** for PR 20987 at commit

[GitHub] spark issue #20961: [SPARK-23823][SQL] Keep origin in transformExpression

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20961 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20961: [SPARK-23823][SQL] Keep origin in transformExpression

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20961 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88957/ Test PASSed. ---

[GitHub] spark issue #20904: [SPARK-23751][ML][PySpark] Kolmogorov-Smirnoff test Pyth...

2018-04-05 Thread yogeshg
Github user yogeshg commented on the issue: https://github.com/apache/spark/pull/20904 lgtm, I'll defer to @jkbradley --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20961: [SPARK-23823][SQL] Keep origin in transformExpression

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20961 **[Test build #88957 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88957/testReport)** for PR 20961 at commit

[GitHub] spark pull request #20904: [SPARK-23751][ML][PySpark] Kolmogorov-Smirnoff te...

2018-04-05 Thread yogeshg
Github user yogeshg commented on a diff in the pull request: https://github.com/apache/spark/pull/20904#discussion_r179639255 --- Diff: python/pyspark/ml/stat.py --- @@ -134,6 +134,63 @@ def corr(dataset, column, method="pearson"): return _java2py(sc,

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179635730 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala --- @@ -53,32 +53,24 @@ class ContinuousSuiteBase

[GitHub] spark issue #20988: [SPARK-23877][SQL]: Use filter predicates to prune parti...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20988 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2017/

[GitHub] spark issue #20988: [SPARK-23877][SQL]: Use filter predicates to prune parti...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20988 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20989 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread madanadit
Github user madanadit commented on the issue: https://github.com/apache/spark/pull/20989 @foxish Can you take a look?. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20989 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20989: [SPARK-23529][K8s] Support mounting hostPath volu...

2018-04-05 Thread madanadit
GitHub user madanadit opened a pull request: https://github.com/apache/spark/pull/20989 [SPARK-23529][K8s] Support mounting hostPath volumes for executors ## What changes were proposed in this pull request? This PR introduces a new config `spark.kubernetes.executor.volumes`

[GitHub] spark issue #20988: [SPARK-23877][SQL]: Use filter predicates to prune parti...

2018-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20988 **[Test build #88959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88959/testReport)** for PR 20988 at commit

[GitHub] spark pull request #20988: [SPARK-23877][SQL]: Use filter predicates to prun...

2018-04-05 Thread rdblue
GitHub user rdblue opened a pull request: https://github.com/apache/spark/pull/20988 [SPARK-23877][SQL]: Use filter predicates to prune partitions in metadata-only queries ## What changes were proposed in this pull request? This updates the OptimizeMetadataOnlyQuery rule

[GitHub] spark pull request #20968: [SPARK-23828][ML][PYTHON]PySpark StringIndexerMod...

2018-04-05 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/20968#discussion_r179623749 --- Diff: python/pyspark/ml/feature.py --- @@ -2342,8 +2342,38 @@ def mean(self): return self._call_java("mean") +class

[GitHub] spark pull request #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Supp...

2018-04-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20937#discussion_r179622965 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/CreateJacksonParser.scala --- @@ -39,11 +40,28 @@ private[sql] object

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179596647 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* --- End

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179603916 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* + *

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179594450 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala --- @@ -43,8 +45,39 @@ object MemoryStream { protected val

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179603105 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* + *

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179604298 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* + *

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179602477 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* + *

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179601495 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* --- End

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179617487 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* + *

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179617590 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* + *

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179599245 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala --- @@ -53,32 +53,24 @@ class ContinuousSuiteBase extends

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179617149 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* + *

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179605302 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* + *

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179618466 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* + *

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r179606525 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,212 @@ +/* + *

  1   2   3   4   5   >