[GitHub] spark pull request #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cle...

2018-11-12 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/22993#discussion_r232847435 --- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala --- @@ -193,6 +194,36 @@ private[spark] class StorageStatus( /** Helper

[GitHub] spark issue #23013: [SPARK-25023] More detailed security guidance for K8S

2018-11-12 Thread mccheah
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/23013 I'm curious as to how much specific K8s guidance we should be giving in Spark documentation. There is such a thing as over-documenting; though we should certainly bias towards writing more

[GitHub] spark issue #23015: [SPARK-26029][BUILD][2.4] Bump previousSparkVersion in M...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23015 thanks, merging to 2.4! Since we have more violates in the master branch, I did not forward port it, and I'll cherry-pick it in another PR. ---

[GitHub] spark pull request #23015: [SPARK-26029][BUILD][2.4] Bump previousSparkVersi...

2018-11-12 Thread cloud-fan
Github user cloud-fan closed the pull request at: https://github.com/apache/spark/pull/23015 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98745/ Test PASSed. ---

[GitHub] spark pull request #23007: [SPARK-26010][R] fix vignette eval with Java 11

2018-11-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23007 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22977 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22977 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22977 **[Test build #98746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98746/testReport)** for PR 22977 at commit

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22977 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22977 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21465: [SPARK-24333][ML][PYTHON]Add fit with validation set to ...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21465 **[Test build #98751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98751/testReport)** for PR 21465 at commit

[GitHub] spark issue #23014: [MINOR][SQL] Add disable bucketedRead workaround when th...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23014 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #23014: [MINOR][SQL] Add disable bucketedRead workaround when th...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23014 **[Test build #98754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98754/testReport)** for PR 23014 at commit

[GitHub] spark issue #23014: [MINOR][SQL] Add disable bucketedRead workaround when th...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23014 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Aggregate expressions shouldn'...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r232905784 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala --- @@ -1556,6 +1556,20 @@ class DatasetSuite extends QueryTest with

[GitHub] spark pull request #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cle...

2018-11-12 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22993#discussion_r232848389 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java --- @@ -67,6 +67,60 @@ unaligned = _unaligned; } + //

[GitHub] spark issue #23009: SPARK-26011: pyspark app with "spark.jars.packages" conf...

2018-11-12 Thread imatiach-msft
Github user imatiach-msft commented on the issue: https://github.com/apache/spark/pull/23009 @srowen , @drdarshan mentioned that it may be better to fix livy instead of spark. The problem is that when using the %%configure magic spark.jars.packages does not add the python files in

[GitHub] spark pull request #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cle...

2018-11-12 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22993#discussion_r232867047 --- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala --- @@ -193,6 +194,36 @@ private[spark] class StorageStatus( /** Helper

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23019 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98743/ Test PASSed. ---

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23019 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...

2018-11-12 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 FYI This is unused code I’m going to remove it https://github.com/apache/spark/blob/master/R/pkg/src-native/string_hash_code.c ---

[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22974 **[Test build #98747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98747/testReport)** for PR 22974 at commit

[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22974 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #23014: [MINOR][SQL] Add disable bucketedRead workaround when th...

2018-11-12 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23014 > The reason is that each bucket file is too big Can you elaborate please? Is it because we don't chunk each file into multiple splits when we read bucketed table? ---

[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22974 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #23006: [SPARK-26007][SQL] DataFrameReader.csv() respects to spa...

2018-11-12 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23006 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21465: [SPARK-24333][ML][PYTHON]Add fit with validation set to ...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21465 **[Test build #98751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98751/testReport)** for PR 21465 at commit

[GitHub] spark issue #21465: [SPARK-24333][ML][PYTHON]Add fit with validation set to ...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21465 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98751/ Test PASSed. ---

[GitHub] spark issue #21465: [SPARK-24333][ML][PYTHON]Add fit with validation set to ...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21465 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in...

2018-11-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22977 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cle...

2018-11-12 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/22993#discussion_r232852693 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java --- @@ -67,6 +67,59 @@ unaligned = _unaligned; } + //

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22977 **[Test build #98750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98750/testReport)** for PR 22977 at commit

[GitHub] spark issue #22518: [SPARK-25482][SQL] Avoid pushdown of subqueries to data ...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22518 I'd like to merge this simple PR first, to address the performance problem (unnecessary subquery execution). Let's create a new ticket for subquery filter pushing to data source, and

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21688 **[Test build #98741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98741/testReport)** for PR 21688 at commit

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98741/ Test PASSed. ---

[GitHub] spark issue #23018: [SPARK-26023][SQL] Dumping truncated plans and generated...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23018 **[Test build #98742 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98742/testReport)** for PR 23018 at commit

[GitHub] spark issue #23018: [SPARK-26023][SQL] Dumping truncated plans and generated...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23018 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98742/ Test PASSed. ---

[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...

2018-11-12 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 In this way, we could postpone R upgrade after Spark 3.0.0 release in Jenkins, and could still test the deprecated R version 3.1. ---

[GitHub] spark pull request #23014: [MINOR][SQL] Add disable bucketedRead workaround ...

2018-11-12 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/23014#discussion_r232885260 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java --- @@ -101,7 +101,8 @@ private void

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22977 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22977 **[Test build #98750 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98750/testReport)** for PR 22977 at commit

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22977 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98750/ Test FAILed. ---

[GitHub] spark pull request #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cle...

2018-11-12 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22993#discussion_r232849634 --- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala --- @@ -193,6 +194,36 @@ private[spark] class StorageStatus( /** Helper

[GitHub] spark pull request #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cle...

2018-11-12 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22993#discussion_r232849773 --- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala --- @@ -193,6 +194,36 @@ private[spark] class StorageStatus( /** Helper

[GitHub] spark pull request #23018: [SPARK-26023][SQL] Dumping truncated plans and ge...

2018-11-12 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/23018#discussion_r232871356 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -469,7 +471,21 @@ abstract class TreeNode[BaseType <:

[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...

2018-11-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23012#discussion_r232882419 --- Diff: docs/index.md --- @@ -31,7 +31,8 @@ Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS). It's easy locally on one

[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...

2018-11-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23012#discussion_r232881732 --- Diff: R/pkg/R/sparkR.R --- @@ -283,6 +283,10 @@ sparkR.session <- function( enableHiveSupport = TRUE, ...) { + if

[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...

2018-11-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23012#discussion_r232882178 --- Diff: docs/index.md --- @@ -31,7 +31,8 @@ Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS). It's easy locally on one

[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...

2018-11-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23012#discussion_r232881594 --- Diff: R/WINDOWS.md --- @@ -3,7 +3,7 @@ To build SparkR on Windows, the following steps are required 1. Install R (>= 3.1) and

[GitHub] spark issue #23007: [SPARK-26010][R] fix vignette eval with Java 11

2018-11-12 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23007 merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21688 **[Test build #98745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98745/testReport)** for PR 21688 at commit

[GitHub] spark issue #23014: [MINOR][SQL] Add disable bucketedRead workaround when th...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23014 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #23014: [MINOR][SQL] Add disable bucketedRead workaround when th...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23014 **[Test build #98748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98748/testReport)** for PR 23014 at commit

[GitHub] spark issue #23014: [MINOR][SQL] Add disable bucketedRead workaround when th...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23014 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request #23014: [MINOR][SQL] Add disable bucketedRead workaround ...

2018-11-12 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/23014#discussion_r232893546 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java --- @@ -101,10 +101,11 @@ private void

[GitHub] spark pull request #22954: [SPARK-25981][R] Enables Arrow optimization from ...

2018-11-12 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22954#discussion_r232895848 --- Diff: R/pkg/R/SQLContext.R --- @@ -172,36 +257,72 @@ getDefaultSqlSource <- function() { createDataFrame <- function(data, schema = NULL,

[GitHub] spark issue #22944: [SPARK-25942][SQL] Aggregate expressions shouldn't be re...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22944 **[Test build #98752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98752/testReport)** for PR 22944 at commit

[GitHub] spark pull request #22518: [SPARK-25482][SQL] Avoid pushdown of subqueries t...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22518#discussion_r232906707 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala --- @@ -47,7 +47,8 @@ private[sql] object

[GitHub] spark pull request #22518: [SPARK-25482][SQL] Avoid pushdown of subqueries t...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22518#discussion_r232906743 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala --- @@ -1268,4 +1269,16 @@ class SubquerySuite extends QueryTest with

[GitHub] spark pull request #22518: [SPARK-25482][SQL] Avoid pushdown of subqueries t...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22518#discussion_r232906652 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala --- @@ -155,15 +155,14 @@ object

[GitHub] spark pull request #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cle...

2018-11-12 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/22993#discussion_r232846926 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java --- @@ -67,6 +67,60 @@ unaligned = _unaligned; } + //

[GitHub] spark pull request #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cle...

2018-11-12 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/22993#discussion_r232847703 --- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala --- @@ -193,6 +194,36 @@ private[spark] class StorageStatus( /** Helper

[GitHub] spark pull request #22952: [SPARK-20568][SS] Rename files which are complete...

2018-11-12 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/spark/pull/22952#discussion_r232869187 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -530,6 +530,8 @@ Here are the details of all the sources in Spark.

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23019 **[Test build #98743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98743/testReport)** for PR 23019 at commit

[GitHub] spark pull request #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22977#discussion_r232886260 --- Diff: project/MimaExcludes.scala --- @@ -164,7 +212,50 @@ object MimaExcludes {

[GitHub] spark issue #22944: [SPARK-25942][SQL] Aggregate expressions shouldn't be re...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22944 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22944: [SPARK-25942][SQL] Aggregate expressions shouldn't be re...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22944 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22518: [SPARK-25482][SQL] Avoid pushdown of subqueries to data ...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22518 BTW can you include a simple benchmark to show this problem? e.g. just run a query in spark-shell, and post the result before and after this PR. ---

[GitHub] spark issue #23006: [SPARK-26007][SQL] DataFrameReader.csv() respects to spa...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23006 **[Test build #98740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98740/testReport)** for PR 23006 at commit

[GitHub] spark issue #23018: [SPARK-26023][SQL] Dumping truncated plans and generated...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23018 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #23009: SPARK-26011: pyspark app with "spark.jars.package...

2018-11-12 Thread shanyu
Github user shanyu commented on a diff in the pull request: https://github.com/apache/spark/pull/23009#discussion_r232868311 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -318,7 +318,7 @@ private[spark] class SparkSubmit extends Logging {

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23019 **[Test build #98744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98744/testReport)** for PR 23019 at commit

[GitHub] spark issue #23014: [MINOR][SQL] Add disable bucketedRead workaround when th...

2018-11-12 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/23014 Yes. The `filePartitions` are the same as the bucket number when `BucketedRead`:

[GitHub] spark pull request #23006: [SPARK-26007][SQL] DataFrameReader.csv() respects...

2018-11-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23006 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Aggregate expressions shouldn'...

2018-11-12 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r232894304 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala --- @@ -1556,6 +1556,20 @@ class DatasetSuite extends QueryTest with

[GitHub] spark issue #23002: [SPARK-26003] Improve SQLAppStatusListener.aggregateMetr...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23002 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22961: [SPARK-25947][SQL] Reduce memory usage in Shuffle...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22961#discussion_r232906123 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala --- @@ -214,13 +214,22 @@ object ShuffleExchangeExec

[GitHub] spark issue #22954: [SPARK-25981][R] Enables Arrow optimization from R DataF...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22954 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22977 since this PR only touches mima, and the jenkins already passed the mima check, I'm going to merge it to master, thanks! ---

[GitHub] spark issue #22954: [SPARK-25981][R] Enables Arrow optimization from R DataF...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22954 **[Test build #98755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98755/testReport)** for PR 22954 at commit

[GitHub] spark issue #22954: [SPARK-25981][R] Enables Arrow optimization from R DataF...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22954 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #23009: SPARK-26011: pyspark app with "spark.jars.packages" conf...

2018-11-12 Thread shanyu
Github user shanyu commented on the issue: https://github.com/apache/spark/pull/23009 @imatiach-msft I think this is a Spark bug not Livy's. Livy just calls SparkSubmit with "spark-internal" type of resource, with "spark.jars.pacakges" setting. SparkSubmit module is the one who

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23019 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98744/ Test PASSed. ---

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23019 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...

2018-11-12 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23012 Nice. Thanks!. BTW Felix, are you maybe worrying about that we happen to upgrade R version in Jenkins to 3.4 and .. we could break lower deprecated R version support in Spark 3.0 I guess?

[GitHub] spark issue #22866: WIP [SPARK-12172][SPARKR] Remove internal-only RDD metho...

2018-11-12 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22866 thx, but DO NOT MERGE - there's some nasty bug I'm still investigating.. --- - To unsubscribe, e-mail:

[GitHub] spark issue #23018: [SPARK-26023][SQL] Dumping truncated plans and generated...

2018-11-12 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23018 Looks fine to me. adding @cloud-fan and @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #23018: [SPARK-26023][SQL] Dumping truncated plans and ge...

2018-11-12 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/23018#discussion_r232883084 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -469,7 +471,21 @@ abstract class TreeNode[BaseType <:

[GitHub] spark pull request #22961: [SPARK-25947][SQL] Reduce memory usage in Shuffle...

2018-11-12 Thread mu5358271
Github user mu5358271 commented on a diff in the pull request: https://github.com/apache/spark/pull/22961#discussion_r232888324 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala --- @@ -214,13 +214,22 @@ object ShuffleExchangeExec

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22977 **[Test build #98746 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98746/testReport)** for PR 22977 at commit

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22977 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98746/ Test FAILed. ---

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22977 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22989: [SPARK-25986][Build] Add rules to ban throw Errors in ap...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22989 **[Test build #98749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98749/testReport)** for PR 22989 at commit

[GitHub] spark issue #22989: [SPARK-25986][Build] Add rules to ban throw Errors in ap...

2018-11-12 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/22989 @srowen Great thanks for your guidance, address all your suggestion in ff234d3 and update the record table in https://github.com/apache/spark/pull/22989#issuecomment-437939830. ---

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22977 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22977 **[Test build #98753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98753/testReport)** for PR 22977 at commit

[GitHub] spark issue #22977: [SPARK-26030][BUILD] Bump previousSparkVersion in MimaBu...

2018-11-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22977 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

  1   2   3   4   5   >