[GitHub] spark issue #19843: [SPARK-22644][ML][TEST][WIP] Make ML testsuite support S...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19843 **[Test build #84287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84287/testReport)** for PR 19843 at commit

[GitHub] spark issue #19839: SPARK-22373 Bump Janino dependency version to fix thread...

2017-11-28 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19839 I think we don't need this since we need to upgrade to the next janino release for the issue related to SPARK-18016. --- - To

[GitHub] spark issue #19843: [SPARK-22644][ML][TEST][WIP] Make ML testsuite support S...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19843 **[Test build #84286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84286/testReport)** for PR 19843 at commit

[GitHub] spark pull request #19843: [SPARK-22644][ML][TEST][WIP] Make ML testsuite su...

2017-11-28 Thread WeichenXu123
GitHub user WeichenXu123 opened a pull request: https://github.com/apache/spark/pull/19843 [SPARK-22644][ML][TEST][WIP] Make ML testsuite support StructuredStreaming test ## What changes were proposed in this pull request? We need to add some helper code to make testing ML

[GitHub] spark issue #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset API

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19805 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84282/ Test PASSed. ---

[GitHub] spark issue #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset API

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19805 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...

2017-11-28 Thread Gschiavon
Github user Gschiavon commented on a diff in the pull request: https://github.com/apache/spark/pull/19793#discussion_r153709924 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/SubmitRestProtocolSuite.scala --- @@ -86,6 +86,8 @@ class SubmitRestProtocolSuite extends

[GitHub] spark issue #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset API

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19805 **[Test build #84282 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84282/testReport)** for PR 19805 at commit

[GitHub] spark issue #19218: [SPARK-21786][SQL] The 'spark.sql.parquet.compression.co...

2017-11-28 Thread fjh100456
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/19218 @gatorsmile I'd tested the performance of 'uncompressed', 'snappy', 'gzip' compression algorithm for parquet, the input data volume is 22MB, 220MB, 1100MB, respectively run 10 times, finally

[GitHub] spark issue #19842: [SPARK-22643][SQL] ColumnarArray should be an immutable ...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19842 **[Test build #84285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84285/testReport)** for PR 19842 at commit

[GitHub] spark pull request #19842: [SPARK-22643][SQL] ColumnarArray should be an imm...

2017-11-28 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/19842 [SPARK-22643][SQL] ColumnarArray should be an immutable view ## What changes were proposed in this pull request? To make `ColumnVector` public, `ColumnarArray` need to be public too, and

[GitHub] spark issue #19842: [SPARK-22643][SQL] ColumnarArray should be an immutable ...

2017-11-28 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19842 cc @michal-databricks @hvanhovell @kiszk @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19812: [SPARK-22598][CORE] ExecutorAllocationManager does not r...

2017-11-28 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19812 Does this failure ". For some reason, all of the 3 executors failed. " happened during task running or before task submission? Besides, if you're running on yarn, yarn will bring new executors to

[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19750 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19750 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84281/ Test PASSed. ---

[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19750 **[Test build #84281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84281/testReport)** for PR 19750 at commit

[GitHub] spark issue #19812: [SPARK-22598][CORE] ExecutorAllocationManager does not r...

2017-11-28 Thread liutang123
Github user liutang123 commented on the issue: https://github.com/apache/spark/pull/19812 Hi @jerryshao , I modified the info of this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19468 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - Basic Sc...

2017-11-28 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19468 For future pull requests, can you create subtasks under https://issues.apache.org/jira/browse/SPARK-18278 ? --- - To unsubscribe,

[GitHub] spark issue #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - Basic Sc...

2017-11-28 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19468 Thanks - merging in master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19840: [SPARK-22640][PYSPARK][YARN]switch python exec on execut...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19840 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84280/ Test FAILed. ---

[GitHub] spark issue #19840: [SPARK-22640][PYSPARK][YARN]switch python exec on execut...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19840 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19840: [SPARK-22640][PYSPARK][YARN]switch python exec on execut...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19840 **[Test build #84280 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84280/testReport)** for PR 19840 at commit

[GitHub] spark issue #19825: [SPARK-22615][SQL] Handle more cases in PropagateEmptyRe...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19825 **[Test build #84284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84284/testReport)** for PR 19825 at commit

[GitHub] spark issue #19841: [SPARK-22642][SQL] the createdTempDir will not be delete...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19841 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19841: [SPARK-22642][SQL] the createdTempDir will not be...

2017-11-28 Thread zuotingbing
GitHub user zuotingbing opened a pull request: https://github.com/apache/spark/pull/19841 [SPARK-22642][SQL] the createdTempDir will not be deleted if an exception occurs ## What changes were proposed in this pull request? We found staging directories will not be

[GitHub] spark issue #19823: [SPARK-22601][SQL] Data load is getting displayed succes...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19823 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19823: [SPARK-22601][SQL] Data load is getting displayed succes...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19823 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84283/ Test FAILed. ---

[GitHub] spark issue #19821: [WIP][SPARK-22608][SQL] add new API to CodeGeneration.sp...

2017-11-28 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19821 Sure, I have resolved the conflict in my environment. I will commit soon. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19823: [SPARK-22601][SQL] Data load is getting displayed succes...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19823 **[Test build #84283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84283/testReport)** for PR 19823 at commit

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84278/ Test PASSed. ---

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19651 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19651 **[Test build #84278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84278/testReport)** for PR 19651 at commit

[GitHub] spark issue #19834: [SPARK-22585][Core] Path in addJar is not url encoded

2017-11-28 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19834 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #18714: [SPARK-20236][SQL] runtime partition overwrite

2017-11-28 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/18714 ping, very interested in this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset A...

2017-11-28 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19805#discussion_r153693534 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -537,9 +536,55 @@ class Dataset[T] private[sql]( */ @Experimental

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19651 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84279/ Test FAILed. ---

[GitHub] spark pull request #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset A...

2017-11-28 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19805#discussion_r153693511 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -537,9 +536,55 @@ class Dataset[T] private[sql]( */ @Experimental

[GitHub] spark pull request #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset A...

2017-11-28 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19805#discussion_r153693403 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -518,13 +518,12 @@ class Dataset[T] private[sql]( * the logical plan of

[GitHub] spark pull request #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset A...

2017-11-28 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19805#discussion_r153694085 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -537,9 +536,55 @@ class Dataset[T] private[sql]( */ @Experimental

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19651 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19651 **[Test build #84277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84277/testReport)** for PR 19651 at commit

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84277/ Test PASSed. ---

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19651 **[Test build #84279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84279/testReport)** for PR 19651 at commit

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19651 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19823: [SPARK-22601][SQL] Data load is getting displayed succes...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19823 **[Test build #84283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84283/testReport)** for PR 19823 at commit

[GitHub] spark issue #19823: [SPARK-22601][SQL] Data load is getting displayed succes...

2017-11-28 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19823 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19823: [SPARK-22601][SQL] Data load is getting displayed...

2017-11-28 Thread sujith71955
Github user sujith71955 commented on a diff in the pull request: https://github.com/apache/spark/pull/19823#discussion_r153693386 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -2392,5 +2392,13 @@ abstract class DDLSuite extends

[GitHub] spark pull request #19823: [SPARK-22601][SQL] Data load is getting displayed...

2017-11-28 Thread sujith71955
Github user sujith71955 commented on a diff in the pull request: https://github.com/apache/spark/pull/19823#discussion_r153693359 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -2392,5 +2392,13 @@ abstract class DDLSuite extends

[GitHub] spark issue #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset API

2017-11-28 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19805 @ferdonline Could you file a JIRA issue and add the id to the title like `[SPARK-xxx][PYTHON][SQL] ...`? --- - To unsubscribe,

[GitHub] spark issue #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset API

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19805 **[Test build #84282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84282/testReport)** for PR 19805 at commit

[GitHub] spark issue #19805: [PYTHON][SQL] Adding localCheckpoint to Dataset API

2017-11-28 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19805 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19792: [SPARK-22566][PYTHON] Better error message for `_...

2017-11-28 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19792#discussion_r153692220 --- Diff: python/pyspark/sql/types.py --- @@ -1108,19 +1109,23 @@ def _has_nulltype(dt): return isinstance(dt, NullType) -def

[GitHub] spark issue #19833: [SPARK-22605][SQL] SQL write job should also set Spark t...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19833 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19833: [SPARK-22605][SQL] SQL write job should also set Spark t...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19833 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84276/ Test PASSed. ---

[GitHub] spark issue #19833: [SPARK-22605][SQL] SQL write job should also set Spark t...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19833 **[Test build #84276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84276/testReport)** for PR 19833 at commit

[GitHub] spark issue #19835: [SPARK-21866][ML][PYTHON][FOLLOWUP] Few cleanups and tes...

2017-11-28 Thread imatiach-msft
Github user imatiach-msft commented on the issue: https://github.com/apache/spark/pull/19835 the changes look good to me, the extra verification logic for arguments is a great addition --- - To unsubscribe, e-mail:

[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19750 **[Test build #84281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84281/testReport)** for PR 19750 at commit

[GitHub] spark pull request #19835: [SPARK-21866][ML][PYTHON][FOLLOWUP] Few cleanups ...

2017-11-28 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/19835#discussion_r153689403 --- Diff: python/pyspark/ml/image.py --- @@ -146,7 +163,12 @@ def toImage(self, array, origin=""): mode = ocvTypes["CV_8UC4"]

[GitHub] spark pull request #19835: [SPARK-21866][ML][PYTHON][FOLLOWUP] Few cleanups ...

2017-11-28 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/19835#discussion_r153689356 --- Diff: python/pyspark/ml/tests.py --- @@ -1836,6 +1836,24 @@ def test_read_images(self): self.assertEqual(ImageSchema.imageFields,

[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.

2017-11-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19750 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19833: [SPARK-22605][SQL] SQL write job should also set ...

2017-11-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19833#discussion_r153687993 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala --- @@ -106,6 +105,13 @@ class

[GitHub] spark issue #19840: [SPARK-22640][PYSPARK][YARN]switch python exec in execut...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19840 **[Test build #84280 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84280/testReport)** for PR 19840 at commit

[GitHub] spark pull request #19840: [SPARK-22640][PYSPARK][YARN]switch python exec in...

2017-11-28 Thread yaooqinn
GitHub user yaooqinn opened a pull request: https://github.com/apache/spark/pull/19840 [SPARK-22640][PYSPARK][YARN]switch python exec in executor side ## What changes were proposed in this pull request? ``` PYSPARK_PYTHON=~/anaconda3/envs/py3/bin/python \

[GitHub] spark pull request #19836: [SPARK-22618][CORE] Catch exception in removeRDD ...

2017-11-28 Thread brad-kaiser
Github user brad-kaiser commented on a diff in the pull request: https://github.com/apache/spark/pull/19836#discussion_r153686556 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -159,11 +160,18 @@ class BlockManagerMasterEndpoint(

[GitHub] spark pull request #19783: [SPARK-21322][SQL] support histogram in filter ca...

2017-11-28 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19783#discussion_r153686107 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/EstimationUtils.scala --- @@ -114,4 +114,197 @@ object

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19651 **[Test build #84279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84279/testReport)** for PR 19651 at commit

[GitHub] spark issue #19839: SPARK-22373 Bump Janino dependency version to fix thread...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19839 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19839: SPARK-22373 Bump Janino dependency version to fix...

2017-11-28 Thread Victsm
GitHub user Victsm opened a pull request: https://github.com/apache/spark/pull/19839 SPARK-22373 Bump Janino dependency version to fix thread safety issue… … with Janino when compiling generated code. ## What changes were proposed in this pull request? Bump up

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19651 **[Test build #84278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84278/testReport)** for PR 19651 at commit

[GitHub] spark pull request #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new O...

2017-11-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19651#discussion_r153683384 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcDeserializer.scala --- @@ -0,0 +1,304 @@ +/* + *

[GitHub] spark pull request #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new O...

2017-11-28 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19651#discussion_r153682330 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcDeserializer.scala --- @@ -0,0 +1,304 @@ +/* + * Licensed to

[GitHub] spark pull request #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cache...

2017-11-28 Thread zzl1787
Github user zzl1787 commented on a diff in the pull request: https://github.com/apache/spark/pull/19129#discussion_r153682329 --- Diff: docs/sql-programming-guide.md --- @@ -1587,6 +1580,10 @@ options. Note that this is different from the Hive behavior. - As a

[GitHub] spark pull request #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new O...

2017-11-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19651#discussion_r153682051 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcDeserializer.scala --- @@ -0,0 +1,226 @@ +/* + *

[GitHub] spark pull request #19714: [SPARK-22489][SQL] Shouldn't change broadcast joi...

2017-11-28 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19714#discussion_r153680715 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -153,6 +151,27 @@ abstract class SparkStrategies extends

[GitHub] spark pull request #19714: [SPARK-22489][SQL] Shouldn't change broadcast joi...

2017-11-28 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19714#discussion_r153679188 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -91,10 +91,10 @@ abstract class SparkStrategies extends

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-28 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r153678912 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed to

[GitHub] spark pull request #19823: [SPARK-22601][SQL] Data load is getting displayed...

2017-11-28 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19823#discussion_r153678875 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -2392,5 +2392,13 @@ abstract class DDLSuite extends QueryTest

[GitHub] spark issue #19651: [SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileF...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19651 **[Test build #84277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84277/testReport)** for PR 19651 at commit

[GitHub] spark pull request #19823: [SPARK-22601][SQL] Data load is getting displayed...

2017-11-28 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19823#discussion_r153678491 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -2392,5 +2392,13 @@ abstract class DDLSuite extends QueryTest

[GitHub] spark issue #19831: [SPARK-22626][SQL] Wrong Hive table statistics may trigg...

2017-11-28 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19831 Besides, if the size stats `totalSize` or `rawDataSize` is wrong, the problem also exists whether CBO is enabled or not. We need to change that in the title too. ---

[GitHub] spark pull request #19714: [SPARK-22489][SQL] Shouldn't change broadcast joi...

2017-11-28 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19714#discussion_r153677941 --- Diff: docs/sql-programming-guide.md --- @@ -1492,6 +1492,61 @@ that these options will be deprecated in future release as more optimizations ar

[GitHub] spark issue #19831: [SPARK-22626][SQL] Wrong Hive table statistics may trigg...

2017-11-28 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19831 BTW, the case here is not about join reorder, it's actually about broadcast decision. Could you update the title of this PR? ---

[GitHub] spark pull request #19714: [SPARK-22489][SQL] Shouldn't change broadcast joi...

2017-11-28 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19714#discussion_r153677668 --- Diff: docs/sql-programming-guide.md --- @@ -1492,6 +1492,61 @@ that these options will be deprecated in future release as more optimizations ar

[GitHub] spark pull request #19831: [SPARK-22626][SQL] Wrong Hive table statistics ma...

2017-11-28 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19831#discussion_r153677676 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala --- @@ -418,7 +418,7 @@ private[hive] class HiveClientImpl(

[GitHub] spark pull request #19831: [SPARK-22626][SQL] Wrong Hive table statistics ma...

2017-11-28 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19831#discussion_r153677300 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala --- @@ -418,7 +418,7 @@ private[hive] class HiveClientImpl(

[GitHub] spark issue #19833: [SPARK-22605][SQL] SQL write job should also set Spark t...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19833 **[Test build #84276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84276/testReport)** for PR 19833 at commit

[GitHub] spark issue #19821: [WIP][SPARK-22608][SQL] add new API to CodeGeneration.sp...

2017-11-28 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19821 @kiszk Can you fix the conflict? now we can add a middle-advanced version: ``` def splitExpressions( expressions: Seq[String], funcName: String, extraArguments:

[GitHub] spark pull request #19838: [SPARK-22638][SS]Use a separate query for Streami...

2017-11-28 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/19838#discussion_r153674694 --- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala --- @@ -87,7 +87,9 @@ private[spark] class LiveListenerBus(conf:

[GitHub] spark pull request #19717: [SPARK-18278] [Submission] Spark on Kubernetes - ...

2017-11-28 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r15367 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/Client.scala --- @@ -0,0 +1,236 @@ +/* + * Licensed

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19813 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19813 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84275/ Test FAILed. ---

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19813 **[Test build #84275 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84275/testReport)** for PR 19813 at commit

[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate query for StreamingQuery...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19838 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84274/ Test PASSed. ---

[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate query for StreamingQuery...

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19838 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate query for StreamingQuery...

2017-11-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19838 **[Test build #84274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84274/testReport)** for PR 19838 at commit

[GitHub] spark issue #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - Basic Sc...

2017-11-28 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/19468 LGTM, thanks for the awesome work! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19594: [SPARK-21984] [SQL] Join estimation based on equi...

2017-11-28 Thread ron8hu
Github user ron8hu commented on a diff in the pull request: https://github.com/apache/spark/pull/19594#discussion_r153665547 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/JoinEstimationSuite.scala --- @@ -67,6 +68,205 @@ class

[GitHub] spark pull request #19594: [SPARK-21984] [SQL] Join estimation based on equi...

2017-11-28 Thread ron8hu
Github user ron8hu commented on a diff in the pull request: https://github.com/apache/spark/pull/19594#discussion_r153665092 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/JoinEstimationSuite.scala --- @@ -67,6 +68,205 @@ class

[GitHub] spark issue #19837: [SPARK-22637][SQL] Only refresh a logical plan once.

2017-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19837 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

  1   2   3   4   >