[GitHub] spark pull request #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-20 Thread actuaryzhang
Github user actuaryzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/16344#discussion_r93289668 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -592,6 +629,59 @@ object

[GitHub] spark pull request #16356: [SPARK-18949] [SQL] Add repairTable API to Catalo...

2016-12-20 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/16356 [SPARK-18949] [SQL] Add repairTable API to Catalog ### What changes were proposed in this pull request? Currently, we only have a SQL interface for recovering all the partitions in the

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add repairTable API to Catalog

2016-12-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16356 We have two SQL equivalent commands: - ALTER TABLE table RECOVER PARTITIONS; - MSCK REPAIR TABLE table; I am not good at naming. How about `recoverPartitions`? --- If your

[GitHub] spark pull request #16357: [SPARK-18928][branch-2.0]Check TaskContext.isInte...

2016-12-20 Thread JoshRosen
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/16357 [SPARK-18928][branch-2.0]Check TaskContext.isInterrupted() in FileScanRDD, JDBCRDD & UnsafeSorter This is a branch-2.0 backport of #16340; the original description follows: ## What

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16337 yea those sound great. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16337 If the uniqueness is a problem, we can also just update SQLQueryTestSuite to use the path rather than just file name. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #16355: [SPARK-16473][MLLIB] Fix BisectingKMeans Algorith...

2016-12-20 Thread imatiach-msft
GitHub user imatiach-msft opened a pull request: https://github.com/apache/spark/pull/16355 [SPARK-16473][MLLIB] Fix BisectingKMeans Algorithm failing in edge case [SPARK-16473][MLLIB] Fix BisectingKMeans Algorithm failing in edge case where no children exist in updateAssignments

[GitHub] spark pull request #16352: [SPARK-18947][SQL] SQLContext.tableNames should n...

2016-12-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16352#discussion_r93300223 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala --- @@ -747,7 +747,7 @@ class SQLContext private[sql](val sparkSession:

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16336 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add repairTable API to Catalog

2016-12-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16356 What is the SQL equivalent command? MSCK? Should we match that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-20 Thread actuaryzhang
Github user actuaryzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/16344#discussion_r93290858 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -592,6 +629,59 @@ object

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2016-12-20 Thread merlintang
Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @cloud-fan @gatorsmile I have backport the code from #16134, can you verify and backport this to spark 1.6.x? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #15996: [SPARK-18567][SQL] Simplify CreateDataSourceTableAsSelec...

2016-12-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15996 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16336 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70415/ Test PASSed. ---

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16336 **[Test build #70415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70415/testReport)** for PR 16336 at commit

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add repairTable API to Catalog

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16356 **[Test build #70416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70416/testReport)** for PR 16356 at commit

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add repairTable API to Catalog

2016-12-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16356 Sure, will do. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add repairTable API to Catalog

2016-12-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16356 We should also add the Python API. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16354: [SPARK-18886][Scheduler][WIP] Adjust Delay scheduling to...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16354 **[Test build #70412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70412/testReport)** for PR 16354 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2016-12-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 Let me backport it to Spark 2.0.x at first. We need to follow the backport ordering --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-20 Thread nsyca
Github user nsyca commented on the issue: https://github.com/apache/spark/pull/16337 We can live with the uniqueness at this point. I'd like to get your feedback (positive, I hope) on the structure of the directories and the naming convention of the test files so we can move forward

[GitHub] spark issue #15980: [SPARK-18528][SQL] Fix a bug to initialise an iterator o...

2016-12-20 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/15980 lgtm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2016-12-20 Thread merlintang
Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @gatorsmile Great! thanks so much, because I was pinged multiple times for this bug. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #16357: [SPARK-18928][branch-2.0]Check TaskContext.isInterrupted...

2016-12-20 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/16357 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark pull request #16354: [SPARK-18886][Scheduler][WIP] Adjust Delay schedu...

2016-12-20 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/16354 [SPARK-18886][Scheduler][WIP] Adjust Delay scheduling to prevent under-utilization of cluster ## What changes were proposed in this pull request? This is a significant change to delay

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2016-12-20 Thread nsyca
Github user nsyca commented on the issue: https://github.com/apache/spark/pull/16337 I have tested a few runs on `SQLQueryTestSuite` to confirm it allows to have sub-directories under `sql/core/src/test/resources/sql-tests/[inputs|results]` to group test files further. By reading

[GitHub] spark pull request #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-20 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16344#discussion_r93291042 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -592,6 +629,59 @@ object GeneralizedLinearRegression

[GitHub] spark pull request #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-20 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16344#discussion_r93290854 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -242,7 +275,7 @@ class GeneralizedLinearRegression

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add repairTable API to Catalog

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16356 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70416/ Test FAILed. ---

[GitHub] spark issue #16357: [SPARK-18928][branch-2.0]Check TaskContext.isInterrupted...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16357 **[Test build #70417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70417/consoleFull)** for PR 16357 at commit

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add repairTable API to Catalog

2016-12-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16356 Yea recoverPartitions sound a lot better. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add repairTable API to Catalog

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16356 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add repairTable API to Catalog

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16356 **[Test build #70416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70416/testReport)** for PR 16356 at commit

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16343 **[Test build #70413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70413/testReport)** for PR 16343 at commit

[GitHub] spark pull request #16322: [SPARK-18908][SS] Creating StreamingQueryExceptio...

2016-12-20 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/16322#discussion_r93296609 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala --- @@ -282,7 +289,7 @@ class StreamExecution(

[GitHub] spark issue #16355: [SPARK-16473][MLLIB] Fix BisectingKMeans Algorithm faili...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16355 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16343 **[Test build #70414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70414/testReport)** for PR 16343 at commit

[GitHub] spark issue #16354: [SPARK-18886][Scheduler][WIP] Adjust Delay scheduling to...

2016-12-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16354 @mridulm @markhamstra @kayousterhout This is *not* ready to merge -- it needs some cleanup and more tests -- but I thought that seeing an implementation might help think through the design. I

[GitHub] spark pull request #16354: [SPARK-18886][Scheduler][WIP] Adjust Delay schedu...

2016-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16354#discussion_r93292503 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -549,11 +546,15 @@ class TaskSetManagerSuite extends SparkFunSuite

[GitHub] spark pull request #15996: [SPARK-18567][SQL] Simplify CreateDataSourceTable...

2016-12-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15996#discussion_r93298271 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -364,48 +365,149 @@ final class DataFrameWriter[T] private[sql](ds:

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

2016-12-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16336 Thank you, @srowen ! I will update the `README.md` , too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16336 **[Test build #70415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70415/testReport)** for PR 16336 at commit

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add repairTable API to Catalog

2016-12-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16356 cc @rxin @cloud-fan @ericl --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16343 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16304: [SPARK-18894][SS] Fix event time watermark delay thresho...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16304 **[Test build #70420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70420/testReport)** for PR 16304 at commit

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70414/ Test PASSed. ---

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16343 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70425/testReport)** for PR 16322 at commit

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

2016-12-20 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16322 @marmbrus updated as per our offline discussion. I also added a new method `assertAwaitThread` to avoid the user calling the awaitXXX methods inside the stream thread (e.g., in the

[GitHub] spark issue #16357: [SPARK-18928][branch-2.0]Check TaskContext.isInterrupted...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16357 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70417/ Test PASSed. ---

[GitHub] spark issue #16357: [SPARK-18928][branch-2.0]Check TaskContext.isInterrupted...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16357 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #16361: [SPARK-18952] Regex strings not properly escaped ...

2016-12-20 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/16361#discussion_r93347240 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala --- @@ -48,28 +48,30 @@ class

[GitHub] spark pull request #16357: [SPARK-18928][branch-2.0]Check TaskContext.isInte...

2016-12-20 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/16357#discussion_r93321229 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java --- @@ -256,6 +259,14 @@ public boolean hasNext() {

[GitHub] spark pull request #16357: [SPARK-18928][branch-2.0]Check TaskContext.isInte...

2016-12-20 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/16357#discussion_r93323151 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala --- @@ -88,7 +88,15 @@ class FileScanRDD(

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add recoverPartitions API to Catalog

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16356 **[Test build #70423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70423/testReport)** for PR 16356 at commit

[GitHub] spark issue #16359: [SPARK-18951] Upgrade com.thoughtworks.paranamer/paranam...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16359 **[Test build #70424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70424/testReport)** for PR 16359 at commit

[GitHub] spark issue #16358: [SPARK-18761][branch-2.0] Introduce "task reaper" to ove...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16358 **[Test build #70419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70419/consoleFull)** for PR 16358 at commit

[GitHub] spark issue #16361: [SPARK-18952] Regex strings not properly escaped in code...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16361 **[Test build #70432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70432/testReport)** for PR 16361 at commit

[GitHub] spark pull request #16361: [SPARK-18952] Regex strings not properly escaped ...

2016-12-20 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16361#discussion_r93346662 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/RowBasedHashMapGenerator.scala --- @@ -43,28 +43,30 @@ class

[GitHub] spark issue #16361: [SPARK-18952] Regex strings not properly escaped in code...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16361 **[Test build #70433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70433/testReport)** for PR 16361 at commit

[GitHub] spark issue #16253: [SPARK-18537][Web UI] Add a REST api to serve spark stre...

2016-12-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16253 Looks like others are not really interested in reviewing. I'll take another closer look at the model tomorrow, just to make sure. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16343 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70413/ Test FAILed. ---

[GitHub] spark issue #16354: [SPARK-18886][Scheduler][WIP] Adjust Delay scheduling to...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16354 **[Test build #70412 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70412/testReport)** for PR 16354 at commit

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add recoverPartitions API to Catalog

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16356 **[Test build #70422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70422/testReport)** for PR 16356 at commit

[GitHub] spark pull request #16356: [SPARK-18949] [SQL] Add recoverPartitions API to ...

2016-12-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16356#discussion_r93329801 --- Diff: project/MimaExcludes.scala --- @@ -37,7 +37,9 @@ object MimaExcludes { // Exclude rules for 2.2.x lazy val v22excludes =

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add recoverPartitions API to Catalog

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16356 **[Test build #70418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70418/testReport)** for PR 16356 at commit

[GitHub] spark issue #16360: [SPARK-18234][SS] Made update mode public

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16360 **[Test build #70429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70429/testReport)** for PR 16360 at commit

[GitHub] spark issue #16358: [SPARK-18761][branch-2.0] Introduce "task reaper" to ove...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16358 **[Test build #70419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70419/consoleFull)** for PR 16358 at commit

[GitHub] spark pull request #16263: [SPARK-18281][SQL][PySpark] Remove timeout for re...

2016-12-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16263 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #16356: [SPARK-18949] [SQL] Add recoverPartitions API to ...

2016-12-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16356#discussion_r93335790 --- Diff: python/pyspark/sql/catalog.py --- @@ -258,6 +258,11 @@ def refreshTable(self, tableName): """Invalidate and refresh all the cached

[GitHub] spark pull request #15666: [SPARK-11421] [Core][Python][R] Added ability for...

2016-12-20 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/15666#discussion_r93338899 --- Diff: core/src/main/scala/org/apache/spark/TestUtils.scala --- @@ -164,6 +164,27 @@ private[spark] object TestUtils {

[GitHub] spark pull request #16360: Made update mode public

2016-12-20 Thread tdas
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/16360 Made update mode public ## What changes were proposed in this pull request? Made update mode public. As part of that here are the changes. - Update DatastreamWriter to accept "update"

[GitHub] spark issue #16211: [SPARK-18576][PYTHON] Add basic TaskContext information ...

2016-12-20 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/16211 Maybe @davies has some cycles now that 2.1 is out :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add recoverPartitions API to Catalog

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16356 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add recoverPartitions API to Catalog

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16356 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70418/ Test FAILed. ---

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16343 **[Test build #70431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70431/testReport)** for PR 16343 at commit

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

2016-12-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16336 Thank you for review, @holdenk ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16253: [SPARK-18537][Web UI] Add a REST api to serve spark stre...

2016-12-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16253 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add recoverPartitions API to Catalog

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16356 **[Test build #70418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70418/testReport)** for PR 16356 at commit

[GitHub] spark pull request #16357: [SPARK-18928][branch-2.0]Check TaskContext.isInte...

2016-12-20 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16357#discussion_r93323712 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java --- @@ -256,6 +259,14 @@ public boolean hasNext() {

[GitHub] spark pull request #16357: [SPARK-18928][branch-2.0]Check TaskContext.isInte...

2016-12-20 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16357#discussion_r93326959 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java --- @@ -256,6 +259,14 @@ public boolean hasNext() {

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add recoverPartitions API to Catalog

2016-12-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16356 We can laso merge this in branch-2.1. So let's do 2.1.1 as since version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #16359: [SPARK-18951] Upgrade com.thoughtworks.paranamer/...

2016-12-20 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16359#discussion_r93335284 --- Diff: pom.xml --- @@ -179,7 +179,7 @@ 4.5.3 1.1 2.52.0 -2.8 +2.6 --- End diff -- Yep, looks

[GitHub] spark pull request #16356: [SPARK-18949] [SQL] Add recoverPartitions API to ...

2016-12-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16356#discussion_r93337049 --- Diff: python/pyspark/sql/catalog.py --- @@ -258,6 +258,11 @@ def refreshTable(self, tableName): """Invalidate and refresh all the cached

[GitHub] spark issue #16360: Made update mode public

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16360 **[Test build #70428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70428/testReport)** for PR 16360 at commit

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16343 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16343 **[Test build #70427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70427/testReport)** for PR 16343 at commit

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16343 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70427/ Test FAILed. ---

[GitHub] spark pull request #16361: [SPARK-18952] Regex strings not properly escaped ...

2016-12-20 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16361#discussion_r93346634 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala --- @@ -48,28 +48,30 @@ class

[GitHub] spark issue #16263: [SPARK-18281][SQL][PySpark] Remove timeout for reading d...

2016-12-20 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/16263 Merging this into master and 2.1 branch, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #16282: [SPARK-18588][SS][Kafka]Create a new KafkaConsume...

2016-12-20 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/16282#discussion_r93331357 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala --- @@ -536,11 +570,12 @@ private[kafka010] object

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16343 **[Test build #70421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70421/testReport)** for PR 16343 at commit

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16343 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70421/ Test FAILed. ---

[GitHub] spark issue #16343: [FLAKY-TEST][DO NOT MERGE] InputStreamsSuite.socket inpu...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16343 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

2016-12-20 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/16336 I think for new Python or Scala developers there is a benefit in skipping the R docs because they require some extra setup work to build and having the process simplified would be good for

[GitHub] spark issue #16356: [SPARK-18949] [SQL] Add recoverPartitions API to Catalog

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16356 **[Test build #70426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70426/testReport)** for PR 16356 at commit

[GitHub] spark issue #16342: [SPARK-18927][SS] MemorySink for StructuredStreaming can...

2016-12-20 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16342 LGTM. Thanks! Merging to master and 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16360: [SPARK-18234][SS] Made update mode public

2016-12-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16360 **[Test build #70430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70430/testReport)** for PR 16360 at commit

[GitHub] spark issue #16358: [SPARK-18761][branch-2.0] Introduce "task reaper" to ove...

2016-12-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16358 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-20 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @srowen Thanks for the comments. Really helpful. I have made a new commit that addresses the issues you raised: - I think the use of a global family object does not work well for the

  1   2   3   4   5   >