[GitHub] spark issue #15202: Backport SPARK-17599 and SPARK-17569 to Spark 2.0 branch

2016-09-22 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15202 Please separate these as two pull requests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15179: [SPARK-10835] [ML] Change Output of NGram to Array(Strin...

2016-09-22 Thread hhbyyh
Github user hhbyyh commented on the issue: https://github.com/apache/spark/pull/15179 Sorry, I meant unit test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15202: Backport SPARK-17599 and SPARK-17569 to Spark 2.0 branch

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15202 **[Test build #65790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65790/consoleFull)** for PR 15202 at commit

[GitHub] spark issue #15202: Backport SPARK-17599 and SPARK-17569 to Spark 2.0 branch

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15202 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14359: [SPARK-16719][ML] Random Forests should communicate fewe...

2016-09-22 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/14359 Thanks @hhbyyh and @sethah ! I agree that a later PR could be more careful about which trees are completed in which order and test this more thoroughly. But I hope this takes us 80% of

[GitHub] spark issue #15205: [SPARK-16240][ML] ML persistence backward compatibility ...

2016-09-22 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/15205 To reviewers: This code was taken and modified from [#14112]. @GayathriMurali should be the primary author when we merge this into branch-2.0 I'll merge this once tests pass. --- If

[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15206 **[Test build #65801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65801/consoleFull)** for PR 15206 at commit

[GitHub] spark issue #15202: Backport SPARK-17599 and SPARK-17569 to Spark 2.0 branch

2016-09-22 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15202 @rxin this one just added the unit test of #15153 into branch 2.0 + #15122. Do you still insist on separating them? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...

2016-09-22 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/15206 @zsxwing Will this break streaming jobs if someone upgrades their Spark version, since they won't be able to deserialize the class correctly? --- If your project is set up for it, you can reply to

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 @tdas moving this conversation back to the PR that's linked from the public jira > yeah, i am trying to figure out all the options and write up something to so that we are clear on the

[GitHub] spark pull request #15090: [SPARK-17073] [SQL] generate column-level statist...

2016-09-22 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/15090#discussion_r80161814 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -32,19 +34,74 @@ package

[GitHub] spark issue #15090: [SPARK-17073] [SQL] generate column-level statistics

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15090 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65793/ Test FAILed. ---

[GitHub] spark issue #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated Stats In...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14971 **[Test build #65795 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65795/consoleFull)** for PR 14971 at commit

[GitHub] spark issue #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated Stats In...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14971 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15206 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65803/ Test PASSed. ---

[GitHub] spark issue #15202: Backport SPARK-17599 and SPARK-17569 to Spark 2.0 branch

2016-09-22 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/15202 @rxin The only change that was needed was: https://github.com/apache/spark/pull/15202/files#diff-e82a44dc550d2a0a92e44d1ec2ecabccR137 Which was equivalent to the two master branch changes

[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/10212 **[Test build #65787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65787/consoleFull)** for PR 10212 at commit

[GitHub] spark issue #15202: Backport SPARK-17599 and SPARK-17569 to Spark 2.0 branch

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15202 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65790/ Test PASSed. ---

[GitHub] spark issue #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated Stats In...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14971 **[Test build #65795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65795/consoleFull)** for PR 14971 at commit

[GitHub] spark pull request #15203: [TEST][SPARK-17569] Make the unit test added for ...

2016-09-22 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15203 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15206 **[Test build #65803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65803/consoleFull)** for PR 15206 at commit

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14659 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65799/ Test PASSed. ---

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14659 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15204: [SPARK-17639][build] Add jce.jar to buildclasspath when ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15204 **[Test build #65808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65808/consoleFull)** for PR 15204 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 > I agree that if/when we add that ability to add existing partitions midstream we'd probably need to add two offsets in to the SQL offset for new partitions. It's not just existing

[GitHub] spark issue #15179: [SPARK-10835] [ML] Change Output of NGram to Array(Strin...

2016-09-22 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/15179 Accepting more input types SGTM too (with unit tests). The PR title and description (and perhaps the JIRA too) should be updated. Thanks! --- If your project is set up for it, you can reply to

[GitHub] spark issue #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to register Jav...

2016-09-22 Thread zjffdu
Github user zjffdu commented on the issue: https://github.com/apache/spark/pull/9766 Rebase the PR, @davies @JoshRosen Could you help to review it ? Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #14359: [SPARK-16719][ML] Random Forests should communicate fewe...

2016-09-22 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/14359 LGTM pending tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...

2016-09-22 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15206 /cc @jerryshao --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark pull request #15206: [SPARK-17640][SQL]Avoid using -1 as the default b...

2016-09-22 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/15206 [SPARK-17640][SQL]Avoid using -1 as the default batchId for FileStreamSource.FileEntry ## What changes were proposed in this pull request? Avoid using -1 as the default batchId for

[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS...

2016-09-22 Thread Sherry302
Github user Sherry302 commented on a diff in the pull request: https://github.com/apache/spark/pull/14659#discussion_r80161383 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Task.scala --- @@ -54,7 +54,10 @@ private[spark] abstract class Task[T]( val partitionId:

[GitHub] spark issue #15205: [SPARK-16240][ML] ML persistence backward compatibility ...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15205 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65800/ Test FAILed. ---

[GitHub] spark issue #15205: [SPARK-16240][ML] ML persistence backward compatibility ...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15205 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15205: [SPARK-16240][ML] ML persistence backward compatibility ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15205 **[Test build #65800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65800/consoleFull)** for PR 15205 at commit

[GitHub] spark issue #14359: [SPARK-16719][ML] Random Forests should communicate fewe...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14359 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65798/ Test PASSed. ---

[GitHub] spark issue #14359: [SPARK-16719][ML] Random Forests should communicate fewe...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14359 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14359: [SPARK-16719][ML] Random Forests should communicate fewe...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14359 **[Test build #65798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65798/consoleFull)** for PR 14359 at commit

[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics

2016-09-22 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/10212#discussion_r80165121 --- Diff: core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json --- @@ -6,6 +6,7 @@ "numCompleteTasks" : 8,

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 > For streaming you already know what the global order is, because you know when you asked for A and B. I agree that we should probably remove the comparable requirement from Offset in favor of

[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics

2016-09-22 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/10212#discussion_r80165213 --- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala --- @@ -1097,7 +1100,9 @@ private[spark] object JsonProtocolSuite extends

[GitHub] spark issue #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated Stats In...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14971 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65795/ Test PASSed. ---

[GitHub] spark issue #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated Stats In...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14971 **[Test build #65796 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65796/consoleFull)** for PR 14971 at commit

[GitHub] spark issue #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated Stats In...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14971 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65796/ Test PASSed. ---

[GitHub] spark pull request #15207: [SPARK-17643] Remove comparable requirement from ...

2016-09-22 Thread marmbrus
GitHub user marmbrus opened a pull request: https://github.com/apache/spark/pull/15207 [SPARK-17643] Remove comparable requirement from Offset For some sources, it is difficult to provide a global ordering based only on the data in the offset. Since we don't use comparison for

[GitHub] spark issue #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated Stats In...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14971 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65783/ Test PASSed. ---

[GitHub] spark pull request #15203: [TEST][SPARK-17569] Make the unit test added for ...

2016-09-22 Thread brkyvz
GitHub user brkyvz opened a pull request: https://github.com/apache/spark/pull/15203 [TEST][SPARK-17569] Make the unit test added for SPARK-17569 work again ## What changes were proposed in this pull request? A

[GitHub] spark issue #15203: [TEST][SPARK-17569] Make the unit test added for SPARK-1...

2016-09-22 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/15203 cc @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-09-22 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/14079 @kayousterhout @tgravescs sorry for the long delay from me. I've addressed most the feedback. But I haven't looked at separating out the blacklist logic into a separate class inside TaskSetManager

[GitHub] spark issue #15090: [SPARK-17073] [SQL] generate column-level statistics

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15090 **[Test build #65793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65793/consoleFull)** for PR 15090 at commit

[GitHub] spark pull request #15204: [SPARK-17639][build] Add jce.jar to buildclasspat...

2016-09-22 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/15204 [SPARK-17639][build] Add jce.jar to buildclasspath when building. This was missing, preventing code that uses javax.crypto to properly compile in Spark. You can merge this pull request into a

[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics

2016-09-22 Thread jisookim0513
Github user jisookim0513 commented on a diff in the pull request: https://github.com/apache/spark/pull/10212#discussion_r80156532 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -759,7 +761,15 @@ private[spark] object JsonProtocol { return

[GitHub] spark pull request #15090: [SPARK-17073] [SQL] generate column-level statist...

2016-09-22 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15090#discussion_r80156525 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -32,19 +34,74 @@ package

[GitHub] spark pull request #15034: [SPARK-16240][ML] ML persistence backward compati...

2016-09-22 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15034 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...

2016-09-22 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15206 > @zsxwing Will this break streaming jobs if someone upgrades their Spark version, since they won't be able to deserialize the class correctly? @brkyvz No. This doesn't change the file

[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...

2016-09-22 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15206 @brkyvz right now we cannot support upgrading anyway since the execution metadata uses Java serialization. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #15199: [SPARK-17635][SQL] Remove hardcode "agg_plan" in HashAgg...

2016-09-22 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15199 This was introduced by https://github.com/apache/spark/pull/14176. It should not affect Branch-2.0 --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...

2016-09-22 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/15206 LGTM, thanks for the fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15102 Comparable requirement removed in #15207. > I think in the absence of prior information about the position in a topicpartition, you start a new batch on topic B starting from wherever the

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15102 @tdas I think as long as marmbrus' PR to remove comparable from the interface works for sane variations of subscription changes it's the best way to go. I'm honestly fine with someone getting

[GitHub] spark pull request #15174: [SPARK-17502] [17609] [SQL] [Backport] [2.0] Fix ...

2016-09-22 Thread gatorsmile
Github user gatorsmile closed the pull request at: https://github.com/apache/spark/pull/15174 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS and YA...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14659 **[Test build #65799 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65799/consoleFull)** for PR 14659 at commit

[GitHub] spark issue #15174: [SPARK-17502] [17609] [SQL] [Backport] [2.0] Fix Multipl...

2016-09-22 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15174 Let me close it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15041: [SPARK-17488][CORE] TakeAndOrder will OOM when the data ...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15041 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15199: [SPARK-17635][SQL] Remove hardcode "agg_plan" in HashAgg...

2016-09-22 Thread yucai
Github user yucai commented on the issue: https://github.com/apache/spark/pull/15199 Thanks all, it should be fixed in master only, my mistake. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #15041: [SPARK-17488][CORE] TakeAndOrder will OOM when the data ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15041 **[Test build #65809 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65809/consoleFull)** for PR 15041 at commit

[GitHub] spark issue #15041: [SPARK-17488][CORE] TakeAndOrder will OOM when the data ...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15041 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65809/ Test FAILed. ---

[GitHub] spark pull request #15089: [SPARK-15621] [SQL] Support spilling for Python U...

2016-09-22 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/15089#discussion_r80174300 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/RowQueue.scala --- @@ -0,0 +1,278 @@ +/* +* Licensed to the Apache Software

[GitHub] spark issue #15204: [SPARK-17639][build] Add jce.jar to buildclasspath when ...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15204 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65802/ Test PASSed. ---

[GitHub] spark issue #15204: [SPARK-17639][build] Add jce.jar to buildclasspath when ...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15204 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15204: [SPARK-17639][build] Add jce.jar to buildclasspath when ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15204 **[Test build #65794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65794/consoleFull)** for PR 15204 at commit

[GitHub] spark issue #15205: [SPARK-16240][ML] ML persistence backward compatibility ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15205 **[Test build #65800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65800/consoleFull)** for PR 15205 at commit

[GitHub] spark issue #15202: Backport SPARK-17599 and SPARK-17569 to Spark 2.0 branch

2016-09-22 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15202 OK that's fine. Merging in. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15204: [SPARK-17639][build] Add jce.jar to buildclasspath when ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15204 **[Test build #65802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65802/consoleFull)** for PR 15204 at commit

[GitHub] spark pull request #15090: [SPARK-17073] [SQL] generate column-level statist...

2016-09-22 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/15090#discussion_r80165466 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala --- @@ -32,19 +34,74 @@ package

[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15206 **[Test build #65801 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65801/consoleFull)** for PR 15206 at commit

[GitHub] spark issue #15204: [SPARK-17639][build] Add jce.jar to buildclasspath when ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15204 **[Test build #65802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65802/consoleFull)** for PR 15204 at commit

[GitHub] spark issue #15199: [SPARK-17635][SQL] Remove hardcode "agg_plan" in HashAgg...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15199 **[Test build #65788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65788/consoleFull)** for PR 15199 at commit

[GitHub] spark issue #15034: [SPARK-16240][ML] ML persistence backward compatibility ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15034 **[Test build #65789 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65789/consoleFull)** for PR 15034 at commit

[GitHub] spark issue #15034: [SPARK-16240][ML] ML persistence backward compatibility ...

2016-09-22 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/15034 I'll go ahead and merge this. Thanks @hhbyyh for reviewing it! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #14359: [SPARK-16719][ML] Random Forests should communicate fewe...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14359 **[Test build #65798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65798/consoleFull)** for PR 14359 at commit

[GitHub] spark issue #15203: [TEST][SPARK-17569] Make the unit test added for SPARK-1...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15203 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15203: [TEST][SPARK-17569] Make the unit test added for SPARK-1...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15203 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65792/ Test PASSed. ---

[GitHub] spark pull request #15199: [SPARK-17635][SQL] Remove hardcode "agg_plan" in ...

2016-09-22 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15199 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to register Jav...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/9766 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65797/ Test PASSed. ---

[GitHub] spark issue #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to register Jav...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/9766 **[Test build #65797 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65797/consoleFull)** for PR 9766 at commit

[GitHub] spark issue #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to register Jav...

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/9766 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15208: [SPARK-17641][SQL] Collect_list/Collect_set should not c...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15208 **[Test build #65806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65806/consoleFull)** for PR 15208 at commit

[GitHub] spark issue #15203: [TEST][SPARK-17569] Make the unit test added for SPARK-1...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15203 **[Test build #65792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65792/consoleFull)** for PR 15203 at commit

[GitHub] spark issue #13458: [SPARK-15717][GraphX] Cannot perform RDD operations on a...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13458 **[Test build #3288 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3288/consoleFull)** for PR 13458 at commit

[GitHub] spark pull request #15102: [SPARK-17346][SQL] Add Kafka source for Structure...

2016-09-22 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/15102#discussion_r80154723 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala --- @@ -0,0 +1,474 @@ +/* + * Licensed to the

[GitHub] spark issue #9766: [SPARK-11775][PYSPARK][SQL] Allow PySpark to register Jav...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/9766 **[Test build #65797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65797/consoleFull)** for PR 9766 at commit

[GitHub] spark pull request #10212: [SPARK-12221] add cpu time to metrics

2016-09-22 Thread jisookim0513
Github user jisookim0513 commented on a diff in the pull request: https://github.com/apache/spark/pull/10212#discussion_r80156386 --- Diff: core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json --- @@ -6,6 +6,7 @@ "numCompleteTasks" :

[GitHub] spark issue #15203: [TEST][SPARK-17569] Make the unit test added for SPARK-1...

2016-09-22 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15203 LGTM. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15203: [TEST][SPARK-17569] Make the unit test added for SPARK-1...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15203 **[Test build #65792 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65792/consoleFull)** for PR 15203 at commit

[GitHub] spark issue #15090: [SPARK-17073] [SQL] generate column-level statistics

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15090 **[Test build #65793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65793/consoleFull)** for PR 15090 at commit

[GitHub] spark issue #15090: [SPARK-17073] [SQL] generate column-level statistics

2016-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15090 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15205: [SPARK-16240][ML] ML persistence backward compatibility ...

2016-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15205 **[Test build #65804 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65804/consoleFull)** for PR 15205 at commit

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

2016-09-22 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15102 > "I want to be able to add a topicpartition mid stream, but I don't want to start it from the beginning." I see, I was thinking only of new topics that appear that match your pattern. I

  1   2   3   4   5   >