[GitHub] spark issue #14617: [SPARK-17019][Core] Expose on-heap and off-heap memory u...

2017-03-20 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/14617 Thanks @tgravescs and @squito for your comments. Based on @tgravescs 's point, looks like making them as a table column is more valid. So I will revert back to use column and combine

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-20 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r107050116 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala --- @@ -965,9 +965,27 @@ class JDBCSuite extends SparkFunSuite

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-20 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r107050148 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/TableScanSuite.scala --- @@ -416,4 +416,22 @@ class TableScanSuite extends DataSourceTest

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r107050121 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -1860,4 +1861,115 @@ class HiveDDLSuite }

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r107050048 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/TableScanSuite.scala --- @@ -416,4 +416,22 @@ class TableScanSuite extends DataSourceTest

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-20 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r107049979 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -2178,4 +2178,138 @@ abstract class DDLSuite extends

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-20 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r107049953 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -296,6 +311,51 @@ class SessionCatalog( }

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r107050001 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala --- @@ -965,9 +965,27 @@ class JDBCSuite extends SparkFunSuite

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r107049695 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -2178,4 +2178,138 @@ abstract class DDLSuite extends

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r107049525 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -2178,4 +2178,138 @@ abstract class DDLSuite extends

[GitHub] spark pull request #17350: [SPARK-20017][SQL] change the nullability of func...

2017-03-20 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/17350#discussion_r107049220 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala --- @@ -251,6 +251,9 @@ class ComplexTypeSuite

[GitHub] spark issue #17350: [SPARK-20017][SQL] change the nullability of function 'S...

2017-03-20 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/17350 @hvanhovell could you check? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17335: [SPARK-19995][Hive][Yarn] Using real user to initialize ...

2017-03-20 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17335 Thanks @yaooqinn , that's really an issue here. That was my concern when I had this fix, since we wrap the whole `SessionState.start` with real user, it means all the operations inside this

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r107048698 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -296,6 +311,51 @@ class SessionCatalog( }

[GitHub] spark issue #17306: [SPARK-19979][MLLIB] Allow multiple pipelines when tunin...

2017-03-20 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/17306 Yeah, that's true in this case. You could just build the grids separately and combine them like this ```scala val pipeline1_grid = new ParamGridBuilder()

[GitHub] spark pull request #17246: [SPARK-19906][SS][DOCS] Documentation describing ...

2017-03-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17246 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17354: [SPARK-20024] [SQL] [test-maven] SessionCatalog reset ne...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17354 **[Test build #74915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74915/testReport)** for PR 17354 at commit

[GitHub] spark pull request #17354: [SPARK-20024] [SQL] [test-maven] SessionCatalog A...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17354#discussion_r107047761 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -226,6 +226,7 @@ class SessionCatalog(

[GitHub] spark issue #17246: [SPARK-19906][SS][DOCS] Documentation describing how to ...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17246 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17354: [SPARK-20024] [SQL] [test-maven] SessionCatalog A...

2017-03-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17354#discussion_r107047089 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -226,6 +226,7 @@ class SessionCatalog(

[GitHub] spark issue #17246: [SPARK-19906][SS][DOCS] Documentation describing how to ...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17246 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74913/ Test PASSed. ---

[GitHub] spark issue #17246: [SPARK-19906][SS][DOCS] Documentation describing how to ...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17246 **[Test build #74913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74913/testReport)** for PR 17246 at commit

[GitHub] spark issue #17219: [SPARK-19876][SS][WIP] OneTime Trigger Executor

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17219 **[Test build #74914 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74914/testReport)** for PR 17219 at commit

[GitHub] spark pull request #17219: [SPARK-19876][SS][WIP] OneTime Trigger Executor

2017-03-20 Thread tcondie
Github user tcondie commented on a diff in the pull request: https://github.com/apache/spark/pull/17219#discussion_r107045633 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/Trigger.scala --- @@ -38,6 +38,51 @@ sealed trait Trigger /** * ::

[GitHub] spark issue #17246: [SPARK-19906][SS][DOCS] Documentation describing how to ...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17246 **[Test build #74913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74913/testReport)** for PR 17246 at commit

[GitHub] spark issue #16971: [SPARK-19573][SQL] Make NaN/null handling consistent in ...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16971 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16971: [SPARK-19573][SQL] Make NaN/null handling consistent in ...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16971 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74908/ Test PASSed. ---

[GitHub] spark issue #16971: [SPARK-19573][SQL] Make NaN/null handling consistent in ...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16971 **[Test build #74908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74908/testReport)** for PR 16971 at commit

[GitHub] spark pull request #17297: [SPARK-14649][CORE] DagScheduler should not run d...

2017-03-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/17297#discussion_r107044660 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -929,12 +946,22 @@ class DAGScheduler( } }

[GitHub] spark pull request #17297: [SPARK-14649][CORE] DagScheduler should not run d...

2017-03-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/17297#discussion_r107044272 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -803,6 +810,16 @@ class DAGScheduler(

[GitHub] spark issue #17354: [SPARK-20024] [SQL] [test-maven] SessionCatalog API setC...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17354 cc @cloud-fan @yhuai @windpiger --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17354: [SPARK-20024] [SQL] [test-maven] SessionCatalog A...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17354#discussion_r107043195 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -226,6 +226,7 @@ class SessionCatalog(

[GitHub] spark issue #17354: [SPARK-20024] [SQL] [test-maven] SessionCatalog API setC...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17354 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17354: [SPARK-20024] [SQL] [test-maven] SessionCatalog API setC...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17354 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74904/ Test PASSed. ---

[GitHub] spark pull request #17297: [SPARK-14649][CORE] DagScheduler should not run d...

2017-03-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/17297#discussion_r107040190 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -418,6 +424,15 @@ private[spark] class MapOutputTrackerMaster(conf:

[GitHub] spark issue #17366: [SPARK-19970][SQL][BRANCH-1.6] Table owner should be USE...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17366 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17366: [SPARK-19970][SQL][BRANCH-1.6] Table owner should be USE...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74907/ Test PASSed. ---

[GitHub] spark issue #17366: [SPARK-19970][SQL][BRANCH-1.6] Table owner should be USE...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17366 **[Test build #74907 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74907/consoleFull)** for PR 17366 at commit

[GitHub] spark issue #17295: [SPARK-19556][core] Do not encrypt block manager data in...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17295 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17295: [SPARK-19556][core] Do not encrypt block manager data in...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17295 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74905/ Test PASSed. ---

[GitHub] spark issue #17295: [SPARK-19556][core] Do not encrypt block manager data in...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17295 **[Test build #74905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74905/testReport)** for PR 17295 at commit

[GitHub] spark issue #17295: [SPARK-19556][core] Do not encrypt block manager data in...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17295 **[Test build #74911 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74911/testReport)** for PR 17295 at commit

[GitHub] spark issue #17295: [SPARK-19556][core] Do not encrypt block manager data in...

2017-03-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17295 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #17295: [SPARK-19556][core] Do not encrypt block manager data in...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17295 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17295: [SPARK-19556][core] Do not encrypt block manager data in...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17295 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74906/ Test FAILed. ---

[GitHub] spark issue #17295: [SPARK-19556][core] Do not encrypt block manager data in...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17295 **[Test build #74906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74906/testReport)** for PR 17295 at commit

[GitHub] spark issue #17361: [SPARK-20030][SS][WIP]Event-time-based timeout for MapGr...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17361 **[Test build #74910 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74910/testReport)** for PR 17361 at commit

[GitHub] spark issue #17250: [SPARK-19911][STREAMING] Add builder interface for Kines...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17250 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17250: [SPARK-19911][STREAMING] Add builder interface for Kines...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17250 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74909/ Test PASSed. ---

[GitHub] spark issue #17250: [SPARK-19911][STREAMING] Add builder interface for Kines...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17250 **[Test build #74909 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74909/testReport)** for PR 17250 at commit

[GitHub] spark pull request #17354: [SPARK-20024] [SQL] [test-maven] SessionCatalog A...

2017-03-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/17354#discussion_r107032174 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -226,6 +226,7 @@ class SessionCatalog(

[GitHub] spark issue #17364: [SPARK-20038] [SQL]: FileFormatWriter.ExecuteWriteTask.r...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17364 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17364: [SPARK-20038] [SQL]: FileFormatWriter.ExecuteWriteTask.r...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17364 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74903/ Test PASSed. ---

[GitHub] spark issue #17364: [SPARK-20038] [SQL]: FileFormatWriter.ExecuteWriteTask.r...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17364 **[Test build #74903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74903/testReport)** for PR 17364 at commit

[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...

2017-03-20 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16499#discussion_r107028767 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1048,7 +1065,7 @@ private[spark] class BlockManager( try {

[GitHub] spark issue #17306: [SPARK-19979][MLLIB] Allow multiple pipelines when tunin...

2017-03-20 Thread leifker
Github user leifker commented on the issue: https://github.com/apache/spark/pull/17306 Sorry of the delayed response @BryanCutler, that's pretty neat, however this will perform unneeded work as it will execute nonsensical combinations of parameters. For example, if pipeline2 is

[GitHub] spark issue #12004: [SPARK-7481] [build] Add spark-cloud module to pull in o...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12004 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #12004: [SPARK-7481] [build] Add spark-cloud module to pull in o...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12004 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74899/ Test PASSed. ---

[GitHub] spark issue #12004: [SPARK-7481] [build] Add spark-cloud module to pull in o...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12004 **[Test build #74899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74899/testReport)** for PR 12004 at commit

[GitHub] spark issue #17250: [SPARK-19911][STREAMING] Add builder interface for Kines...

2017-03-20 Thread budde
Github user budde commented on the issue: https://github.com/apache/spark/pull/17250 @brkyvz Updated the PR to remove ```defaultMessageHandler()``` from ```KinesisUtils``` in favor of keeping this method in ```KinesisInputDStream```. My thought here was that this would be a better

[GitHub] spark issue #17250: [SPARK-19911][STREAMING] Add builder interface for Kines...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17250 **[Test build #74909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74909/testReport)** for PR 17250 at commit

[GitHub] spark issue #12004: [SPARK-7481] [build] Add spark-cloud module to pull in o...

2017-03-20 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/12004 The latest patch embraces the fact that 2.6 is the base hadoop version so the `hadoop-aws` JAR is always pulled in, dependencies set up. One thing to bear in mind here that the [Phase I

[GitHub] spark pull request #17250: [SPARK-19911][STREAMING] Add builder interface fo...

2017-03-20 Thread budde
Github user budde commented on a diff in the pull request: https://github.com/apache/spark/pull/17250#discussion_r107025906 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -71,7 +75,256 @@ private[kinesis] class

[GitHub] spark issue #14731: [SPARK-17159] [streaming]: optimise check for new files ...

2017-03-20 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14731 Any more comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16971: [SPARK-19573][SQL] Make NaN/null handling consistent in ...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16971 **[Test build #74908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74908/testReport)** for PR 16971 at commit

[GitHub] spark issue #17364: [SPARK-20038] [SQL]: FileFormatWriter.ExecuteWriteTask.r...

2017-03-20 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/17364 Note that as [the exception handler](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala#L244) tries to close

[GitHub] spark issue #16971: [SPARK-19573][SQL] Make NaN/null handling consistent in ...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16971 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #17315: [SPARK-19949][SQL] unify bad record handling in CSV and ...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17315 LGTM except a major comment about FAIL_FAST_MODE. Looks great! Glad to see such a code clean! --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #17366: [SPARK-19970][SQL][BRANCH-1.6] Table owner should be USE...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17366 **[Test build #74907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74907/consoleFull)** for PR 17366 at commit

[GitHub] spark pull request #17366: [SPARK-19970][SQL][BRANCH-1.6] Table owner should...

2017-03-20 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/17366 [SPARK-19970][SQL][BRANCH-1.6] Table owner should be USER instead of PRINCIPAL in kerberized clusters ## What changes were proposed in this pull request? In the kerberized hadoop

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17315: [SPARK-19949][SQL] unify bad record handling in C...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17315#discussion_r107020700 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala --- @@ -0,0 +1,80 @@ +/* + * Licensed to the

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74897/ Test PASSed. ---

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74897/testReport)** for PR 17343 at commit

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-20 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r107019056 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/StarJoinReorderSuite.scala --- @@ -0,0 +1,580 @@ +/* + *

[GitHub] spark pull request #17297: [SPARK-14649][CORE] DagScheduler should not run d...

2017-03-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/17297#discussion_r107018874 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -378,15 +382,17 @@ private[spark] class MapOutputTrackerMaster(conf:

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-20 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r107018483 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/StarJoinReorderSuite.scala --- @@ -0,0 +1,580 @@ +/* + *

[GitHub] spark pull request #17297: [SPARK-14649][CORE] DagScheduler should not run d...

2017-03-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/17297#discussion_r107018555 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -378,15 +382,17 @@ private[spark] class MapOutputTrackerMaster(conf:

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-20 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r107018102 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,340 @@ package

[GitHub] spark pull request #17297: [SPARK-14649][CORE] DagScheduler should not run d...

2017-03-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/17297#discussion_r107017201 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1265,64 +1280,11 @@ class DAGScheduler( val

[GitHub] spark issue #17365: [SPARK-19962] [MLlib] [WIP] add DictVectorizer to ml.fea...

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17365 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #17315: [SPARK-19949][SQL] unify bad record handling in C...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17315#discussion_r107014888 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala --- @@ -233,81 +187,41 @@ class UnivocityParser(

[GitHub] spark pull request #17365: [SPARK-19962] [MLlib] [WIP] add DictVectorizer to...

2017-03-20 Thread yupbank
GitHub user yupbank opened a pull request: https://github.com/apache/spark/pull/17365 [SPARK-19962] [MLlib] [WIP] add DictVectorizer to ml.feature ## What changes were proposed in this pull request? add a new estimator`DictVectorizer` and transformer `DictVectorizerModel` for

[GitHub] spark pull request #17315: [SPARK-19949][SQL] unify bad record handling in C...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17315#discussion_r107014069 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala --- @@ -46,85 +46,39 @@ class UnivocityParser(

[GitHub] spark pull request #17315: [SPARK-19949][SQL] unify bad record handling in C...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17315#discussion_r107013706 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -435,14 +442,20 @@ class DataFrameReader private[sql](sparkSession:

[GitHub] spark pull request #17315: [SPARK-19949][SQL] unify bad record handling in C...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17315#discussion_r107013316 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala --- @@ -0,0 +1,80 @@ +/* + * Licensed to the

[GitHub] spark pull request #17315: [SPARK-19949][SQL] unify bad record handling in C...

2017-03-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17315#discussion_r107013197 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala --- @@ -0,0 +1,80 @@ +/* + * Licensed to the

[GitHub] spark pull request #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-20 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17170#discussion_r107013042 --- Diff: R/pkg/R/mllib_fpm.R --- @@ -0,0 +1,153 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

[GitHub] spark pull request #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-20 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17170#discussion_r107012724 --- Diff: R/pkg/R/mllib_fpm.R --- @@ -0,0 +1,144 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74893/ Test PASSed. ---

[GitHub] spark pull request #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-20 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17170#discussion_r107010797 --- Diff: R/pkg/R/mllib_fpm.R --- @@ -0,0 +1,153 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

[GitHub] spark issue #17295: [SPARK-19556][core] Do not encrypt block manager data in...

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17295 **[Test build #74906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74906/testReport)** for PR 17295 at commit

[GitHub] spark pull request #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-20 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17170#discussion_r107009625 --- Diff: R/pkg/R/mllib_fpm.R --- @@ -0,0 +1,153 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74893 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74893/testReport)** for PR 17343 at commit

[GitHub] spark pull request #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-20 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17170#discussion_r107009205 --- Diff: R/pkg/R/mllib_fpm.R --- @@ -0,0 +1,153 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

[GitHub] spark pull request #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-20 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17170#discussion_r107009471 --- Diff: R/pkg/R/mllib_fpm.R --- @@ -0,0 +1,153 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

[GitHub] spark pull request #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-20 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17170#discussion_r107011970 --- Diff: mllib/src/main/scala/org/apache/spark/ml/r/FPGrowthWrapper.scala --- @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-20 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17170#discussion_r107010057 --- Diff: R/pkg/R/mllib_fpm.R --- @@ -0,0 +1,153 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

[GitHub] spark pull request #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-20 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17170#discussion_r107011745 --- Diff: mllib/src/main/scala/org/apache/spark/ml/r/FPGrowthWrapper.scala --- @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software

<    1   2   3   4   5   6   7   >