[GitHub] spark issue #17120: [SPARK-19715][Structured Streaming] Option to Strip Path...

2017-03-03 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/17120 Note streams can be very long running, so this isn't about some short window. It could even be that I'm moving to a different bucket (but don't want to loose my exactly once guarantees of a very

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104250125 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -159,6 +165,10 @@ private[hive] class

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104250011 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -169,16 +179,22 @@ private[hive] class

[GitHub] spark issue #17120: [SPARK-19715][Structured Streaming] Option to Strip Path...

2017-03-03 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/17120 I know that's the *current* use case, but I'm thinking about future confusion, especially as the use case you espoused, "move from s3n to s3a within the same window" isn't likely to be that

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread budde
Github user budde commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104249753 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -159,6 +165,10 @@ private[hive] class

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread budde
Github user budde commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104249556 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -169,16 +179,22 @@ private[hive] class

[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16476 It seems MySQL do need all arguments in `FIELD` have same type(or can be converted to same type): http://www.w3resource.com/mysql/string-functions/mysql-field-function.php I think this

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread budde
Github user budde commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104249293 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -217,6 +235,62 @@ private[hive] class

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread budde
Github user budde commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104248644 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -195,17 +211,19 @@ private[hive] class

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread budde
Github user budde commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104248243 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -597,6 +597,16 @@ private[spark] class HiveExternalCatalog(conf:

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread budde
Github user budde commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104247964 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -195,17 +211,19 @@ private[hive] class

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104245300 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -195,17 +211,19 @@ private[hive] class

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104245195 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -195,17 +211,19 @@ private[hive] class

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104244968 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -169,16 +179,22 @@ private[hive] class

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104244767 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -217,6 +235,62 @@ private[hive] class

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104244137 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -159,6 +165,10 @@ private[hive] class

[GitHub] spark issue #17091: DriverEndpoint#makeOffers race against CoarseGrainedSche...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17091 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73859/ Test PASSed. ---

[GitHub] spark issue #17091: DriverEndpoint#makeOffers race against CoarseGrainedSche...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17091 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17091: DriverEndpoint#makeOffers race against CoarseGrainedSche...

2017-03-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17091 **[Test build #73859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73859/testReport)** for PR 17091 at commit

[GitHub] spark pull request #16944: [SPARK-19611][SQL] Introduce configurable table s...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16944#discussion_r104243616 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -597,6 +597,16 @@ private[spark] class

[GitHub] spark pull request #16910: [SPARK-19575][SQL]Reading from or writing to a hi...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16910#discussion_r104242881 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -1588,6 +1590,153 @@ class HiveDDLSuite }

[GitHub] spark issue #17061: [SPARK-13446] [SQL] Support reading data from Hive 2.0.1...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17061 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17061: [SPARK-13446] [SQL] Support reading data from Hive 2.0.1...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17061 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73860/ Test PASSed. ---

[GitHub] spark issue #17061: [SPARK-13446] [SQL] Support reading data from Hive 2.0.1...

2017-03-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17061 **[Test build #73860 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73860/testReport)** for PR 17061 at commit

[GitHub] spark pull request #17075: [SPARK-19727][SQL] Fix for round function that mo...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17075#discussion_r104236324 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/MathFunctionsSuite.scala --- @@ -233,6 +233,18 @@ class MathFunctionsSuite extends QueryTest with

[GitHub] spark pull request #17075: [SPARK-19727][SQL] Fix for round function that mo...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17075#discussion_r104236016 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/MathFunctionsSuite.scala --- @@ -233,6 +233,18 @@ class MathFunctionsSuite extends QueryTest with

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-03 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16867 more brainstorming: (1) you could lazily update your median collection (whether its a treeset or median heap). First you'd just dump tasks into an array, and then when you query for the

[GitHub] spark pull request #17075: [SPARK-19727][SQL] Fix for round function that mo...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17075#discussion_r104234498 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala --- @@ -223,6 +224,18 @@ final class Decimal extends Ordered[Decimal] with

[GitHub] spark pull request #17094: [SPARK-19762][ML] Hierarchy for consolidating ML ...

2017-03-03 Thread sethah
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/17094#discussion_r104234261 --- Diff: mllib/src/main/scala/org/apache/spark/ml/optim/loss/DifferentiableRegularization.scala --- @@ -0,0 +1,68 @@ +/* + * Licensed to the

[GitHub] spark pull request #16696: [SPARK-19350] [SQL] Cardinality estimation of Lim...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16696#discussion_r104232961 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala --- @@ -773,14 +773,20 @@ case class

[GitHub] spark issue #16854: [WIP][SPARK-15463][SQL] Add an API to load DataFrame fro...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16854 does `def csv(csvDataset: Dataset[String])` need to support whole file? I think the JSON one doesn't support it either. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request #16910: [SPARK-19575][SQL]Reading from or writing to a hi...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16910#discussion_r104231197 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -1588,6 +1596,147 @@ class HiveDDLSuite }

[GitHub] spark pull request #17148: [SPARK-17075][SQL][followup] fix filter estimatio...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17148#discussion_r104230245 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -469,22 +487,28 @@ case

[GitHub] spark pull request #17148: [SPARK-17075][SQL][followup] fix filter estimatio...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17148#discussion_r104229797 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -414,53 +422,63 @@ case

[GitHub] spark pull request #17148: [SPARK-17075][SQL][followup] fix filter estimatio...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17148#discussion_r104229412 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -414,53 +422,63 @@ case

[GitHub] spark pull request #17148: [SPARK-17075][SQL][followup] fix filter estimatio...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17148#discussion_r104228902 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -323,7 +331,7 @@ case

[GitHub] spark pull request #17148: [SPARK-17075][SQL][followup] fix filter estimatio...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17148#discussion_r104228741 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -249,17 +251,19 @@ case

[GitHub] spark pull request #17148: [SPARK-17075][SQL][followup] fix filter estimatio...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17148#discussion_r104228522 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -101,21 +101,23 @@ case

[GitHub] spark pull request #17148: [SPARK-17075][SQL][followup] fix filter estimatio...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17148#discussion_r104228432 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -101,21 +101,23 @@ case

[GitHub] spark pull request #17061: [SPARK-13446] [SQL] Support reading data from Hiv...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17061#discussion_r104227900 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala --- @@ -98,7 +98,10 @@ class VersionsSuite extends QueryTest with

[GitHub] spark issue #17140: [SPARK-19796][CORE] Fix serialization of long property v...

2017-03-03 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/17140 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #17061: [SPARK-13446] [SQL] Support reading data from Hive 2.0.1...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17061 we should also update `InsertIntoHiveTable`, it will be great if we can improve it to fail compilation when adding new hive versions. --- If your project is set up for it, you can reply to this

[GitHub] spark pull request #17061: [SPARK-13446] [SQL] Support reading data from Hiv...

2017-03-03 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/17061#discussion_r104226078 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala --- @@ -62,6 +62,13 @@ package object client {

[GitHub] spark pull request #17061: [SPARK-13446] [SQL] Support reading data from Hiv...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17061#discussion_r104225709 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala --- @@ -62,6 +62,13 @@ package object client {

[GitHub] spark issue #17144: [SPARK-19803][TEST] flaky BlockManagerReplicationSuite t...

2017-03-03 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/17144 Also @uncleGen would you mind filing a JIRA for the second failed test case? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #17144: [SPARK-19803][TEST] flaky BlockManagerReplicationSuite t...

2017-03-03 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/17144 cc @shubhamchopra who wrote the original code and @joshrosen who did the main review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #17110: [SPARK-19635][ML] DataFrame-based API for chi square tes...

2017-03-03 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/17110 Actually, synced with @thunterdb and will update design doc to put everything under a "Statistics" object. I'll wait until https://github.com/apache/spark/pull/17108 gets merged. --- If your

[GitHub] spark pull request #17123: [SPARK-19781][ML] Handle NULLs as well as NaNs in...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17123#discussion_r104224870 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/Bucketizer.scala --- @@ -105,20 +106,24 @@ final class Bucketizer @Since("1.4.0")

[GitHub] spark issue #17144: [SPARK-19803][TEST] flaky BlockManagerReplicationSuite t...

2017-03-03 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/17144 I'm not really the right person to review this code, but that being said, I'm not crazy about this fix, because 1s is kind of a long time to consistently wait. It's better for tests to

[GitHub] spark pull request #16696: [SPARK-19350] [SQL] Cardinality estimation of Lim...

2017-03-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16696#discussion_r104223997 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/StatsConfSuite.scala --- @@ -1,64 +0,0 @@ -/* - * Licensed

[GitHub] spark issue #17061: [SPARK-13446] [SQL] Support reading data from Hive 2.0.1...

2017-03-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17061 **[Test build #73860 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73860/testReport)** for PR 17061 at commit

[GitHub] spark issue #17061: [SPARK-13446] [SQL] Support reading data from Hive 2.0.1...

2017-03-03 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17061 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #17056: [SPARK-17495] [SQL] Support Decimal type in Hive-...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17056#discussion_r10449 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -732,6 +743,48 @@ object HiveHashFunction extends

[GitHub] spark issue #16944: [SPARK-19611][SQL] Introduce configurable table schema i...

2017-03-03 Thread budde
Github user budde commented on the issue: https://github.com/apache/spark/pull/16944 @viirya – Latest update incorporates your feedback. Thanks! @cloud-fan, @ericl – Any chance you could look at this updated PR? --- If your project is set up for it, you can reply to

[GitHub] spark issue #16944: [SPARK-19611][SQL] Introduce configurable table schema i...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16944 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73858/ Test PASSed. ---

[GitHub] spark issue #16944: [SPARK-19611][SQL] Introduce configurable table schema i...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16944 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17056: [SPARK-17495] [SQL] Support Decimal type in Hive-...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17056#discussion_r104221836 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -732,6 +743,48 @@ object HiveHashFunction extends

[GitHub] spark issue #16944: [SPARK-19611][SQL] Introduce configurable table schema i...

2017-03-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16944 **[Test build #73858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73858/testReport)** for PR 16944 at commit

[GitHub] spark issue #17120: [SPARK-19715][Structured Streaming] Option to Strip Path...

2017-03-03 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/17120 The use case here is when you have truly unique filenames (i.e. they contain a guid). This is actually pretty common in my experience. We definitely shouldn't turn this on by default, but as

[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2017-03-03 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/16061 +1 I think we should close this PR to avoid confusion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2017-03-03 Thread mccheah
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/16061 Work on this feature has moved to https://github.com/apache-spark-on-k8s/spark. The exact diff we are working with is this: https://github.com/apache-spark-on-k8s/spark/pull/1. Feel free to

[GitHub] spark pull request #17110: [SPARK-19635][ML] DataFrame-based API for chi squ...

2017-03-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17110#discussion_r104220095 --- Diff: mllib/src/test/scala/org/apache/spark/ml/stat/ChiSquareSuite.scala --- @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17110: [SPARK-19635][ML] DataFrame-based API for chi squ...

2017-03-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17110#discussion_r104220074 --- Diff: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquare.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #17110: [SPARK-19635][ML] DataFrame-based API for chi squ...

2017-03-03 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17110#discussion_r104220081 --- Diff: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquare.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #17091: DriverEndpoint#makeOffers race against CoarseGrainedSche...

2017-03-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17091 **[Test build #73859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73859/testReport)** for PR 17091 at commit

[GitHub] spark issue #17044: [SPARK-19718][SS]Handle more interrupt cases properly fo...

2017-03-03 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17044 @tdas could you take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17091: DriverEndpoint#makeOffers race against CoarseGrainedSche...

2017-03-03 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17091 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark pull request #17107: [SPARK-19774] StreamExecution should call stop() ...

2017-03-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17107 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15666: [SPARK-11421] [Core][Python][R] Added ability for addJar...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15666 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15666: [SPARK-11421] [Core][Python][R] Added ability for addJar...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15666 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73856/ Test FAILed. ---

[GitHub] spark issue #15666: [SPARK-11421] [Core][Python][R] Added ability for addJar...

2017-03-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15666 **[Test build #73856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73856/testReport)** for PR 15666 at commit

[GitHub] spark issue #17107: [SPARK-19774] StreamExecution should call stop() on sour...

2017-03-03 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17107 Thanks. LGTM. Merging to master and 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17091: DriverEndpoint#makeOffers race against CoarseGrai...

2017-03-03 Thread jxiang
Github user jxiang commented on a diff in the pull request: https://github.com/apache/spark/pull/17091#discussion_r104214804 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -222,12 +222,17 @@ class

[GitHub] spark pull request #17133: [SPARK-19793] Use clock.getTimeMillis when mark t...

2017-03-03 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/17133#discussion_r104214744 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala --- @@ -75,6 +75,8 @@ class TaskInfo( } private[spark] def

[GitHub] spark issue #17056: [SPARK-17495] [SQL] Support Decimal type in Hive-hash

2017-03-03 Thread tejasapatil
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/17056 @cloud-fan ping !! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #17114: [SPARK-19758][SQL] Resolving timezone aware expre...

2017-03-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17114#discussion_r104211876 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -146,7 +146,7 @@ class Analyzer(

[GitHub] spark pull request #17125: [SPARK-19211][SQL] Explicitly prevent Insert into...

2017-03-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17125#discussion_r104211301 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -1329,6 +1329,16 @@ class SparkSqlAstBuilder(conf: SQLConf)

[GitHub] spark pull request #17125: [SPARK-19211][SQL] Explicitly prevent Insert into...

2017-03-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17125#discussion_r104211285 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -1329,6 +1329,16 @@ class SparkSqlAstBuilder(conf: SQLConf)

[GitHub] spark issue #17123: [SPARK-19781][ML] Handle NULLs as well as NaNs in Bucket...

2017-03-03 Thread crackcell
Github user crackcell commented on the issue: https://github.com/apache/spark/pull/17123 @srowen @cloud-fan Please review my code. Thanks. :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #17125: [SPARK-19211][SQL] Explicitly prevent Insert into...

2017-03-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17125#discussion_r104210669 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala --- @@ -484,6 +485,23 @@ abstract class SQLViewSuite extends

[GitHub] spark pull request #16842: [SPARK-19304] [Streaming] [Kinesis] fix kinesis s...

2017-03-03 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/16842#discussion_r104210579 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -193,9 +201,10 @@ class

[GitHub] spark pull request #17125: [SPARK-19211][SQL] Explicitly prevent Insert into...

2017-03-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17125#discussion_r104210054 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala --- @@ -484,6 +485,23 @@ abstract class SQLViewSuite extends

[GitHub] spark pull request #17125: [SPARK-19211][SQL] Explicitly prevent Insert into...

2017-03-03 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17125#discussion_r104210022 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala --- @@ -484,6 +485,23 @@ abstract class SQLViewSuite extends

[GitHub] spark issue #17125: [SPARK-19211][SQL] Explicitly prevent Insert into View o...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17125 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73857/ Test PASSed. ---

[GitHub] spark issue #17125: [SPARK-19211][SQL] Explicitly prevent Insert into View o...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17125 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17125: [SPARK-19211][SQL] Explicitly prevent Insert into View o...

2017-03-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17125 **[Test build #73857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73857/testReport)** for PR 17125 at commit

[GitHub] spark issue #17152: [SPARK-18389][SQL] Disallow cyclic view reference

2017-03-03 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17152 Based on our current impl of view, `createOrReplaceTempView` can still trigger the cyclic view reference. Right? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Increase default refuse_seconds tim...

2017-03-03 Thread mgummelt
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17031 @skonto I completely agree that this is a cluster-wide issue, but unfortunately that's the state of things. In the long-term, optimistic offers in Mesos should fix this. --- If your project is

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Increase default refuse_seconds tim...

2017-03-03 Thread mgummelt
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17031 @srowen Yes, most of the code is refactoring that I came across when solving this. If that's going to delay this being merged, please let me know and I can remove the refactoring. --- If your

[GitHub] spark issue #15158: [SPARK-17603] [SQL] Utilize Hive-generated Statistics Fo...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15158 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15158: [SPARK-17603] [SQL] Utilize Hive-generated Statistics Fo...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15158 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73853/ Test PASSed. ---

[GitHub] spark issue #17117: [SPARK-10780][ML] Support initial model for KMeans.

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17117 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17149: [SPARK-19257][SQL]location for table/partition/database ...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17149 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73855/ Test PASSed. ---

[GitHub] spark issue #17117: [SPARK-10780][ML] Support initial model for KMeans.

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17117 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73852/ Test PASSed. ---

[GitHub] spark issue #15158: [SPARK-17603] [SQL] Utilize Hive-generated Statistics Fo...

2017-03-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15158 **[Test build #73853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73853/consoleFull)** for PR 15158 at commit

[GitHub] spark issue #17149: [SPARK-19257][SQL]location for table/partition/database ...

2017-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17149 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17117: [SPARK-10780][ML] Support initial model for KMeans.

2017-03-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17117 **[Test build #73852 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73852/testReport)** for PR 17117 at commit

[GitHub] spark issue #17149: [SPARK-19257][SQL]location for table/partition/database ...

2017-03-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17149 **[Test build #73855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73855/testReport)** for PR 17149 at commit

[GitHub] spark pull request #16061: [SPARK-18278] [Scheduler] Support native submissi...

2017-03-03 Thread jaceklaskowski
Github user jaceklaskowski commented on a diff in the pull request: https://github.com/apache/spark/pull/16061#discussion_r104206250 --- Diff: kubernetes/pom.xml --- @@ -0,0 +1,54 @@ + + +http://maven.apache.org/POM/4.0.0;

[GitHub] spark pull request #16061: [SPARK-18278] [Scheduler] Support native submissi...

2017-03-03 Thread jaceklaskowski
Github user jaceklaskowski commented on a diff in the pull request: https://github.com/apache/spark/pull/16061#discussion_r104206163 --- Diff: kubernetes/pom.xml --- @@ -0,0 +1,54 @@ + + +http://maven.apache.org/POM/4.0.0;

<    1   2   3   4   5   6   7   >