[GitHub] spark issue #17083: [SPARK-19750][UI][branch-2.1] Fix redirect issue from ht...

2017-02-27 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17083 Due to the change of (https://github.com/apache/spark/pull/16625), the issue is obsolete. So it effects spark 2.1 and 2.0. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #17071: [SPARK-15615][SQL][BUILD][FOLLOW-UP] Replace deprecated ...

2017-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17071 (I put a test here - https://github.com/apache/spark/pull/17071/files#diff-7e47859dbd409cc39f2908615fbd07ffR419) --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #17068: [SPARK-19709][SQL] Read empty file with CSV data ...

2017-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/17068#discussion_r103214603 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala --- @@ -40,7 +41,19 @@ private[csv] object

[GitHub] spark pull request #16809: [SPARK-19463][SQL]refresh cache after the InsertI...

2017-02-27 Thread windpiger
Github user windpiger commented on a diff in the pull request: https://github.com/apache/spark/pull/16809#discussion_r103185139 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala --- @@ -132,6 +132,9 @@ case class

[GitHub] spark pull request #16990: [SPARK-19660][CORE][SQL] Replace the configuratio...

2017-02-27 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/16990#discussion_r103185158 --- Diff: sql/hive/src/test/resources/ql/src/test/queries/clientpositive/smb_mapjoin_25.q --- @@ -19,7 +19,7 @@ select * from (select a.key from

[GitHub] spark pull request #14731: [SPARK-17159] [streaming]: optimise check for new...

2017-02-27 Thread uncleGen
Github user uncleGen commented on a diff in the pull request: https://github.com/apache/spark/pull/14731#discussion_r103187577 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -140,7 +137,7 @@ class FileInputDStream[K, V, F <:

[GitHub] spark issue #16971: [SPARK-19573][SQL] Make NaN/null handling consistent in ...

2017-02-27 Thread zhengruifeng
Github user zhengruifeng commented on the issue: https://github.com/apache/spark/pull/16971 ping @MLnick @gatorsmile @thunterdb --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17059: [SPARK-19733][ML]Removed unnecessary castings and refact...

2017-02-27 Thread datumbox
Github user datumbox commented on the issue: https://github.com/apache/spark/pull/17059 @srowen: Thanks for the comments. We are getting there. :) I will handle the Long case as you suggest. If you think people use SQL decimal types, I can include them at the end of

[GitHub] spark pull request #16990: [SPARK-19660][CORE][SQL] Replace the configuratio...

2017-02-27 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/16990#discussion_r103183030 --- Diff: sql/hive/src/test/resources/ql/src/test/queries/clientpositive/smb_mapjoin_25.q --- @@ -19,7 +19,7 @@ select * from (select a.key from

[GitHub] spark issue #17039: [SPARK-19710][SQL][TESTS] Fix ordering of rows in query ...

2017-02-27 Thread robbinspg
Github user robbinspg commented on the issue: https://github.com/apache/spark/pull/17039 @gatorsmile I'm glad it wasn't just me that found it complex ;-) I've modified the patch to remove an unnecessary change as that query was not ordered and the test suite code handles

[GitHub] spark issue #15505: [SPARK-18890][CORE] Move task serialization from the Tas...

2017-02-27 Thread witgo
Github user witgo commented on the issue: https://github.com/apache/spark/pull/15505 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #16990: [SPARK-19660][CORE][SQL] Replace the configuratio...

2017-02-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16990#discussion_r103180859 --- Diff: sql/hive/src/test/resources/ql/src/test/queries/clientpositive/smb_mapjoin_25.q --- @@ -19,7 +19,7 @@ select * from (select a.key from

[GitHub] spark issue #17083: [SPARK-19750][UI][branch-2.1] Fix redirect issue from ht...

2017-02-27 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17083 Was this fixed otherwise in master, or did some other change make it obsolete? just trying to link this to whatever reason it's only a problem in 2.1, for the record. --- If your project is set up

[GitHub] spark pull request #14731: [SPARK-17159] [streaming]: optimise check for new...

2017-02-27 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/14731#discussion_r103183646 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -140,7 +137,7 @@ class FileInputDStream[K, V, F

[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser

2017-02-27 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/17080 LGTM. Verified option name in `org.apache.hadoop.fs.s3a.Constants` file; env var name in `com.amazonaws.SDKGlobalConfiguration' --- If your project is set up for it, you can reply to

[GitHub] spark issue #17039: [SPARK-19710][SQL][TESTS] Fix ordering of rows in query ...

2017-02-27 Thread robbinspg
Github user robbinspg commented on the issue: https://github.com/apache/spark/pull/17039 Jenkins retest please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #16990: [SPARK-19660][CORE][SQL] Replace the configuratio...

2017-02-27 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/16990#discussion_r103200223 --- Diff: python/pyspark/tests.py --- @@ -1515,12 +1515,12 @@ def test_oldhadoop(self): conf = {

[GitHub] spark pull request #16990: [SPARK-19660][CORE][SQL] Replace the configuratio...

2017-02-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/16990#discussion_r103179898 --- Diff: python/pyspark/tests.py --- @@ -1515,12 +1515,12 @@ def test_oldhadoop(self): conf = {

[GitHub] spark pull request #14731: [SPARK-17159] [streaming]: optimise check for new...

2017-02-27 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/14731#discussion_r103184528 --- Diff: docs/streaming-programming-guide.md --- @@ -615,35 +615,114 @@ which creates a DStream from text data received over a TCP socket

[GitHub] spark issue #16867: [WIP][SPARK-16929] Improve performance when check specul...

2017-02-27 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/16867 @squito Thanks a lot for your comments : ) >When check speculatable tasks in TaskSetManager, current code scan all task infos and sort durations of successful tasks in O(NlogN) time

[GitHub] spark issue #17071: [SPARK-15615][SQL][BUILD][FOLLOW-UP] Replace deprecated ...

2017-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17071 Sure, sounds better and I can't find a reason to not follow. Let me maybe add single small Java one somewhere because the deprecated Java one calls the deprecated Scala one. --- If your

[GitHub] spark issue #17059: [SPARK-19733][ML]Removed unnecessary castings and refact...

2017-02-27 Thread datumbox
Github user datumbox commented on the issue: https://github.com/apache/spark/pull/17059 Ignore my comment about duplicate code. It can be written to avoid it. I will investigate handling the SQL decimal types as you recommended and I will update the code tonight. --- If your

[GitHub] spark pull request #16990: [SPARK-19660][CORE][SQL] Replace the configuratio...

2017-02-27 Thread jcamachor
Github user jcamachor commented on a diff in the pull request: https://github.com/apache/spark/pull/16990#discussion_r103184073 --- Diff: sql/hive/src/test/resources/ql/src/test/queries/clientpositive/smb_mapjoin_25.q --- @@ -19,7 +19,7 @@ select * from (select a.key from

[GitHub] spark pull request #17076: [SPARK-19745][ML] SVCAggregator captures coeffici...

2017-02-27 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/17076#discussion_r103187723 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala --- @@ -440,19 +440,9 @@ private class LinearSVCAggregator(

[GitHub] spark issue #17083: [SPARK-19750][UI][branch-2.1] Fix redirect issue from ht...

2017-02-27 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17083 Not sure why Jenkins test cannot be started automatically. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #17090: [Spark-19535][ML] RecommendForAllUsers RecommendForAllIt...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17090 **[Test build #73543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73543/testReport)** for PR 17090 at commit

[GitHub] spark issue #16959: [SPARK-19631][CORE] OutputCommitCoordinator should not a...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16959 **[Test build #73544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73544/testReport)** for PR 16959 at commit

[GitHub] spark issue #17052: [SPARK-19690][SS] Join a streaming DataFrame with a batc...

2017-02-27 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/17052 working on unit test failure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17012 **[Test build #73548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73548/testReport)** for PR 17012 at commit

[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17012 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73548/ Test FAILed. ---

[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17012 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17082: [SPARK-19749][SS] Name socket source with a meaningful n...

2017-02-27 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17082 Thanks! LGTM. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16774: [SPARK-19357][ML] Adding parallel model evaluation in ML...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16774 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16774: [SPARK-19357][ML] Adding parallel model evaluation in ML...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16774 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73545/ Test PASSed. ---

[GitHub] spark pull request #17093: [SPARK-19761][SQL]create InMemoryFileIndex with a...

2017-02-27 Thread windpiger
GitHub user windpiger opened a pull request: https://github.com/apache/spark/pull/17093 [SPARK-19761][SQL]create InMemoryFileIndex with an empty rootPaths when set PARALLEL_PARTITION_DISCOVERY_THRESHOLD to zero failed ## What changes were proposed in this pull request? If

[GitHub] spark issue #16989: [WIP][SPARK-19659] Fetch big blocks to disk when shuffle...

2017-02-27 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/16989 @squito I've uploaded a design doc to jira, please take a look when you have time :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #17092: [SPARK-18450][ML] Scala API Change for LSH AND-amplifica...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17092 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17047: [SPARK-19720][SPARK SUBMIT] Redact sensitive info...

2017-02-27 Thread markgrover
Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/17047#discussion_r103367049 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -2574,13 +2575,30 @@ private[spark] object Utils extends Logging {

[GitHub] spark issue #17092: [SPARK-18450][ML] Scala API Change for LSH AND-amplifica...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17092 **[Test build #73550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73550/testReport)** for PR 17092 at commit

[GitHub] spark issue #17092: [SPARK-18450][ML] Scala API Change for LSH AND-amplifica...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17092 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73550/ Test PASSed. ---

[GitHub] spark issue #17079: [SPARK-19748][SQL]refresh function has a wrong order to ...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17079 **[Test build #73546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73546/testReport)** for PR 17079 at commit

[GitHub] spark issue #17094: [SPARK-19762][ML][WIP] Hierarchy for consolidating ML ag...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17094 **[Test build #73557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73557/testReport)** for PR 17094 at commit

[GitHub] spark issue #17052: [SPARK-19690][SS] Join a streaming DataFrame with a batc...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17052 **[Test build #73558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73558/testReport)** for PR 17052 at commit

[GitHub] spark issue #17095: [SPARK-19763][SQL]qualified external datasource table lo...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17095 **[Test build #73556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73556/testReport)** for PR 17095 at commit

[GitHub] spark pull request #17079: [SPARK-19748][SQL]refresh function has a wrong or...

2017-02-27 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17079#discussion_r103373646 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala --- @@ -178,6 +178,33 @@ class FileIndexSuite extends

[GitHub] spark issue #17079: [SPARK-19748][SQL]refresh function has a wrong order to ...

2017-02-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17079 LGTM except two minor comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17079: [SPARK-19748][SQL]refresh function has a wrong or...

2017-02-27 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17079#discussion_r103373620 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala --- @@ -178,6 +178,33 @@ class FileIndexSuite extends

[GitHub] spark issue #17059: [SPARK-19733][ML]Removed unnecessary castings and refact...

2017-02-27 Thread imatiach-msft
Github user imatiach-msft commented on the issue: https://github.com/apache/spark/pull/17059 @datumbox I like the changes, I just had a minor concern about the code where we call v.intValue and then compare this to v.doubleValue -- due to precision issues, I'm not sure if this is

[GitHub] spark issue #17015: [SPARK-19678][SQL] remove MetastoreRelation

2017-02-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17015 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #10307: [SPARK-12334][SQL][PYSPARK] Support read from multiple i...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/10307 **[Test build #73567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73567/testReport)** for PR 10307 at commit

[GitHub] spark pull request #17015: [SPARK-19678][SQL] remove MetastoreRelation

2017-02-27 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17015#discussion_r103387597 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeColumnCommand.scala --- @@ -40,38 +38,24 @@ case class

[GitHub] spark pull request #17079: [SPARK-19748][SQL]refresh function has a wrong or...

2017-02-27 Thread windpiger
Github user windpiger commented on a diff in the pull request: https://github.com/apache/spark/pull/17079#discussion_r103357633 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala --- @@ -178,6 +178,34 @@ class FileIndexSuite extends

[GitHub] spark pull request #17062: [SPARK-17495] [SQL] Support date, timestamp and i...

2017-02-27 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/17062#discussion_r103357272 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala --- @@ -169,6 +171,96 @@ class

[GitHub] spark pull request #17062: [SPARK-17495] [SQL] Support date, timestamp and i...

2017-02-27 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/17062#discussion_r103357588 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala --- @@ -169,6 +171,96 @@ class

[GitHub] spark pull request #17062: [SPARK-17495] [SQL] Support date, timestamp and i...

2017-02-27 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/17062#discussion_r103300592 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala --- @@ -169,6 +171,96 @@ class

[GitHub] spark pull request #16715: [Spark-18080][ML][PYTHON] Python API & Examples f...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16715#discussion_r103357342 --- Diff: python/pyspark/ml/feature.py --- @@ -120,6 +122,196 @@ def getThreshold(self): return self.getOrDefault(self.threshold)

[GitHub] spark pull request #17062: [SPARK-17495] [SQL] Support date, timestamp and i...

2017-02-27 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/17062#discussion_r103281696 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala --- @@ -169,6 +171,96 @@ class

[GitHub] spark pull request #17062: [SPARK-17495] [SQL] Support date, timestamp and i...

2017-02-27 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/17062#discussion_r103300013 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala --- @@ -169,6 +171,96 @@ class

[GitHub] spark pull request #17062: [SPARK-17495] [SQL] Support date, timestamp and i...

2017-02-27 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/17062#discussion_r103357472 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala --- @@ -169,6 +171,96 @@ class

[GitHub] spark pull request #17062: [SPARK-17495] [SQL] Support date, timestamp and i...

2017-02-27 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/17062#discussion_r103300293 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala --- @@ -169,6 +171,96 @@ class

[GitHub] spark issue #17062: [SPARK-17495] [SQL] Support date, timestamp and interval...

2017-02-27 Thread tejasapatil
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/17062 @gatorsmile : can you please review this PR ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #17012: [SPARK-19677][SS] Renaming a file atop an existin...

2017-02-27 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/17012#discussion_r103359940 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala --- @@ -274,7 +274,9 @@ private[state]

[GitHub] spark pull request #17082: [SPARK-19749][SS] Name socket source with a meani...

2017-02-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17082 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r103362319 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -123,19 +123,36 @@ case class Not(child:

[GitHub] spark issue #16917: [SPARK-19529][BRANCH-1.6] Backport PR #16866 to branch-1...

2017-02-27 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/16917 Let's use a meaningful title in future :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17093: [SPARK-19761][SQL]create InMemoryFileIndex with a...

2017-02-27 Thread windpiger
Github user windpiger commented on a diff in the pull request: https://github.com/apache/spark/pull/17093#discussion_r103377566 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala --- @@ -178,6 +179,12 @@ class FileIndexSuite extends

[GitHub] spark pull request #13036: [SPARK-15243][ML][SQL][PYSPARK] Param methods sho...

2017-02-27 Thread sethah
Github user sethah closed the pull request at: https://github.com/apache/spark/pull/13036 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #13036: [SPARK-15243][ML][SQL][PYSPARK] Param methods should use...

2017-02-27 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/13036 @holdenk please feel free to take this over. Can't find time to work on it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r103354299 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala --- @@ -40,19 +42,179 @@ abstract class

[GitHub] spark pull request #14273: [SPARK-9140] [ML] Replace TimeTracker by MultiSto...

2017-02-27 Thread MechCoder
Github user MechCoder closed the pull request at: https://github.com/apache/spark/pull/14273 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #16959: [SPARK-19631][CORE] OutputCommitCoordinator shoul...

2017-02-27 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16959#discussion_r103354382 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -111,13 +115,13 @@ private[spark] class

[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...

2017-02-27 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17012 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark pull request #17092: [SPARK-18450][ML] Scala API Change for LSH AND-am...

2017-02-27 Thread Yunni
GitHub user Yunni opened a pull request: https://github.com/apache/spark/pull/17092 [SPARK-18450][ML] Scala API Change for LSH AND-amplification ## What changes were proposed in this pull request? Implemented a new Param numHashFunctions as the dimension of AND-amplification

[GitHub] spark pull request #17012: [SPARK-19677][SS] Renaming a file atop an existin...

2017-02-27 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/17012#discussion_r103361529 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala --- @@ -682,6 +684,21 @@ private[state] object

[GitHub] spark pull request #16715: [Spark-18080][ML][PYTHON] Python API & Examples f...

2017-02-27 Thread Yunni
Github user Yunni commented on a diff in the pull request: https://github.com/apache/spark/pull/16715#discussion_r103361528 --- Diff: python/pyspark/ml/feature.py --- @@ -120,6 +122,196 @@ def getThreshold(self): return self.getOrDefault(self.threshold)

[GitHub] spark issue #17092: [SPARK-18450][ML] Scala API Change for LSH AND-amplifica...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17092 **[Test build #73550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73550/testReport)** for PR 17092 at commit

[GitHub] spark issue #17093: [SPARK-19761][SQL]create InMemoryFileIndex with an empty...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17093 **[Test build #73552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73552/testReport)** for PR 17093 at commit

[GitHub] spark issue #17093: [SPARK-19761][SQL]create InMemoryFileIndex with an empty...

2017-02-27 Thread windpiger
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/17093 cc @cloud-fan @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #17090: [Spark-19535][ML] RecommendForAllUsers RecommendF...

2017-02-27 Thread sueann
Github user sueann commented on a diff in the pull request: https://github.com/apache/spark/pull/17090#discussion_r103366357 --- Diff: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala --- @@ -285,6 +285,43 @@ class ALSModel private[ml] (

[GitHub] spark issue #17090: [Spark-19535][ML] RecommendForAllUsers RecommendForAllIt...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17090 **[Test build #73553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73553/testReport)** for PR 17090 at commit

[GitHub] spark pull request #17094: [SPARK-19762][ML] Hierarchy for consolidating ML ...

2017-02-27 Thread sethah
GitHub user sethah opened a pull request: https://github.com/apache/spark/pull/17094 [SPARK-19762][ML] Hierarchy for consolidating ML aggregator/loss code ## What changes were proposed in this pull request? JIRA:

[GitHub] spark issue #17088: [SPARK-19753][CORE] All shuffle files on a host should b...

2017-02-27 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17088 >> This is quite drastic for a fetch failure : spark already has mechanisms in place to detect executor/host failure - which take care of these failure modes. Unfortunately, mechanisms

[GitHub] spark issue #16959: [SPARK-19631][CORE] OutputCommitCoordinator should not a...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16959 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73544/ Test PASSed. ---

[GitHub] spark issue #16959: [SPARK-19631][CORE] OutputCommitCoordinator should not a...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16959 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17094: [SPARK-19762][ML][WIP] Hierarchy for consolidating ML ag...

2017-02-27 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/17094 Jenkins test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #17095: [SPARK-19763][SQL]qualified external datasource t...

2017-02-27 Thread windpiger
GitHub user windpiger opened a pull request: https://github.com/apache/spark/pull/17095 [SPARK-19763][SQL]qualified external datasource table location stored in catalog ## What changes were proposed in this pull request? If we create a external datasource table with a

[GitHub] spark pull request #17015: [SPARK-19678][SQL] remove MetastoreRelation

2017-02-27 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17015#discussion_r103383279 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -349,36 +350,41 @@ object CatalogTypes {

[GitHub] spark pull request #17015: [SPARK-19678][SQL] remove MetastoreRelation

2017-02-27 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17015#discussion_r103387529 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeColumnCommand.scala --- @@ -90,10 +74,10 @@ object AnalyzeColumnCommand

[GitHub] spark issue #17079: [SPARK-19748][SQL]refresh function has a wrong order to ...

2017-02-27 Thread windpiger
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/17079 there is no related test case for InMemoryFileIndex with FileStatusCache. When I do this [PR](https://github.com/apache/spark/pull/17081), and add a fileStatusCache in DataSource, I found this

[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17012 **[Test build #73548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73548/testReport)** for PR 17012 at commit

[GitHub] spark issue #17047: [SPARK-19720][SPARK SUBMIT] Redact sensitive information...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17047 **[Test build #73554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73554/testReport)** for PR 17047 at commit

[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...

2017-02-27 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/16819 @vanzin What do you think about current approach? I have tested on a same Spark hive-thriftserver, the `spark.dynamicAllocation.maxExecutors` wiil decrease if I kill 4 NodeManager: ```

[GitHub] spark issue #17094: [SPARK-19762][ML][WIP] Hierarchy for consolidating ML ag...

2017-02-27 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/17094 ping @MLnick @jkbradley --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #17093: [SPARK-19761][SQL]create InMemoryFileIndex with an empty...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17093 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17093: [SPARK-19761][SQL]create InMemoryFileIndex with an empty...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17093 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73552/ Test PASSed. ---

[GitHub] spark issue #17083: [SPARK-19750][UI][branch-2.1] Fix redirect issue from ht...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17083 **[Test build #73551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73551/testReport)** for PR 17083 at commit

[GitHub] spark issue #17052: [SPARK-19690][SS] Join a streaming DataFrame with a batc...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17052 **[Test build #73559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73559/testReport)** for PR 17052 at commit

[GitHub] spark issue #17052: [SPARK-19690][SS] Join a streaming DataFrame with a batc...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17052 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17052: [SPARK-19690][SS] Join a streaming DataFrame with a batc...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17052 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73559/ Test FAILed. ---

[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17067 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

  1   2   3   4   5   6   7   >