[GitHub] spark issue #4066: [SPARK-4879] Use driver to coordinate Hadoop output commi...

2017-01-04 Thread matrixlibing
Github user matrixlibing commented on the issue: https://github.com/apache/spark/pull/4066 SPARK-4879 also happened when use saveAsNewAPIHadoopFile. Why does not support the saveAsNewAPIHadoopFile function? --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #14725: [SPARK-17161] [PYSPARK][ML] Add PySpark-ML JavaWrapper c...

2017-01-04 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14725 Maybe it could be cleared up a bit with a good docstring? Although if the result is too confusing to be used then it's probably not worth doing. --- If your project is set up for it, you can reply

[GitHub] spark pull request #15211: [SPARK-14709][ML] spark.ml API for linear SVM

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15211#discussion_r94723792 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala --- @@ -0,0 +1,554 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16296 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16296 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70904/ Test FAILed. ---

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16296 **[Test build #70904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70904/testReport)** for PR 16296 at commit

[GitHub] spark issue #12135: [SPARK-14352][SQL] approxQuantile should support multi c...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12135 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70902/ Test PASSed. ---

[GitHub] spark issue #12135: [SPARK-14352][SQL] approxQuantile should support multi c...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12135 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #12135: [SPARK-14352][SQL] approxQuantile should support multi c...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12135 **[Test build #70902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70902/testReport)** for PR 12135 at commit

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16432 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70910/ Test PASSed. ---

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16432 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16432 **[Test build #70910 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70910/testReport)** for PR 16432 at commit

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16474 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16474 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70903/ Test PASSed. ---

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16474 **[Test build #70903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70903/testReport)** for PR 16474 at commit

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16296 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70905/ Test FAILed. ---

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16296 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16296 **[Test build #70905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70905/testReport)** for PR 16296 at commit

[GitHub] spark pull request #16464: [SPARK-19066][SparkR]:SparkR LDA doesn't set opti...

2017-01-04 Thread wangmiao1981
Github user wangmiao1981 commented on a diff in the pull request: https://github.com/apache/spark/pull/16464#discussion_r94721880 --- Diff: mllib/src/main/scala/org/apache/spark/ml/r/LDAWrapper.scala --- @@ -172,6 +187,8 @@ private[r] object LDAWrapper extends

[GitHub] spark pull request #16417: [SPARK-19014][SQL] support complex aggregate buff...

2017-01-04 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16417#discussion_r94721694 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala --- @@ -92,46 +89,53 @@ object

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 Weird... Not sure why the build failed. The build works in my local environment. cc @srowen @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request #16417: [SPARK-19014][SQL] support complex aggregate buff...

2017-01-04 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16417#discussion_r94721395 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -327,24 +359,28 @@ class CodegenContext {

[GitHub] spark issue #16347: [SPARK-18934][SQL] Writing to dynamic partitions does no...

2017-01-04 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16347 Maybe we should make DataFrameWriter.sortBy work here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16432 **[Test build #70910 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70910/testReport)** for PR 16432 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70908/ Test FAILed. ---

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70908/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #16417: [SPARK-19014][SQL] support complex aggregate buffer in H...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16417 **[Test build #70909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70909/testReport)** for PR 16417 at commit

[GitHub] spark pull request #16417: [SPARK-19014][SQL] support complex aggregate buff...

2017-01-04 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16417#discussion_r94719867 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java --- @@ -303,6 +331,15 @@ public void setDecimal(int ordinal,

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16417: [SPARK-19014][SQL] support complex aggregate buffer in H...

2017-01-04 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16417 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #16417: [SPARK-19014][SQL] support complex aggregate buffer in H...

2017-01-04 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16417 Jenkins looks unstable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70908/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory ...

2017-01-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15819#discussion_r94719375 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala --- @@ -216,5 +219,37 @@ class VersionsSuite extends SparkFunSuite

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70907/ Test FAILed. ---

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70907 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70907/consoleFull)** for PR 15819 at commit

[GitHub] spark pull request #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory ...

2017-01-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15819#discussion_r94719123 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -54,6 +63,63 @@ case class InsertIntoHiveTable(

[GitHub] spark pull request #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory ...

2017-01-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15819#discussion_r94719028 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala --- @@ -216,5 +219,37 @@ class VersionsSuite extends SparkFunSuite

[GitHub] spark issue #16470: [SPARK-19033][Core] Add admin acls for history server

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16470 **[Test build #70906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70906/testReport)** for PR 16470 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70907/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16470: [SPARK-19033][Core] Add admin acls for history server

2017-01-04 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/16470 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

2017-01-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16337 I compared the results and confirmed the results are consistent. LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94718428 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala --- @@ -0,0 +1,232 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #16451: [SPARK-18922][SQL][CORE][STREAMING][TESTS] Fix all ident...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16451 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70899/ Test PASSed. ---

[GitHub] spark issue #16451: [SPARK-18922][SQL][CORE][STREAMING][TESTS] Fix all ident...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16451 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #16460: [SPARK-19058][SQL] fix partition related behavior...

2017-01-04 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16460 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #16460: [SPARK-19058][SQL] fix partition related behaviors with ...

2017-01-04 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16460 thanks for the view, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16451: [SPARK-18922][SQL][CORE][STREAMING][TESTS] Fix all ident...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16451 **[Test build #70899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70899/testReport)** for PR 16451 at commit

[GitHub] spark pull request #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN s...

2017-01-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16337#discussion_r94718082 --- Diff: sql/core/src/test/resources/sql-tests/results/subquery/in-subquery/in-group-by.sql.out --- @@ -0,0 +1,357 @@ +-- Automatically generated

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94717710 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala --- @@ -0,0 +1,232 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #16347: [SPARK-18934][SQL] Writing to dynamic partitions does no...

2017-01-04 Thread junegunn
Github user junegunn commented on the issue: https://github.com/apache/spark/pull/16347 @chpritchard-expedia The patch here fixes the problem. I don't think it's possible to workaround the issue by using Spark API in some different ways, because we can't completely avoid memory

[GitHub] spark issue #13252: [SPARK-15473][SQL] CSV data source writes header for emp...

2017-01-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/13252 Let me suggest a generalized way latter because it does not look a clean fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request #13252: [SPARK-15473][SQL] CSV data source writes header ...

2017-01-04 Thread HyukjinKwon
Github user HyukjinKwon closed the pull request at: https://github.com/apache/spark/pull/13252 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94717542 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala --- @@ -0,0 +1,232 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94717473 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala --- @@ -0,0 +1,232 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #16470: [SPARK-19033][Core] Add admin acls for history server

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16470 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16470: [SPARK-19033][Core] Add admin acls for history server

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16470 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70900/ Test FAILed. ---

[GitHub] spark issue #16470: [SPARK-19033][Core] Add admin acls for history server

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16470 **[Test build #70900 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70900/testReport)** for PR 16470 at commit

[GitHub] spark issue #14284: [SPARK-16633] [SPARK-16642] [SPARK-16721] [SQL] Fixes th...

2017-01-04 Thread chengat1314
Github user chengat1314 commented on the issue: https://github.com/apache/spark/pull/14284 @hvanhovell Nice, thank you very much! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #14725: [SPARK-17161] [PYSPARK][ML] Add PySpark-ML JavaWrapper c...

2017-01-04 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/14725 Thanks @holdenk for taking a look! Yeah, I think you're right about the issues trying to infer a type. It would be nice if there was some easy way to specify a primitive type since that would

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94717245 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala --- @@ -0,0 +1,232 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #16435: [SPARK-19027][SQL] estimate size of object buffer...

2017-01-04 Thread cloud-fan
Github user cloud-fan closed the pull request at: https://github.com/apache/spark/pull/16435 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #16435: [SPARK-19027][SQL] estimate size of object buffer for ob...

2017-01-04 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16435 closing this as it's very hard to estimate the size and does not provide much benefit for end users. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #16460: [SPARK-19058][SQL] fix partition related behaviors with ...

2017-01-04 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/16460 looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94717055 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala --- @@ -0,0 +1,232 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15671 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15671 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70901/ Test PASSed. ---

[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15671 **[Test build #70901 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70901/testReport)** for PR 15671 at commit

[GitHub] spark pull request #16464: [SPARK-19066][SparkR]:SparkR LDA doesn't set opti...

2017-01-04 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16464#discussion_r94716683 --- Diff: mllib/src/main/scala/org/apache/spark/ml/r/LDAWrapper.scala --- @@ -172,6 +187,8 @@ private[r] object LDAWrapper extends

[GitHub] spark pull request #16460: [SPARK-19058][SQL] fix partition related behavior...

2017-01-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16460#discussion_r94716609 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala --- @@ -74,12 +69,30 @@ case class

[GitHub] spark issue #16460: [SPARK-19058][SQL] fix partition related behaviors with ...

2017-01-04 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16460 cc @ericl anymore comments on this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94716584 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala --- @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #16468: [SPARK-19074][SS][DOCS] Updated Structured Streaming Pro...

2017-01-04 Thread david-weiluo-ren
Github user david-weiluo-ren commented on the issue: https://github.com/apache/spark/pull/16468 @tdas It says “However, note that all of the operations applicable on static DataFrames/Datasets are not supported in streaming DataFrames/Datasets yet” in

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16296 **[Test build #70905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70905/testReport)** for PR 16296 at commit

[GitHub] spark issue #16422: [SPARK-17642] [SQL] support DESC EXTENDED/FORMATTED tabl...

2017-01-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16422 Column-level security can block users to access the specific columns, but this command `DESC EXTENDED/FORMATTED COLUMN` might not be part of the design/solution. --- If your project is set up

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94716372 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala --- @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94716281 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala --- @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #16296: [SPARK-18885][SQL] unify CREATE TABLE syntax for data so...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16296 **[Test build #70904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70904/testReport)** for PR 16296 at commit

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94716047 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala --- @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #16472: [SPARK-18877][SQL][BACKPORT-2.0] `CSVInferSchema.inferFi...

2017-01-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16472 Thanks! Merged to Spark 2.0. Could you please close it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94715820 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala --- @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #16451: [SPARK-18922][SQL][CORE][STREAMING][TESTS] Fix all ident...

2017-01-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16451 I just checked each except for the one below is passed in Windows via concatenated [full-log](https://gist.github.com/HyukjinKwon/2d199ac9156c380015ad5a71f77866be). It seems

[GitHub] spark pull request #15415: [SPARK-14501][ML] spark.ml API for FPGrowth

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15415#discussion_r94715503 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala --- @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #16308: [SPARK-18936][SQL] Infrastructure for session local time...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70896/ Test PASSed. ---

[GitHub] spark issue #16308: [SPARK-18936][SQL] Infrastructure for session local time...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16308: [SPARK-18936][SQL] Infrastructure for session local time...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16308 **[Test build #70896 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70896/testReport)** for PR 16308 at commit

[GitHub] spark pull request #16401: [SPARK-18998] [SQL] Add a cbo conf to switch betw...

2017-01-04 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/16401#discussion_r94714976 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala --- @@ -95,6 +96,29 @@ abstract class LogicalPlan extends

[GitHub] spark issue #12135: [SPARK-14352][SQL] approxQuantile should support multi c...

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on the issue: https://github.com/apache/spark/pull/12135 @jkbradley Updated. Thanks for reviewing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16474 **[Test build #70903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70903/testReport)** for PR 16474 at commit

[GitHub] spark pull request #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work f...

2017-01-04 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/16474 [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parquet ## What changes were proposed in this pull request? We have a config `spark.sql.files.ignoreCorruptFiles` which can be used to

[GitHub] spark issue #12135: [SPARK-14352][SQL] approxQuantile should support multi c...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12135 **[Test build #70902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70902/testReport)** for PR 12135 at commit

[GitHub] spark issue #16429: [SPARK-19019][PYTHON] Fix hijacked `collections.namedtup...

2017-01-04 Thread azmras
Github user azmras commented on the issue: https://github.com/apache/spark/pull/16429 @cxww107 Try to update both patched files in the following locations /usr/local/Cellar/apache-spark/2.1.0/libexec/python/pyspark

[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...

2017-01-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15671 **[Test build #70901 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70901/testReport)** for PR 15671 at commit

[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on the issue: https://github.com/apache/spark/pull/15671 @jkbradley Update according to your comments, including adding `quantileProbabilities` and `docConcentration`. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,L...

2017-01-04 Thread zhengruifeng
Github user zhengruifeng commented on a diff in the pull request: https://github.com/apache/spark/pull/15671#discussion_r94712844 --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala --- @@ -905,7 +911,10 @@ class LDA @Since("1.6.0") ( case m:

[GitHub] spark issue #16347: [SPARK-18934][SQL] Writing to dynamic partitions does no...

2017-01-04 Thread chpritchard-expedia
Github user chpritchard-expedia commented on the issue: https://github.com/apache/spark/pull/16347 @junegunn I ran into the same issue, using partitionBy; missed it completely during my testing. Would you share the workaround you used? I wasn't able to understand it from your Apache

[GitHub] spark issue #12775: [SPARK-14958][Core] Failed task not handled when there's...

2017-01-04 Thread lirui-intel
Github user lirui-intel commented on the issue: https://github.com/apache/spark/pull/12775 I think the failure is due to one more [skipped

[GitHub] spark issue #15314: [SPARK-17747][ML] WeightCol support non-double numeric d...

2017-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15314 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

  1   2   3   4   5   >