spark git commit: [SPARK-20815][SPARKR] NullPointerException in RPackageUtils#checkManifestForR

2017-05-22 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master a2460be9c -> 4dbb63f08 [SPARK-20815][SPARKR] NullPointerException in RPackageUtils#checkManifestForR ## What changes were proposed in this pull request? - Add a null check to RPackageUtils#checkManifestForR so that jars w/o manifests

spark git commit: [SPARK-20815][SPARKR] NullPointerException in RPackageUtils#checkManifestForR

2017-05-22 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.2 d8328d8d1 -> ddc199eef [SPARK-20815][SPARKR] NullPointerException in RPackageUtils#checkManifestForR ## What changes were proposed in this pull request? - Add a null check to RPackageUtils#checkManifestForR so that jars w/o manifests

spark git commit: [SPARK-20763][SQL][BACKPORT-2.1] The function of `month` and `day` return the value which is not we expected.

2017-05-22 Thread ueshin
Repository: spark Updated Branches: refs/heads/branch-2.1 f5ef07625 -> f4538c95f [SPARK-20763][SQL][BACKPORT-2.1] The function of `month` and `day` return the value which is not we expected. What changes were proposed in this pull request? This PR is to backport #17997 to Spark 2.1 when

spark git commit: [SPARK-17410][SPARK-17284] Move Hive-generated Stats Info to HiveClientImpl

2017-05-22 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 9b0910193 -> a2460be9c [SPARK-17410][SPARK-17284] Move Hive-generated Stats Info to HiveClientImpl ### What changes were proposed in this pull request? After we adding a new field `stats` into `CatalogTable`, we should not expose

spark git commit: [SPARK-20751][SQL][FOLLOWUP] Add cot test in MathExpressionsSuite

2017-05-22 Thread lixiao
Repository: spark Updated Branches: refs/heads/master df64fa79d -> 9b0910193 [SPARK-20751][SQL][FOLLOWUP] Add cot test in MathExpressionsSuite ## What changes were proposed in this pull request? Add cot test in MathExpressionsSuite as

spark git commit: [SPARK-20814][MESOS] Restore support for spark.executor.extraClassPath.

2017-05-22 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.2 2fd613875 -> d8328d8d1 [SPARK-20814][MESOS] Restore support for spark.executor.extraClassPath. Restore code that was removed as part of SPARK-17979, but instead of using the deprecated env variable name to propagate the class path, use

spark git commit: [SPARK-20814][MESOS] Restore support for spark.executor.extraClassPath.

2017-05-22 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 4be337583 -> df64fa79d [SPARK-20814][MESOS] Restore support for spark.executor.extraClassPath. Restore code that was removed as part of SPARK-17979, but instead of using the deprecated env variable name to propagate the class path, use a

spark git commit: [SPARK-15767][ML][SPARKR] Decision Tree wrapper in SparkR

2017-05-22 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master 363091100 -> 4be337583 [SPARK-15767][ML][SPARKR] Decision Tree wrapper in SparkR ## What changes were proposed in this pull request? support decision tree in R ## How was this patch tested? added tests Author: Zheng RuiFeng

spark git commit: [SPARK-20756][YARN] yarn-shuffle jar references unshaded guava

2017-05-22 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.0 9b145c664 -> 4dd34d004 [SPARK-20756][YARN] yarn-shuffle jar references unshaded guava and contains scala classes ## What changes were proposed in this pull request? This change ensures that all references to guava from within the yarn

spark git commit: [SPARK-20756][YARN] yarn-shuffle jar references unshaded guava

2017-05-22 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.1 c3a986b19 -> f5ef07625 [SPARK-20756][YARN] yarn-shuffle jar references unshaded guava and contains scala classes ## What changes were proposed in this pull request? This change ensures that all references to guava from within the yarn

spark git commit: [SPARK-20756][YARN] yarn-shuffle jar references unshaded guava

2017-05-22 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.2 a0bf5c47c -> 2fd613875 [SPARK-20756][YARN] yarn-shuffle jar references unshaded guava and contains scala classes ## What changes were proposed in this pull request? This change ensures that all references to guava from within the yarn

spark git commit: [SPARK-20756][YARN] yarn-shuffle jar references unshaded guava

2017-05-22 Thread vanzin
Repository: spark Updated Branches: refs/heads/master cfca01136 -> 363091100 [SPARK-20756][YARN] yarn-shuffle jar references unshaded guava and contains scala classes ## What changes were proposed in this pull request? This change ensures that all references to guava from within the yarn

spark git commit: [SPARK-20764][ML][PYSPARK] Fix visibility discrepancy with numInstances and degreesOfFreedom in LR and GLR - Python version

2017-05-22 Thread yliang
Repository: spark Updated Branches: refs/heads/branch-2.2 a57553279 -> a0bf5c47c [SPARK-20764][ML][PYSPARK] Fix visibility discrepancy with numInstances and degreesOfFreedom in LR and GLR - Python version ## What changes were proposed in this pull request? SPARK-20097 exposed

spark git commit: [SPARK-20764][ML][PYSPARK] Fix visibility discrepancy with numInstances and degreesOfFreedom in LR and GLR - Python version

2017-05-22 Thread yliang
Repository: spark Updated Branches: refs/heads/master f3ed62a38 -> cfca01136 [SPARK-20764][ML][PYSPARK] Fix visibility discrepancy with numInstances and degreesOfFreedom in LR and GLR - Python version ## What changes were proposed in this pull request? SPARK-20097 exposed degreesOfFreedom

spark git commit: [SPARK-20831][SQL] Fix INSERT OVERWRITE data source tables with IF NOT EXISTS

2017-05-22 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 81f63c892 -> a57553279 [SPARK-20831][SQL] Fix INSERT OVERWRITE data source tables with IF NOT EXISTS ### What changes were proposed in this pull request? Currently, we have a bug when we specify `IF NOT EXISTS` in `INSERT OVERWRITE`

spark git commit: [SPARK-20831][SQL] Fix INSERT OVERWRITE data source tables with IF NOT EXISTS

2017-05-22 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 2597674bc -> f3ed62a38 [SPARK-20831][SQL] Fix INSERT OVERWRITE data source tables with IF NOT EXISTS ### What changes were proposed in this pull request? Currently, we have a bug when we specify `IF NOT EXISTS` in `INSERT OVERWRITE` data

spark git commit: [SPARK-20801] Record accurate size of blocks in MapStatus when it's above threshold.

2017-05-22 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 c4b16dcca -> 81f63c892 [SPARK-20801] Record accurate size of blocks in MapStatus when it's above threshold. ## What changes were proposed in this pull request? Currently, when number of reduces is above 2000,

spark git commit: [SPARK-20801] Record accurate size of blocks in MapStatus when it's above threshold.

2017-05-22 Thread wenchen
Repository: spark Updated Branches: refs/heads/master aea73be1b -> 2597674bc [SPARK-20801] Record accurate size of blocks in MapStatus when it's above threshold. ## What changes were proposed in this pull request? Currently, when number of reduces is above 2000, HighlyCompressedMapStatus is

spark git commit: [SPARK-20813][WEB UI] Fixed Web UI executor page tab search by status not working

2017-05-22 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 50dba3053 -> c4b16dcca [SPARK-20813][WEB UI] Fixed Web UI executor page tab search by status not working ## What changes were proposed in this pull request? On status column of the table, I removed the condition that forced only the

spark git commit: [SPARK-20813][WEB UI] Fixed Web UI executor page tab search by status not working

2017-05-22 Thread srowen
Repository: spark Updated Branches: refs/heads/master f1ffc6e71 -> aea73be1b [SPARK-20813][WEB UI] Fixed Web UI executor page tab search by status not working ## What changes were proposed in this pull request? On status column of the table, I removed the condition that forced only the

spark git commit: [SPARK-20609][CORE] Run the SortShuffleSuite unit tests have residual spark_* system directory

2017-05-22 Thread srowen
Repository: spark Updated Branches: refs/heads/master 190d8b0b6 -> f1ffc6e71 [SPARK-20609][CORE] Run the SortShuffleSuite unit tests have residual spark_* system directory ## What changes were proposed in this pull request? This PR solution to run the SortShuffleSuite unit tests have

spark git commit: [SPARK-20591][WEB UI] Succeeded tasks num not equal in all jobs page and job detail page on spark web ui when speculative task(s) exist.

2017-05-22 Thread srowen
Repository: spark Updated Branches: refs/heads/master be846db48 -> 190d8b0b6 [SPARK-20591][WEB UI] Succeeded tasks num not equal in all jobs page and job detail page on spark web ui when speculative task(s) exist. ## What changes were proposed in this pull request? Modified succeeded num in

spark git commit: [SPARK-20506][DOCS] Add HTML links to highlight list in MLlib guide for 2.2

2017-05-22 Thread mlnick
Repository: spark Updated Branches: refs/heads/branch-2.2 af1ff8b00 -> 50dba3053 [SPARK-20506][DOCS] Add HTML links to highlight list in MLlib guide for 2.2 Quick follow up to #17996 - forgot to add the HTML links to the relevant sections of the guide in the highlights list. ## How was this

spark git commit: [SPARK-20506][DOCS] Add HTML links to highlight list in MLlib guide for 2.2

2017-05-22 Thread mlnick
Repository: spark Updated Branches: refs/heads/master 06dda1d58 -> be846db48 [SPARK-20506][DOCS] Add HTML links to highlight list in MLlib guide for 2.2 Quick follow up to #17996 - forgot to add the HTML links to the relevant sections of the guide in the highlights list. ## How was this

spark git commit: [SPARK-20687][MLLIB] mllib.Matrices.fromBreeze may crash when converting from Breeze sparse matrix

2017-05-22 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 e9804b3d4 -> c3a986b19 [SPARK-20687][MLLIB] mllib.Matrices.fromBreeze may crash when converting from Breeze sparse matrix ## What changes were proposed in this pull request? When two Breeze SparseMatrices are operated, the result

spark git commit: [SPARK-20687][MLLIB] mllib.Matrices.fromBreeze may crash when converting from Breeze sparse matrix

2017-05-22 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 41d8d2165 -> af1ff8b00 [SPARK-20687][MLLIB] mllib.Matrices.fromBreeze may crash when converting from Breeze sparse matrix ## What changes were proposed in this pull request? When two Breeze SparseMatrices are operated, the result

spark git commit: [SPARK-20687][MLLIB] mllib.Matrices.fromBreeze may crash when converting from Breeze sparse matrix

2017-05-22 Thread srowen
Repository: spark Updated Branches: refs/heads/master a2b3b6762 -> 06dda1d58 [SPARK-20687][MLLIB] mllib.Matrices.fromBreeze may crash when converting from Breeze sparse matrix ## What changes were proposed in this pull request? When two Breeze SparseMatrices are operated, the result matrix

spark git commit: [SPARK-19089][SQL] Add support for nested sequences

2017-05-22 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 cfd1bf0be -> 41d8d2165 [SPARK-19089][SQL] Add support for nested sequences ## What changes were proposed in this pull request? Replaced specific sequence encoders with generic sequence encoder to enable nesting of sequences. Does

spark git commit: [SPARK-19089][SQL] Add support for nested sequences

2017-05-22 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 833c8d415 -> a2b3b6762 [SPARK-19089][SQL] Add support for nested sequences ## What changes were proposed in this pull request? Replaced specific sequence encoders with generic sequence encoder to enable nesting of sequences. Does not

spark git commit: [SPARK-20770][SQL] Improve ColumnStats

2017-05-22 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 3c9eef35a -> 833c8d415 [SPARK-20770][SQL] Improve ColumnStats ## What changes were proposed in this pull request? This PR improves the implementation of `ColumnStats` by using the following appoaches. 1. Declare subclasses of