spark git commit: [SPARK-18726][SQL] resolveRelation for FileFormat DataSource don't need to listFiles twice

2017-03-02 Thread wenchen
Repository: spark Updated Branches: refs/heads/master e24f21b5f -> 982f3223b [SPARK-18726][SQL] resolveRelation for FileFormat DataSource don't need to listFiles twice ## What changes were proposed in this pull request? Currently when we resolveRelation for a `FileFormat DataSource` without

spark git commit: [SPARK-19779][SS] Delete needless tmp file after restart structured streaming job

2017-03-02 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.1 3a7591ad5 -> 1237aaea2 [SPARK-19779][SS] Delete needless tmp file after restart structured streaming job ## What changes were proposed in this pull request? [SPARK-19779](https://issues.apache.org/jira/browse/SPARK-19779) The PR

spark git commit: [SPARK-19779][SS] Delete needless tmp file after restart structured streaming job

2017-03-02 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.0 491b47a16 -> 73801880f [SPARK-19779][SS] Delete needless tmp file after restart structured streaming job ## What changes were proposed in this pull request? [SPARK-19779](https://issues.apache.org/jira/browse/SPARK-19779) The PR

spark git commit: [SPARK-19602][SQL][TESTS] Add tests for qualified column names

2017-03-02 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 93ae176e8 -> f37bb1430 [SPARK-19602][SQL][TESTS] Add tests for qualified column names ## What changes were proposed in this pull request? - Add tests covering different scenarios with qualified column names - Please see Section 2 in the

spark git commit: [SPARK-19779][SS] Delete needless tmp file after restart structured streaming job

2017-03-02 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master f37bb1430 -> e24f21b5f [SPARK-19779][SS] Delete needless tmp file after restart structured streaming job ## What changes were proposed in this pull request? [SPARK-19779](https://issues.apache.org/jira/browse/SPARK-19779) The PR

spark git commit: [SPARK-19745][ML] SVCAggregator captures coefficients in its closure

2017-03-02 Thread yliang
Repository: spark Updated Branches: refs/heads/master 8417a7ae6 -> 93ae176e8 [SPARK-19745][ML] SVCAggregator captures coefficients in its closure ## What changes were proposed in this pull request? JIRA: [SPARK-19745](https://issues.apache.org/jira/browse/SPARK-19745) Reorganize

spark git commit: [SPARK-19750][UI][BRANCH-2.1] Fix redirect issue from http to https

2017-03-02 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.0 e30fe1c6a -> 491b47a16 [SPARK-19750][UI][BRANCH-2.1] Fix redirect issue from http to https ## What changes were proposed in this pull request? If spark ui port (4040) is not set, it will choose port number 0, this will make https

spark git commit: [SPARK-19750][UI][BRANCH-2.1] Fix redirect issue from http to https

2017-03-02 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.1 27347b5f2 -> 3a7591ad5 [SPARK-19750][UI][BRANCH-2.1] Fix redirect issue from http to https ## What changes were proposed in this pull request? If spark ui port (4040) is not set, it will choose port number 0, this will make https

spark git commit: [SPARK-19276][CORE] Fetch Failure handling robust to user error handling

2017-03-02 Thread kayousterhout
Repository: spark Updated Branches: refs/heads/master 433d9eb61 -> 8417a7ae6 [SPARK-19276][CORE] Fetch Failure handling robust to user error handling ## What changes were proposed in this pull request? Fault-tolerance in spark requires special handling of shuffle fetch failures. The

spark git commit: [SPARK-19631][CORE] OutputCommitCoordinator should not allow commits for already failed tasks

2017-03-02 Thread kayousterhout
Repository: spark Updated Branches: refs/heads/master 5ae3516bf -> 433d9eb61 [SPARK-19631][CORE] OutputCommitCoordinator should not allow commits for already failed tasks ## What changes were proposed in this pull request? Previously it was possible for there to be a race between a task

spark git commit: [SPARK-19720][CORE] Redact sensitive information from SparkSubmit console

2017-03-02 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 9cca3dbf4 -> 5ae3516bf [SPARK-19720][CORE] Redact sensitive information from SparkSubmit console ## What changes were proposed in this pull request? This change redacts senstive information (based on `spark.redaction.regex` property) from

spark git commit: [SPARK-19345][ML][DOC] Add doc for "coldStartStrategy" usage in ALS

2017-03-02 Thread mlnick
Repository: spark Updated Branches: refs/heads/master 50c08e82f -> 9cca3dbf4 [SPARK-19345][ML][DOC] Add doc for "coldStartStrategy" usage in ALS [SPARK-14489](https://issues.apache.org/jira/browse/SPARK-14489) added the ability to skip `NaN` predictions during `ALSModel.transform`. This PR

spark git commit: [SPARK-19766][SQL][BRANCH-2.0] Constant alias columns in INNER JOIN should not be folded by FoldablePropagation rule

2017-03-02 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 c9c45d97b -> e30fe1c6a [SPARK-19766][SQL][BRANCH-2.0] Constant alias columns in INNER JOIN should not be folded by FoldablePropagation rule This PR fix for branch-2.0 Refer #17099 gatorsmile Author: Stan Zhai

spark git commit: [SPARK-19704][ML] AFTSurvivalRegression should support numeric censorCol

2017-03-02 Thread mlnick
Repository: spark Updated Branches: refs/heads/master 625cfe09e -> 50c08e82f [SPARK-19704][ML] AFTSurvivalRegression should support numeric censorCol ## What changes were proposed in this pull request? make `AFTSurvivalRegression` support numeric censorCol ## How was this patch tested?

spark git commit: [SPARK-19733][ML] Removed unnecessary castings and refactored checked casts in ALS.

2017-03-02 Thread mlnick
Repository: spark Updated Branches: refs/heads/master 8d6ef895e -> 625cfe09e [SPARK-19733][ML] Removed unnecessary castings and refactored checked casts in ALS. ## What changes were proposed in this pull request? The original ALS was performing unnecessary casting to the user and item ids

spark git commit: [SPARK-18352][DOCS] wholeFile JSON update doc and programming guide

2017-03-02 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master d2a879762 -> 8d6ef895e [SPARK-18352][DOCS] wholeFile JSON update doc and programming guide ## What changes were proposed in this pull request? Update doc for R, programming guide. Clarify default behavior for all languages. ## How was