GitHub user jagadeesanas2 opened a pull request:
https://github.com/apache/spark/pull/15728
[SPARK-18133] [branch-2.0] [Examples] [ML] [Python ML Pipeline Example has
syntax errors]
## What changes were proposed in this pull request?
[Fix] [branch-2.0] In Python 3, there is only one integer type (i.e., int),
which mostly behaves like the long type in Python 2. Since Python 3 won't
accept "L", so removed "L" in all examples.
## How was this patch tested?
Unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ibmsoe/spark SPARK-18133_2.0
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15728.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15728
commit 191d99692dc4315c371b566e3a9c5b114876ee49
Author: Wenchen Fan
Date: 2016-09-01T00:54:59Z
[SPARK-17180][SPARK-17309][SPARK-17323][SQL][2.0] create AlterViewAsCommand
to handle ALTER VIEW AS
## What changes were proposed in this pull request?
Currently we use `CreateViewCommand` to implement ALTER VIEW AS, which has
3 bugs:
1. SPARK-17180: ALTER VIEW AS should alter temp view if view name has no
database part and temp view exists
2. SPARK-17309: ALTER VIEW AS should issue exception if view does not exist.
3. SPARK-17323: ALTER VIEW AS should keep the previous table properties,
comment, create_time, etc.
The root cause is, ALTER VIEW AS is quite different from CREATE VIEW, we
need different code path to handle them. However, in `CreateViewCommand`, there
is no way to distinguish ALTER VIEW AS and CREATE VIEW, we have to introduce
extra flag. But instead of doing this, I think a more natural way is to
separate the ALTER VIEW AS logic into a new command.
backport https://github.com/apache/spark/pull/14874 to 2.0
## How was this patch tested?
new tests in SQLViewSuite
Author: Wenchen Fan
Closes #14893 from cloud-fan/minor4.
commit 8711b451d727074173748418a47cec210f84f2f7
Author: Junyang Qian
Date: 2016-09-01T04:28:53Z
[SPARKR][MINOR] Fix windowPartitionBy example
## What changes were proposed in this pull request?
The usage in the original example is incorrect. This PR fixes it.
## How was this patch tested?
Manual test.
Author: Junyang Qian
Closes #14903 from junyangq/SPARKR-FixWindowPartitionByDoc.
(cherry picked from commit d008638fbedc857c1adc1dff399d427b8bae848e)
Signed-off-by: Shivaram Venkataraman
commit 6281b74b6965ffcd0600844cea168cbe71ca8248
Author: Shixiong Zhu
Date: 2016-09-01T06:25:20Z
[SPARK-17318][TESTS] Fix ReplSuite replicating blocks of object with class
defined in repl again
## What changes were proposed in this pull request?
After digging into the logs, I noticed the failure is because in this test,
it starts a local cluster with 2 executors. However, when SparkContext is
created, executors may be still not up. When one of the executor is not up
during running the job, the blocks won't be replicated.
This PR just adds a wait loop before running the job to fix the flaky test.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu
Closes #14905 from zsxwing/SPARK-17318-2.
(cherry picked from commit 21c0a4fe9d8e21819ba96e7dc2b1f2999d3299ae)
Signed-off-by: Shixiong Zhu
commit 13bacd7308c42c92f42fbc3ffbee9a13282668a9
Author: Tejas Patil
Date: 2016-09-01T16:49:43Z
[SPARK-17271][SQL] Planner adds un-necessary Sort even if child ordeâ¦
## What changes were proposed in this pull request?
Ports https://github.com/apache/spark/pull/14841 and
https://github.com/apache/spark/pull/14910 from `master` to `branch-2.0`
Jira : https://issues.apache.org/jira/browse/SPARK-17271
Planner is adding un-needed SORT operation due to bug in the way comparison
for `SortOrder` is done at
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala#L253
`SortOrder` needs to be compared semantically because `Expression` within
two `SortOrder` can be "semantically equal" but not literally equal objects.
eg. In case of `sql("SELECT * FROM table1 a JOIN table2 b ON
a.col1=b.col1")`
Expression in required SortOrder:
```
AttributeReference(
name = "col1",
dataType = LongType,
nullable = false
) (exprId = exprId,
qualifier = Some("a")
)
```
Expression in child SortOrder:
```
AttributeReference(
name = "col1",
dataType = LongType,
nullable = false
