spark git commit: [SPARK-20857][SQL] Generic resolved hint node

2017-05-23 Thread rxin 
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 dbb068f4f -> d20c64695


[SPARK-20857][SQL] Generic resolved hint node

## What changes were proposed in this pull request?
This patch renames BroadcastHint to ResolvedHint (and Hint to UnresolvedHint) 
so the hint framework is more generic and would allow us to introduce other 
hint types in the future without introducing new hint nodes.

## How was this patch tested?
Updated test cases.

Author: Reynold Xin 

Closes #18072 from rxin/SPARK-20857.

(cherry picked from commit 0d589ba00b5d539fbfef5174221de046a70548cd)
Signed-off-by: Reynold Xin 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d20c6469
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d20c6469
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d20c6469

Branch: refs/heads/branch-2.2
Commit: d20c6469565c4f7687f9af14a6f12a775b0c6e62
Parents: dbb068f
Author: Reynold Xin 
Authored: Tue May 23 18:44:49 2017 +0200
Committer: Reynold Xin 
Committed: Tue May 23 18:45:08 2017 +0200

--
 .../spark/sql/catalyst/analysis/Analyzer.scala  |  2 +-
 .../sql/catalyst/analysis/CheckAnalysis.scala   |  2 +-
 .../sql/catalyst/analysis/ResolveHints.scala| 12 ++---
 .../sql/catalyst/optimizer/Optimizer.scala  |  2 +-
 .../sql/catalyst/optimizer/expressions.scala|  2 +-
 .../spark/sql/catalyst/parser/AstBuilder.scala  |  4 +-
 .../spark/sql/catalyst/planning/patterns.scala  |  4 +-
 .../sql/catalyst/plans/logical/Statistics.scala |  5 ++
 .../plans/logical/basicLogicalOperators.scala   | 22 +
 .../sql/catalyst/plans/logical/hints.scala  | 49 
 .../catalyst/analysis/ResolveHintsSuite.scala   | 41 
 .../catalyst/optimizer/ColumnPruningSuite.scala |  5 +-
 .../optimizer/FilterPushdownSuite.scala |  4 +-
 .../optimizer/JoinOptimizationSuite.scala   |  4 +-
 .../sql/catalyst/parser/PlanParserSuite.scala   | 15 +++---
 .../BasicStatsEstimationSuite.scala |  2 +-
 .../scala/org/apache/spark/sql/Dataset.scala|  2 +-
 .../spark/sql/execution/SparkStrategies.scala   |  2 +-
 .../scala/org/apache/spark/sql/functions.scala  |  5 +-
 .../execution/joins/BroadcastJoinSuite.scala| 14 +++---
 20 files changed, 118 insertions(+), 80 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/d20c6469/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
--
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index 5be67ac..9979642 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -1311,7 +1311,7 @@ class Analyzer(
 
 // Category 1:
 // BroadcastHint, Distinct, LeafNode, Repartition, and SubqueryAlias
-case _: BroadcastHint | _: Distinct | _: LeafNode | _: Repartition | 
_: SubqueryAlias =>
+case _: ResolvedHint | _: Distinct | _: LeafNode | _: Repartition | _: 
SubqueryAlias =>
 
 // Category 2:
 // These operators can be anywhere in a correlated subquery.

http://git-wip-us.apache.org/repos/asf/spark/blob/d20c6469/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
--
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
index ea4560a..2e3ac3e 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
@@ -399,7 +399,7 @@ trait CheckAnalysis extends PredicateHelper {
  |in operator ${operator.simpleString}
""".stripMargin)
 
-  case _: Hint =>
+  case _: UnresolvedHint =>
 throw new IllegalStateException(
   "Internal error: logical hint operator should have been removed 
during analysis")
 

http://git-wip-us.apache.org/repos/asf/spark/blob/d20c6469/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala
--
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala
index df688fa..9dfd84c 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/

spark git commit: [SPARK-20857][SQL] Generic resolved hint node

2017-05-23 Thread rxin 
Repository: spark
Updated Branches:
  refs/heads/master ad09e4ca0 -> 0d589ba00


[SPARK-20857][SQL] Generic resolved hint node

## What changes were proposed in this pull request?
This patch renames BroadcastHint to ResolvedHint (and Hint to UnresolvedHint) 
so the hint framework is more generic and would allow us to introduce other 
hint types in the future without introducing new hint nodes.

## How was this patch tested?
Updated test cases.

Author: Reynold Xin 

Closes #18072 from rxin/SPARK-20857.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0d589ba0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0d589ba0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0d589ba0

Branch: refs/heads/master
Commit: 0d589ba00b5d539fbfef5174221de046a70548cd
Parents: ad09e4c
Author: Reynold Xin 
Authored: Tue May 23 18:44:49 2017 +0200
Committer: Reynold Xin 
Committed: Tue May 23 18:44:49 2017 +0200

--
 .../spark/sql/catalyst/analysis/Analyzer.scala  |  2 +-
 .../sql/catalyst/analysis/CheckAnalysis.scala   |  2 +-
 .../sql/catalyst/analysis/ResolveHints.scala| 12 ++---
 .../sql/catalyst/optimizer/Optimizer.scala  |  2 +-
 .../sql/catalyst/optimizer/expressions.scala|  2 +-
 .../spark/sql/catalyst/parser/AstBuilder.scala  |  4 +-
 .../spark/sql/catalyst/planning/patterns.scala  |  4 +-
 .../sql/catalyst/plans/logical/Statistics.scala |  5 ++
 .../plans/logical/basicLogicalOperators.scala   | 22 +
 .../sql/catalyst/plans/logical/hints.scala  | 49 
 .../catalyst/analysis/ResolveHintsSuite.scala   | 41 
 .../catalyst/optimizer/ColumnPruningSuite.scala |  5 +-
 .../optimizer/FilterPushdownSuite.scala |  4 +-
 .../optimizer/JoinOptimizationSuite.scala   |  4 +-
 .../sql/catalyst/parser/PlanParserSuite.scala   | 15 +++---
 .../BasicStatsEstimationSuite.scala |  2 +-
 .../scala/org/apache/spark/sql/Dataset.scala|  2 +-
 .../spark/sql/execution/SparkStrategies.scala   |  2 +-
 .../scala/org/apache/spark/sql/functions.scala  |  5 +-
 .../execution/joins/BroadcastJoinSuite.scala| 14 +++---
 20 files changed, 118 insertions(+), 80 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/0d589ba0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
--
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index d58b8ac..d130962 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -1336,7 +1336,7 @@ class Analyzer(
 
 // Category 1:
 // BroadcastHint, Distinct, LeafNode, Repartition, and SubqueryAlias
-case _: BroadcastHint | _: Distinct | _: LeafNode | _: Repartition | 
_: SubqueryAlias =>
+case _: ResolvedHint | _: Distinct | _: LeafNode | _: Repartition | _: 
SubqueryAlias =>
 
 // Category 2:
 // These operators can be anywhere in a correlated subquery.

http://git-wip-us.apache.org/repos/asf/spark/blob/0d589ba0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
--
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
index ea4560a..2e3ac3e 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
@@ -399,7 +399,7 @@ trait CheckAnalysis extends PredicateHelper {
  |in operator ${operator.simpleString}
""".stripMargin)
 
-  case _: Hint =>
+  case _: UnresolvedHint =>
 throw new IllegalStateException(
   "Internal error: logical hint operator should have been removed 
during analysis")
 

http://git-wip-us.apache.org/repos/asf/spark/blob/0d589ba0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala
--
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala
index df688fa..9dfd84c 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/cat