Repository: spark
Updated Branches:
refs/heads/master b50b34f56 -> cb324f611
[SPARK-17425][SQL] Override sameResult in HiveTableScanExec to make
ReuseExchange work in text format table
## What changes were proposed in this pull request?
The PR will override the `sameResult` in `HiveTableScanExec` to make
`ReuseExchange` work in text format table.
## How was this patch tested?
# SQL
```sql
SELECT * FROM src t1
JOIN src t2 ON t1.key = t2.key
JOIN src t3 ON t1.key = t3.key;
```
# Before
```
== Physical Plan ==
*BroadcastHashJoin [key#30], [key#34], Inner, BuildRight
:- *BroadcastHashJoin [key#30], [key#32], Inner, BuildRight
: :- *Filter isnotnull(key#30)
: : +- HiveTableScan [key#30, value#31], MetastoreRelation default, src
: +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int,
false] as bigint)))
: +- *Filter isnotnull(key#32)
: +- HiveTableScan [key#32, value#33], MetastoreRelation default, src
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false]
as bigint)))
+- *Filter isnotnull(key#34)
+- HiveTableScan [key#34, value#35], MetastoreRelation default, src
```
# After
```
== Physical Plan ==
*BroadcastHashJoin [key#2], [key#6], Inner, BuildRight
:- *BroadcastHashJoin [key#2], [key#4], Inner, BuildRight
: :- *Filter isnotnull(key#2)
: : +- HiveTableScan [key#2, value#3], MetastoreRelation default, src
: +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int,
false] as bigint)))
: +- *Filter isnotnull(key#4)
: +- HiveTableScan [key#4, value#5], MetastoreRelation default, src
+- ReusedExchange [key#6, value#7], BroadcastExchange
HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)))
```
cc: davies cloud-fan
Author: Yadong Qi <[email protected]>
Closes #14988 from watermen/SPARK-17425.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cb324f61
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cb324f61
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cb324f61
Branch: refs/heads/master
Commit: cb324f61150c962aeabf0a779f6a09797b3d5072
Parents: b50b34f
Author: Yadong Qi <[email protected]>
Authored: Thu Sep 22 13:04:42 2016 +0800
Committer: Wenchen Fan <[email protected]>
Committed: Thu Sep 22 13:04:42 2016 +0800
----------------------------------------------------------------------
.../spark/sql/hive/execution/HiveTableScanExec.scala | 15 +++++++++++++++
1 file changed, 15 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/cb324f61/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala
----------------------------------------------------------------------
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala
index a716a3e..231f204 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala
@@ -164,4 +164,19 @@ case class HiveTableScanExec(
}
override def output: Seq[Attribute] = attributes
+
+ override def sameResult(plan: SparkPlan): Boolean = plan match {
+ case other: HiveTableScanExec =>
+ val thisPredicates = partitionPruningPred.map(cleanExpression)
+ val otherPredicates = other.partitionPruningPred.map(cleanExpression)
+
+ val result = relation.sameResult(other.relation) &&
+ output.length == other.output.length &&
+ output.zip(other.output)
+ .forall(p => p._1.name == p._2.name && p._1.dataType ==
p._2.dataType) &&
+ thisPredicates.length == otherPredicates.length &&
+ thisPredicates.zip(otherPredicates).forall(p =>
p._1.semanticEquals(p._2))
+ result
+ case _ => false
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
