Repository: spark Updated Branches: refs/heads/branch-1.1 b3d763b0b -> 77116875f
[SQL] [SPARK-3236] Reading Parquet tables from Metastore mangles location Currently we do `relation.hiveQlTable.getDataLocation.getPath`, which returns the path-part of the URI (e.g., "s3n://my-bucket/my-path" => "/my-path"). We should do `relation.hiveQlTable.getDataLocation.toString` instead, as a URI's toString returns a faithful representation of the full URI, which can later be passed into a Hadoop Path. Author: Aaron Davidson <[email protected]> Closes #2150 from aarondav/parquet-location and squashes the following commits: 459f72c [Aaron Davidson] [SQL] [SPARK-3236] Reading Parquet tables from Metastore mangles location (cherry picked from commit cc275f4b7910f6d0ad266a43bac2fdae58e9739e) Signed-off-by: Michael Armbrust <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/77116875 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/77116875 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/77116875 Branch: refs/heads/branch-1.1 Commit: 77116875f4184e0a637d9d7fd5b1dfeaabe0c9d3 Parents: b3d763b Author: Aaron Davidson <[email protected]> Authored: Wed Aug 27 15:05:47 2014 -0700 Committer: Michael Armbrust <[email protected]> Committed: Wed Aug 27 15:06:04 2014 -0700 ---------------------------------------------------------------------- .../src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/77116875/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala ---------------------------------------------------------------------- diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala index 10fa831..47e24f0 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala @@ -135,7 +135,7 @@ private[hive] trait HiveStrategies { .fakeOutput(projectList.map(_.toAttribute)):: Nil } else { hiveContext - .parquetFile(relation.hiveQlTable.getDataLocation.getPath) + .parquetFile(relation.hiveQlTable.getDataLocation.toString) .lowerCase .where(unresolvedOtherPredicates) .select(unresolvedProjection:_*) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
