Repository: spark Updated Branches: refs/heads/master 088b11ec5 -> 52ae95257
[SPARK-9974] [BUILD] [SQL] Makes sure com.twitter:parquet-hadoop-bundle:1.6.0 is in SBT assembly jar PR #7967 enables Spark SQL to persist Parquet tables in Hive compatible format when possible. One of the consequence is that, we have to set input/output classes to `MapredParquetInputFormat`/`MapredParquetOutputFormat`, which rely on com.twitter:parquet-hadoop:1.6.0 bundled with Hive 1.2.1. When loading such a table in Spark SQL, `o.a.h.h.ql.metadata.Table` first loads these input/output format classes, and thus classes in com.twitter:parquet-hadoop:1.6.0. However, the scope of this dependency is defined as "runtime", and is not packaged into Spark assembly jar. This results in a `ClassNotFoundException`. This issue can be worked around by asking users to add parquet-hadoop 1.6.0 via the `--driver-class-path` option. However, considering Maven build is immune to this problem, I feel it can be confusing and inconvenient for users. So this PR fixes this issue by changing scope of parquet-hadoop 1.6.0 to "compile". Author: Cheng Lian <[email protected]> Closes #8198 from liancheng/spark-9974/bundle-parquet-1.6.0. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/52ae9525 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/52ae9525 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/52ae9525 Branch: refs/heads/master Commit: 52ae952574f5d641a398dd185e09e5a79318c8a9 Parents: 088b11e Author: Cheng Lian <[email protected]> Authored: Mon Aug 17 17:25:14 2015 -0700 Committer: Reynold Xin <[email protected]> Committed: Mon Aug 17 17:25:14 2015 -0700 ---------------------------------------------------------------------- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/52ae9525/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index cfd7d32..9bfca1c 100644 --- a/pom.xml +++ b/pom.xml @@ -1598,7 +1598,7 @@ <groupId>com.twitter</groupId> <artifactId>parquet-hadoop-bundle</artifactId> <version>${hive.parquet.version}</version> - <scope>runtime</scope> + <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.flume</groupId> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
