[
https://issues.apache.org/jira/browse/IMPALA-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498790#comment-17498790
]
ASF subversion and git services commented on IMPALA-10871:
----------------------------------------------------------
Commit 4186727fe6cc35a10a0565214d1c593b9006d862 in impala's branch
refs/heads/master from Fucun Chu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4186727 ]
IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2
Like IMPALA-8369, this patch adds a compatibility shim in fe so that
Impala can interoperate with Hive 3.1.2. we need adds a new
Metastoreshim class under compat-apache-hive-3 directory. These shim
classes implement method which are different in cdp-hive-3 vs
apache-hive-3 and are used by front end code. At the build time, based
on the environment variable IMPALA_HIVE_DIST_TYPE one of the two shims
is added to as source using the fe/pom.xml build plugin.
Some codes that directly use Hive 4 APIs need to be ignored in
compilation, eg. fe/src/main/java/org/apache/impala/catalog/metastore/.
Use Maven profile to ignore some codes, profile will automatically
activated based on the IMPALA_HIVE_DIST_TYPE.
Testing:
1. Code compiles and runs against both HMS-3 and ASF-HMS-3
2. Ran full-suite of tests against HMS-3
3. Running full-tests against ASF-HMS-3 will need more work
supporting Tez in the mini-cluster (for dataloading) and HMS
transaction support. This will be on-going effort and test failures
on ASF-Hive-3 will be fixed in additional sub-tasks.
Notes:
1. Patch uses a custom build of Apache Hive to be deployed in
mini-cluster. This build has the fixes for HIVE-21569, HIVE-20038.
This hack will be added to the build script in additional sub-tasks.
Change-Id: I9f08db5f6da735ac431819063060941f0941f606
Reviewed-on: http://gerrit.cloudera.org:8080/17774
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add MetastoreShim to support Apache Hive 3.x
> --------------------------------------------
>
> Key: IMPALA-10871
> URL: https://issues.apache.org/jira/browse/IMPALA-10871
> Project: IMPALA
> Issue Type: Task
> Reporter: Quanlong Huang
> Assignee: Fucun Chu
> Priority: Major
>
> Impala 3.x supports both Hive 2 and 3 by a MetastoreShim that is chose at
> compilation time:
> {code:bash}
> fe/src/compat-hive-2
> └── java/org/apache
> ├── hadoop/hive/common/ValidWriteIdList.java
> └── impala/compat/MetastoreShim.java
> fe/src/compat-hive-3
> └── java/org/apache/impala/compat
> ├── HiveMetadataFormatUtils.java
> └── MetastoreShim.java {code}
> In pom.xml:
> {code:xml}
> <plugin>
> <groupId>org.codehaus.mojo</groupId>
> <artifactId>build-helper-maven-plugin</artifactId>
> <version>1.5</version>
> <executions>
> <!-- Tell maven about our generated files -->
> <execution>
> <id>add-source</id>
> <phase>generate-sources</phase>
> <goals>
> <goal>add-source</goal>
> </goals>
> <configuration>
> <sources>
> ...
>
> <source>${project.basedir}/src/compat-hive-${hive.major.version}/java</source>
> </sources>
> </configuration>
> </execution>
> </executions>
> </plugin>
> {code}
> Now MetastoreShim evolves to only support CDP Hive which contains some
> breaking changes of Hive 4.x.
> To support Apache Hive 3.x, we need another MetastoreShim. Some codes that
> directly use Hive 4 APIs may need to be ignored in compilation, e.g.
> fe/src/main/java/org/apache/impala/catalog/metastore/*.java
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
