Hi Akila, I think getting Impala to compile against Hive 3.1 would be a modest amount of effort. Most of it revolves around the IMPALA_HIVE_VERSION, and it would require the Hive source (to get hive_metastore.thrift) and a maven repository with the Hive 3.1 artifacts (which I think would be available in the Apache maven repository).
So far, there has not been an effort to support vanilla versions of Hadoop/Hive/etc. Contributions are definitely welcome. I think it would be useful for it to be easy for Impala to be deployed against vanilla deployments, but it would also a very large amount of effort to maintain. Thanks, Joe On Wed, Dec 18, 2019 at 8:54 PM Akila Wajirasena <akila.wajiras...@gmail.com> wrote: > Thanks you Joe. I was able to start get it work by using the Hive version > that is downloaded when building Impala. > How ever my other questions are; > Is it possible the use the 3.1 Hive version to build Impala? > Also is there any plans to support vanilla version of Hive and Hadoop for > Impala? > I saw most of the dependencies are downloaded from Cloudera repos. > > Thanks & Regards, > Akila > > > On Thu, Dec 19, 2019 at 6:37 AM Joe McDonnell <joemcdonn...@cloudera.com> > wrote: > >> Hi Akila, >> >> USE_CDP_HIVE=true uses a Hive based off of Hive's master branch, which >> has changed quite a bit since Hive 3.1. create_table_req was introduced in >> HIVE-21838: >> >> https://github.com/apache/hive/commit/1fb0107ea522730196cd3bff7b1c4a7273f25291#diff-96182be0c29b7ceda2b60e931c70ecc4R1966 >> It looks like Hive's branch-3.1 does not have HIVE-21838, so it doesn't >> understand create_table_req. >> >> Thanks, >> Joe >> >> >> >> >> On Wed, Dec 18, 2019 at 3:47 AM Akila Wajirasena < >> akila.wajiras...@gmail.com> wrote: >> >>> Hi, >>> >>> I am trying to build impala with hive 3.1 support by enabling >>> USE_CDP_HIVE=true. >>> Build finishes with any issues and impala starts and connect to the hive >>> 3.1 standalone meta store without any issue. >>> >>> However when I try to create a table using impala shell I get the >>> following error; >>> Has anyone encountered a similar issue? >>> >>> Query: CREATE TABLE test_123(id INT, name STRING) >>> ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive >>> Metastore: >>> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req' >>> >>> I1218 11:10:13.229060 62868 jni-util.cc:288] >>> org.apache.impala.common.ImpalaRuntimeException: Error making 'createTable' >>> RPC to Hive Metastore: >>> at >>> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:2107) >>> at >>> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:1936) >>> at >>> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:330) >>> at >>> org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:167) >>> Caused by: org.apache.thrift.TApplicationException: Invalid method name: >>> 'create_table_req' >>> at >>> org.apache.thrift.TApplicationException.read(TApplicationException.java:111) >>> at >>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) >>> at >>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1698) >>> at >>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1685) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3359) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:990) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:975) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at >>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208) >>> at com.sun.proxy.$Proxy4.createTable(Unknown Source) >>> at >>> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:2086) >>> ... 3 more >>> I1218 11:10:13.230059 62868 status.cc:124] ImpalaRuntimeException: Error >>> making 'createTable' RPC to Hive Metastore: >>> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req' >>> @ 0xbc38a9 >>> @ 0x129139e >>> @ 0xbb0317 >>> @ 0xb9f0e9 >>> @ 0xca58cc >>> @ 0xca3e4c >>> @ 0xb91059 >>> @ 0x1028a10 >>> @ 0x101e68e >>> @ 0x101f521 >>> @ 0x131928f >>> @ 0x1319e39 >>> @ 0x1a5f329 >>> @ 0x7fc4313d0e24 >>> @ 0x7fc42de2b34c >>> E1218 11:10:13.230078 62868 catalog-server.cc:112] >>> ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore: >>> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req' >>> I1218 11:10:13.230155 62868 catalog-server.cc:116] ExecDdl(): >>> response=TDdlExecResponse { >>> 01: result (struct) = TCatalogUpdateResult { >>> 01: catalog_service_id (struct) = TUniqueId { >>> 01: hi (i64) = 0, >>> 02: lo (i64) = 0, >>> }, >>> 02: version (i64) = 0, >>> 03: status (struct) = TStatus { >>> 01: status_code (i32) = 2, >>> 02: error_msgs (list) = list<string>[1] { >>> [0] = "ImpalaRuntimeException: Error making 'createTable' RPC to >>> Hive Metastore: \nCAUSED BY: TApplicationException: Invalid method name: >>> 'create_table_req'", >>> }, >>> }, >>> 04: is_invalidate (bool) = false, >>> }, >>> } >>> >>> >>> -- >>> Regards >>> Akila Wajirasena >>> >>> >>> >>> > > > > >
