[
https://issues.apache.org/jira/browse/IMPALA-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367628#comment-17367628
]
Vihang Karajgaonkar edited comment on IMPALA-10759 at 6/22/21, 7:24 PM:
------------------------------------------------------------------------
I took a brief look at this yesterday and I found that the issue happens when
Impala is using 0.9.3 thrift version and hive is using 0.13.0 version. This
happens because the hashCode method in the thrift generated code for HMS
objects like Partition changes when you change thrift from 0.9.3 to 0.13.0.
Specifically the hashCode values for the primitive fields like long now use
TBaseHelper.hashCode(long) instead of the old way of add it to a ArrayList and
then comparing the hashCode.
For example, in case of writeId field of the partition which is definied as a
i64 in the thrift file, the hashCode is computed using TBaseHelper as seen in
the diffs here
https://github.com/apache/hive/commit/1945e2f67e5b09cdda40146b87e1ba492f897196#diff-505c537842790dadd6f182b07b0b216be40e050588941213220b4ae3622bd0faR877
I don't think there is a good way to "fix" this. This should get fixed
automatically when Impala uses 0.11.0 thrift version. I confirmed that the
build where we saw this failure did not have the Impala thrift version as
0.11.0.
was (Author: vihangk1):
I took a brief look at this yesterday and I found that the issue happens when
Impala is using 0.9.3 thrift version and hive is using 0.13.0 version. This
happens because the {noformat}hashCode{noformat} method in the thrift generated
code for HMS objects like Partition changes when you change thrift from 0.9.3
to 0.13.0. Specifically the hashCode values for the primitive fields like long
now use {noformat}TBaseHelper.hashCode{noformat} instead of the old way of add
it to a ArrayList and then comparing the hashCode.
For example, in case of writeId field of the partition which is definied as a
i64 in the thrift file, the hashCode is computed using TBaseHelper as seen in
the diffs here
https://github.com/apache/hive/commit/1945e2f67e5b09cdda40146b87e1ba492f897196#diff-505c537842790dadd6f182b07b0b216be40e050588941213220b4ae3622bd0faR877
I don't think there is a good way to "fix" this. This should get fixed
automatically when Impala uses 0.11.0 thrift version. I confirmed that the
build where we saw this failure did not have the Impala thrift version as
0.11.0.
> MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
> ----------------------------------------------------------------------------
>
> Key: IMPALA-10759
> URL: https://issues.apache.org/jira/browse/IMPALA-10759
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Yongzhi Chen
> Assignee: Vihang Karajgaonkar
> Priority: Critical
>
> impala-cdpd-master-core
> EnableCatalogdHmsCacheFlagTest.testEnableCatalogdCachingFlag test fails with
> following stack:
> {noformat}
> Exception in thread "pool-470-thread-1" java.lang.NoSuchMethodError:
> org.apache.thrift.TBaseHelper.hashCode(J)I
> at
> org.apache.hadoop.hive.metastore.api.Partition.hashCode(Partition.java:971)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at
> org.apache.impala.catalog.CatalogHmsAPIHelper.loadAndSetFileMetadataFromFs(CatalogHmsAPIHelper.java:527)
> at
> org.apache.impala.catalog.metastore.MetastoreServiceHandler.get_partitions_by_names_req(MetastoreServiceHandler.java:1443)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServiceHandler.get_partitions_by_names_req(CatalogMetastoreServiceHandler.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServer$TimingInvocationHandler.invoke(CatalogMetastoreServer.java:223)
> at com.sun.proxy.$Proxy87.get_partitions_by_names_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20087)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20066)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
