[jira] [Commented] (IMPALA-10759) MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
[
https://issues.apache.org/jira/browse/IMPALA-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376565#comment-17376565
]
Csaba Ringhofer commented on IMPALA-10759:
--
[~asherman] Impala already uses Thrift 0.11.0 on asf master:
https://github.com/apache/impala/commit/94f67a343294430b53da58d1af60e659af18545a
Note that the Hive version used in asf Impala still uses Thrift 0.9.3, so it is
possible for some older Thrift related issues to leak in.
> MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
>
>
> Key: IMPALA-10759
> URL: https://issues.apache.org/jira/browse/IMPALA-10759
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Reporter: Yongzhi Chen
>Assignee: Vihang Karajgaonkar
>Priority: Critical
>
> impala-cdpd-master-core
> EnableCatalogdHmsCacheFlagTest.testEnableCatalogdCachingFlag test fails with
> following stack:
> {noformat}
> Exception in thread "pool-470-thread-1" java.lang.NoSuchMethodError:
> org.apache.thrift.TBaseHelper.hashCode(J)I
> at
> org.apache.hadoop.hive.metastore.api.Partition.hashCode(Partition.java:971)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at
> org.apache.impala.catalog.CatalogHmsAPIHelper.loadAndSetFileMetadataFromFs(CatalogHmsAPIHelper.java:527)
> at
> org.apache.impala.catalog.metastore.MetastoreServiceHandler.get_partitions_by_names_req(MetastoreServiceHandler.java:1443)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServiceHandler.get_partitions_by_names_req(CatalogMetastoreServiceHandler.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServer$TimingInvocationHandler.invoke(CatalogMetastoreServer.java:223)
> at com.sun.proxy.$Proxy87.get_partitions_by_names_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20087)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20066)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10759) MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
[
https://issues.apache.org/jira/browse/IMPALA-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368403#comment-17368403
]
Andrew Sherman commented on IMPALA-10759:
-
[~vihangk1] thanks for the analysis. When will Impala use the 0.11.0 thrift
version?
> MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
>
>
> Key: IMPALA-10759
> URL: https://issues.apache.org/jira/browse/IMPALA-10759
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Reporter: Yongzhi Chen
>Assignee: Vihang Karajgaonkar
>Priority: Critical
>
> impala-cdpd-master-core
> EnableCatalogdHmsCacheFlagTest.testEnableCatalogdCachingFlag test fails with
> following stack:
> {noformat}
> Exception in thread "pool-470-thread-1" java.lang.NoSuchMethodError:
> org.apache.thrift.TBaseHelper.hashCode(J)I
> at
> org.apache.hadoop.hive.metastore.api.Partition.hashCode(Partition.java:971)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at
> org.apache.impala.catalog.CatalogHmsAPIHelper.loadAndSetFileMetadataFromFs(CatalogHmsAPIHelper.java:527)
> at
> org.apache.impala.catalog.metastore.MetastoreServiceHandler.get_partitions_by_names_req(MetastoreServiceHandler.java:1443)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServiceHandler.get_partitions_by_names_req(CatalogMetastoreServiceHandler.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServer$TimingInvocationHandler.invoke(CatalogMetastoreServer.java:223)
> at com.sun.proxy.$Proxy87.get_partitions_by_names_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20087)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20066)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10759) MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
[
https://issues.apache.org/jira/browse/IMPALA-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367785#comment-17367785
]
Andrew Sherman commented on IMPALA-10759:
-
For the record this happens in
org.apache.impala.catalog.metastore.CatalogHmsFileMetadataTest.testFileMetadataForPartitions
org.apache.impala.catalog.metastore.CatalogHmsFileMetadataTest.testFileMetadataForTable
org.apache.impala.catalog.metastore.EnableCatalogdHmsCacheFlagTest.testEnableCatalogdCachingFlag
> MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
>
>
> Key: IMPALA-10759
> URL: https://issues.apache.org/jira/browse/IMPALA-10759
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Reporter: Yongzhi Chen
>Assignee: Vihang Karajgaonkar
>Priority: Critical
>
> impala-cdpd-master-core
> EnableCatalogdHmsCacheFlagTest.testEnableCatalogdCachingFlag test fails with
> following stack:
> {noformat}
> Exception in thread "pool-470-thread-1" java.lang.NoSuchMethodError:
> org.apache.thrift.TBaseHelper.hashCode(J)I
> at
> org.apache.hadoop.hive.metastore.api.Partition.hashCode(Partition.java:971)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at
> org.apache.impala.catalog.CatalogHmsAPIHelper.loadAndSetFileMetadataFromFs(CatalogHmsAPIHelper.java:527)
> at
> org.apache.impala.catalog.metastore.MetastoreServiceHandler.get_partitions_by_names_req(MetastoreServiceHandler.java:1443)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServiceHandler.get_partitions_by_names_req(CatalogMetastoreServiceHandler.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServer$TimingInvocationHandler.invoke(CatalogMetastoreServer.java:223)
> at com.sun.proxy.$Proxy87.get_partitions_by_names_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20087)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20066)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10759) MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
[
https://issues.apache.org/jira/browse/IMPALA-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367628#comment-17367628
]
Vihang Karajgaonkar commented on IMPALA-10759:
--
I took a brief look at this yesterday and I found that the issue happens when
Impala is using 0.9.3 thrift version and hive is using 0.13.0 version. This
happens because the {noformat}hashCode{noformat} method in the thrift generated
code for HMS objects like Partition changes when you change thrift from 0.9.3
to 0.13.0. Specifically the hashCode values for the primitive fields like long
now use {noformat}TBaseHelper.hashCode{noformat} instead of the old way of add
it to a ArrayList and then comparing the hashCode.
For example, in case of writeId field of the partition which is definied as a
i64 in the thrift file, the hashCode is computed using TBaseHelper as seen in
the diffs here
https://github.com/apache/hive/commit/1945e2f67e5b09cdda40146b87e1ba492f897196#diff-505c537842790dadd6f182b07b0b216be40e050588941213220b4ae3622bd0faR877
I don't think there is a good way to "fix" this. This should get fixed
automatically when Impala uses 0.11.0 thrift version. I confirmed that the
build where we saw this failure did not have the Impala thrift version as
0.11.0.
> MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
>
>
> Key: IMPALA-10759
> URL: https://issues.apache.org/jira/browse/IMPALA-10759
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Reporter: Yongzhi Chen
>Assignee: Vihang Karajgaonkar
>Priority: Critical
>
> impala-cdpd-master-core
> EnableCatalogdHmsCacheFlagTest.testEnableCatalogdCachingFlag test fails with
> following stack:
> {noformat}
> Exception in thread "pool-470-thread-1" java.lang.NoSuchMethodError:
> org.apache.thrift.TBaseHelper.hashCode(J)I
> at
> org.apache.hadoop.hive.metastore.api.Partition.hashCode(Partition.java:971)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at
> org.apache.impala.catalog.CatalogHmsAPIHelper.loadAndSetFileMetadataFromFs(CatalogHmsAPIHelper.java:527)
> at
> org.apache.impala.catalog.metastore.MetastoreServiceHandler.get_partitions_by_names_req(MetastoreServiceHandler.java:1443)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServiceHandler.get_partitions_by_names_req(CatalogMetastoreServiceHandler.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServer$TimingInvocationHandler.invoke(CatalogMetastoreServer.java:223)
> at com.sun.proxy.$Proxy87.get_partitions_by_names_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20087)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20066)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10759) MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
[
https://issues.apache.org/jira/browse/IMPALA-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367550#comment-17367550
]
Andrew Sherman commented on IMPALA-10759:
-
Hi [~vihangk1] can you take a look at this? Please reassign as necessary
> MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
>
>
> Key: IMPALA-10759
> URL: https://issues.apache.org/jira/browse/IMPALA-10759
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Reporter: Yongzhi Chen
>Assignee: Vihang Karajgaonkar
>Priority: Critical
>
> impala-cdpd-master-core
> EnableCatalogdHmsCacheFlagTest.testEnableCatalogdCachingFlag test fails with
> following stack:
> {noformat}
> Exception in thread "pool-470-thread-1" java.lang.NoSuchMethodError:
> org.apache.thrift.TBaseHelper.hashCode(J)I
> at
> org.apache.hadoop.hive.metastore.api.Partition.hashCode(Partition.java:971)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at
> org.apache.impala.catalog.CatalogHmsAPIHelper.loadAndSetFileMetadataFromFs(CatalogHmsAPIHelper.java:527)
> at
> org.apache.impala.catalog.metastore.MetastoreServiceHandler.get_partitions_by_names_req(MetastoreServiceHandler.java:1443)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServiceHandler.get_partitions_by_names_req(CatalogMetastoreServiceHandler.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServer$TimingInvocationHandler.invoke(CatalogMetastoreServer.java:223)
> at com.sun.proxy.$Proxy87.get_partitions_by_names_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20087)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20066)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10759) MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
[
https://issues.apache.org/jira/browse/IMPALA-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17366934#comment-17366934
]
Andrew Sherman commented on IMPALA-10759:
-
[~sql_forever] can you take a look at this? It seemed to appear on 16-June and
you committed on 15-June. And you changed PlannerTest. Maybe just a coincidence
but please take a look and reassign if you can find someone better.
> MetastoreServiceHandler.get_partitions_by_names_req throws NoSuchMethodError
>
>
> Key: IMPALA-10759
> URL: https://issues.apache.org/jira/browse/IMPALA-10759
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Reporter: Yongzhi Chen
>Assignee: Qifan Chen
>Priority: Critical
>
> impala-cdpd-master-core
> EnableCatalogdHmsCacheFlagTest.testEnableCatalogdCachingFlag test fails with
> following stack:
> {noformat}
> Exception in thread "pool-470-thread-1" java.lang.NoSuchMethodError:
> org.apache.thrift.TBaseHelper.hashCode(J)I
> at
> org.apache.hadoop.hive.metastore.api.Partition.hashCode(Partition.java:971)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at
> org.apache.impala.catalog.CatalogHmsAPIHelper.loadAndSetFileMetadataFromFs(CatalogHmsAPIHelper.java:527)
> at
> org.apache.impala.catalog.metastore.MetastoreServiceHandler.get_partitions_by_names_req(MetastoreServiceHandler.java:1443)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServiceHandler.get_partitions_by_names_req(CatalogMetastoreServiceHandler.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.impala.catalog.metastore.CatalogMetastoreServer$TimingInvocationHandler.invoke(CatalogMetastoreServer.java:223)
> at com.sun.proxy.$Proxy87.get_partitions_by_names_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20087)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_names_req.getResult(ThriftHiveMetastore.java:20066)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
