[
https://issues.apache.org/jira/browse/HIVE-23145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-23145:
-------------------------------
Description:
Expression is not parsable in most of the cases. Current API
*get_partitions_by_expr* anticipates this and provide a fallback mechanism.
This basically deserialize the provided expression, fetches all partition names
for the table, prune partition names using the expression and then uses the
names to fetch required partition data.
Note that this expect serialized expression instead of string.
This need to be done for both Direct SQL and JDO path.
e.g. Following error is thrown for tpcds query 55 which provide expression
*<column> IS NOT NULL filter*
*ERROR*
{code:java}
MetaException(message:Error parsing partition filter; lexer error: null;
exception NoViableAltException(13@[]))MetaException(message:Error parsing
partition filter; lexer error: null; exception NoViableAltException(13@[])) at
org.apache.hadoop.hive.metastore.PartFilterExprUtil.getFilterParser(PartFilterExprUtil.java:154)
at
org.apache.hadoop.hive.metastore.ObjectStore$15.initExpressionTree(ObjectStore.java:4339)
at
org.apache.hadoop.hive.metastore.ObjectStore$15.canUseDirectSql(ObjectStore.java:4319)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.start(ObjectStore.java:4021)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3985)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionSpecsByFilterAndProjection(ObjectStore.java:4395)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at
com.sun.proxy.$Proxy26.getPartitionSpecsByFilterAndProjection(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_with_specs(HiveMetaStore.java:5356)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy27.get_partitions_with_specs(Unknown Source) at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21620)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21604)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
at java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
{code}
was:
Expression is not parsable in most of the cases. Current API
*get_partitions_by_expr* anticipates this and provide a fallback mechanism.
This basically deserialize the provided expression, fetches all partition names
for the table, prune partition names using the expression and then uses the
names to fetch required partition data.
Note that this expect serialized expression instead of string.
This need to be done for both Direct SQL and JDO path.
e.g. Following error is thrown for tpcds query 55 which provide expression
*<column> IS NOT NULL filter*
*ERROR*
{code:java}
MetaException(message:Error parsing partition filter; lexer error: null;
exception NoViableAltException(13@[]))MetaException(message:Error parsing
partition filter; lexer error: null; exception NoViableAltException(13@[])) at
org.apache.hadoop.hive.metastore.PartFilterExprUtil.getFilterParser(PartFilterExprUtil.java:154)
at
org.apache.hadoop.hive.metastore.ObjectStore$15.initExpressionTree(ObjectStore.java:4339)
at
org.apache.hadoop.hive.metastore.ObjectStore$15.canUseDirectSql(ObjectStore.java:4319)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.start(ObjectStore.java:4021)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3985)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionSpecsByFilterAndProjection(ObjectStore.java:4395)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at
com.sun.proxy.$Proxy26.getPartitionSpecsByFilterAndProjection(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_with_specs(HiveMetaStore.java:5356)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy27.get_partitions_with_specs(Unknown Source) at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21620)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21604)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
at java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
{code}
Instead of throwing an error this API should do what *listPartitionsByExpr* does
{code:java}
{code}
> get_partitions_with_specs fails if filter expression is not parsable
> --------------------------------------------------------------------
>
> Key: HIVE-23145
> URL: https://issues.apache.org/jira/browse/HIVE-23145
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
> Affects Versions: 4.0.0
> Reporter: Vineet Garg
> Priority: Major
>
> Expression is not parsable in most of the cases. Current API
> *get_partitions_by_expr* anticipates this and provide a fallback mechanism.
> This basically deserialize the provided expression, fetches all partition
> names for the table, prune partition names using the expression and then uses
> the names to fetch required partition data.
> Note that this expect serialized expression instead of string.
> This need to be done for both Direct SQL and JDO path.
> e.g. Following error is thrown for tpcds query 55 which provide expression
> *<column> IS NOT NULL filter*
> *ERROR*
> {code:java}
> MetaException(message:Error parsing partition filter; lexer error: null;
> exception NoViableAltException(13@[]))MetaException(message:Error parsing
> partition filter; lexer error: null; exception NoViableAltException(13@[]))
> at
> org.apache.hadoop.hive.metastore.PartFilterExprUtil.getFilterParser(PartFilterExprUtil.java:154)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$15.initExpressionTree(ObjectStore.java:4339)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$15.canUseDirectSql(ObjectStore.java:4319)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.start(ObjectStore.java:4021)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3985)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionSpecsByFilterAndProjection(ObjectStore.java:4395)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy26.getPartitionSpecsByFilterAndProjection(Unknown
> Source) at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_with_specs(HiveMetaStore.java:5356)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.get_partitions_with_specs(Unknown Source) at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21620)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21604)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
