[
https://issues.apache.org/jira/browse/HIVE-19040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19040:
----------------------------
Description:
In the HiveMetaStore implementation of {{public PartitionsByExprResult
get_partitions_by_expr(PartitionsByExprRequest req) throws TException}} , an
expression is serialized into byte array from the client side and passed
through PartitionsByExprRequest. Then HMS will deserialize back into the
expression and filter the partitions by it.
Such partition filtering expression can contain various UDFs. If there are some
changes to one of the UDFs between different Hive versions, HS2 on the older
version will serialize the expression in old format which won't be able to be
deserialized by HMS on the newer version. One example of that is, GenericUDFIn
class adds {{transient}} to the field constantInSet which will cause such
incompatibility.
One approach I'm thinking of is, instead of converting the expression object to
byte array, we can pass the expression string directly.
was:
In the HiveMetaStore implementation of {{public PartitionsByExprResult
get_partitions_by_expr(PartitionsByExprRequest req) throws TException}} , an
expression is serialized into byte array from the client side and passed
through PartitionsByExprRequest. Then HMS will deserialize back into the
expression and filter the partitions by it.
Such partition filtering expression can contain various UDFs. If there are some
changes to one of the UDFs between different Hive versions, HS2 on the older
version will serialize the expression in old format which won't be able to be
deserialized by HMS on the newer version. One example of that is, GenericUDFIn
class adds {{transient}} to the field constantInSet which will cause such
incompatibility.
One approach I'm thinking is, instead of converting the expression object to
byte array, we can pass the expression string directly.
> get_partitions_by_expr() implementation in HiveMetaStore causes backward
> incompatibility easily
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-19040
> URL: https://issues.apache.org/jira/browse/HIVE-19040
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
> Affects Versions: 2.0.0
> Reporter: Aihua Xu
> Priority: Major
>
> In the HiveMetaStore implementation of {{public PartitionsByExprResult
> get_partitions_by_expr(PartitionsByExprRequest req) throws TException}} , an
> expression is serialized into byte array from the client side and passed
> through PartitionsByExprRequest. Then HMS will deserialize back into the
> expression and filter the partitions by it.
> Such partition filtering expression can contain various UDFs. If there are
> some changes to one of the UDFs between different Hive versions, HS2 on the
> older version will serialize the expression in old format which won't be able
> to be deserialized by HMS on the newer version. One example of that is,
> GenericUDFIn class adds {{transient}} to the field constantInSet which will
> cause such incompatibility.
> One approach I'm thinking of is, instead of converting the expression object
> to byte array, we can pass the expression string directly.
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
