[
https://issues.apache.org/jira/browse/HAWQ-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415589#comment-15415589
]
Shivram Mani edited comment on HAWQ-992 at 8/10/16 9:57 PM:
------------------------------------------------------------
HiveText,HiveRc and HiveOrc uses a fragmenter that doens't send back the
partition properties - colummn types, serailaization info etc) and uses the
hawq DDL column types
Even though the fragmenter can permit hawq types such as varchar char without
length, the Hive type checker TypeInfoUtils doesn't permit using types without
length.
For example using varchar in hawq tagble resutls in the following exception
"java.lang.Exception: java.lang.IllegalArgumentException: varchar type is
specified without length"
{code}
Caused by: java.lang.IllegalArgumentException: varchar type is specified
without length:
string,string,int,double,decimal,timestamp,float,bigint,boolean,small
at
org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:403)
at
org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseTypeInfos(TypeInfoUtils.java:305)
at
org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfosFromTypeString(TypeInfoUtils.java:765)
at org.apache.hadoop.hive.ql.io.orc.OrcSerde.initialize(OrcSerde.java:104)
at
org.apache.hawq.pxf.plugins.hive.HiveORCSerdeResolver.initSerde(HiveORCSerdeResolver.java:147)
at
org.apache.hawq.pxf.plugins.hive.HiveResolver.<init>(HiveResolver.java:106)
at
org.apache.hawq.pxf.plugins.hive.HiveORCSerdeResolver.<init>(HiveORCSerdeResolver.java:70)
{code}
For this reason we will have to enforce strict type check including
precision/length. The only other option will be to send back the partition
properties as part of the userData. This is done in makeUserData in fragmenter.
was (Author: shivram):
Even though the fragmenter can permitm hawq types such as varchar char without
length, the Hive type checker TypeInfoUtils doesn't permit using types without
length.
For example using varchar in hawq tagble resutls in the following exception
"java.lang.Exception: java.lang.IllegalArgumentException: varchar type is
specified without length"
{code}
Caused by: java.lang.IllegalArgumentException: varchar type is specified
without length:
string,string,int,double,decimal,timestamp,float,bigint,boolean,small
at
org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:403)
at
org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseTypeInfos(TypeInfoUtils.java:305)
at
org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfosFromTypeString(TypeInfoUtils.java:765)
at org.apache.hadoop.hive.ql.io.orc.OrcSerde.initialize(OrcSerde.java:104)
at
org.apache.hawq.pxf.plugins.hive.HiveORCSerdeResolver.initSerde(HiveORCSerdeResolver.java:147)
at
org.apache.hawq.pxf.plugins.hive.HiveResolver.<init>(HiveResolver.java:106)
at
org.apache.hawq.pxf.plugins.hive.HiveORCSerdeResolver.<init>(HiveORCSerdeResolver.java:70)
{code}
> PXF Hive data type check in Fragmenter too restrictive
> ------------------------------------------------------
>
> Key: HAWQ-992
> URL: https://issues.apache.org/jira/browse/HAWQ-992
> Project: Apache HAWQ
> Issue Type: Bug
> Components: PXF
> Reporter: Shivram Mani
> Assignee: Shivram Mani
> Fix For: backlog
>
>
> HiveDataFragmenter used by both HiveText and HiveRC profiles has a very
> strict type check.
> Hawq type numeric(10,10) is compatible with hive's decimal(10,10)
> But Hawq type numeric is not compatible with hive's decimal(10,10)
> Similar issue exits with other data types which have variable optional
> arguments. The type check should be modified to allow hawq type that is a
> compabitle type but without optional precision/length arguments to work with
> the corresponding hive type.
> Support following additional hive data types: date, varchar, char
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
