HCatalog should handle boolean fields
-------------------------------------
Key: HCATALOG-364
URL: https://issues.apache.org/jira/browse/HCATALOG-364
Project: HCatalog
Issue Type: Bug
Reporter: Travis Crawford
Assignee: Travis Crawford
Currently HCatalog does not support boolean fields, explicitly looking for them
and throwing an exception:
{code}
Caused by: org.apache.pig.PigException: ERROR 1115: Incompatible type found in
hcat table schema: boolean
at
org.apache.hcatalog.pig.PigHCatUtil.validateHcatFieldFollowsPigRules(PigHCatUtil.java:403)
at
org.apache.hcatalog.pig.PigHCatUtil.validateHCatSchemaFollowsPigRules(PigHCatUtil.java:391)
at
org.apache.hcatalog.pig.PigHCatUtil.validateHcatFieldFollowsPigRules(PigHCatUtil.java:408)
at
org.apache.hcatalog.pig.PigHCatUtil.validateHCatSchemaFollowsPigRules(PigHCatUtil.java:391)
at
org.apache.hcatalog.pig.PigHCatUtil.validateHCatTableSchemaFollowsPigRules(PigHCatUtil.java:422)
at org.apache.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:158)
{code}
This is a big problem as many of our schemas use boolean.
I propose we take the same approach as elephant-bird's
[ThriftToPig.java|https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java#L106]
and translate into an Integer in the pig layer. This approach allows
processing frameworks with boolean support to take advantage of that feature,
and frameworks without boolean support could use the translated value (versions
of pig people are likely to have deployed; newer pig versions with boolean
support could use the type directly). We've been using this technique for some
time and in practice it works well.
Thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
