Francis,
The patch for PIG-2712 will only go into trunk(0.11) because it will
case backward compatibility break with 0.20.2 and pig 0.10 is supported
with 0.20.2. I don't understand why you think it is too much of a
workaround. The proposed fix for 451 is that abortJob() and cleanupJob()
will be doing the same thing till everyone moves to pig 0.11 as pig calls
cleanupJob() instead of abortJob().
commitJob() {
commit work - move from temp to final location, register partitions,
update table schema, etc
}
abortJob(){
cleanupJob()
}
cleanupJob() {
delete temp dir
}
Regards,
Rohini
On Thu, Jul 26, 2012 at 4:07 PM, Francis Christopher Liu <
[email protected]> wrote:
> Hi,
>
> HCat 0.5 compatible with Pig 0.10? Are we making that official?
>
> The reason I'm asking is because of this bug:
>
> https://issues.apache.org/jira/browse/HCATALOG-451
>
> It's caused because pig does not override and call
> OutputCommitter.abortJob() (PIG-2712). The bug can and should be fixed in
> pig, but we may need to fix it in HCatalog to support unfixed versions of
> pig. I'd rather we not have another workaround and have HCatalog 0.5
> support later versions of pig which fixes this bug.
>
> Thoughts?
>
> -Francis
>
>
>
> On 7/23/12 9:32 AM, "Travis Crawford" <[email protected]> wrote:
>
> >On Mon, Jul 23, 2012 at 8:54 AM, Alan Gates <[email protected]>
> wrote:
> >
> >> The issue before pig 0.10 is that the parser won't allow booleans in
> >> certain locations. For example,
> >>
> >> A = load 'foo';
> >> B = foreach A generate mybool; -- where mybool is a boolean
> >>
> >> will fail. Half baked supported is worse than no support. In Pig 0.10
> >>I
> >> think your changes make sense. We could declare that HCat 0.5 works wit
> >> Pig 0.10 and then make these changes on HCat trunk.
> >>
> >>
> >Cool thanks for clearing that up. I like the idea of bumping the required
> >pig version and simply adding boolean support. I think this means we'll
> >have some integration challenges with mismatched antlr versions, but we'll
> >have to address it at some point.
> >
> >--travis
> >
> >
> >
> >
> >> Alan.
> >>
> >> On Jul 20, 2012, at 2:06 PM, Travis Crawford wrote:
> >>
> >> > Hey all -
> >> >
> >> > I'm taking a look at boolean support and started by making just this
> >> simple
> >> > change. Basically just remove the bits that throw exceptions for
> >>boolean
> >> > fields.
> >> >
> >> > Using pig 0.9 I ran a simple query against a table with booleans and
> >>it
> >> > worked just fine. DataType.BOOLEAN already exists (even in pig 0.8),
> >> > "describe" showed the fields as boolean, and dump printed out
> >>true/false
> >> as
> >> > expected.
> >> >
> >> > Am I missing something? The user seems better off by not throwing an
> >> > exception here because if their pig can handle booleans their query
> >>runs.
> >> >
> >> >
> >> > TW-MBP13-TCrawford:hcatalog travis$ git diff
> >> > diff --git
> >> >
> >>
> >>a/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUtil.
> >>java
> >> >
> >>
> >>b/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUtil.
> >>java
> >> > index 696081f..fe686cd 100644
> >> > ---
> >> >
> >>
> >>a/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUtil.
> >>java
> >> > +++
> >> >
> >>
> >>b/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUtil.
> >>java
> >> > @@ -298,9 +298,7 @@ public class PigHCatUtil {
> >> > }
> >> >
> >> > if (type == Type.BOOLEAN){
> >> > *- errMsg = "HCatalog column type 'BOOLEAN' is not supported in
> >>" +
> >> > - "Pig as a column type";
> >> > - throw new PigException(errMsg, PIG_EXCEPTION_CODE);
> >> > + return DataType.BOOLEAN;
> >> > * }
> >> >
> >> > errMsg = "HCatalog column type '"+ type.toString() +"' is not
> >> > supported in Pig as a column type";
> >> > @@ -409,7 +407,6 @@ public class PigHCatUtil {
> >> > // We don't do type promotion/demotion.
> >> > case SMALLINT:
> >> > case TINYINT:
> >> > *- case BOOLEAN:
> >> > * throw new PigException("Incompatible type found in hcat
> >>table
> >> > schema: "+hcatField, PigHCatUtil.PIG_EXCEPTION_CODE);
> >> > case ARRAY:
> >> >
> >> > validateHCatSchemaFollowsPigRules(hcatField.getArrayElementSchema());
> >> > TW-MBP13-TCrawford:hcatalog travis$
> >> >
> >> > --travis
>
>
