The outputSchema is set to Long
90 @Override
91 public Schema outputSchema(Schema input) {
92 return new Schema(new
Schema.FieldSchema(getSchemaName(this.getClass
().getName().toLowerCase(), input), DataType.CHARARRAY));
93 }
The describe output is
eventData: {event: chararray,publication: chararray,deviceType:
chararray,adID: chararray,mcc: chararray,eventTimestamp:
long,startTime: long}
but when I try to do filter eventData by eventTimestamp < starttime,
it throws the error.
I am guessing there is a diff between long and Long ... totally a
guess. To prove this, I change it to return a chararray and
everything work, so it further suggests it is type mismatch issue.
On Wed, Dec 15, 2010 at 5:23 PM, Dmitriy Ryaboy <[email protected]> wrote:
> What is the describe output?
>
> Are you setting the outputSchema appropriately in ISOToUnixInSecond ?
>
> (btw.. you probably want to call that InSeconds, plural)
>
> D
>
> On Wed, Dec 15, 2010 at 2:00 PM, John Hui <[email protected]> wrote:
> > To give more context, the ISOToUnixInSecond return UnixTime in second.
> The
> > return value of this function is Long
> >
> > 75 @Override
> > 76 public Long exec(Tuple input) throws IOException
> > 77 {
> > 78 if (input == null || input.size() < 1) {
> > 79 return null;
> > 80 }
> > 81
> > 82 // Set the time to default or the output is in UTC
> > 83 DateTimeZone.setDefault(DateTimeZone.UTC);
> > 84
> > 85 DateTime result = new DateTime(input.get(0).toString());
> > 86
> > 87 return (result.getMillis() / 1000);
> > 88 }
> >
> >
> > I think somehow pig is not converting the Long into the proper long type
> in
> > the pig script.
> >
> > ISOToUnixInSecond('$STARTDATETIME') AS startTime:long
> >
> > Hence during the comparion, it treat the Long as a string value ...
> >
> > On Wed, Dec 15, 2010 at 4:28 PM, John Hui <[email protected]> wrote:
> >
> >> This is actually, please ignore the code section below, Thanks!
> >>
> >> 7 eventData = FOREACH flattenData GENERATE B1::event AS event,
> >> B1::publication AS publication, B1::deviceType AS deviceType, B1::adID
> AS
> >> adID, B1::mcc AS mcc, B1::event_timestamp AS eventTimestamp:long,
> >> ISOToUnixInSecond('$STARTDATETIME') AS startTime:long;
> >> 8
> >> 9 eventData = FILTER eventData BY (event == 'adImpression') AND
> >> (eventTimesta mp <= startTime);
> >> 10
> >> 11 DESCRIBE eventData;
> >> 12
> >> 13 B = GROUP eventData BY (event, publication, deviceType, adID, mcc);
> >>
> >>
> >> On Wed, Dec 15, 2010 at 4:21 PM, John Hui <[email protected]> wrote:
> >>
> >>> I am having a hard time getting comparison to work. I am comparing from
> >>> two long values but I keep on getting a cast long to String error
> >>>
> >>> Backend error message
> >>> ---------------------
> >>> java.lang.ClassCastException: java.lang.Long cannot be cast to
> java.lang.String
> >>> at java.lang.String.compareTo(String.java:92)
> >>> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.LTOrEqualToExpr.doComparison(LTOrEqualToExpr.java:152)
> >>> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.LTOrEqualToExpr.getNext(LTOrEqualToExpr.java:114)
> >>> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POAnd.getNext(POAnd.java:83)
> >>> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.getNext(POFilter.java:148)
> >>> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:272)
> >>> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:255)
> >>> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:232)
> >>> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:227)
> >>> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:52)
> >>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:644)
> >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:320)
> >>> at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
> >>> at java.security.AccessController.doPrivileged(Native Method)
> >>> at javax.security.auth.Subject.doAs(Subject.java:396)
> >>> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
> >>> at org.apache.hadoop.mapred.Child.main(Child.java:211
> >>>
> >>>
> >>> this is my pig code
> >>>
> >>> 7 eventData = FOREACH flattenData GENERATE B1::event AS event,
> >>> B1::publicatio n AS publication, B1::deviceType AS deviceType,
> B1::adID
> >>> AS adID, B1::mcc A S mcc, B1::event_timestamp AS
> eventTimestamp:long,
> >>> ISOToUnixInSecond('$STAR TDATETIME') AS startTime:long;
> >>> 8 9 ILLUSTRATE eventData;
> >>> 10 11 eventData = FILTER eventData BY (startTime <= startTime);
> >>> 12 --eventData = FILTER eventData BY (event == 'adImpression') AND
> >>> (eventTimes tamp <= startTime);
> >>> 13
> >>> 14 DESCRIBE eventData;
> >>>
> >>> Here is the output
> >>>
> >>> eventData: {event: chararray,publication: chararray,deviceType:
> chararray,adID: chararray,mcc: chararray,eventTimestamp: long,startTime:
> long}
> >>>
> >>>
> >>>
> >>
> >
>
