Re: Comparison between long

Thu, 16 Dec 2010 22:22:47 -0800 

On Thursday 16 December 2010 03:58 AM, John Hui wrote:

The outputSchema is set to Long

  90     @Override
  91     public Schema outputSchema(Schema input) {
  92         return new Schema(new
Schema.FieldSchema(getSchemaName(this.getClass
  ().getName().toLowerCase(), input), DataType.CHARARRAY));
  93     }


This sets it to chararray - not Long ...
Maybe root cause of the issue ?

Did you try casting to long ? (if this is not the problem).


Regards,
Mridul



The describe output is

eventData: {event: chararray,publication: chararray,deviceType:
chararray,adID: chararray,mcc: chararray,eventTimestamp:
long,startTime: long}


but when I try to do filter eventData by eventTimestamp<  starttime,
it throws the error.


I am guessing there is a diff between long and Long ... totally a
guess.  To prove this, I change it to return a chararray and
everything work, so it further suggests it is type mismatch issue.




On Wed, Dec 15, 2010 at 5:23 PM, Dmitriy Ryaboy<[email protected]>  wrote:


What is the describe output?

Are you setting the outputSchema appropriately in ISOToUnixInSecond ?

(btw.. you probably want to call that InSeconds, plural)

D

On Wed, Dec 15, 2010 at 2:00 PM, John Hui<[email protected]>  wrote:

To give more context, the ISOToUnixInSecond return UnixTime in second.

  The

return value of this function is Long

  75     @Override
  76     public Long exec(Tuple input) throws IOException
  77     {
  78         if (input == null || input.size()<  1) {
  79             return null;
  80         }
  81
  82         // Set the time to default or the output is in UTC
  83         DateTimeZone.setDefault(DateTimeZone.UTC);
  84
  85         DateTime result = new DateTime(input.get(0).toString());
  86
  87         return (result.getMillis() / 1000);
  88     }


I think somehow pig is not converting the Long into the proper long type

in

the pig script.

ISOToUnixInSecond('$STARTDATETIME') AS startTime:long

Hence during the comparion, it treat the Long as a string value ...

On Wed, Dec 15, 2010 at 4:28 PM, John Hui<[email protected]>  wrote:


This is actually, please ignore the code section below, Thanks!

7 eventData = FOREACH flattenData GENERATE B1::event AS event,
B1::publication AS publication, B1::deviceType AS deviceType, B1::adID

AS

adID, B1::mcc AS mcc, B1::event_timestamp AS eventTimestamp:long,
ISOToUnixInSecond('$STARTDATETIME') AS startTime:long;
8
9 eventData = FILTER eventData BY (event == 'adImpression') AND
(eventTimesta    mp<= startTime);
10
11 DESCRIBE eventData;
12
13 B = GROUP eventData BY (event, publication, deviceType, adID, mcc);


On Wed, Dec 15, 2010 at 4:21 PM, John Hui<[email protected]>  wrote:


I am having a hard time getting comparison to work. I am comparing from
two long values but I keep on getting a cast long to String error

Backend error message
---------------------
java.lang.ClassCastException: java.lang.Long cannot be cast to

java.lang.String

      at java.lang.String.compareTo(String.java:92)
      at

org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.LTOrEqualToExpr.doComparison(LTOrEqualToExpr.java:152)

      at

org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.LTOrEqualToExpr.getNext(LTOrEqualToExpr.java:114)

      at

org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POAnd.getNext(POAnd.java:83)

      at

org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.getNext(POFilter.java:148)

      at

org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:272)

      at

org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:255)

      at

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:232)

      at

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:227)

      at

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:52)

      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:644)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:320)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at

org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)

      at org.apache.hadoop.mapred.Child.main(Child.java:211


this is my pig code

   7 eventData = FOREACH flattenData GENERATE B1::event AS event,
B1::publicatio    n AS publication, B1::deviceType AS deviceType,

B1::adID

AS adID, B1::mcc A    S mcc, B1::event_timestamp AS

eventTimestamp:long,

ISOToUnixInSecond('$STAR    TDATETIME') AS startTime:long;
   8   9 ILLUSTRATE eventData;
  10  11 eventData = FILTER eventData BY (startTime<= startTime);
  12 --eventData = FILTER eventData BY (event == 'adImpression') AND
(eventTimes    tamp<= startTime);
  13
  14 DESCRIBE eventData;

Here is the output

eventData: {event: chararray,publication: chararray,deviceType:

chararray,adID: chararray,mcc: chararray,eventTimestamp: long,startTime:
long}

Reply via email to