Here it is
Backend error message
---------------------
org.apache.pig.backend.executionengine.ExecException: ERROR 2106:
Error while computing max in Initial
at org.apache.pig.builtin.FloatMax$Initial.exec(FloatMax.java:84)
at org.apache.pig.builtin.FloatMax$Initial.exec(FloatMax.java:64)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:216)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:253)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:332)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:284)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:256)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:267)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:262)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassCastException: java.lang.String cannot be
cast to java.lang.Float
at org.apache.pig.builtin.FloatMax$Initial.exec(FloatMax.java:76)
... 19 more
Pig Stack Trace
---------------
ERROR 2997: Unable to recreate exception from backed error:
org.apache.pig.backend.executionengine.ExecException: ERROR 2106:
Error while computing max in Initial
org.apache.pig.backend.executionengine.ExecException: ERROR 2997:
Unable to recreate exception from backed error:
org.apache.pig.backend.executionengine.ExecException: ERROR 2106:
Error while computing max in Initial
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:343)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1314)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1299)
at org.apache.pig.PigServer.execute(PigServer.java:1286)
On Tue, Dec 20, 2011 at 2:44 PM, Dmitriy Ryaboy <[email protected]> wrote:
> Can you take a look at the failed map tasks and paste their stack traces?
>
> On Tue, Dec 20, 2011 at 2:40 PM, Grig Gheorghiu
> <[email protected]> wrote:
>> Thanks Dmitriy! I made some progress by using MAX without the
>> piggybank declaration, but I still got an error:
>>
>> 2011-12-20 22:35:25,832 [main] INFO org.apache.pig.Main - Logging
>> error messages to: /home/hadoop/pigscripts/pig_1324420525827.log
>> 2011-12-20 22:35:26,139 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> Connecting to hadoop file system at: hdfs://10.110.209.25:9000
>> 2011-12-20 22:35:26,615 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> Connecting to map-reduce job tracker at: 10.110.209.25:9001
>> 2011-12-20 22:35:27,815 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>> script: GROUP_BY,FILTER
>> 2011-12-20 22:35:29,109 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler
>> - File concatenation threshold: 100 optimistic? false
>> 2011-12-20 22:35:29,121 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer
>> - Choosing to move algebraic foreach to combiner
>> 2011-12-20 22:35:29,148 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> - MR plan size before optimization: 1
>> 2011-12-20 22:35:29,148 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> - MR plan size after optimization: 1
>> 2011-12-20 22:35:29,293 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
>> added to the job
>> 2011-12-20 22:35:29,308 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - mapred.job.reduce.markreset.buffer.percent is not set, set to
>> default 0.3
>> 2011-12-20 22:35:37,231 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - Setting up single store job
>> 2011-12-20 22:35:37,276 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=0
>> 2011-12-20 22:35:37,276 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - Neither PARALLEL nor default parallelism is set for this job.
>> Setting number of reducers to 1
>> 2011-12-20 22:35:37,328 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 1 map-reduce job(s) waiting for submission.
>> 2011-12-20 22:35:37,345 [Thread-5] INFO
>> org.apache.hadoop.mapred.JobClient - Default number of map tasks: null
>> 2011-12-20 22:35:37,345 [Thread-5] INFO
>> org.apache.hadoop.mapred.JobClient - Setting default number of map
>> tasks based on cluster size to : 32
>> 2011-12-20 22:35:37,346 [Thread-5] INFO
>> org.apache.hadoop.mapred.JobClient - Default number of reduce tasks: 1
>> 2011-12-20 22:35:37,829 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 0% complete
>> 2011-12-20 22:35:38,083 [Thread-5] INFO
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input
>> paths to process : 1
>> 2011-12-20 22:35:38,083 [Thread-5] INFO
>> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
>> input paths to process : 1
>> 2011-12-20 22:35:38,093 [Thread-5] INFO
>> com.hadoop.compression.lzo.GPLNativeCodeLoader - Loaded native gpl
>> library
>> 2011-12-20 22:35:38,095 [Thread-5] WARN
>> com.hadoop.compression.lzo.LzoCodec - Could not find build properties
>> file with revision hash
>> 2011-12-20 22:35:38,095 [Thread-5] INFO
>> com.hadoop.compression.lzo.LzoCodec - Successfully loaded &
>> initialized native-lzo library [hadoop-lzo rev UNKNOWN]
>> 2011-12-20 22:35:38,102 [Thread-5] WARN
>> org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native
>> library is available
>> 2011-12-20 22:35:38,102 [Thread-5] INFO
>> org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native
>> library loaded
>> 2011-12-20 22:35:38,105 [Thread-5] INFO
>> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
>> input paths (combined) to process : 1
>> 2011-12-20 22:35:38,921 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - HadoopJobId: job_201112192006_0026
>> 2011-12-20 22:35:38,922 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - More information at:
>> http://10.110.209.25:9100/jobdetails.jsp?jobid=job_201112192006_0026
>> 2011-12-20 22:36:28,616 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - job job_201112192006_0026 has failed! Stop running all dependent
>> jobs
>> 2011-12-20 22:36:28,616 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 100% complete
>> 2011-12-20 22:36:28,633 [main] ERROR
>> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
>> recreate exception from backed error:
>> org.apache.pig.backend.executionengine.ExecException: ERROR 2106:
>> Error while computing max in Initial
>> 2011-12-20 22:36:28,633 [main] ERROR
>> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s)
>> failed!
>> 2011-12-20 22:36:28,635 [main] INFO
>> org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
>>
>> HadoopVersion PigVersion UserId StartedAt FinishedAt
>> Features
>> 0.20.205 0.9.1-amzn hadoop 2011-12-20 22:35:29 2011-12-20
>> 22:36:28 GROUP_BY,FILTER
>>
>> Failed!
>>
>> Failed Jobs:
>> JobId Alias Feature Message Outputs
>> job_201112192006_0026
>> AGGREGATES,DATE_URL,FILTERED_DATE_URL,GROUP_BY_DATE_URL,LOGS_BASE,RAW_LOGS
>> GROUP_BY,COMBINER Message:
>> Job failed! Error - # of failed Map Tasks exceeded allowed limit.
>> FailedCount: 1. LastFailedTask:
>> task_201112192006_0026_m_000000
>> s3://mapreduce.bucket/nginx/pigoutput/nginx_201112202235,
>>
>> Input(s):
>> Failed to read data from "s3://mapreduce.bucket/nginx/*test*.gz"
>>
>> Output(s):
>> Failed to produce result in
>> "s3://mapreduce.bucket/nginx/pigoutput/nginx_201112202235"
>>
>> Counters:
>> Total records written : 0
>> Total bytes written : 0
>> Spillable Memory Manager spill count : 0
>> Total bags proactively spilled: 0
>> Total records proactively spilled: 0
>>
>> Job DAG:
>> job_201112192006_0026
>>
>>
>> 2011-12-20 22:36:28,635 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - Failed!
>> 2011-12-20 22:36:29,181 [main] ERROR
>> org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Unable to
>> recreate exception from backed error:
>> org.apache.pig.backend.executionengine.ExecException: ERROR 2106:
>> Error while computing max in Initial
>> Details at logfile: /home/hadoop/pigscripts/pig_1324420525827.log
>>
>> On Tue, Dec 20, 2011 at 2:32 PM, Dmitriy Ryaboy <[email protected]> wrote:
>>> Try just using MAX (no piggybank). If that doesn't work, try FloatMax
>>> (still no piggybank).
>>>
>>>
>>> On Tue, Dec 20, 2011 at 2:19 PM, Grig Gheorghiu
>>> <[email protected]> wrote:
>>>> Hello,
>>>>
>>>> Noob here. I am trying to analyze some Nginx log files and get some
>>>> aggregate stats based on date and URL. Here is the beginning of a Pig
>>>> script I have (I am running this in Elastic MapReduce, with Pig
>>>> 0.9.1):
>>>>
>>>>
>>>> REGISTER file:/home/hadoop/lib/pig/piggybank.jar;
>>>> DEFINE EXTRACT org.apache.pig.piggybank.evaluation.string.EXTRACT();
>>>> DEFINE MAX org.apache.pig.piggybank.evaluation.math.MAX();
>>>> RAW_LOGS = LOAD '$INPUT' as (line:chararray);
>>>> LOGS_BASE = FOREACH RAW_LOGS GENERATE
>>>>
>>>> FLATTEN(
>>>> EXTRACT(line, '(\\S+) - -
>>>> \\[([^/]+)\\/([^/]+)\\/(\\d+):(\\d+):(\\d+):(\\d+)
>>>> ([+-]\\d+)\\]\\s+"([^"]+)"\\s+(\\d+)\\s+(\\d+)\\s+"([^"]+)"\\s+"([^"]+)"\\s+"([^"]+)"\\s+(\\S+)')
>>>> )
>>>> AS (
>>>> ip: chararray,
>>>> day: chararray,
>>>> month: chararray,
>>>> year: chararray,
>>>> hour: chararray,
>>>> minute: chararray,
>>>> second: chararray,
>>>> tzoffset: chararray,
>>>> url: chararray,
>>>> status: chararray,
>>>> bytes: chararray,
>>>> referrer: chararray,
>>>> useragent: chararray,
>>>> xfwd: chararray,
>>>> reqtime: float
>>>> );
>>>> DATE_URL = FOREACH LOGS_BASE GENERATE year, month, day, url, reqtime;
>>>> FILTERED_DATE_URL = FILTER DATE_URL BY NOT url IS NULL;
>>>> GROUP_BY_DATE_URL = GROUP FILTERED_DATE_URL BY (year, month, day, url);
>>>>
>>>> Now I would like to get the MAX of the request time. If I do this:
>>>>
>>>> AGGREGATES = FOREACH GROUP_BY_DATE_URL GENERATE FLATTEN(group) as
>>>> (year, month, day, url), MAX(FILTERED_DATE_URL.reqtime) as maxreqtime;
>>>> STORE AGGREGATES INTO '$OUTPUT';
>>>>
>>>> I get this error
>>>>
>>>> 2011-12-20 22:16:32,147 [main] ERROR org.apache.pig.tools.grunt.Grunt
>>>> - ERROR 1045:
>>>> <file /home/hadoop/pigscripts/nginx_access_log.pig, line 31, column
>>>> 91> Could not infer the matching function for
>>>> org.apache.pig.piggybank.evaluation.math.MAX as multiple or none of
>>>> them fit. Please use an explicit cast.
>>>>
>>>> If I do this cast:
>>>>
>>>> AGGREGATES = FOREACH GROUP_BY_DATE_URL GENERATE FLATTEN(group) as
>>>> (year, month, day, url), MAX((float)FILTERED_DATE_URL.reqtime) as
>>>> maxreqtime;
>>>>
>>>> I get another error:
>>>>
>>>> 2011-12-20 22:18:35,115 [main] ERROR org.apache.pig.tools.grunt.Grunt
>>>> - ERROR 1052:
>>>> <file /home/hadoop/pigscripts/nginx_access_log.pig, line 31, column
>>>> 96> Cannot cast bag with schema :bag{:tuple(reqtime:float)} to float
>>>>
>>>> Any help would be greatly appreciated.
>>>>
>>>> Grig
