Hi All, My PIG jobs are failing since yesterday which was completed successfully in the past. I would appreciate any pointers on the possible root cause. Here is the console log from the job and the dump of my environment values.
Thanks, Raman ================== grunt> STORE trec INTO '/apps/sq/ryakkala/trec_fp_us_clicks_desc1' USING PigStorage(); 2010-08-27 07:03:16,240 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for odl: $1, $2, $4, $5, $6, $8 2010-08-27 07:03:16,240 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for odl 2010-08-27 07:03:16,242 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for oil: $2, $3, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $51, $53, $55 2010-08-27 07:03:16,242 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for oil 2010-08-27 07:03:16,243 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for qif: $0, $2 2010-08-27 07:03:16,243 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for qif 2010-08-27 07:03:16,402 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: Store( hdfs://srwaishdc1nn0001/apps/sq/ryakkala/trec_fp_us_clicks_desc1:PigStorage) - 1-2213 Operator Key: 1-2213) 2010-08-27 07:03:16,442 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 4 time(s). 2010-08-27 07:03:16,446 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning MULTI_LEAF_MAP 2 time(s). 2010-08-27 07:03:16,461 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 5 2010-08-27 07:03:16,461 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 5 2010-08-27 07:03:16,507 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2010-08-27 07:03:17,861 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2010-08-27 07:03:17,916 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2010-08-27 07:03:19,150 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2010-08-27 07:03:19,191 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2010-08-27 07:03:20,395 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2010-08-27 07:03:20,421 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 3 map-reduce job(s) waiting for submission. 2010-08-27 07:03:20,424 [Thread-16] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2010-08-27 07:03:20,923 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2010-08-27 07:03:25,950 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8621/job.jar retrying... 2010-08-27 07:03:26,353 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8621/job.jar retrying... 2010-08-27 07:03:27,667 [Thread-16] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 7850 2010-08-27 07:03:54,399 [Thread-16] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2010-08-27 07:03:59,359 [Thread-16] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1300 2010-08-27 07:04:15,397 [Thread-16] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2010-08-27 07:04:20,525 [Thread-16] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 2010-08-27 07:04:20,574 [Thread-16] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 200 2010-08-27 07:04:26,184 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.split retrying... 2010-08-27 07:04:26,586 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.split retrying... 2010-08-27 07:04:26,988 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.split retrying... 2010-08-27 07:04:27,390 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.split retrying... 2010-08-27 07:04:27,792 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.split retrying... 2010-08-27 07:04:28,194 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.split retrying... 2010-08-27 07:04:28,596 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.split retrying... 2010-08-27 07:04:28,998 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.split retrying... 2010-08-27 07:04:34,438 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:34,840 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:35,242 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:35,644 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:36,046 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:36,448 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:36,850 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:37,252 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:37,654 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:38,056 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:38,458 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:38,860 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:39,262 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:39,664 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:40,066 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:40,468 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:40,870 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:41,272 [Thread-16] INFO org.apache.hadoop.hdfs.DFSClient - Could not complete file /tmp/hadoop-hadoop/mapred/system/job_201007221306_8623/job.xml retrying... 2010-08-27 07:04:48,911 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201007221306_8621 2010-08-27 07:04:48,911 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_8621 2010-08-27 07:04:48,911 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201007221306_8622 2010-08-27 07:04:48,911 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_8622 2010-08-27 07:04:48,911 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201007221306_8623 2010-08-27 07:04:48,911 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_8623 2010-08-27 07:04:48,914 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1% complete 2010-08-27 07:05:03,709 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 4% complete 2010-08-27 07:05:08,257 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 6% complete 2010-08-27 07:05:08,760 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 8% complete 2010-08-27 07:05:10,777 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete 2010-08-27 07:05:11,282 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 12% complete 2010-08-27 07:05:13,304 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 15% complete 2010-08-27 07:05:18,342 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 17% complete 2010-08-27 07:05:23,429 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 18% complete 2010-08-27 07:05:28,472 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 20% complete 2010-08-27 07:05:46,630 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 28% complete 2010-08-27 07:06:04,019 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 31% complete 2010-08-27 07:07:34,303 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 41% complete 2010-08-27 07:13:44,914 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 58% complete 2010-08-27 07:20:03,257 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 59% complete 2010-08-27 07:20:28,389 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2010-08-27 07:20:29,578 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2010-08-27 07:20:29,581 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2010-08-27 07:20:29,587 [Thread-44] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2010-08-27 07:20:29,954 [Thread-44] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 2000 2010-08-27 07:20:29,955 [Thread-44] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 2000 2010-08-27 07:20:31,040 [Thread-44] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 2000 2010-08-27 07:20:31,041 [Thread-44] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 2000 2010-08-27 07:20:40,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201007221306_8626 2010-08-27 07:20:40,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_8626 2010-08-27 07:22:15,269 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2010-08-27 07:22:16,468 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2010-08-27 07:22:16,473 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2010-08-27 07:22:16,478 [Thread-54] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2010-08-27 07:22:16,832 [Thread-54] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 2000 2010-08-27 07:22:16,833 [Thread-54] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 2000 2010-08-27 07:22:17,721 [Thread-54] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 2000 2010-08-27 07:22:17,721 [Thread-54] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 2000 2010-08-27 07:22:29,084 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201007221306_8627 2010-08-27 07:22:29,084 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_8627 2010-08-27 07:32:29,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 92% complete 2010-08-27 07:35:35,239 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2010-08-27 07:35:35,240 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map reduce job(s) failed! 2010-08-27 07:35:39,699 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed to produce result in: "hdfs://srwaishdc1nn0001/apps/sq/ryakkala/trec_fp_us_clicks_desc1" 2010-08-27 07:35:41,622 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Some jobs have failed! Stop running all dependent jobs grunt> grunt> grunt> grunt> grunt> quit [ryakk...@srwaishdc1gn0001 ~]$ env HOSTNAME=srwaishdc1gn0001 SHELL=/bin/bash TERM=vt100 HISTSIZE=1000 HADOOP_HOME=/usr/lib/hadoop SSH_CLIENT=10.254.24.100 2212 22 KDE_NO_IPV6=1 PIGDIR=/export/home/ryakkala/hadoop/pig-0.7.0 PIG_OPTS=-Dmapred.child.java.opts=-Xmx2048m SSH_TTY=/dev/pts/24 bis=/inbound/sq/bis USER=ryakkala LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: HADOOPSITEPATH=/etc/hadoop/conf/core-site.xml OOZIE_URL=http://localhost:8080/oozie SSH_AUTH_SOCK=/tmp/ssh-IzHeiw5668/agent.5668 KDEDIR=/usr PATH=/export/home/ryakkala/hadoop/pig-0.7.0/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/export/home/ryakkala/bin MAIL=/var/spool/mail/ryakkala PWD=/export/home/ryakkala INPUTRC=/etc/inputrc JAVA_HOME=/usr/java/jdk1.6.0_16/ KDE_IS_PRELINKED=1 LANG=en_US.UTF-8 PIG_CLASSPATH=/usr/lib/hadoop-0.20/lib/hadoop-lzo-0.4.3.jar:/etc/hadoop/conf HADOOP_CONF_DIR=/usr/lib/hadoop/conf SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass HOME=/export/home/ryakkala SHLVL=2 LOGNAME=ryakkala SSH_CONNECTION=10.254.24.100 2212 10.110.217.53 22 PIG_HEAPSIZE=30000 LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 _=/bin/env
