Hi Daniel, >From the console output, I can see Pig ran against Hadoop 1.x as root while it ran against Hadoop 2.x as watrous. I don't know what your setup is like, but please double check what version of Hadoop jars are present in classpath. Running "pig -secretDebugCmd" should show HADOOP_CLASSPATH.
Apparently, your Pig jar is compiled against Hadoop 1.x, so it only works against Hadoop 1.x (as root). Thanks, Cheolsoo On Wed, Nov 27, 2013 at 11:01 AM, Watrous, Daniel <[email protected]>wrote: > Hi, > > I've been following the book Hadoop Definitive Guide and a few other > resources while exploring PIG. I am able to run as root, but not as a > regular user. I am running in local mode. I've found a few links, but they > haven't helped me solve my issue. The most relevant seems to be: > http://answers.mapr.com/questions/5135/cannot-run-pig-in-local-mode > > My first experiment is just to load in a text file and dump it to the > console. The files exist and are owned by the user running PIG. I show the > failed attempt as a regular user followed by a successful run as root on > the same system. Any help is appreciated. > > Here's what I'm trying: > [watrous@c0003913 ~]$ pig -x local > 2013-11-27 18:53:18,691 [main] INFO org.apache.pig.Main - Apache Pig > version 0.12.0 (r1529718) compiled Oct 07 2013, 12:20:14 > 2013-11-27 18:53:18,692 [main] INFO org.apache.pig.Main - Logging error > messages to: /home/watrous/pig_1385578398689.log > 2013-11-27 18:53:18,747 [main] INFO org.apache.pig.impl.util.Utils - > Default bootup file /home/watrous/.pigbootup not found > 2013-11-27 18:53:19,025 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > deprecated. Instead, use fs.defaultFS > 2013-11-27 18:53:19,025 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is > deprecated. Instead, use mapreduce.jobtracker.address > 2013-11-27 18:53:19,027 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting to hadoop file system at: file:/// > 2013-11-27 18:53:19,031 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.used.genericoptionsparser is deprecated. Instead, use > mapreduce.client.genericoptionsparser.used > 2013-11-27 18:53:19,451 [main] WARN > org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 2013-11-27 18:53:19,517 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum > is deprecated. Instead, use dfs.bytes-per-checksum > 2013-11-27 18:53:19,519 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > deprecated. Instead, use fs.defaultFS > grunt> ls /home/watrous/input/ > file:/home/watrous/input/sample.txt<r 1> 530 > grunt> cat /home/watrous/input/sample.txt > > 0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999 > > 0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999 > > 0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999 > > 0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999 > > 0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999 > grunt> records = load '/home/watrous/input/sample.txt'; > 2013-11-27 18:53:25,086 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - job.end.retry.interval > is deprecated. Instead, use mapreduce.job.end-notification.retry.interval > 2013-11-27 18:53:25,086 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.tracker.retiredjobs.cache.size is deprecated. Instead, use > mapreduce.jobtracker.retiredjobs.cache.size > 2013-11-27 18:53:25,086 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.task.profile.reduces is deprecated. Instead, use > mapreduce.task.profile.reduces > 2013-11-27 18:53:25,086 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.reuse.jvm.num.tasks is deprecated. Instead, use > mapreduce.job.jvm.numtasks > 2013-11-27 18:53:25,086 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.reduce.tasks.speculative.execution is deprecated. Instead, use > mapreduce.reduce.speculative > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.task.tracker.http.address is deprecated. Instead, use > mapreduce.tasktracker.http.address > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.task.tracker.report.address is deprecated. Instead, use > mapreduce.tasktracker.report.address > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.healthChecker.interval is deprecated. Instead, use > mapreduce.tasktracker.healthchecker.interval > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.child.tmp is > deprecated. Instead, use mapreduce.task.tmp.dir > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.tasktracker.taskmemorymanager.monitoring-interval is deprecated. > Instead, use mapreduce.tasktracker.taskmemorymanager.monitoringinterval > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.shuffle.connect.timeout is deprecated. Instead, use > mapreduce.reduce.shuffle.connect.timeout > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.speculative.execution.speculativeCap is deprecated. Instead, use > mapreduce.job.speculative.speculativecap > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - io.sort.spill.percent > is deprecated. Instead, use mapreduce.map.sort.spill.percent > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.shuffle.input.buffer.percent is deprecated. Instead, use > mapreduce.reduce.shuffle.input.buffer.percent > 2013-11-27 18:53:25,087 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.skip.map.max.skip.records is deprecated. Instead, use > mapreduce.map.skip.maxrecords > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.task.profile.maps is deprecated. Instead, use > mapreduce.task.profile.maps > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.merge.recordsBeforeProgress is deprecated. Instead, use > mapreduce.task.merge.progress.records > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.tracker.http.address is deprecated. Instead, use > mapreduce.jobtracker.http.address > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.compress.map.output is deprecated. Instead, use > mapreduce.map.output.compress > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.userlog.retain.hours is deprecated. Instead, use > mapreduce.job.userlog.retain.hours > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.speculative.execution.slowNodeThreshold is deprecated. Instead, use > mapreduce.job.speculative.slownodethreshold > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.tasktracker.reduce.tasks.maximum is deprecated. Instead, use > mapreduce.tasktracker.reduce.tasks.maximum > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.jobtracker.restart.recover is deprecated. Instead, use > mapreduce.jobtracker.restart.recover > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.reduce.child.log.level is deprecated. Instead, use > mapreduce.reduce.log.level > 2013-11-27 18:53:25,088 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.inmem.merge.threshold is deprecated. Instead, use > mapreduce.reduce.merge.inmem.threshold > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks is > deprecated. Instead, use mapreduce.job.reduces > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.acls.enabled is > deprecated. Instead, use mapreduce.cluster.acls.enabled > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.tasktracker.dns.nameserver is deprecated. Instead, use > mapreduce.tasktracker.dns.nameserver > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.submit.replication is deprecated. Instead, use > mapreduce.client.submit.file.replication > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.map.tasks.speculative.execution is deprecated. Instead, use > mapreduce.map.speculative > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.map.child.log.level is deprecated. Instead, use > mapreduce.map.log.level > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.map.max.attempts > is deprecated. Instead, use mapreduce.map.maxattempts > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.shuffle.merge.percent is deprecated. Instead, use > mapreduce.reduce.shuffle.merge.percent > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.tracker.jobhistory.lru.cache.size is deprecated. Instead, use > mapreduce.jobtracker.jobhistory.lru.cache.size > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.job.queue.nameis > deprecated. Instead, use mapreduce.job.queuename > 2013-11-27 18:53:25,089 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.tracker.persist.jobstatus.hours is deprecated. Instead, use > mapreduce.jobtracker.persist.jobstatus.hours > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.healthChecker.script.timeout is deprecated. Instead, use > mapreduce.tasktracker.healthchecker.script.timeout > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.tasktracker.map.tasks.maximum is deprecated. Instead, use > mapreduce.tasktracker.map.tasks.maximum > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use > mapreduce.reduce.markreset.buffer.percent > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > jobclient.completion.poll.interval is deprecated. Instead, use > mapreduce.client.completion.pollinterval > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.tracker.persist.jobstatus.dir is deprecated. Instead, use > mapreduce.jobtracker.persist.jobstatus.dir > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.reduce.slowstart.completed.maps is deprecated. Instead, use > mapreduce.job.reduce.slowstart.completedmaps > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - io.sort.mb is > deprecated. Instead, use mapreduce.task.io.sort.mb > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - dfs.umaskmode is > deprecated. Instead, use fs.permissions.umask-mode > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.jobtracker.instrumentation is deprecated. Instead, use > mapreduce.jobtracker.instrumentation > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > topology.node.switch.mapping.impl is deprecated. Instead, use > net.topology.node.switch.mapping.impl > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.output.compression.type is deprecated. Instead, use > mapreduce.output.fileoutputformat.compress.type > 2013-11-27 18:53:25,090 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.skip.attempts.to.start.skipping is deprecated. Instead, use > mapreduce.task.skip.start.attempts > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.task.tracker.task-controller is deprecated. Instead, use > mapreduce.tasktracker.taskcontroller > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.userlog.limit.kb > is deprecated. Instead, use mapreduce.task.userlog.limit.kb > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use > mapreduce.job.committer.setup.cleanup.needed > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.map.tasks is > deprecated. Instead, use mapreduce.job.maps > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.local.dir.minspacekill is deprecated. Instead, use > mapreduce.tasktracker.local.dir.minspacekill > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapreduce.jobtracker.split.metainfo.maxsize is deprecated. Instead, use > mapreduce.job.split.metainfo.maxsize > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > jobclient.progress.monitor.poll.interval is deprecated. Instead, use > mapreduce.client.progressmonitor.pollinterval > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.min.split.size > is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.output.compression.codec is deprecated. Instead, use > mapreduce.output.fileoutputformat.compress.codec > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile is > deprecated. Instead, use mapreduce.task.profile > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.reduce.parallel.copies is deprecated. Instead, use > mapreduce.reduce.shuffle.parallelcopies > 2013-11-27 18:53:25,091 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - io.sort.factor is > deprecated. Instead, use mapreduce.task.io.sort.factor > 2013-11-27 18:53:25,092 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.task.timeout is > deprecated. Instead, use mapreduce.task.timeout > 2013-11-27 18:53:25,092 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.heartbeats.in.second is deprecated. Instead, use > mapreduce.jobtracker.heartbeats.in.second > 2013-11-27 18:53:25,092 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.map.output.compression.codec is deprecated. Instead, use > mapreduce.map.output.compress.codec > 2013-11-27 18:53:25,092 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.task.cache.levels is deprecated. Instead, use > mapreduce.jobtracker.taskcache.levels > 2013-11-27 18:53:25,092 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.tasktracker.dns.interface is deprecated. Instead, use > mapreduce.tasktracker.dns.interface > 2013-11-27 18:53:25,092 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.max.tracker.failures is deprecated. Instead, use > mapreduce.job.maxtaskfailures.per.tracker > 2013-11-27 18:53:25,092 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - dfs.df.interval is > deprecated. Instead, use fs.df.interval > 2013-11-27 18:53:25,092 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.tasktracker.tasks.sleeptime-before-sigkill is deprecated. Instead, > use mapreduce.tasktracker.tasks.sleeptimebeforesigkill > 2013-11-27 18:53:25,092 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.max.tracker.blacklists is deprecated. Instead, use > mapreduce.jobtracker.tasktracker.maxblacklists > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - jobclient.output.filter > is deprecated. Instead, use mapreduce.client.output.filter > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.jobtracker.taskScheduler is deprecated. Instead, use > mapreduce.jobtracker.taskscheduler > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - job.end.retry.attempts > is deprecated. Instead, use mapreduce.job.end-notification.retry.attempts > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.speculative.execution.slowTaskThreshold is deprecated. Instead, use > mapreduce.job.speculative.slowtaskthreshold > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.tasktracker.indexcache.mb is deprecated. Instead, use > mapreduce.tasktracker.indexcache.mb > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > tasktracker.http.threads is deprecated. Instead, use > mapreduce.tasktracker.http.threads > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.tracker.handler.count is deprecated. Instead, use > mapreduce.jobtracker.handler.count > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - keep.failed.task.files > is deprecated. Instead, use mapreduce.task.files.preserve.failedtasks > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress > is deprecated. Instead, use mapreduce.output.fileoutputformat.compress > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.jobtracker.job.history.block.size is deprecated. Instead, use > mapreduce.jobtracker.jobhistory.block.size > 2013-11-27 18:53:25,093 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.skip.reduce.max.skip.groups is deprecated. Instead, use > mapreduce.reduce.skip.maxgroups > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > topology.script.number.args is deprecated. Instead, use > net.topology.script.number.args > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.local.dir.minspacestart is deprecated. Instead, use > mapreduce.tasktracker.local.dir.minspacestart > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.jobtracker.maxtasks.per.job is deprecated. Instead, use > mapreduce.jobtracker.maxtasks.perjob > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.reduce.max.attempts is deprecated. Instead, use > mapreduce.reduce.maxattempts > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.shuffle.read.timeout is deprecated. Instead, use > mapreduce.reduce.shuffle.read.timeout > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.tasktracker.instrumentation is deprecated. Instead, use > mapreduce.tasktracker.instrumentation > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.tasktracker.expiry.interval is deprecated. Instead, use > mapreduce.jobtracker.expire.trackers.interval > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.tracker.persist.jobstatus.active is deprecated. Instead, use > mapreduce.jobtracker.persist.jobstatus.active > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.temp.dir is > deprecated. Instead, use mapreduce.cluster.temp.dir > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - hadoop.native.lib is > deprecated. Instead, use io.native.lib.available > 2013-11-27 18:53:25,094 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapreduce.job.counters.limit is deprecated. Instead, use > mapreduce.job.counters.max > 2013-11-27 18:53:25,095 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.reduce.input.buffer.percent is deprecated. Instead, use > mapreduce.reduce.input.buffer.percent > grunt> dump records; > 2013-11-27 18:53:31,060 [main] INFO > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > script: UNKNOWN > 2013-11-27 18:53:31,118 [main] INFO > org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - > {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, > DuplicateForEachColumnRewrite, GroupByConstParallelSetter, > ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, > MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, > PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], > RULES_DISABLED=[FilterLogicExpressionSimplifier]} > 2013-11-27 18:53:31,234 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > File concatenation threshold: 100 optimistic? false > 2013-11-27 18:53:31,292 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size before optimization: 1 > 2013-11-27 18:53:31,292 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size after optimization: 1 > 2013-11-27 18:53:31,322 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - session.id is > deprecated. Instead, use dfs.metrics.session-id > 2013-11-27 18:53:31,323 [main] INFO > org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with > processName=JobTracker, sessionId= > 2013-11-27 18:53:31,367 [main] WARN > org.apache.pig.backend.hadoop20.PigJobControl - falling back to default > JobControl (not using hadoop 0.20 ?) > java.lang.NoSuchFieldException: runnerState > at java.lang.Class.getDeclaredField(Class.java:1899) > at > org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51) > at > org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:98) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:287) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:190) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1322) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307) > at org.apache.pig.PigServer.storeEx(PigServer.java:978) > at org.apache.pig.PigServer.store(PigServer.java:942) > at org.apache.pig.PigServer.openIterator(PigServer.java:855) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) > at org.apache.pig.Main.run(Main.java:541) > at org.apache.pig.Main.main(Main.java:156) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > 2013-11-27 18:53:31,371 [main] INFO > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added > to the job > 2013-11-27 18:53:31,378 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 > 2013-11-27 18:53:31,449 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - Setting up single store job > 2013-11-27 18:53:31,456 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - Map only job, skipping reducer estimation > 2013-11-27 18:53:31,499 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 1 map-reduce job(s) waiting for submission. > 2013-11-27 18:53:31,515 [JobControl] INFO > org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics > with processName=JobTracker, sessionId= - already initialized > 2013-11-27 18:53:31,524 [JobControl] ERROR > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while trying > to run jobs. > java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186) > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) > at > org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) > at > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240) > at > org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121) > at java.lang.Thread.run(Thread.java:679) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270) > 2013-11-27 18:53:31,525 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 0% complete > 2013-11-27 18:53:31,533 [main] WARN > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to > stop immediately on failure. > 2013-11-27 18:53:31,533 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - job null has failed! Stop running all dependent jobs > 2013-11-27 18:53:31,534 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 100% complete > 2013-11-27 18:53:31,545 [main] ERROR > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to > recreate exception from backend error: Unexpected System Error Occured: > java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186) > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) > at > org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) > at > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240) > at > org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121) > at java.lang.Thread.run(Thread.java:679) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270) > > 2013-11-27 18:53:31,546 [main] ERROR > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed! > 2013-11-27 18:53:31,546 [main] INFO > org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats > reported below may be incomplete > 2013-11-27 18:53:31,547 [main] INFO > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: > > HadoopVersion PigVersion UserId StartedAt FinishedAt > Features > 2.2.0 0.12.0 watrous 2013-11-27 18:53:31 2013-11-27 18:53:31 > UNKNOWN > > Failed! > > Failed Jobs: > JobId Alias Feature Message Outputs > N/A records MAP_ONLY Message: Unexpected System Error Occured: > java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186) > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) > at > org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) > at > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240) > at > org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121) > at java.lang.Thread.run(Thread.java:679) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270) > file:/tmp/temp-884153250/tmp-1708053986, > > Input(s): > Failed to read data from "/home/watrous/input/sample.txt" > > Output(s): > Failed to produce result in "file:/tmp/temp-884153250/tmp-1708053986" > > Job DAG: > null > > > 2013-11-27 18:53:31,547 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Failed! > 2013-11-27 18:53:31,553 [main] ERROR org.apache.pig.tools.grunt.Grunt - > ERROR 1066: Unable to open iterator for alias records > Details at logfile: /home/watrous/pig_1385578398689.log > grunt> > > The exact same process works fine it I'm root: > [watrous@c0003913 ~]$ sudo su > [root@c0003913 watrous]# pig -x local > which: no hadoop in > (/sbin:/bin:/usr/sbin:/usr/bin:/opt/mtp/bin:/home/watrous/pig-0.12.0/bin) > 2013-11-27 18:58:34,166 [main] INFO org.apache.pig.Main - Apache Pig > version 0.12.0 (r1529718) compiled Oct 07 2013, 12:20:14 > 2013-11-27 18:58:34,167 [main] INFO org.apache.pig.Main - Logging error > messages to: /home/watrous/pig_1385578714163.log > 2013-11-27 18:58:34,184 [main] INFO org.apache.pig.impl.util.Utils - > Default bootup file /root/.pigbootup not found > 2013-11-27 18:58:34,321 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting to hadoop file system at: file:/// > grunt> records = load '/home/watrous/input/sample.txt'; > grunt> dump records; > 2013-11-27 18:59:17,019 [main] INFO > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > script: UNKNOWN > 2013-11-27 18:59:17,082 [main] INFO > org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - > {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, > DuplicateForEachColumnRewrite, GroupByConstParallelSetter, > ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, > MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, > PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], > RULES_DISABLED=[FilterLogicExpressionSimplifier]} > 2013-11-27 18:59:17,185 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > File concatenation threshold: 100 optimistic? false > 2013-11-27 18:59:17,236 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size before optimization: 1 > 2013-11-27 18:59:17,237 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size after optimization: 1 > 2013-11-27 18:59:17,278 [main] INFO > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added > to the job > 2013-11-27 18:59:17,301 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 > 2013-11-27 18:59:17,338 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - Setting up single store job > 2013-11-27 18:59:17,344 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - Map only job, skipping reducer estimation > 2013-11-27 18:59:17,375 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 1 map-reduce job(s) waiting for submission. > 2013-11-27 18:59:17,387 [JobControl] WARN > org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 2013-11-27 18:59:17,392 [JobControl] WARN > org.apache.hadoop.mapred.JobClient - No job jar file set. User classes > may not be found. See JobConf(Class) or JobConf#setJar(String). > 2013-11-27 18:59:17,450 [JobControl] INFO > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths > to process : 1 > 2013-11-27 18:59:17,450 [JobControl] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total > input paths to process : 1 > 2013-11-27 18:59:17,459 [JobControl] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total > input paths (combined) to process : 1 > 2013-11-27 18:59:17,812 [Thread-3] INFO > org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0 > 2013-11-27 18:59:17,862 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Using ResourceCalculatorPlugin : > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@79ca209e > 2013-11-27 18:59:17,877 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - HadoopJobId: job_local_0001 > 2013-11-27 18:59:17,877 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Processing aliases records > 2013-11-27 18:59:17,877 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - detailed locations: M: records[1,10] C: R: > 2013-11-27 18:59:17,884 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 0% complete > 2013-11-27 18:59:17,921 [Thread-3] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader > - Current split being processed file:/home/watrous/input/sample.txt:0+530 > 2013-11-27 18:59:17,979 [Thread-3] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map > - Aliases being processed per job phase (AliasName[line,offset]): M: > records[1,10] C: R: > 2013-11-27 18:59:17,995 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Task:attempt_local_0001_m_000000_0 is done. And is in the process of > commiting > 2013-11-27 18:59:18,000 [Thread-3] INFO > org.apache.hadoop.mapred.LocalJobRunner - > 2013-11-27 18:59:18,001 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Task attempt_local_0001_m_000000_0 is allowed to commit now > 2013-11-27 18:59:18,004 [Thread-3] INFO > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output > of task 'attempt_local_0001_m_000000_0' to > file:/tmp/temp-1635315632/tmp501386857 > 2013-11-27 18:59:20,734 [Thread-3] INFO > org.apache.hadoop.mapred.LocalJobRunner - > 2013-11-27 18:59:20,751 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Task 'attempt_local_0001_m_000000_0' done. > 2013-11-27 18:59:20,893 [main] WARN > org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for > job job_local_0001 > 2013-11-27 18:59:20,896 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 100% complete > 2013-11-27 18:59:20,896 [main] INFO > org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats > reported below may be incomplete > 2013-11-27 18:59:20,898 [main] INFO > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: > > HadoopVersion PigVersion UserId StartedAt FinishedAt > Features > 1.0.0 0.12.0 root 2013-11-27 18:59:17 2013-11-27 18:59:20 > UNKNOWN > > Success! > > Job Stats (time in seconds): > JobId Alias Feature Outputs > job_local_0001 records MAP_ONLY > file:/tmp/temp-1635315632/tmp501386857, > > Input(s): > Successfully read records from: "/home/watrous/input/sample.txt" > > Output(s): > Successfully stored records in: "file:/tmp/temp-1635315632/tmp501386857" > > Job DAG: > job_local_0001 > > > 2013-11-27 18:59:20,905 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Success! > 2013-11-27 18:59:20,908 [main] WARN > org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already > been initialized > 2013-11-27 18:59:20,911 [main] INFO > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths > to process : 1 > 2013-11-27 18:59:20,911 [main] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total > input paths to process : 1 > > (0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999) > > (0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999) > > (0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999) > > (0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999) > > (0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999) > grunt> >
