Last week I used the user "root" to start the Hadoop's dfs & mapreduce and
run the Embedded-Pig Java code. It was running fine.
This week I'd like to perform the same task by using a non-root
user: charlie.
After changing a few directories' user permission settings, now I'm able to
use the user "charlie" to start the Hadoop's dfs & mapreduce with no errors.
However, when I use the user "charlie" to run the Embedded-Pig Java code,
it keeps complaining the permission of the hadoop.tmp.dir that I set
to /opt/hdfs/tmp/ in core-stie.xml:
java.io.FileNotFoundException:
/opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (Permission
denied)
I've checked the permission for the following directories:
bash-3.2$ ls -lt /opt/hdfs/tmp
total 4
drwxr-xr-x 3 charlie comusers 4096 Apr 16 19:30 mapred
bash-3.2$ ls -lt /opt/hdfs/tmp/mapred
total 4
drwxr-xr-x 2 charlie comusers 4096 Apr 16 19:30 local
bash-3.2$ ls -lt /opt/hdfs/tmp/mapred/local
total 0
I need some direction about what goes wrong. I've googled those keywords
but found nothing. Any help would be appreciated!
I've attached the pig output as follows. Hope the info would be helpful.
12/04/16 19:31:28 INFO executionengine.HExecutionEngine: Connecting to
hadoop file system at: hdfs://hadoop-namenode:9000
12/04/16 19:31:29 INFO pigstats.ScriptState: Pig features used in the
script: HASH_JOIN,GROUP_BY,FILTER,CROSS
12/04/16 19:31:29 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
12/04/16 19:31:29 INFO mapReduceLayer.MRCompiler: File concatenation
threshold: 100 optimistic? false
12/04/16 19:31:30 INFO mapReduceLayer.CombinerOptimizer: Choosing to move
algebraic foreach to combiner
12/04/16 19:31:30 INFO
mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite:
POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO
mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite:
POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO
mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite:
POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO
mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite:
POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO
mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite:
POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO
mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite:
POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
before optimization: 11
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of
total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of
total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0
map-reduce splittees.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of
total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of
total 2 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of
total 2 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 1
map-reduce splittees.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 1 out of
total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
after optimization: 10
12/04/16 19:31:30 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:30 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:30 INFO pigstats.ScriptState: Pig script settings are added
to the job
12/04/16 19:31:30 WARN pigstats.ScriptState: unable to read pigs manifest
file
12/04/16 19:31:30 INFO mapReduceLayer.JobControlCompiler:
mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
12/04/16 19:31:35 INFO mapReduceLayer.JobControlCompiler: Setting up multi
store job
12/04/16 19:31:35 INFO mapReduceLayer.JobControlCompiler:
BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=957600
12/04/16 19:31:35 INFO mapReduceLayer.JobControlCompiler: Neither PARALLEL
nor default parallelism is set for this job. Setting number of reducers to 1
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
job(s) waiting for submission.
12/04/16 19:31:35 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO input.FileInputFormat: Total input paths to process
: 1
12/04/16 19:31:35 INFO util.MapRedUtil: Total input paths to process : 1
12/04/16 19:31:35 INFO util.MapRedUtil: Total input paths (combined) to
process : 1
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: 0% complete
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: job null has
failed! Stop running all dependent jobs
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: 100% complete
12/04/16 19:31:36 WARN mapReduceLayer.Launcher: There is no log file to
write to.
12/04/16 19:31:36 ERROR mapReduceLayer.Launcher: Backend error message
during job submission
java.io.FileNotFoundException:
/opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (Permission
denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:180)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:176)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:234)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:368)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:465)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:372)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1216)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:92)
at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:373)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:662)
12/04/16 19:31:36 ERROR pigstats.SimplePigStats: ERROR 2997: Unable to
recreate exception from backend error: java.io.FileNotFoundException:
/opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (Permission
denied)
12/04/16 19:31:36 ERROR pigstats.PigStatsUtil: 1 map reduce job(s) failed!
12/04/16 19:31:36 WARN pigstats.ScriptState: unable to read pigs manifest
file
12/04/16 19:31:36 INFO pigstats.SimplePigStats: Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2 charlie 2012-04-16 19:31:30 2012-04-16 19:31:36
HASH_JOIN,GROUP_BY,FILTER,CROSS
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
N/A events,events1,grouped MULTI_QUERY Message:
java.io.FileNotFoundException:
/opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (Permission
denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:180)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:176)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:234)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:368)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:465)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:372)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1216)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:92)
at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:373)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:662)
Input(s):
Failed to read data from
"/grapevine/analysis/recommendation/input/article_based/all_grapevine_events.txt"
Output(s):
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
null -> null,null,
null -> null,
null -> null,
null -> null,null,
null -> null,
null -> null,null,
null -> null,
null -> null,
null -> null,
null
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: Failed!
