The JobClient is trying to download the profile output to the local directory. It seems like you don't have write permissions in the current working directory where you are running the JobClient. Please check that.
HTH. +Vinod Hortonworks Inc. http://hortonworks.com/ On Thu, Mar 8, 2012 at 3:13 PM, Leonardo Urbina <lurb...@mit.edu> wrote: > Does anyone have any idea how to solve this problem? Regardless of whether > I'm using plain HPROF or profiling through Starfish, I am getting the same > error: > > Exception in thread "main" java.io.FileNotFoundException: > attempt_201203071311_0004_m_ > 000000_0.profile (Permission denied) > at java.io.FileOutputStream.open(Native Method) > at java.io.FileOutputStream.<init>(FileOutputStream.java:194) > at java.io.FileOutputStream.<init>(FileOutputStream.java:84) > at > org.apache.hadoop.mapred.JobClient.downloadProfile(JobClient.java:1226) > at > org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1302) > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1251) > at > com.BitSight.hadoopAggregator.AggregatorDriver.run(AggregatorDriver.java:89) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at > com.BitSight.hadoopAggregator.AggregatorDriver.main(AggregatorDriver.java:94) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > > But I can't find what permissions to change to fix this issue. Any ideas? > Thanks in advance, > > Best, > -Leo > > > On Wed, Mar 7, 2012 at 3:52 PM, Leonardo Urbina <lurb...@mit.edu> wrote: > >> Thanks, >> -Leo >> >> >> On Wed, Mar 7, 2012 at 3:47 PM, Jie Li <ji...@cs.duke.edu> wrote: >> >>> Hi Leo, >>> >>> Thanks for pointing out the outdated README file. Glad to tell you that >>> we >>> do support the old API in the latest version. See here: >>> >>> http://www.cs.duke.edu/starfish/previous.html >>> >>> Welcome to join our mailing list and your questions will reach more of our >>> group members. >>> >>> Jie >>> >>> On Wed, Mar 7, 2012 at 3:37 PM, Leonardo Urbina <lurb...@mit.edu> wrote: >>> >>> > Hi Jie, >>> > >>> > According to the Starfish README, the hadoop programs must be written >>> using >>> > the new Hadoop API. This is not my case (I am using MultipleInputs among >>> > other non-new API supported features). Is there any way around this? >>> > Thanks, >>> > >>> > -Leo >>> > >>> > On Wed, Mar 7, 2012 at 3:19 PM, Jie Li <ji...@cs.duke.edu> wrote: >>> > >>> > > Hi Leonardo, >>> > > >>> > > You might want to try Starfish which supports the memory profiling as >>> > well >>> > > as cpu/disk/network profiling for the performance tuning. >>> > > >>> > > Jie >>> > > ------------------ >>> > > Starfish is an intelligent performance tuning tool for Hadoop. >>> > > Homepage: www.cs.duke.edu/starfish/ >>> > > Mailing list: http://groups.google.com/group/hadoop-starfish >>> > > >>> > > >>> > > On Wed, Mar 7, 2012 at 2:36 PM, Leonardo Urbina <lurb...@mit.edu> >>> wrote: >>> > > >>> > > > Hello everyone, >>> > > > >>> > > > I have a Hadoop job that I run on several GBs of data that I am >>> trying >>> > to >>> > > > optimize in order to reduce the memory consumption as well as >>> improve >>> > the >>> > > > speed. I am following the steps outlined in Tom White's "Hadoop: The >>> > > > Definitive Guide" for profiling using HPROF (p161), by setting the >>> > > > following properties in the JobConf: >>> > > > >>> > > > job.setProfileEnabled(true); >>> > > > >>> > > > >>> job.setProfileParams("-agentlib:hprof=cpu=samples,heap=sites,depth=6," >>> > + >>> > > > "force=n,thread=y,verbose=n,file=%s"); >>> > > > job.setProfileTaskRange(true, "0-2"); >>> > > > job.setProfileTaskRange(false, "0-2"); >>> > > > >>> > > > I am trying to run this locally on a single pseudo-distributed >>> install >>> > of >>> > > > hadoop (0.20.2) and it gives the following error: >>> > > > >>> > > > Exception in thread "main" java.io.FileNotFoundException: >>> > > > attempt_201203071311_0004_m_000000_0.profile (Permission denied) >>> > > > at java.io.FileOutputStream.open(Native Method) >>> > > > at java.io.FileOutputStream.<init>(FileOutputStream.java:194) >>> > > > at java.io.FileOutputStream.<init>(FileOutputStream.java:84) >>> > > > at >>> > > > >>> org.apache.hadoop.mapred.JobClient.downloadProfile(JobClient.java:1226) >>> > > > at >>> > > > >>> > > >>> > >>> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1302) >>> > > > at >>> > org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1251) >>> > > > at >>> > > > >>> > > > >>> > > >>> > >>> com.BitSight.hadoopAggregator.AggregatorDriver.run(AggregatorDriver.java:89) >>> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>> > > > at >>> > > > >>> > > > >>> > > >>> > >>> com.BitSight.hadoopAggregator.AggregatorDriver.main(AggregatorDriver.java:94) >>> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> > > > at >>> > > > >>> > > > >>> > > >>> > >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> > > > at >>> > > > >>> > > > >>> > > >>> > >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> > > > at java.lang.reflect.Method.invoke(Method.java:597) >>> > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>> > > > >>> > > > However, I can access these logs directly from the tasktracker's >>> logs >>> > > > (through the web UI). For the sakes of running this locally, I >>> could >>> > > just >>> > > > ignore this error, however I want to be able to profile the job once >>> > > > deployed to our hadoop cluster and need to be able to automatically >>> > > > retrieve these logs. Do I need to change the permissions in HDFS to >>> > allow >>> > > > for this? Any ideas on how to fix this? Thanks in advance, >>> > > > >>> > > > Best, >>> > > > -Leo >>> > > > >>> > > > -- >>> > > > Leo Urbina >>> > > > Massachusetts Institute of Technology >>> > > > Department of Electrical Engineering and Computer Science >>> > > > Department of Mathematics >>> > > > lurb...@mit.edu >>> > > > >>> > > >>> > >>> > >>> > >>> > -- >>> > Leo Urbina >>> > Massachusetts Institute of Technology >>> > Department of Electrical Engineering and Computer Science >>> > Department of Mathematics >>> > lurb...@mit.edu >>> > >>> >> >> >> >> -- >> Leo Urbina >> Massachusetts Institute of Technology >> Department of Electrical Engineering and Computer Science >> Department of Mathematics >> lurb...@mit.edu >> >> > > > -- > Leo Urbina > Massachusetts Institute of Technology > Department of Electrical Engineering and Computer Science > Department of Mathematics > lurb...@mit.edu
