Thanks Marcus and Zhu. I set them explicitly in my program for now. The config approach as suggested by Zhu makes sense and wil try that too.
Sid
On Thu, Jun 7, 2012 at 7:07 AM, GUOJUN Zhu <guojun_...@freddiemac.com>wrote:
>
> You can also define a new config file ("mymapred.xml"), and put all the
> properties you want to change there. Then you can do "hadoop --config
> mymapred.xml jar {yourjar} {classname}". Also, I add the extra properties
> with a space between -D and property name, "hadoop --config mymapred.xml
> jar {yourjar} {classname} -D mapred.create.symlink=yes .... {your
> program arguments}". Hadoop program is somewhat picky when interperate the
> arguments. In your runner, try to print out all the arguments, if the
> "-D..." goes there, you know that hadoop does not pick up those property
> override.
>
> Zhu, Guojun
> Modeling Sr Graduate
> 571-3824370
> guojun_...@freddiemac.com
> Financial Engineering
> Freddie Mac
>
>
> *Marcos Ortiz <mlor...@uci.cu>*
>
> 06/07/2012 09:16 AM
> Please respond to
> mapreduce-user@hadoop.apache.org
>
> To
> mapreduce-user@hadoop.apache.org
> cc
> Sid Kumar <sqlsid...@gmail.com>
> Subject
> Re: override mapred-site.xml from command line
>
>
>
>
>
>
> On 06/06/2012 07:44 PM, Sid Kumar wrote:
> I am able to set it via the API.
> Configuration.setBoolean(mapred.output.compress,true). This works!
>
> But the -D from the command line still doesn't work. Any idea what I may
> be missing here?
>
> Some additional info - Also when I try running the -D on command line on a
> local cluster (pseudo distributed mode) it works, but when I try it on a
> fully distributed cluster running jobs from a client machine it doesn't
> work. Is there a different way for setting it in this case - in hadoop-env
> perhaps?
>
> Thanks
> Sid
>
> On Wed, Jun 6, 2012 at 4:06 PM, Sid Kumar
> <*sqlsid...@gmail.com*<sqlsid...@gmail.com>>
> wrote:
> Mayank,
> I dont have a final tag for that property set. I looked at the
> mapred-default.xml in the src/mapred folder and that doesn't have a final
> tag too. Should I set it explicitly to false?
> You should do it explicitly.
> You should read the excellent blog post from Lars Francke where he did a
> great job explaining parameter by parameter and why is recommendable to set
> them to final.*
> **
> http://gbif.blogspot.com/2011/01/setting-up-hadoop-cluster-part-1-manual.html
> *<http://gbif.blogspot.com/2011/01/setting-up-hadoop-cluster-part-1-manual.html>
>
> Regards
>
> Sid
>
>
> On Wed, Jun 6, 2012 at 3:50 PM, Mayank Bansal
> <*may...@apache.org*<may...@apache.org>>
> wrote:
> Check your mapred site xml if these parameters have <final>true</final>
>
> making final to false should solve your problem.
>
>
> On Wed, Jun 6, 2012 at 3:41 PM, Sid Kumar
> <*sqlsid...@gmail.com*<sqlsid...@gmail.com>>
> wrote:
> Hi,
> I am trying to override mapred-site.xml (more specifically
> mapred.compress.map.output
> and mapred.output.compression.
> codec) from the command line when I
> execute the jar.
> I have been using hadoop jar <jarname> <class> -
> Dmapred.compress.map.output=true and
> -Dmapred.output.compression.codec=org.apache.hadoop.io.SnappyCodec
>
> The above doesnt work as the job.xml for the jar still uses the default
> properties and not the one i specify here. Is there a different approach to
> override these properties. I am submitting jobs from a client machine that
> has the same version of configuration files as my cluster.
>
> Thanks
>
> Sid
>
>
>
>
>
> --
> Marcos Luis OrtÃz Valmaseda
> Data Engineer && Sr. System Administrator at UCI
> *http://marcosluis2186.posterous.com*<http://marcosluis2186.posterous.com/>
> *http://www.linkedin.com/in/marcosluis2186*<http://www.linkedin.com/in/marcosluis2186>
> Twitter: @marcosluis2186
>
> <http://www.uci.cu/>
>
>
<<image/gif>>
