:50 PM, Adaryl "Bob" Wakefield, MBA <
adaryl.wakefi...@hotmail.com> wrote:
> What is the rule for determining how many nodes should be in your
> initial cluster?
> B.
>
--
Thanks
Devaraj K
wrote:
> I read a bit on documentation on yarn memory tuning and found that
>
> It is suggested to set mapreduce.map.java.opts = 0.8 *
> mapreduce.map.memory.mb.
>
> I am wondering why is 0.8, but not 0.9 or higher?
>
> --
> Chen Song
>
>
--
Thanks
Devaraj K
(TextInputFormat.class);
>
> job.setOutputFormatClass(TextOutputFormat.class);
>
>
>
> FileInputFormat.addInputPath(job, new Path(args[0]));
>
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
>
>
>
>
> job.submit();
>
>
>
> job.waitForCompletion(true);
>
>
>
>
>
> }
>
>
>
> }
>
>
>
>
>
>
>
> Regards,
>
> Chhaya Vishwakarma
>
>
>
> --
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>
--
Thanks
Devaraj K
her two stay in accepted status,why ,the default queue
> only 50% capacity used,i do not know why?
>
>
> yarn.resourcemanager.scheduler.class
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
>
>
--
Thanks
Devaraj K
Your Job is running in local mode, that's why you don't see in the RM UI/Job
History.
Can you change 'mapreduce.framework.name' configuration value to 'yarn', it
will show in RM UI.
Thanks
Devaraj k
From: ch huang [mailto:justlo...@gmail.com]
Sent: 11
It seems filesystem is closed in the Task. Do you see any error in
Datanode/Namenode(HDFS) log files?
Thanks
Devaraj k
From: heyamin [mailto:heya...@jiandan100.cn]
Sent: 11 September 2013 14:28
To: user@hadoop.apache.org
Cc: user-unsubscr...@hadoop.apache.org
Subject: help!!!,what is happened
You can find the Application Master/Job History link for each application in RM
web UI. The default port for RM Web UI is 8088. From the Application Master/Job
History UI, you can find the tasks status/progress.
Thanks
Devaraj k
From: ch huang [mailto:justlo...@gmail.com]
Sent: 11 September
help you better..
Thanks
Devaraj k
From: Rahul Bhattacharjee [mailto:rahul.rec@gmail.com]
Sent: 06 September 2013 09:43
To: user@hadoop.apache.org
Subject: Re: Question related to resource allocation in Yarn!
I could progress a bit on this.
I was not setting responseId while asking for co
Please send a mail to
user-unsubscr...@hadoop.apache.org<mailto:user-unsubscr...@hadoop.apache.org>
for unsubscribe.
Thanks
Devaraj k
From: berty...@gmail.com [mailto:berty...@gmail.com] On Behalf Of Bert Yuan
Sent: 04 September 2013 07:43
To: user@hadoop.apache.org
Subject: unsubscribe
Perhaps you can try writing the same yarn application using these steps.
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
Thanks
Devaraj k
From: Punnoose, Roshan [mailto:rashan.punnr...@merck.com]
Sent: 29 August 2013 19:43
To: user
Please ask this question in u...@hbase.apache.org, you would get better
response there.
Thanks
Devaraj k
-Original Message-
From: Lanati, Matteo [mailto:matteo.lan...@lrz.de]
Sent: 29 August 2013 14:03
To:
Subject: HBase client with security
Hi all,
I set up Hadoop (1.2.0
Could you check whether the Job is getting submitted to Job Tracker or is it
running in local mode. You can verify the same by seeing the Job Id.
Thanks
Devaraj k
From: ch huang [mailto:justlo...@gmail.com]
Sent: 19 August 2013 14:00
To: user@hadoop.apache.org
Subject: why mapred job can not
html
http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/db/DBOutputFormat.html
Thanks
Devaraj k
From: jamal sasha [mailto:jamalsha...@gmail.com]
Sent: 17 August 2013 00:47
To: user@hadoop.apache.org
Subject: Things to keep in mind when writing to a db
Hi,
I am wonderin
One task can use only one slot, It cannot use more than one slot. If the task
is Map task then it will use one map slot and if the task is reduce task the it
will use one reduce slot from the configured ones.
Thanks
Devaraj k
From: Azuryy Yu [mailto:azury...@gmail.com]
Sent: 08 August 2013 08
Can you find out the reason for going Data node down from Data Node log? Do you
get any exception in the client when you try to put the file in HDFS.
Thanks
Devaraj k
From: Manickam P [mailto:manicka...@outlook.com]
Sent: 06 August 2013 15:07
To: user@hadoop.apache.org
Subject: Hadoop datanode
oject-dist/hadoop-common/ClusterSetup.html
Thanks
Devaraj k
From: Pierre-Francois Laquerre [mailto:pierre.franc...@nec-labs.com]
Sent: 01 August 2013 22:57
To: user@hadoop.apache.org
Subject: incorrect staging area path in 2.0.5
I recently updated from 1.0.4 to 2.0.5. Since then, streaming jobs have
If you want to write a mapreduce Job, you need to have basic knowledge on core
Java. You can get many resources in the internet for that.
If you face any problems related to Hadoop, you could ask here for help.
Thanks
Devaraj k
From: jamal sasha [mailto:jamalsha...@gmail.com]
Sent: 31 July
you have only two words in a line, you can modify the
code to get these directly instead of iterating multiple times.
Thanks
Devaraj k
From: jamal sasha [mailto:jamalsha...@gmail.com]
Sent: 31 July 2013 23:40
To: user@hadoop.apache.org
Subject: java.util.NoSuchElementException
Hi,
I am getting
You can write custom key/value classes by implementing
org.apache.hadoop.io.Writable interface for your Job.
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/Writable.html
Thanks
Devaraj k
From: jamal sasha [mailto:jamalsha...@gmail.com]
Sent: 30 July 2013 10:27
To: user
ove reduce function you can add logical condition to avoid extra ','
at end of each value line.
Thanks
Devaraj k
From: manish dunani [mailto:manishd...@gmail.com]
Sent: 27 July 2013 10:02
To: user@hadoop.apache.org
Subject: Want to Sort the values in one line using map reduce
Hi,
I ha
gument
to configure() method which you implement in Mapper as you mentioned below.
Thanks
Devaraj k
From: Tanniru Govardhan [mailto:govardhan5...@gmail.com]
Sent: 26 July 2013 21:06
To: user@hadoop.apache.org; dunaniman...@gmail.com
Subject: Re: Context Object in Map Reduce
Thanks Manish. you are
hanks
Devaraj k
From: Krishna Kishore Bonagiri [mailto:write2kish...@gmail.com]
Sent: 25 July 2013 16:09
To: user@hadoop.apache.org
Subject: Node manager crashing when running an app requiring 100 containers on
hadoop-2.1.0-beta RC0
Hi,
I am running an application against hadoop-2.1.0-beta RC, and m
You can try writing the mapreduce job for this. In the Job, you can filter the
records in Mapper based on the where condition regex and then perform the join
in the Reducer.
Please refer the classes present in hadoop-datajoin module to get an idea how
to implement the join job.
Thanks
Devaraj
, you can ask the same in Hive mailing list.
Thanks
Devaraj k
From: Balamurali [mailto:balamurali...@gmail.com]
Sent: 23 July 2013 12:42
To: user
Subject: Re: Copy data from Mainframe to HDFS
Hi,
I configured hadoop-1.0.3, hbase-0.92.1 and hive-0.10.0 .
Created table in HBase.Inserted
'mapred.task.timeout' is deprecated configuration. You can use
'mapreduce.task.timeout' property to do the same.
You could set this configuration while submitting the Job using
org.apache.hadoop.conf.Configuration.setLong(String name, long value) API from
conf or JobCon
Hi Pradeep,
Please send mail to subscribe mail ids, after subscription if you have any
queries you can reach to the corresponding lists. You can find the subscribe
mail ids in this page.
http://hadoop.apache.org/mailing_lists.html
Thanks
Devaraj k
From: Pradeep Singh [mailto:hadoop.guy0
which internally extends FileOutputFormat, that's why you see in the below
exception still it is using the FileOutputFormat.
Thanks
Devaraj k
From: Max Lebedev [mailto:ma...@actionx.com]
Sent: 18 July 2013 01:03
To: user@hadoop.apache.org
Subject: Re: Incrementally adding to existing output dir
depend on the input data for
the Job, that's why you see no of mappers is proportional to the Job input size.
http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/InputFormat.html#getSplits(org.apache.hadoop.mapreduce.JobContext)
Thanks
Devaraj k
From: Austin Chungath [mailto:
MultipleOutputs class for implementation details how it could be done.
Thanks
Devaraj k
From: Max Lebedev [mailto:ma...@actionx.com]
Sent: 16 July 2013 23:33
To: user@hadoop.apache.org
Subject: Incrementally adding to existing output directory
Hi
I'm trying to figure out how to incrementally a
Hi,
In the given image, I see there are some failed/killed map& reduce task
attempts. Could you check why these are failing, you can check further based on
the fail/kill reason.
Thanks
Devaraj k
From: kira.w...@xiaoi.com [mailto:kira.w...@xiaoi.com]
Sent: 16 July 2013 12:57
To:
Hi,
It doesn’t consider where the map’s ran to schedule the reducers because
reducers need to contact all the mappers for the map o/p’s. It schedules
reducers wherever the slots available.
Thanks
Devaraj k
From: Felix.徐 [mailto:ygnhz...@gmail.com]
Sent: 16 July 2013 09:25
To: user
u can tune your Job configuration
according your cluster configurations.
Thanks
Devaraj k
From: Shekhar Sharma [mailto:shekhar2...@gmail.com]
Sent: 15 July 2013 07:32
To: user@hadoop.apache.org
Subject: Re: Map slots and Reduce slots
Sorry for the wrong properties name, i meant the same..
I unde
nds on which type of tasks you are going to run and load of the
task. Normally each task requires one core to execute, no of concurrent tasks
can be configured based on this. And memory required for the task depends on
how much data it is going to process.
Thanks
Devaraj k
From: Shekhar Sharma [
I think, there is mismatch of jar’s coming in the classpath for the map tasks
when it runs in different machines. You can find out this, by giving some
unique name for your Mapper class, Job Submit class and then submit the Job.
Thanks
Devaraj k
From: Ramya S [mailto:ram...@suntecgroup.com
Could you tell, what is the Map Output Value class you are configuring while
submitting Job and what is the type of the value writing from the Mapper. If
both of these mismatches then it will trow the below error.
Thanks
Devaraj k
From: Ramya S [mailto:ram...@suntecgroup.com]
Sent: 12 July
You need to send a mail to user-unsubscr...@hadoop.apache.org for unsubscribe.
http://hadoop.apache.org/mailing_lists.html#User
Thanks
Devaraj k
-Original Message-
From: Margusja [mailto:mar...@roo.ee]
Sent: 12 July 2013 14:26
To: user@hadoop.apache.org
Subject: unsubscribe
this JIRA
https://issues.apache.org/jira/browse/MAPREDUCE-3193.
You can give input dir to the Job which doesn't have nested dir's or you can
make use of the old FileInputFormat API to read files recursively in the sub
dir's.
Thanks
Devaraj k
-Original Message-
From: Liu
Hi Jay,
Here client is trying to create a staging directory in local file system,
which actually should create in HDFS.
Could you check whether do you have configured "fs.defaultFS" configuration in
client with the HDFS.
Thanks
Devaraj k
From: Jay Vyas [mailto:jayunit...@gmail
Hi Andrew,
You could make use of hadoop data join classes to perform the join or you can
refer these classes for better idea to perform join.
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-datajoin
Thanks
Devaraj k
From: Botelho, Andrew [mailto:andrew.bote...@emc.com
configuration "mapreduce.shuffle.port" in all Node Mangers yarn-site.xml file.
Thanks
Devaraj k
From: ch huang [mailto:justlo...@gmail.com]
Sent: 11 July 2013 07:46
To: user@hadoop.apache.org
Subject: yarn Failed to bind to: 0.0.0.0/0.0.0.0:8080
i have 3 NM, on the box of one of NM ,the 8080 PORT has alrea
share/hadoop/yarn/*:/opt/hadoop/hadoop-2.0.0-cdh4.3.0/share/hadoop/mapreduce2/lib/*:/opt/hadoop/hadoop-2.0.0-cdh4.3.0/share/hadoop/mapreduce/*"
If this is not the MRAppMaster process classpath, can you try to get the
MRAppMaster java process classpath.
Thanks
Devaraj k
From: Yu, Libo [m
time for any clue.
Thanks
Devaraj k
From: Andrei [mailto:faithlessfri...@gmail.com]
Sent: 10 July 2013 17:32
To: user@hadoop.apache.org
Subject: ConnectionException in container, happens only sometimes
Hi,
I'm running CDH4.3 installation of Hadoop with the following simple setup:
master-ho
'yarn.nodemanager.address' is not required to submit the Job, it will be
required only in NM side.
Thanks
Devaraj k
From: Azuryy Yu [mailto:azury...@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-
e.xml");
// Can you try printing the properties 'yarn.resourcemanager.address' &
'mapreduce.framework.name' values and check they are coming as per the config
files
JobConf conf = new JobConf(conf1);
Thanks
Devaraj k
From: Azuryy Yu [mailto:azury..
Hi,
Are you trying to stop the DFS with same user or different user?
Could you check whether these processes are running or not using 'jps' or 'ps' .
Thanks
Devaraj k
From: YouPeng Yang [mailto:yypvsxf19870...@gmail.com]
Sent: 10 July 2013 11:01
To: user@hadoop.apache.org
for NM
"yarn.resourcemanager.resource-tracker.address" with RM resource tracker
address.
Thanks
Devaraj k
From: ch huang [mailto:justlo...@gmail.com]
Sent: 10 July 2013 08:36
To: user@hadoop.apache.org
Subject: can not start yarn
i am testing mapreducev2 ,i find i start NM error
here
e things you can select which API's to use.
Thanks
Devaraj k
From: Ahmed Eldawy [mailto:aseld...@gmail.com]
Sent: 05 July 2013 00:00
To: user@hadoop.apache.org
Subject: Which InputFormat to use?
Hi I'm developing a new set of InputFormats that are used for a project I'm
doing. I fou
And also could you check whether the client is connecting to NameNode or any
failure in connecting to the NN.
Thanks
Devaraj k
From: Azuryy Yu [mailto:azury...@gmail.com]
Sent: 05 July 2013 09:15
To: user@hadoop.apache.org
Subject: Re: Decomssion datanode - no response
I added
u can
update the file with new hosts and refreshNodes command can be issued, then
newly updated the DN's will be decommissioned.
Thanks
Devaraj k
From: Azuryy Yu [mailto:azury...@gmail.com]
Sent: 05 July 2013 08:48
To: user@hadoop.apache.org
Subject: Re: Decomssion datanode - no response
Than
Hi Kishore,
hadoop-2.1.0 beta release is in voting process now.
You can try out from hadoop-2.1.0 beta RC
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc0/ or you could check
the same with trunk build.
Thanks
Devaraj k
From: Krishna Kishore Bonagiri [mailto:write2kish...@gmail.com
Do you see any log related to this in Name Node logs when you issue
refreshNodes dfsadmin command?
Thanks
Devaraj k
From: Azuryy Yu [mailto:azury...@gmail.com]
Sent: 05 July 2013 08:12
To: user@hadoop.apache.org
Subject: Decomssion datanode - no response
Hi,
I am using hadoop-2.0.5-alpha, and
Hi Steven,
For subscribing to this list, you need to send mail to
user-subscr...@hadoop.apache.org<mailto:user-subscr...@hadoop.apache.org>.
Please find the all mailing lists here
http://hadoop.apache.org/mailing_lists.html
Thanks
Devaraj k
From: Steven Fuller [mailto:sful...@cloude
#x27; work directories, called container_${contid}, will
be subdirectories of this.
yarn.nodemanager.local-dirs
${hadoop.tmp.dir}/nm-local-dir
Thanks
Devaraj k
From: John Lilley [mailto:john.lil...@redpoint.net]
Sent: 02 July 2013 02:08
To: user@hadoop.apache.org
Subject:
If you are 100% sure that all the node data nodes are available and healthy for
that period of time, you can choose the replication factor as 1 or <3.
Thanks
Devaraj k
From: John Lilley [mailto:john.lil...@redpoint.net]
Sent: 02 July 2013 04:40
To: user@hadoop.apache.org
Subject:
'yarn.nodemanager.local-dirs' configuration) accordingly
with the app id & container id, and this will be cleaned up after the app
completion. You need to make use of this persisted data before completing the
application.
Thanks
Devaraj k
From: John Lilley [mailto:john.lil...@redpoint.net]
Can you ask this HBase question in the HBase user mailing list?
Thanks
Devaraj k
From: ch huang [mailto:justlo...@gmail.com]
Sent: 26 June 2013 14:30
To: user@hadoop.apache.org
Subject: region server can not start
i change zookeeper port from 2181 to 2281 , region server can not start
2013-06
Can you ask this HBase question in the HBase user mailing list?
Thanks
Devaraj k
From: ch huang [mailto:justlo...@gmail.com]
Sent: 26 June 2013 14:52
To: user@hadoop.apache.org
Subject: master node abnormal ,help
when i start master node ,it not work,anyone can help?
2013-06-26 17:17:52,552
You need to update the jars in client side with the jars which server uses.
Thanks
Devaraj k
From: ch huang [mailto:justlo...@gmail.com]
Sent: 26 June 2013 14:04
To: user@hadoop.apache.org
Subject: eclipse connect problem in CDH3u4 Protocol
ora.apache.hadoop.hdfs.protocal.ClientProtocol version
Could you check the logs for the hadoop processes, are they started
successfully or any problem while starting?
Thanks
Devaraj k
From: ch huang [mailto:justlo...@gmail.com]
Sent: 26 June 2013 12:38
To: user@hadoop.apache.org
Subject:
hi i build a new hadoop cluster ,but i can not ACCESS hdfs
Hi,
Could you check the network usage in the cluster when this problem occurs?
Probably it is causing due to high network usage.
Thanks
Devaraj k
From: blah blah [mailto:tmp5...@gmail.com]
Sent: 26 June 2013 05:39
To: user@hadoop.apache.org
Subject: Yarn HDFS and Yarn Exceptions when
I agree, for getting status/counters we need HS. I mean Job can finish without
HS also.
Thanks
Devaraj k
From: Alejandro Abdelnur [mailto:t...@cloudera.com]
Sent: 25 June 2013 18:05
To: common-u...@hadoop.apache.org
Subject: Re: Job end notification does not always work (Hadoop 2.x)
Devaraj
failing task log files, you will get more idea who
is causing the problem.
Thanks
Devaraj k
From: Ramya S [mailto:ram...@suntecgroup.com]
Sent: 25 June 2013 15:50
To: user@hadoop.apache.org
Subject: RE: Error:java heap size
Hi,
I have set the properties in mapred-site.xml as follows
u are allocating 2GB for map container, you can probably pass the same
value as –Xmx for the mapreduce.map.java.opts and same way for reducer as well.
Thanks
Devaraj k
From: Ramya S [mailto:ram...@suntecgroup.com]
Sent: 25 June 2013 14:39
To: user@hadoop.apache.org
Subject: RE: Error:java heap siz
Hi Ramya,
Where did you get the java heap size error?
Could you see the error in client side/RM/AM log? What is the detailed error?
Thanks
Devaraj k
From: Ramya S [mailto:ram...@suntecgroup.com]
Sent: 25 June 2013 13:10
To: user@hadoop.apache.org
Subject: Error:java heap size
Hi,
I am using
It is a good start Lokesh.
Can you go through this page http://wiki.apache.org/hadoop/HowToContribute for
the steps and guidelines to contribute.
Thanks
Devaraj k
From: Lokesh Basu [mailto:lokesh.b...@gmail.com]
Sent: 25 June 2013 11:03
To: user@hadoop.apache.org
Subject: After importing into
Could you check this page for any of these possible cause in your cluster.
http://wiki.apache.org/hadoop/CouldOnlyBeReplicatedTo
Thanks
Devaraj k
From: Yuzhang Han [mailto:yuzhanghan1...@gmail.com]
Sent: 25 June 2013 09:34
To: user@hadoop.apache.org
Subject: Re: "could only be replicated
It is not mandatory to have running HS in the cluster. Still the user can
submit the job without HS in the cluster, and user may expect the Job/App End
Notification.
Thanks
Devaraj k
From: Alejandro Abdelnur [mailto:t...@cloudera.com]
Sent: 24 June 2013 21:42
To: user@hadoop.apache.org
Cc
tween AM and NM.
I feel it would be better to move End-Notification responsibility to RM as Yarn
Service because it ensures 100% notification and also useful for other types of
applications as well.
Thanks
Devaraj K
From: Ravi Prakash [mailto:ravi...@ymail.com]
Sent: 23 June 2013 19:01
To:
Do you see any problem in JT or TT for Job not running?
Thanks
Devaraj K
From: Pavan Kumar Polineni [mailto:smartsunny...@gmail.com]
Sent: 23 June 2013 19:20
To: user@hadoop.apache.org; Ravi Prakash
Subject: Re: MapReduce job not running - i think i keep all correct
configuration.
Hi ravi
he maximum size of physical mem of a container? By the parameter
>of 'mapred.child.java.opts'?
It can be set based on the resources requested for that container.
Thanks
Devaraj K
From: sam liu [mailto:samliuhad...@gmail.com]
Sent: 19 June 2013 08:16
To: user@hadoop.apache.org
Subject:
connecting
remotely, you need to update the DS application code accordingly.
Thanks
Devaraj K
From: Curtis Ullerich [mailto:curtisuller...@gmail.com]
Sent: 18 June 2013 08:19
To: user@hadoop.apache.org
Subject: Debugging YARN AM
Hi all,
I can successfully debug the MapReduce ApplicationMaster in
/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
Thanks
Devaraj K
From: Rahul Bhattacharjee [mailto:rahul.rec
Hi,
You need to send a mail to user-unsubscr...@hadoop.apache.org for
unsubscribing from this list.
http://hadoop.apache.org/mailing_lists.html#User
Thanks
Devaraj K
On 6/17/13, Manny Vazquez wrote:
>
>
> Manuel Vazquez, BI Administrator, IS Engineering
> Apollo Group | Apollo
iguration.
Thanks&Regards
Devaraj K
From: Maysam Hossein Yabandeh [mailto:myaban...@qf.org.qa]
Sent: 12 June 2013 21:16
To: user@hadoop.apache.org
Cc: Maysam Hossein Yabandeh
Subject: Assigning the same partition number to the mapper output
Hi,
I was wondering if it is possible in h
Can you check the Job Tracker and task Tracker log files, whether any problem
while starting the Task Tracker or any problem while connecting to Job
tracker...
Thanks
Devaraj
From: Vikas Jadhav [mailto:vikascjadha...@gmail.com]
Sent: 13 June 2013 12:22
To: user@hadoop.apache.org
Subject: JobTra
Can you go through this issue
https://issues.apache.org/jira/browse/HADOOP-7489, It is discussed and provided
some workarounds for this problem.
Thanks
Devaraj
From: Subho Banerjee [subs.z...@gmail.com]
Sent: Monday, August 13, 2012 10:47 AM
To: user@hadoop
76 matches
Mail list logo
