Re: query number of mappers in 0.22 mapreduce API

Please file a JIRA for adding this to JobContext. Do let this thread know the JIRA ID once you've filed it (under MAPREDUCE). MAPREDUCE-3968

Re: query number of mappers in 0.22 mapreduce API

Radim, This does not exist yet as a direct API but its a fair thing to demand from a submitted job configuration (as that does hold the truthful value). Please file a JIRA for adding this to JobContext. Do let this thread know the JIRA ID once you've filed it (under MAPREDUCE). For now, do: job.g

query number of mappers in 0.22 mapreduce API

I switched to new mapreduce API. I need a replacement for job.getNumMapTasks()) in job driver.

Re: number of running reducers exceed max number of mappers on cluster

For how to control Pig's parallelism: http://pig.apache.org/docs/r0.9.1/perf.html#parallel For your 'running' task count bug: https://issues.apache.org/jira/browse/MAPREDUCE-486 (Which would mean that only 5 may be 'running', while the rest may have already completed and were awaiting finaliza

Re: #number of mappers

Indeed, that was the problem, thanks! Keren On Mon, Nov 28, 2011 at 10:46 PM, Harsh J wrote: > Keren, > > Seems to work alright for me once I set these and restart each > TaskTracker. Perhaps the problem is merely that you're missing an opening > XML tag for the second configuration, in that co

Re: #number of mappers

Keren, Seems to work alright for me once I set these and restart each TaskTracker. Perhaps the problem is merely that you're missing an opening XML tag for the second configuration, in that copy-paste below? On Tue, Nov 29, 2011 at 8:44 AM, Keren Ouaknine wrote: > Thanks Harsh, the property in

Re: #number of mappers

From: Keren Ouaknine To: mapreduce-user@hadoop.apache.org Sent: Monday, November 28, 2011 10:14 PM Subject: Re: #number of mappers Thanks Harsh, the property in already set to one on all the nodes (see below). What do you reckon

Re: #number of mappers

Thanks Harsh, the property in already set to one on all the nodes (see below). What do you reckon? mapred.tasktracker.map.tasks.maximum 1 The maximum number of map tasks that will be run simultaneously by a task tracker. mapred.tasktracker.reduce.tasks.maximum 1 The maxim

Re: #number of mappers

Keren, What was your configured property at the TaskTrackers? The right property is mapred.tasktracker.{map,reduce}.tasks.maximum, and applies to each TaskTracker individually (so all your TT mapred-site.xml need to carry that property to limit their slots). On 25-Nov-2011, at 8:45 AM, Keren O

#number of mappers

Hello, I configured mapred-site.xml with one mapper, still the web ui shows: Map shows capacity 40 = 2 x #machines. I was expecting it to be 20. Any idea? Thanks, Keren Running Map TasksRunning Reduce TasksTotal SubmissionsNodesOccupied Map SlotsOccupied Reduce Slots Reserved Map SlotsReserved

Re: increasing number of mappers.

argument to job, but there is no difference. Job still runs on 2 mappers. How split size works? Is max split size used for reading or writing files? it works like this?: set maxsplitsize, write files and you will get bunch of seq files as output. then you will get same number of mappers as

Re: increasing number of mappers.

e this?: set maxsplitsize, write files and you will get bunch > of seq files as output. then you will get same number of mappers as input > files. >

Re: increasing number of mappers.

i found following usefull post on this topic http://yaseminavcular.blogspot.com/2011/06/how-to-set-number-of-maps-with-hadoop.html its more clear than wiki page.

Re: increasing number of mappers.

bunch > of seq files as output. then you will get same number of mappers as input > files. >

increasing number of mappers.

files? it works like this?: set maxsplitsize, write files and you will get bunch of seq files as output. then you will get same number of mappers as input files.

Computing the number of mappers when CombineFileInputFormat is used (Reloaded)

Hello1    I would like to know how Hadoop is computing the number of mappers when CombineFileInputFormat is used? I have read the API specification for CombineFileInputFormat (http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/CombineFileInputFormat.html), but

Computing the number of mappers when CombineFileInputFormat is used

Hello! I would like to know how Hadoop is computing the number of mappers when CombineFileInputFormat is used? I have read the API specification for CombineFileInputFormat (http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/CombineFileInputFormat.html), but

Re: Control the number of Mappers

More to your need, (I had missed this earlier), >>The number of cores is not something I know in advance, so writing a special >>InputFormat might be tricky, unless I can query Hadoop for the available # of >>cores You dont have to write a fancy InputFormat. Once you have an (correct) implementa

Re: Control the number of Mappers

Thanks, I'll take a look On Thu, Nov 25, 2010 at 10:20 PM, Shrijeet Paliwal wrote: > Shai, > > You will have to implement MultiFileInputFormat > > and > set that has your input format. > You ma

Re: Control the number of Mappers

Shai, You will have to implement MultiFileInputFormat and set that has your input format. You may find http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/examples/MultiFileWordCoun

Re: Control the number of Mappers

Ah, In that case this should answer your question: http://wiki.apache.org/hadoop/HowManyMapsAndReduces 2010/11/25 Shai Erera : > I wasn't talking about how to configure the cluster to not invoke more than > a certain # of Mappers simultaneously. Instead, I'd like to configure a > (certain) job t

Re: Control the number of Mappers

I wasn't talking about how to configure the cluster to not invoke more than a certain # of Mappers simultaneously. Instead, I'd like to configure a (certain) job to invoke exactly N Mappers, where N is the number of cores in the cluster. Irregardless of the size of the data. This is not critical if

Re: Control the number of Mappers

Hi, 2010/11/25 Shai Erera : > Is there a way to make MapReduce create exactly N Mappers? More > specifically, if say my data can be split to 200 Mappers, and I have only > 100 cores, how can I ensure only 100 Mappers will be created? The number of > cores is not something I know in advance, so wri

Control the number of Mappers

Hi Is there a way to make MapReduce create exactly N Mappers? More specifically, if say my data can be split to 200 Mappers, and I have only 100 cores, how can I ensure only 100 Mappers will be created? The number of cores is not something I know in advance, so writing a special InputFormat might

Re: Setting the number of mappers to 0

develop a MR application. Due to the kind of application I am > trying to develop, the mapper is a dummy (passes it’s input to it’s output) > task and I am only interested in having a partitioner and reducer. > > The MR framework allows us to set the number of reducers to 0. Is there a &

Setting the number of mappers to 0

here a way to set the number of mappers to 0? Basically, I want to avoid the overhead of creating mapper tasks and calling the map function once per record. Is it feasibly to modify the MR framework so as not the create the mapper tasks and have the partitioners read the input data from HDF

Re: Question about setting the number of mappers.

/19/10 10:23 AM, "Teryl Taylor" wrote: >> >>> Hi guys, >>> >>> Thanks for the answers. Michael, yes you are right, that is what I >> guess, >>> I'm looking for...how to reduce the number of mappers running >>> simultaneously. The sy

Re: Question about setting the number of mappers.

"Teryl Taylor" wrote: > > > Hi guys, > > > > Thanks for the answers. Michael, yes you are right, that is what I > guess, > > I'm looking for...how to reduce the number of mappers running > > simultaneously. The system is running really slo

Re: Question about setting the number of mappers.

What is the value of: mapred.tasktracker.map.tasks.maximum mapred.tasktracker.reduce.tasks.maximum On 1/19/10 10:23 AM, "Teryl Taylor" wrote: > Hi guys, > > Thanks for the answers. Michael, yes you are right, that is what I guess, > I'm looking for...how to r

Re: Question about setting the number of mappers.

Hi guys, Thanks for the answers. Michael, yes you are right, that is what I guess, I'm looking for...how to reduce the number of mappers running simultaneously. The system is running really slow and I think it might be due to constant thread context switching because of so many Mappers ru

RE: Question about setting the number of mappers.

@hadoop.apache.org [mailto:mapreduce-user-return-292-michael.clements=disney@hadoop.apache.org] On Behalf Of Jeff Zhang Sent: Monday, January 18, 2010 4:54 PM To: mapreduce-user@hadoop.apache.org Subject: Re: Question about setting the number of mappers. Hi Teryl The number of mapper is

Re: Question about setting the number of mappers.

Hi, >>so I wanted to try and lower the number to 10 and see how the performance is The number of mappers is provided as only a hint to the framework, it is not guaranteed to be that number. >>I have been digging around in the hadoop source code and it looks like the >>JobCl

Re: Question about setting the number of mappers.

b, submitSplitFile); > } > job.set("mapred.job.split.file", submitSplitFile.toString()); > job.setNumMapTasks(maps); > > // Write job file to JobTracker's fs > FSDataOutputStream out = > FileSystem.create(fs, submitJobFile, > new FsPermission(JOB_FILE_PERMISSION)); > > try { > job.writeXml(out); > } finally { > out.close(); >. > > 737,0-1 39% > } > > > *** > > Is there anything I can do to get the number of mappers to be more > flexible? > > > Cheers, > > Teryl > > -- Best Regards Jeff Zhang

Question about setting the number of mappers.

); try { job.writeXml(out); } finally { out.close(); . 737,0-1 39% } *************** Is there anything I can do to get the number of mappers to be more flexible? Cheers, Teryl