Map outputs=2
Reduce input groups=993814
Reduce input records=994179
Reduce output records=993814
Reduce shuffle bytes=83049802
Shuffled Maps =2
Spilled Records=2861309
SPLIT_RAW_BYTES=200
--
Harsh J
www.harshj.com
I get per-node count ?
You could get it by passing that entire command to the slaves?
Something like slaves.sh '/usr/java/default/bin/jps | /bin/grep Child
| /usr/bin/wc -l' should work I think, instead of aggregating in your
shell..
--
Harsh J
www.harshj.com
I've not heard of one, but there was an idea filed for the same quite
a while ago @ HADOOP-5005
The only other option I could suggest is MiniMRCluster, but I believe
it wouldn't help that much.
--
Harsh J
www.harshj.com
://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.2/src/test/org/apache/hadoop/mapred/lib/aggregate/
Summary:
http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/mapred/lib/aggregate/package-summary.html
--
Harsh J
www.harshj.com
group=Map-Reduce Framework name=Map output
records296754600/counter
Thanks
--
Harsh J
www.harshj.com
.
On Wed, Feb 2, 2011 at 9:09 AM, Ted Yu yuzhih...@gmail.com wrote:
If map task(s) were retried (mapred.map.max.attempts times), how would these
two counters be affected ?
Thanks
On Tue, Feb 1, 2011 at 7:31 PM, Harsh J qwertyman...@gmail.com wrote:
HDFS_BYTES_READ is a FileSystem interface counter
)
break;
writer.append(samples[k], nullValue);
last = k;
}
--
Best Regards
Jeff Zhang
--
Harsh J
. To reply, visit:
https://reviews.apache.org/r/885/#review808
---
On 2011-06-12 14:00:13, Harsh J wrote:
---
This is an automatically generated e-mail. To reply, visit:
https
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/887/
---
Review request for hadoop-mapreduce.
Summary
---
In case of a task failure,
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/
---
(Updated 2011-06-16 12:24:09.519284)
Review request for hadoop-mapreduce.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1091/#review1109
---
,
+Vinod
--
Harsh J
can also provide very
brief summaries on why something doesn't matter anymore - to help
contributors out in spending their time better on future contributions
:)
--
Harsh J
. Why should the calculation of the InputSplit happen on the
client? JobTracker is a high-end machine, can't the calculation happen on
the JobTracker?
http://stackoverflow.com/questions/7546064/hadoop-high-cpu-load-on-client-side-after-committing-jobs
Thanks,
Praveen
--
Harsh J
that will clean up configs and improve the documentation
of them. Even if we have to rewire or rewrite a lot of the Configuration
class to make things work properly.
--Bobby Evans
On 12/5/11 11:54 AM, Harsh J ha...@cloudera.com wrote:
Praveen,
(Inline.)
On 05-Dec-2011, at 10:14 PM
Praveen,
Obviously, a clean target will wipe out all your existing build directories,
and hence the other things start from scratch. That is your slowdown-causer.
Just remove the clean from that command and you're good to go.
On 07-Dec-2011, at 6:37 PM, Praveen Sripati wrote:
Alejandro,
Deepak,
Um, unsure what you are specifically looking for, but there's an example at
http://hadoop.apache.org/common/docs/stable/mapred_tutorial.html and more under
your downloaded source tarball; And for HBase, look at
http://hbase.apache.org/book.html#mapreduce
On 13-Dec-2011, at 2:40 PM,
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1091/#review4166
---
Please mark as submitted, if this was committed. I could not find the
open shared object file: No such file or
directory
You need the dir that contains libprotobuf.so.7 on your
LD_LIBRARY_PATH. For instance, if /usr/local/lib carries it, ensure
you do:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
Before you run the maven build.
--
Harsh J
Customer Ops
://wiki.apache.org/hadoop/HowToContribute for more details.
Please feel free to mail the lists with any specific questions you
have as you go ahead!
--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about
, brisk mylinq...@gmail.com wrote:
Hi Harsh,
So by trunk, do you mean 1.0.0 version?
Thanks,
Ethan
On Wed, Feb 1, 2012 at 7:34 AM, Harsh J ha...@cloudera.com wrote:
Pavlos,
0.21 was abandoned and you'd hardly find support for it from the major
part of the community. What are you looking
/src/webapps/job/
--
Merged
/hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job:r1304063
--
Harsh J
that all the
changes are that sub-directory.
Regards,
Nicholas
- Original Message -
From: Harsh J ha...@cloudera.com
To: mapreduce-dev@hadoop.apache.org; Tsz Wo Sze szets...@yahoo.com
Cc: hdfs dev hdfs-...@hadoop.apache.org
Sent: Thursday, March 22, 2012 9:45 PM
Subject: Re: svn
=%28contribut%29
--
Harsh J
1.0.0 on my system , i wanted to
add more information in the heartbeat of the tasktracker , i am not sure
how to do it ? please can u help me ?
Thank you,
Ashwini
--
Harsh J
will be pluggable implementation something like
resourcecalculatorplugin.
--
Harsh J
, it will take a while to load 1000 bugs in your browser
-
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1913 / Virus Database: 2411/4958 - Release Date: 04/25/12
--
Harsh J
, Gourav Sengupta gourav.had...@gmail.com wrote:
hi,
i am looking towards learning and contributing to this project.
any help will be a great
gourav
--
Harsh J
at
JobTracker if
+ exceeded. Added mapreduce.job.max.split.locations property.
(cdouglas)
+
--
Harsh J
around 100 errors and one of them is
mentioned in the subject. I may be doing something fundamentally stupid
since I am new to JAVA programming.
I will be grateful for any help on this.
Thanks and Regards,
Gourav
--
Harsh J
back. - Piet Hein
(via Tom White)
--
Harsh J
Ixsight Technologies Pvt. Ltd.
--
Harsh J
Done, other than the minor nits I've commented on there, I think it is
good to go in. Thanks!
On Sat, Jul 7, 2012 at 4:49 AM, mio...@internet.is wrote:
Hi everyone,
Could someone please review HDFS-2988.
Miomir Boljanovic
--
Harsh J
Hey Miomir,
Commented.
On Wed, Jul 11, 2012 at 1:05 PM, mio...@internet.is wrote:
Hi everyone,
Could someone update the status of MAPREDUCE-2825.
Class TestRackAwareTaskPlacement doesn't seem to exist anymore?
Miomir Boljanovic
--
Harsh J
://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
Harsh J
turned to be bad design pattern?
--
Harsh J
.
During the executions of MapReduce, I see that *file.out* is created on Map
nodes, so my question is how does a reducer
know what part of file.out to fetch? Is the *file.out.index* play any role?
Any help is appreciated .Thanks
--With Regards
Pavan Kulkarni
--
Harsh J
the DataNode, which
only sees generic requests made by the DFSClient. So, what I'd like to know
is how requests originating from the TaskTracker make their way to the
DataNode. Any help would be appreciated.
Adrian
--
Harsh J
?
Bikas
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Tuesday, September 18, 2012 9:37 PM
To: mapreduce-dev@hadoop.apache.org
Subject: Re: More than one RM in YARN?
Chris,
Not yet, it is coming:
https://issues.apache.org/jira/browse/MAPREDUCE-4326 should go in
first
Welcome to the development community, Lamosi! Take a look at
http://wiki.apache.org/hadoop/HowToContribute to get started.
On Fri, Sep 28, 2012 at 7:00 AM, Lamosi lamosi2...@gmail.com wrote:
Because I want to learn someting about hadoop, so I want to subscribe to
the list,tank you.
--
Harsh
to choose reduce nodes
does not help reduce network traffic. If JobTracker is fully aware of
locations of every map output, why not take advantage of this topology
knowledge?
So, is there anyone who knows where to develop such codes upon? Many thanks.
Regards.
--
Jiwei
--
Harsh J
not aware of anyone working on this. I did attempt it myself once,
pre-MR2 days, but we wound up deciding it is unsuitable to support
this ourselves.
--
Harsh J
Rodrigo Kepe*
Graduando em Ciência da Computação - UFPR
Analista de Desenvolvimento na Siemens Enterprise.
--
Harsh J
,*
*National Institute of Technology,*
*Tiruchirappalli - 620015.*
*+91-9941506562*
--
Harsh J
--
Alejandro
--
Harsh J
,
Is there a documented list of API incompatibility between version 1.0 and
version 2.0 of APIs.
--
Have a Nice Day!
Lohit
--
Harsh J
deprecate these configs.
Thanks,
+Vinod
On Feb 8, 2013, at 2:15 PM, Sandy Ryza wrote:
Is there a reason why mapreduce.jobtracker.address is not deprecated to
yarn.resourcemanager.address?
thanks,
Sandy
--
Alejandro
--
Harsh J
with.
For example JobContext was class in 1.0 while Interface in 2.0. So, user
has to change code.
I was more interested in such cases (if it was documented)
2013/2/8 Harsh J ha...@cloudera.com
Hey Lohit,
It may not be exactly in the form you're looking for, but take a look
at Dave's API Evolution page
that not sound
right to you Harsh?
Was there a conclusion on: may I deprecate it, or is there a larger
decision being waited on?
-Sandy
On Fri, Feb 8, 2013 at 9:35 PM, Harsh J ha...@cloudera.com wrote:
Yes, there isn't a classic version of mapreduce.framework.name now
either. Its mostly being used
and precommit build is clean.
2. Start a discussion on adding Jenkins precommit builds on windows and
how to integrate that with the existing commit process.
Let me know if you have any questions.
Regards,
Suresh
--
http://hortonworks.com/download/
--
Harsh J
,
Ling Kun
--
http://www.lingcc.com
--
http://www.lingcc.com
--
Harsh J
more
2013-03-21 21:04:56,979 INFO org.apache.hadoop.hdfs.DFSClient:
read.type=dfs block=blk_972265969568196888_2183
src=/home/ec2-user/Hadoop/tmp/mapred/staging/ec2-user/.staging/job_201303212104_0005/job.splitmetainfo
Xiao Yu
--
Harsh J
Thank you.
--
Thanx and Regards
Vikas Jadhav
--
Harsh J
but couldn't find related to assigning the bug or may be I missed
out somewhere.
Thanks in advance.
Regards,
Niranjan
--
Harsh J
Hortonworks Inc.
http://hortonworks.com/
--
Harsh J
in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/
The maven artifacts are available via repository.apache.org.
Please try the release and vote; the vote will run for the usual 7 days.
thanks,
Tom Graves
--
Harsh J
a JIRA?
--
Jay Vyas
http://jayunit100.blogspot.com
--
Harsh J
updates if this makes sense :)
Would love to bring the minimrcluster class up to date!
On Apr 18, 2013, at 12:49 AM, Harsh J ha...@cloudera.com wrote:
Why do you imagine a test case would need the Configured and Tool
interfaces, which are more useful for actual client apps?
Because
from some light googling that, at some point
** some version of JobConf was deprecated, and that maybe again it was
undeprecated or maybe moved**
Will have to look into this more formally to really determine what's going on.
On Apr 18, 2013, at 1:16 AM, Harsh J ha...@cloudera.com
); *in
sortAndSpill method but no changes seen in result. I have expected to get
unsorted keys as mapper output but was not like that.
My question is did i perform right action to disable sort or something else
needs to be done ?
Samaneh
--
Harsh J
is appropriate version for me ?
Samaneh
On Tue, Apr 23, 2013 at 10:01 AM, Harsh J ha...@cloudera.com wrote:
Note that the Reducer also does a merge sort over the acquired map
data; but am not sure if you looked there as well.
Mariappan's pointed JIRA (MAPREDUCE-4807) is already available
, I build one of the Hadoop
version(I don't remember which one but was successful in building with ant
but now I got this error, file missing.
Regards
thoihen
Regards
thoihen
--
Harsh J
,*
*National Institute of Technology,*
*Tiruchirappalli - 620015.*
*+91-9941506562*
--
Harsh J
should be left on top of mapred.child.java.opts (or other similar JVM opts)
for
the container memory itself?
I would like to have separated java option settings for map/reduce parts.
Something like
mapred.map.child.java.opts mapred.reduce.child.java.opts.
--
Harsh J
While I think work should be done to make the numbers nearby, we should
ideally raise the JVM heap value than lower the memory.mb resource request
of MR tasks. Otherwise, with YARN, users will start seeing more containers
per node than before.
Also good to raise heap plus the sort buffer memory
, that some reduce
work begin before all maps ends their job.
The very first question is which project from svn is more adequate to work
on/develop?
Second, is there a possibility to implement such a thing and if yes which
classes/packages should I look in?
thanks,
Elma
--
Harsh J
--
Harsh J
.
Thanks in advance.
DR
--
Harsh J
.
Sincerely,
Sreejith R
P. S: I had asked another question recently. I got very good answers. I
was so busy I couldn't even thank them. Special thanks to Kun Ling
--
Harsh J
MRv2's code base is tightly
coupled with YARN currently. Thoughts?
- Tsuyoshi
--
Harsh J
== 1.01 SPLIT_SLOP, so it
just becomes one input splits!!)
Thank you for reading my very long question!
--
Harsh J
received this
communication in error,please
immediately notify the sender by return e-mail, and delete the original
message and all copies from
your system. Thank you.
---
--
Harsh J
-9941506562*
--
Harsh J
: Exiting with
status 1
I really appreciate if any one help about these problems.
Thanks in advance,
--
Behrooz
--
Harsh J
--
Harsh J
Ted's URL fetches you a mirrored repository from GitHub.com instead, which
is absolutely fine to use as an alternate. There might have been an
infrastructure issue when you were attempting to access the original Apache
git repository.
In trying the regular Apache git repository now, it appears to
Components: jobtracker
Affects Versions: 0.21.0
Environment: Not environment related.
Reporter: Harsh J Chouraria
Priority: Trivial
Fix For: 0.21.1
Attachments: repeated.lines.JIP.r1.diff
In the ctor of JobInProgress class that's used
: Improvement
Components: job submission
Affects Versions: 0.20.1
Reporter: Harsh J Chouraria
Assignee: Harsh J Chouraria
Fix For: 0.23.0
MultipleOutputs right now requires for Key/Value classes to utilize the
Writable and WritableComparable interfaces
Reporter: Harsh J Chouraria
Assignee: Harsh J Chouraria
To reproduce, have a mapred.local.dir property set to a few directories. Before
starting up the JT, set one of these directories' permission as 'd-',
and then start the JT/TT. The JT, although it tries to ignore this directory
/Reduce
Issue Type: Improvement
Components: documentation
Affects Versions: 0.21.0
Reporter: Harsh J Chouraria
Assignee: Harsh J Chouraria
Priority: Minor
Fix For: 0.22.0, 0.23.0
The new Partitioner interface does not extend
[
https://issues.apache.org/jira/browse/MAPREDUCE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-2390.
Resolution: Duplicate
Fix Version/s: 0.22.0
Release Note: (was: Ignore bad
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Affects Versions: 0.20.2
Reporter: Harsh J
Assignee: Harsh J
As discussed on HADOOP-7328, MapReduce can handle serializers in a much better
way in case of bad configuration
[
https://issues.apache.org/jira/browse/MAPREDUCE-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-2286.
Tags: spam
Resolution: Incomplete
Fix Version/s: (was: 0.23.0
[
https://issues.apache.org/jira/browse/MAPREDUCE-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-2605.
Resolution: Not A Problem
Fix Version/s: 0.22.0
Greg,
Per {{StreamJob#packageJobJar
://issues.apache.org/jira/browse/MAPREDUCE-2703
Project: Hadoop Map/Reduce
Issue Type: Task
Components: distcp, documentation
Affects Versions: 0.22.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 0.23.0
[
https://issues.apache.org/jira/browse/MAPREDUCE-133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-133.
---
Resolution: Cannot Reproduce
This doesn't seem to be a problem in 0.20 or trunk anymore.
Running
[
https://issues.apache.org/jira/browse/MAPREDUCE-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-2715.
Resolution: Not A Problem
Geoffrey, so taking a look at stable release today, 0.22 and trunk
: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 0.20.2
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Per the implementation of the TaskTracker instrumentation plugin implementation
(from 2008
[
https://issues.apache.org/jira/browse/MAPREDUCE-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-2068.
Resolution: Not A Problem
Thanks for checking in Arun,
Marking as resolved.
Not a problem
[
https://issues.apache.org/jira/browse/MAPREDUCE-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-2945.
Resolution: Invalid
Arsalan,
The Apache JIRA is for filing issues or opening discussion
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
Fix For: 0.23.0
This is the MR part of removing deprecated RecordIO packages - parented by
HADOOP-7781.
Basically, we need to remove {{/hadoop-mapreduce-project/src/c++/librecordio}}
and all
[
https://issues.apache.org/jira/browse/MAPREDUCE-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J reopened MAPREDUCE-1187:
Was a dupe, not won't fix.
mradmin -refreshNodes should be implemented
[
https://issues.apache.org/jira/browse/MAPREDUCE-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-1187.
Resolution: Duplicate
Dupe of https://issues.apache.org/jira/browse/HADOOP-5643
[
https://issues.apache.org/jira/browse/MAPREDUCE-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-3395.
Resolution: Fixed
Add mapred.disk.healthChecker.interval to mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-2236.
Resolution: Not A Problem
From above comments, apparently not a problem on 0.23+. We may
[
https://issues.apache.org/jira/browse/MAPREDUCE-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-3303.
Resolution: Invalid
Per Owen on the parent JIRA, the deprecated recordio is still in use
[
https://issues.apache.org/jira/browse/MAPREDUCE-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-2703.
Resolution: Won't Fix
Appears to be that DistCp uses Sorter only for filename sorting
[
https://issues.apache.org/jira/browse/MAPREDUCE-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-423.
---
Resolution: Not A Problem
Fix Version/s: (was: 0.24.0)
Not a worry in 0.23
[
https://issues.apache.org/jira/browse/MAPREDUCE-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-1932.
Resolution: Won't Fix
Won't Fix, per Tom and Owen's comments above.
record
[
https://issues.apache.org/jira/browse/MAPREDUCE-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-2165.
Resolution: Won't Fix
Resolving as Won't Fix - See MAPREDUCE-1932 (parent) for the reasoning
1 - 100 of 298 matches
Mail list logo
