that
part-0 will be copied to the HDFS? Who make this file copy happen? The
Reducer child? The TaskTracker which run the reduce task? Or the JobTracker?
Thanks,
yours,
Kun Ling
--
http://www.lingcc.com
--
Harsh J
of there are
lots of small files, I would like to change the default.block.size to
16MB
other than adopting the ways to merge the files into large enough (e.g
using sequencefiles).
I want to ask are there any bad influences or issues?
Regards
--
Harsh J
more stable, easier to use, and less error-prone.
Thanks in advance.
--
Harsh J
,
--
Harsh J
. For my academic project
in Hadoop I need to add a new procedure for encryption in file system class
under moveFromLocal() procedure.Please guide me how to add a new procedure
under moveFromLocal().
Thanking you
Priya C
Amrita School of Engineering
Kollam,India.
--
Priya c
--
Harsh J
for FileOutputCommitter.
I have enabled it by setting this property in conf.
Rahul
On Mon, May 6, 2013 at 9:38 AM, Rahul Bhattacharjee
rahul.rec@gmail.com wrote:
Thanks Harsh for the pointers. I will find out more on this.
On Sun, May 5, 2013 at 11:26 PM, Harsh J ha...@cloudera.com wrote
On Mon, May 6, 2013 at 9:38 AM, Rahul Bhattacharjee
rahul.rec@gmail.com wrote:
Thanks Harsh for the pointers. I will find out more on this.
On Sun, May 5, 2013 at 11:26 PM, Harsh J ha...@cloudera.com wrote:
I can think of a few, most obvious ones:
1. Job didn't succeed and/or the file
for this (no _SUCCESS file)?
Thanks,
Rahul
--
Harsh J
-1.0.4
source code.
Thanks Regards,
Nikhil
--
Harsh J
: Thu May 02 15:05:33 CST 2013
--
Harsh J
these numbers).
On Thu, May 2, 2013 at 3:52 PM, Jack jackcpt...@gmail.com wrote:
Hi，
I check the datanode's write_block_op_avg_time on my cluster. It turns out
the value of write_block_op_avg_time is about 2ms. Is that normal ? The
replication is 3.
Regards,
Jack
--
Harsh J
to delete the data.
--
Harsh J
(otherArgs[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job,
new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
Ideas?
--
Harsh J
and your cluster hangs because of that,
you can increase
the queue length by increasing the value in
/sys/block/device_name/queue/nr_requests.
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Sunday, April 28, 2013 12:03 AM
To: user@hadoop.apache.org
Subject: Re
at 11:37 AM, Harsh J ha...@cloudera.com wrote:
The block scanner is a simple, independent operation of the DN that
runs periodically and does work in small phases, to ensure that no
blocks exist that aren't matching their checksums (its an automatic
data validator) - such that it may report
...@gmail.com wrote:
What are the hadoop properties list all the hadoop properties and please
explain in detail.
Thank you.
--
Harsh J
this.
I'm sure there are other options as well but I'm just not that familiar with
hadoop to know and I was hoping someone out there might be able to help me
out.
Thanks in advance,
-Julian
--
Harsh J
heap size, Now i will monitor the Block
scanning speed.
The 2nd idea is good, but I can not merge the small files(~1 MB) since its
all in hive table partitions.
-Selva
On Wed, May 1, 2013 at 2:25 PM, Harsh J ha...@cloudera.com wrote:
Hi,
Neither block reports nor block scanning should
,*
*National Institute of Technology,*
*Tiruchirappalli - 620015.*
*+91-9941506562*
--
Harsh J
Oops, moving for sure this time :)
On Wed, May 1, 2013 at 10:35 AM, Harsh J ha...@cloudera.com wrote:
Moving the question to Apache Avro's user@ lists. Please use the right
lists for the most relevant answers.
Avro is a different serialization technique that intends to replace
the Writable
, Apr 29, 2013 at 10:50 AM, Harsh J ha...@cloudera.com wrote:
This is rather odd and am unable to reproduce this across several
versions. It may even be something to do with all that static loading
done in the VersionInfo class but am unsure at the moment.
What does java -version print for you
that the M/R job
runs on the cluster and not locally? Or is the configuration location
magically determined?
Thank you.
--
Harsh J
one. I am using GenericOptionsParser as it is part of an example that
I copied. But I don't know why this is considered bad.
Thank you.
--
Regards,
Ted Xu
--
Regards,
Ted Xu
--
Regards, Ted Xu
--
Harsh J
--
Harsh J
/ProjectSuggestions) and was wondering if
anyone has given it a shot. If so, any success?
Tim Mauldin
--
Harsh J
old definition of task jsp-compile
warning: [options] bootstrap class path not set in conjunction with -source
1.6
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Thanks,
Amit.
--
Harsh J
Monitoring - http://sematext.com/spm/index.html
--
Harsh J
the cluster with
start-all.sh,but I found that datanode did not start
--
From Good To Great
--
Harsh J
image.png
to be with it, but then they changed what it was. Now, what I'm with
isn't it, and what's it seems weird and scary to me.
-- Abe (Grandpa) Simpson
--
Harsh J
...@comverse.com. Thank You.”
--
Harsh J
$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2706)
Does this mean something wrong with the configuration on node5? or this is
normal when we test the data over TBs.This is the first time I run data over
TBs
Any suggestion is welcome
BRs
Geelong
--
From Good To Great
--
Harsh
the
documentation).
Is that right?
But with only one server answering all requests, I can't imagine it could be
enough.
I have also read that facebook use that FS so it might be possible, isn't it?
Regards.
Anthony.
--
Harsh J
--
Harsh J
original -
De: Harsh J ha...@cloudera.com
À: user@hadoop.apache.org user@hadoop.apache.org
Envoyé: Dimanche 21 Avril 2013 14:06:45
Objet: Re: HDFS load-balancing
Are you speaking of load balancing of the metadata requests served by
the NameNode? There isn't a direct way right now
.
http://stackoverflow.com/questions/16083497/java-exception-when-creating-or-writing-files-on-mounted-hdfs
.
Now my question is that if I can do file operactions to the mounted hdfs via
java api that I wrote in the url.If not,what should be the proper way to
acomplish that?
--
Harsh J
) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1773993984
File Input Format Counters
Bytes Read=68723
File Output Format Counters
Bytes Written=0
--
Harsh J
. Each disk is 1TB. I wanna upload
2TB file to HDFS.How can I put the file to the namenode and upload to HDFS?
--
Harsh J
to do the unit test? Is there any requriement
for the tool
My hadoop version is :1.0.4
Except the MRunit, any other test tool available?
BRs
Geelong
--
From Good To Great
--
Harsh J
as process 21074. Stop it first.
Is there a way to solve this?
Thank you in advance,
Lixiang Ao
--
Harsh J
to know what will be the code for reading the Sequence
file, as the follwoing is deprecated:
SequenceFile.Reader(fs, path, conf);
Thanks,
Sumit
--
Harsh J
solved my problem.
** **
--
Harsh J
image002.png
with another user, call it:
myusername and not hadoop.
I believe it is related to the user permissions but I can't seem to get it
right.
Thanks for the help,
Amit.
--
Harsh J
of the directory. Still won't start.
I am using hadoop 1.0.4 on linux mint.
Any ideas?
Thanks,
-Julian
--
Harsh J
of submitted jobs.
On Apr 15, 2013 4:10 PM, Harsh J ha...@cloudera.com wrote:
When you say nothing happens; where exactly do you mean? The client
doesn't print anything, or the cluster doesn't run anything?
On Mon, Apr 15, 2013 at 3:36 PM, Amit Sela am...@infolinks.com wrote:
Hi all,
I'm trying
for
the steps to do so...
Thanks and Regards,
Rams
--
Harsh J
0.10.1.
Thanks,
Chris
--
Harsh J
Chen
--
Harsh J
task 'attempt_201304120228_0001_m_02_0'
Some please help me to resolve this issue.
I am using hadoop-1.0.2
Thanks,
Sandeep.
--
Harsh J
host5 /tc/rack2
If I am not wrong all hosts have the same levers.
Can you please let me know how to including the default rack value used
upon a mismatch?
Thanks,
Sandeep.
From: Harsh J ha...@cloudera.com
To: user@hadoop.apache.org user@hadoop.apache.org
build the
command/environment to launch our 'task']
--
Harsh J
installed
successfully?
Thanks
Regards
2013/4/10 Harsh J ha...@cloudera.com
Run under hadoop-hdfs project:
mvn install -Drequire.fuse=true
On Wed, Apr 10, 2013 at 7:36 PM, YouPeng Yang yypvsxf19870...@gmail.com
wrote:
Dear All
I want to integrate the FUSE with the Hadoop
something wrong?
Thanks,
Brian
--
Harsh J
as their ApplicationMaster ?
Yes to all of the above. Every MR job currently launches its own app-master.
--
Harsh J
. Can I disable this
feature or give it a very long interval without risk? My hadoop version is
0.20.2.
--
Thanks Regards
Benjamin Liu
--
Harsh J
to 1.0.4 safe? I have nearly 100T data and the data
are very important.
2013/4/10 Harsh J ha...@cloudera.com
Xibin,
I'd recommend upgrading. This was addressed quite a while ago with
HDFS-2379.
On Wed, Apr 10, 2013 at 1:35 PM, Xibin Liu xibin.liu...@gmail.comwrote:
hi all,
My data
.
* *
*
*
*Thanks *
regards
--
Harsh J
%
what can be the possible solution is it the fault of setup or anything
else
please help
--
Harsh J
, at 9:50 AM, Mark static.void@gmail.com wrote:
Forgetting Impala, what format would be best to use with daily logs?
Block-compressed sequence files?
On Apr 8, 2013, at 8:12 PM, Harsh J ha...@cloudera.com wrote:
Hey Mark,
Gzip codec creates extension .gzip, not .deflate (which
CDH4.2 to improve
short circuit read. Is there any plan to apply them to Hadoop 2.0.X
branches? Seems the short circuit read implementation in current Hadoop
2.0.x branches are quite different from HDFS-4354 and HDFS-4416.
Thanks in advance,
Jian
--
Harsh J
in Cluster2 - this job reads the data from Cluster1, 2
Thanks,
--
Best regards,
--
Harsh J
.
-Or does it get the -default- configuration via hard-coded values?
There may be a few cases of hardcodes, missing documentation and
presence in *-default.xml, but they should still be configurable via
(2) and (3).
-Or another mechanism?
--
Harsh J
is that the NameNode need not store anything locally,
but can rely on the edits being stored at the BackupNameNode which
would continuously be receiving it. When restarted, it can grab a
current checkpoint from the BNN and boot up anywhere, since there's no
local storage requirement.
--
Harsh J
not store
anything locally, you mean Primary Name Node do not need to store
checkpoint/journal locally, and only need to keep memory image up-to-date
for edits?
regards,
Lin
On Sun, Apr 7, 2013 at 3:31 PM, Harsh J ha...@cloudera.com wrote:
Hi Lin,
My reply inline.
On Sun, Apr 7, 2013
?
On Sun, Apr 7, 2013 at 4:05 PM, Harsh J ha...@cloudera.com wrote:
Yes, it need not keep an edits (transactions) stream locally cause
those are passed synchronously to the BackupNameNode, which persists
it on its behalf.
On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma lin...@gmail.com wrote:
Thanks
wrote:
I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
Standby Namenode?
--Send from my Sony mobile.
On Apr 7, 2013 9:03 PM, Harsh J ha...@cloudera.com wrote:
BackupNameNode is not present in the maintenance 1.x releases, it is a
feature added to a higher version; you can try
[org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
Oozie Launcher failed, finishing Hadoop job gracefully
Oozie Launcher ends
~Corbett Martin
Software Architect
AbsoluteAR Accounts Receivable Services - An NHIN Solution
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Tuesday
and
installs it as 3.0.0-SNAPSHOT -- but its not clear wether there is a
published version of this
snapshot jar somewhere.
--
Jay Vyas
http://jayunit100.blogspot.com
--
Harsh J
, Harsh J ha...@cloudera.com wrote:
I don't think we publish nightly or rolling jars anywhere on maven
central from trunk builds.
On Sun, Apr 7, 2013 at 2:17 AM, Jay Vyas jayunit...@gmail.com wrote:
Hi guys:
Is there a mvn repo for hadoop's 3.0.0 trunk build?
Clearly the hadoop
/troya.adromeda
Website : www.meruvian.org
--
Harsh J
://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Apr 3, 2013 at 7:45 AM, Harsh J ha...@cloudera.com wrote:
Not quite sure if I got your question. These tidbits may help though,
from what I can understand:
* LocalFileSystem's listing uses Java's APIs for file/dir listing, and
has no concept
. In case a setup goes with option #1 (writing
to NFS, no SNN) , then who does this merging.
Thanks,
Rahul
--
Harsh J
nodes?
2013/4/2 Henry JunYoung KIM henry.jy...@gmail.com
15 for datanodes and 3 for replication factor.
2013. 4. 1., 오후 3:23, varun kumar varun@gmail.com 작성:
How many nodes do you have and replication factor for it.
--
Harsh J
that you've mentioned.
its easier to do so in a rolling manner without need of a
decommission.
to check my understanding, just shutting down 2 of them and then 2 more and
then 2 more without decommissions.
is this correct?
2013. 4. 2., 오후 4:54, Harsh J ha...@cloudera.com 작성:
Note though
for the first time because till now I never tried running MR jobs on local
files.
Thank you so much for your time?
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
--
Harsh J
to the name node so there is no chance it is
the name node.
Please clarify.
Thanks
Sai
--
Harsh J
).
I looked at the directory today, and it's hosting thousands of
directories and more than 8GB of data there.
Is there a way to automatically delete this directory when the job is
done?
Thanks,
JM
--
Harsh J
- The FSDataOutputStream hangs in the out.close() instruction. Why it
hangs? What should I do to solve this?
--
Best regards,
--
Harsh J
,
for file
ip-XXX-XXX-XXX-123.eu-west-1.compute.internal/XXX.XXX.XXX.123:50010:-4664365259588027316,
for block
-4664365259588027316_2050
This error means than it wasn't possible to write on a remote host?
On 27 March 2013 12:24, Harsh J ha...@cloudera.com wrote:
You can try to take
it will not print it. or does it refer to 120 options of
counters in an enum that we can define.
Any help is really appreciated.
Thanks
Sai
--
Harsh J
they different.
Thanks
Sai
--
Harsh J
it will create a file for
each reduce task, and I need a single file. The more general issue here is
how to distribute data produced by reducer to all the mappers?
Thanks.
--
Harsh J
Thanks Arun; I've filed https://issues.apache.org/jira/browse/YARN-511
On Thu, Mar 28, 2013 at 2:41 AM, Arun C Murthy a...@hortonworks.com wrote:
That seems like a bug - we should at least throw a better error.
On Mar 25, 2013, at 5:29 PM, Harsh J wrote:
YARN does not seem to be checking
as it gets
created in STORE command.
What is wrong in this
--
Harsh J
order? Are
there are any configuration options controlling how the input values are
sorted?
I know that the secondary sort way can be used to achieve the same
effect. I am not asking for a workaround.
--
Jingguo
--
Harsh J
of the current stable build (1.1.2 ), is there
any reason to use the CheckPointNode over the BackupNode?
It seems that we need to choose one or the other, and from the docs
it seems like the BackupNode is more efficient in its processes.
--
Regards,
Varun Kumar.P
--
Harsh J
--
Harsh J
average time peaks in Ganglia up to 20-70 seconds
while SecondaryNameNode performs checkpointing.
See attached screenshots please.
I would like to clarify if it is Ok, or not. And what kind of consequences
and risks it could bring up.
--
Best Regards
Ivan Tretyakov
--
Harsh J
...@griddynamics.com wrote:
Thanks Harsh!
My image size is about 3.1 Gb.
Yes, I think feature from HDFS-1457 is what I need, but unfortunately it is
not available in version of hadoop we use.
What kind of risks pose by these peaks.
On Mon, Mar 25, 2013 at 7:31 PM, Harsh J ha...@cloudera.com
)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
at java.lang.Thread.run(Thread.java:662)
Where am I making the mistake?
regards
tmp
--
Harsh J
of method?
Thanks to all.
--
Harsh J
to another reduce.
One more question then,
Creating 2 diffirent job, secondone has only reduce for example, is it
possible to pass first jobs output as argument to second job?
On Sun, Mar 24, 2013 at 5:44 PM, Harsh J ha...@cloudera.com wrote:
You seem to want to re-sort/partition your data
PriviledgedActionException?
Thanks,
--
Harsh J
and generate the packages with native libs as well.
On Mon, Mar 25, 2013 at 9:27 AM, Azuryy Yu azury...@gmail.com wrote:
Hi,
How to get hadoop-2.0.3-alpha native libraries, it was compiled under 32bits
OS in the released package currently.
--
Harsh J
heap.
Appreciate in helping me understand the scenario.
Regards
Nagarjuna K
--
Harsh J
/$DIRECTORY is returning
NULL and that is the reason it throws null value for echo $TestDir and it
goes to else part and displays Directory does not Exist
What is wrong in the above shell script that I have written
-
--
Harsh J
is... SecondaryNameNode is what I should be installing
here. And the SecondaryNameNode is essentially just an earlier version of
the checkpoint node, in terms of functionality. If I understood everything
correctly. Can you confirm?
Thanks,
David
-Original Message-
From: Harsh J [mailto:ha
that they are starting once again.
When i check the logs there are no errors in that.
Could you guys please help me in this
Do you mind sharing the .log and .out files both of the HBase Master alone?
--
Harsh J
is will the setup cleanup run n number of times which
means once for each mapper or for all the mappers they will run only once.
Any help is appreciated.
Thanks
Sai
--
Harsh J
been changed. Most importantly the data
in HDFS is still sitting there.
Is there a way to recover this cluster to a useable state?
thanks
John
--
http://balajin.net/blog
http://flic.kr/balajijegan
--
Harsh J
performs better or worst and I just have something configured wrong?.
Basically my objective isn't performance, it's that the server must
not shut itself down, it can slow down but not shut off.
--
Ted.
--
Harsh J
401 - 500 of 2355 matches
Mail list logo
