Documentation is wrong. Implementation wins.
Could you please file a bug.
Thanks,
--Konstantin
Dima Rzhevskiy wrote:
Hi all
I try get length of file hadoop(RawFilesysten or hdfs) .
In javadoc method org.apache.hadoop.fs.FileStatus.getLen() writtend that
this method return the length of this
These links should help you to rebalance the nodes:
http://developer.yahoo.com/hadoop/tutorial/module2.html#rebalancing
http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Rebalancer
http://hadoop.apache.org/core/docs/current/commands_manual.html#balancer
There are some name-node memory estimates in this jira.
http://issues.apache.org/jira/browse/HADOOP-1687
With 16 GB you can normally have 60 million objects (files
+ blocks) on the name-node. The number of files would depend
on the file to block ratio.
--Konstantin
Brian Bockelman wrote:
On
. Whats the step by step instruction to achieve it?.. i hv google it, got a
lot of different opinions n m totally confused now.
Thanks,
Raakhi
On Tue, May 26, 2009 at 11:27 PM, Konstantin Shvachko s...@yahoo-inc.comwrote:
Hi Rakhi,
This is because your name-node is trying to -importCheckpoint
Hi Rakhi,
This is because your name-node is trying to -importCheckpoint from a directory,
which is locked by secondary name-node.
The secondary node is also running in your case, right?
You should use -importCheckpoint as the last resort, when name-node's
directories
are damaged.
In regular
I just wanted to add to this one other published benchmark
http://developer.yahoo.net/blogs/hadoop/2008/09/scaling_hadoop_to_4000_nodes_a.html
In this example on a very busy cluster of 4000 nodes both read and write
throughputs
were close to the local disk bandwidth.
This benchmark (called
Clarifying: port # is missing in your configuration, should be
property
namefs.default.name/name
valuehdfs://hvcwydev0601:8020/value
/property
where 8020 is your port number.
--Konstantin
Hairong Kuang wrote:
Please try using the port number 8020.
Hairong
On 3/11/09 9:42 AM, Stuart
This is not about the default port.
The port was not specified at all in the original configuration.
--Konstantin
Doug Cutting wrote:
Konstantin Shvachko wrote:
Clarifying: port # is missing in your configuration, should be
property
namefs.default.name/name
valuehdfs://hvcwydev0601:8020
Are you sure you were using 0.19 not 0.20 ?
For 0.17 please check that configuration file hadoop-site.xml exists
in your configuration directory is not empty and points to hdfs rather
than local file system, which it does buy default.
In 0.17 all config variables have been in a common file. 0.19
Yes guys. We observed such problems.
They will be common for 0.18.2 and 0.19.0 exactly as you
described it when data-nodes become unstable.
There were several issues, please take a look
HADOOP-4997 workaround for tmp file handling on DataNodes
HADOOP-4663 - links to other related
HADOOP-4810
Joe,
It looks like you edits file is corrupted or truncated.
Most probably the last modification was not written to it,
when the name-node was turned off. This may happen if the
node crashes depending on the underlying local file system I guess.
Here are some options for you to consider:
- try
Did you look at Zookeeper?
Thanks,
--Konstantin
Sagar Naik wrote:
I would like to implement a locking mechanism across the hdfs cluster
I assume there is no inherent support for it
I was going to do it with files. According to my knowledge, file
creation is an atomic operation. So the
should override
mapred.map.tasks.
nrFiles is the number of the files which will be created and
mapred.map.tasks is the number how many splits will be done by the input
file.
Thanks
Konstantin Shvachko wrote:
Hi tienduc_dinh,
Just a bit of a background, which should help to answer your
Hi, Jonathan.
The problem is that the local drive(s) you use for dfs.name.dir became
unaccessible. So the name-node is not able to persist name-space modifications
anymore, and therefore self terminated.
The rest are the consequences.
This is the core message
2008-12-15 01:49:31,178 FATAL
Hi Jason,
2 million blocks per data-node is not going to work.
There were discussions about it previously, please
check the mail archives.
This means you have a lot of very small files, which
HDFS is not designed to support. A general recommendation
is to group small files into large ones,
scales well! :-)
Sorry for my poor english skill and thanks very much for your help.
Tien Duc Dinh
Konstantin Shvachko wrote:
Hi tienduc_dinh,
Just a bit of a background, which should help to answer your questions.
TestDFSIO mappers perform one operation (read or write) each, measure
the time
Hi tienduc_dinh,
Just a bit of a background, which should help to answer your questions.
TestDFSIO mappers perform one operation (read or write) each, measure
the time taken by the operation and output the following three values:
(I am intentionally omitting some other output stuff.)
- size(i)
-
1) If i set value of dfs.replication to 3 only in hadoop-site.xml of
namenode(master) and
then restart the cluster will this take effect. or i have to change
hadoop-site.xml at all slaves ?
dfs.replication is the name-node parameter, so you need to restart
only the name-node in order to
Sandeep Dhawan wrote:
Hi,
I have a setup of 2-node Hadoop cluster running on Windows using cygwin.
When I open up the web gui to view the number of Live Nodes, it shows 2.
But when I kill the slave node and refreshes the gui, it still shows the
number of Live Nodes as 2.
Its only after
Brian Bockelman wrote:
Hello all,
I'd like to take the datanode's capability to handle multiple
directories to a somewhat-extreme, and get feedback on how well this
might work.
We have a few large RAID servers (12 to 48 disks) which we'd like to
transition to Hadoop. I'd like to mount
This is probably related to HADOOP-4795.
http://issues.apache.org/jira/browse/HADOOP-4795
We are testing it on 0.18 now. Should be committed soon.
Please let know if it is something else.
Thanks,
--Konstantin
Karl Kleinpaste wrote:
We have a cluster comprised of 21 nodes holding a total
Just for the reference these links:
http://wiki.apache.org/hadoop/FAQ#17
http://hadoop.apache.org/core/docs/r0.19.0/hdfs_user_guide.html#DFSAdmin+Command
Decommissioning is not happening at once.
-refreshNodes just starts the process, but does not complete it.
There could be a lot of blocks on
This is very nice.
A suggestion if it is related to the development status.
Do you think guys you can analyze which questions are
discussed most often in the mailing lists, so that we could
update our FAQs based on that.
Thanks,
--Konstantin
Alex Loddengaard wrote:
Some engineers here at
Bagri,
According to the numbers you posted your cluster has 6,000,000 block replicas
and only 12 data-nodes. The blocks are small on average about 78KB according
to fsck. So each node contains about 40GB worth of block data.
But the number of blocks is really huge 500,000 per node. Is my math
to point to dirX and start
new NameNode)?
Thanks,
Tomislav
On Fri, 2008-10-31 at 11:38 -0700, Konstantin Shvachko wrote:
True, dfs.http.address is the NN Web UI address.
This where the NN http server runs. Besides the Web UI there also
a servlet running on that server which is used
much like the one given by Dhruba Borthakur
and augmented by Konstantin Shvachko later in the thread but I never did it
myself.
One thing should be clear though, the NN is and will remain a SPOF (just
like HBase's Master) as long as a distributed manager service (like
Zookeeper) is not plugged
You just start the new data-node as the cluster is running using
bin/hadoop datanode
The configuration on the new data-node should be the same as on other nodes.
The data-node should join the cluster automatically.
Formatting will destroy your file system.
--Konstantin
David Wei wrote:
Well, in
We use TestDFSIO for measuring IO performance on our clusters.
It is called a test, but in fact its a benchmark.
It runs a map-reduce job, which either writes to or reads from files
and collects statistics.
Another thing is that Hadoop automatically collects metrics.
Like number of creates,
Peter,
You are likely to hit memory limitations on the name-node.
With 100 million small files it will need to support 200 mln objects,
which will require roughly 30 GB of RAM on the name-node.
You may also consider hadoop archives or present your files as a
collection of records and use Pig,
NameNodeFailover http://wiki.apache.org/hadoop/NameNodeFailover, with a
SecondaryNameNode http://wiki.apache.org/hadoop/SecondaryNameNode hosted
I think it is wrong, please correct it.
You probably look at some cached results. Both pages do not exist.
The first one was a cause of confusion and
Great!
If you decide to run TestDFSIO on your cluster, please let me know.
I'll run the same on the same scale with hdfs and we can compare the numbers.
--Konstantin
Joel Welling wrote:
That seems to have done the trick! I am now running Hadoop 0.18
straight out of Lustre, without an
I can see 3 reasons for that:
1. dfs.data.dir is pointing to a wrong data-node storage directory, or
2. somebody manually moved directory hadoop into /home/hadoop/dfs/tmp/,
which is supposed to contain only block files named blk_number
3. There is some collision of configuration variables so that
PROTECTED]
On Fri, 2008-08-22 at 15:59 -0700, Konstantin Shvachko wrote:
I think the solution should be easier than Arun and Steve advise.
Lustre is already mounted as a local directory on each cluster machines, right?
Say, it is mounted on /mnt/lustre.
Then you configure hadoop-site.xml and set
I think the solution should be easier than Arun and Steve advise.
Lustre is already mounted as a local directory on each cluster machines, right?
Say, it is mounted on /mnt/lustre.
Then you configure hadoop-site.xml and set
property
namefs.default.name/name
valuefile:///mnt/lustre/value
makes it impossible to use the distributed jar file with any
external application? (Works only with a local recompile)
Thibaut
Konstantin Shvachko wrote:
But you won't get append in 0.18. It was committed for 0.19.
--konstantin
Arun C Murthy wrote:
On Aug 12, 2008, at 11:51 PM, 11 Nov. wrote
But you won't get append in 0.18. It was committed for 0.19.
--konstantin
Arun C Murthy wrote:
On Aug 12, 2008, at 11:51 PM, 11 Nov. wrote:
Hi colleagues,
As you know, the append writer will be available in version 0.18.
We are
here waiting for the feature and want to know the rough
I was wondering around Hadoop wiki and found this page dedicated to name-node
failover.
http://wiki.apache.org/hadoop/NameNodeFailover
I think it is confusing, contradicts other documentation on the subject and
contains incorrect facts. See
On hdfs see
http://wiki.apache.org/hadoop/FAQ#15
In addition to the James's suggestion you can also specify dfs.name.dir
for the name-node to store extra copies of the namespace.
James Moore wrote:
On Tue, Jul 29, 2008 at 6:37 PM, Rafael Turk [EMAIL PROTECTED] wrote:
Hi All,
I´m setting up
You should also run a secondary name-node, which does namespace checkpoints and
shrinks the edits log file.
And this is exactly the case when the checkpoint image comes handy.
http://wiki.apache.org/hadoop/FAQ#7
In the recent release you can start the primary node using the secondary image
Yes this is a known bug.
http://issues.apache.org/jira/browse/HADOOP-1212
You should manually remove current directory from every data-node
after reformatting the name-node and start the cluster again.
I do not believe there is any other way.
Thanks,
--Konstantin
Taeho Kang wrote:
No, I don't
lohit wrote:
1. Can we have multiple files in DFS use different block sizes ?
No, current this might not be possible, we have fixed sized blocks.
Actually you can. HDFS provides api to specify block size
when you create a file. Here is the link
Also HDFS might be critical since to access your data you need to close the
file
Not anymore. Since 0.16 files are readable while being written to.
it as fast as possible. I need to be able to maintain some guaranteed
max. processing time, for example under 3 minutes.
It looks like you do
Did you close those files?
If not they may be empty.
??? wrote:
Dears,
I use hadoop-0.16.4 to do some work and found a error which i can't get the
reasons.
The scenario is like this: In the reduce step, instead of using
OutputCollector to write result, i use FSDataOutputStream to write
Looks like the client machine from which you call -put cannot connect to the
data-nodes.
It could be firewall or wrong configuration parameters that you use for the
client.
Alexander Arimond wrote:
hi,
i'm new in hadoop and im just testing it at the moment.
i set up a cluster with 2 nodes
Would the new archive feature HADOOP-3307 that is currently being developed
help this problem?
http://issues.apache.org/jira/browse/HADOOP-3307
--Konstantin
Subramaniam Krishnan wrote:
We have actually written a custom Multi File Splitter that collapses all
the small files to a single split
Edward,
testDU() writes a 32K file to the local fs and then verifies whether the value
reported by du
changes exactly to the amount written.
Although this is true for most block oriented file systems it might not be true
for some.
I suspect that in your case the file is written to tmpfs, which
Yuri,
The NullPointerException should be fixed as Dhruba proposed.
We do not have any secondary nn web interface as of today.
The http server is used for transferring data between the primary and the
secondary.
I don't see we can display anything useful on the secondary web UI except for
the
Unfortunately we do not have an api for the secondary nn that would allow
browsing the checkpoint.
I agree it would be nice to have one.
Thanks for filing the issue.
--Konstantin
Yuri Pradkin wrote:
On Tuesday 08 April 2008 11:54:35 am Konstantin Shvachko wrote:
If you have anything in mind
Usually a build takes 2 hours or less.
This one is stuck and I don't see changes in the QUEUE OF PENDING PATCHES when
I submit a patch.
I guess something is wrong with Hadson.
Could anybody please check.
--Konstantin
You should use host:port rather than just port.
See HADOOP-2404, and HADOOP-2185.
Ved Prakash wrote:
Hi friends,
I have been trying to start hadoop on the master but it doesn't start the
name node on it, checking the logs I found the following error
hadoop-site.xml listing
configuration
Naama Kraus wrote:
Hi,
Thanks all for the input.
Here are my further questions:
I can consolidate data off-line to have big enough files (64M) or copy to
dfs smaller files and then consolidate using MapReduce.
You can also let MapReduce read your original small local files and write them
André,
You can try to rollback.
You did use upgrade when you switched to the new trunk, right?
--Konstantin
Raghu Angadi wrote:
André Martin wrote:
Hi Raghu,
done: https://issues.apache.org/jira/browse/HADOOP-2873
Subsequent tries did not succeed - so it looks like I need to
re-format the
Yes, please file a bug.
There are file systems with different block sizes out there Linux or Solaris.
Thanks,
--Konstantin
Martin Traverso wrote:
I think I found the issue. The class org.apache.hadoop.fs.DU assumes
1024-byte blocks when reporting usage information:
this.used =
53 matches
Mail list logo
