Re: Too many open files error, which gets resolved after some time
Hi. Thank for the advice, just to clarify: The upgrade of you speak of of cleaning the pipes/epolls more often, is regarding the issue discussed (HADOOP-4346, fixed in my distribution), or it's some other issue? If yes, does it has a ticket I can see, or it should be filled to Jira? Thanks! 2009/6/23 Brian Bockelman bbock...@cse.unl.edu Hey Stas, It sounds like it's technically possible, but it also sounds like a horrible hack: I'd avoid this at all expense. This is how cruft is born. The pipes/epolls are something that eventually get cleaned up - but they don't get cleaned up often enough for your cluster. I would recommend just increasing the limit on the node itself and then wait for an upgrade to solve this. Brian On Jun 23, 2009, at 3:31 AM, Stas Oskin wrote: Hi. Any idea if calling System.gc() periodically will help reducing the amount of pipes / epolls? Thanks for your opinion! 2009/6/22 Stas Oskin stas.os...@gmail.com Ok, seems this issue is already patched in the Hadoop distro I'm using (Cloudera). Any idea if I still should call GC manually/periodically to clean out all the stale pipes / epolls? 2009/6/22 Steve Loughran ste...@apache.org Stas Oskin wrote: Hi. So what would be the recommended approach to pre-0.20.x series? To insure each file is used only by one thread, and then it safe to close the handle in that thread? Regards. good question -I'm not sure. For anythiong you get with FileSystem.get(), its now dangerous to close, so try just setting the reference to null and hoping that GC will do the finalize() when needed
Re: Too many open files error, which gets resolved after some time
Hi. In my testings, I typically opened between 20 and 40 concurrent streams. Regards. 2009/6/23 Raghu Angadi rang...@yahoo-inc.com Stas Oskin wrote: Hi. Any idea if calling System.gc() periodically will help reducing the amount of pipes / epolls? since you have HADOOP-4346, you should not have excessive epoll/pipe fds open. First of all do you still have the problem? If yes, how many hadoop streams do you have at a time? System.gc() won't help if you have HADOOP-4346. Ragu. Thanks for your opinion! 2009/6/22 Stas Oskin stas.os...@gmail.com Ok, seems this issue is already patched in the Hadoop distro I'm using (Cloudera). Any idea if I still should call GC manually/periodically to clean out all the stale pipes / epolls? 2009/6/22 Steve Loughran ste...@apache.org Stas Oskin wrote: Hi. So what would be the recommended approach to pre-0.20.x series? To insure each file is used only by one thread, and then it safe to close the handle in that thread? Regards. good question -I'm not sure. For anythiong you get with FileSystem.get(), its now dangerous to close, so try just setting the reference to null and hoping that GC will do the finalize() when needed
Re: Too many open files error, which gets resolved after some time
Hi. So if I open one stream, it should be 4? 2009/6/23 Raghu Angadi rang...@yahoo-inc.com how many threads do you have? Number of active threads is very important. Normally, #fds = (3 * #threads_blocked_on_io) + #streams 12 per stream is certainly way off. Raghu. Stas Oskin wrote: Hi. In my case it was actually ~ 12 fd's per stream, which included pipes and epolls. Could it be that HDFS opens 3 x 3 (input - output - epoll) fd's per each thread, which make it close to the number I mentioned? Or it always 3 at maximum per thread / stream? Up to 10 sec looks quite the correct number, it seems it gets freed arround this time indeed. Regards. 2009/6/23 Raghu Angadi rang...@yahoo-inc.com To be more accurate, once you have HADOOP-4346, fds for epoll and pipes = 3 * threads blocked on Hadoop I/O Unless you have hundreds of threads at a time, you should not see hundreds of these. These fds stay up to 10sec even after the threads exit. I am a bit confused about your exact situation. Please check number of threads if you still facing the problem. Raghu. Raghu Angadi wrote: since you have HADOOP-4346, you should not have excessive epoll/pipe fds open. First of all do you still have the problem? If yes, how many hadoop streams do you have at a time? System.gc() won't help if you have HADOOP-4346. Ragu. Thanks for your opinion! 2009/6/22 Stas Oskin stas.os...@gmail.com Ok, seems this issue is already patched in the Hadoop distro I'm using (Cloudera). Any idea if I still should call GC manually/periodically to clean out all the stale pipes / epolls? 2009/6/22 Steve Loughran ste...@apache.org Stas Oskin wrote: Hi. So what would be the recommended approach to pre-0.20.x series? To insure each file is used only by one thread, and then it safe to close the handle in that thread? Regards. good question -I'm not sure. For anythiong you get with FileSystem.get(), its now dangerous to close, so try just setting the reference to null and hoping that GC will do the finalize() when needed
Re: Too many open files error, which gets resolved after some time
Hi. I've started doing just that, and indeed the amount of fd's of the DataNode process have reduced significantly. My problem is that my own app, which works with DFS, still have dozens of pipes and epolls open. The usual level seems to be about 300-400 fd's, but when I access the DFS for accessing several files concurrently, this number easily climbs to 700-800. Moreover, the number sometimes seems to be stuck above 1000, and only shutting down the app, at it's start, brings this number back to 300-400. Any idea why this happens, and what else can be released to get it working? Also, every file I open seems to bump the fd count sometimes as high as by 12. Any idea why single file requires so many fd's? Thanks in advance. 2009/6/22 jason hadoop jason.had...@gmail.com Yes. Otherwise the file descriptors will flow away like water. I also strongly suggest having at least 64k file descriptors as the open file limit. On Sun, Jun 21, 2009 at 12:43 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. Thanks for the advice. So you advice explicitly closing each and every file handle that I receive from HDFS? Regards. 2009/6/21 jason hadoop jason.had...@gmail.com Just to be clear, I second Brian's opinion. Relying on finalizes is a very good way to run out of file descriptors. On Sun, Jun 21, 2009 at 9:32 AM, brian.lev...@nokia.com wrote: IMHO, you should never rely on finalizers to release scarce resources since you don't know when the finalizer will get called, if ever. -brian -Original Message- From: ext jason hadoop [mailto:jason.had...@gmail.com] Sent: Sunday, June 21, 2009 11:19 AM To: core-user@hadoop.apache.org Subject: Re: Too many open files error, which gets resolved after some time HDFS/DFS client uses quite a few file descriptors for each open file. Many application developers (but not the hadoop core) rely on the JVM finalizer methods to close open files. This combination, expecially when many HDFS files are open can result in very large demands for file descriptors for Hadoop clients. We as a general rule never run a cluster with nofile less that 64k, and for larger clusters with demanding applications have had it set 10x higher. I also believe there was a set of JVM versions that leaked file descriptors used for NIO in the HDFS core. I do not recall the exact details. On Sun, Jun 21, 2009 at 5:27 AM, Stas Oskin stas.os...@gmail.com wrote: Hi. After tracing some more with the lsof utility, and I managed to stop the growth on the DataNode process, but still have issues with my DFS client. It seems that my DFS client opens hundreds of pipes and eventpolls. Here is a small part of the lsof output: java10508 root 387w FIFO0,6 6142565 pipe java10508 root 388r FIFO0,6 6142565 pipe java10508 root 389u 0,100 6142566 eventpoll java10508 root 390u FIFO0,6 6135311 pipe java10508 root 391r FIFO0,6 6135311 pipe java10508 root 392u 0,100 6135312 eventpoll java10508 root 393r FIFO0,6 6148234 pipe java10508 root 394w FIFO0,6 6142570 pipe java10508 root 395r FIFO0,6 6135857 pipe java10508 root 396r FIFO0,6 6142570 pipe java10508 root 397r 0,100 6142571 eventpoll java10508 root 398u FIFO0,6 6135319 pipe java10508 root 399w FIFO0,6 6135319 pipe I'm using FSDataInputStream and FSDataOutputStream, so this might be related to pipes? So, my questions are: 1) What happens these pipes/epolls to appear? 2) More important, how I can prevent their accumation and growth? Thanks in advance! 2009/6/21 Stas Oskin stas.os...@gmail.com Hi. I have HDFS client and HDFS datanode running on same machine. When I'm trying to access a dozen of files at once from the client, several times in a row, I'm starting to receive the following errors on client, and HDFS browse function. HDFS Client: Could not get block locations. Aborting... HDFS browse: Too many open files I can increase the maximum number of files that can opened, as I have it set to the default 1024, but would like to first solve the problem, as larger value just means it would run out of files again later on. So my questions
Re: Too many open files error, which gets resolved after some time
Hi. So what would be the recommended approach to pre-0.20.x series? To insure each file is used only by one thread, and then it safe to close the handle in that thread? Regards. 2009/6/22 Steve Loughran ste...@apache.org Raghu Angadi wrote: Is this before 0.20.0? Assuming you have closed these streams, it is mostly https://issues.apache.org/jira/browse/HADOOP-4346 It is the JDK internal implementation that depends on GC to free up its cache of selectors. HADOOP-4346 avoids this by using hadoop's own cache. yes, and it's that change that led to my stack traces :( http://jira.smartfrog.org/jira/browse/SFOS-1208
Re: Too many open files error, which gets resolved after some time
Hi Rahid. A question - this issue does not influence Hadoop itself (DataNodes, etc...), but rather influence any application using DFS, correct? If so, without patching iI should either to increase fd limit (which might fill-up as well???), or periodically launch the GC? Regards. 2009/6/22 Raghu Angadi rang...@yahoo-inc.com 64k might help in the sense, you might hit GC before you hit the limit. Otherwise, your only options are to use the patch attached to HADOOP-4346 or run System.gc() occasionally. I think it should be committed to 0.18.4 Raghu. Stas Oskin wrote: Hi. Yes, it happens with 0.18.3. I'm closing now every FSData stream I receive from HDFS, so the number of open fd's in DataNode is reduced. Problem is that my own DFS client still have a high number of fd's open, mostly pipes and epolls. They sometimes quickly drop to the level of ~400 - 500, and sometimes just stuck at ~1000. I'm still trying to find out how well it behaves if I set the maximum fd number to 65K. Regards. 2009/6/22 Raghu Angadi rang...@yahoo-inc.com Is this before 0.20.0? Assuming you have closed these streams, it is mostly https://issues.apache.org/jira/browse/HADOOP-4346 It is the JDK internal implementation that depends on GC to free up its cache of selectors. HADOOP-4346 avoids this by using hadoop's own cache. Raghu. Stas Oskin wrote: Hi. After tracing some more with the lsof utility, and I managed to stop the growth on the DataNode process, but still have issues with my DFS client. It seems that my DFS client opens hundreds of pipes and eventpolls. Here is a small part of the lsof output: java10508 root 387w FIFO0,6 6142565 pipe java10508 root 388r FIFO0,6 6142565 pipe java10508 root 389u 0,100 6142566 eventpoll java10508 root 390u FIFO0,6 6135311 pipe java10508 root 391r FIFO0,6 6135311 pipe java10508 root 392u 0,100 6135312 eventpoll java10508 root 393r FIFO0,6 6148234 pipe java10508 root 394w FIFO0,6 6142570 pipe java10508 root 395r FIFO0,6 6135857 pipe java10508 root 396r FIFO0,6 6142570 pipe java10508 root 397r 0,100 6142571 eventpoll java10508 root 398u FIFO0,6 6135319 pipe java10508 root 399w FIFO0,6 6135319 pipe I'm using FSDataInputStream and FSDataOutputStream, so this might be related to pipes? So, my questions are: 1) What happens these pipes/epolls to appear? 2) More important, how I can prevent their accumation and growth? Thanks in advance! 2009/6/21 Stas Oskin stas.os...@gmail.com Hi. I have HDFS client and HDFS datanode running on same machine. When I'm trying to access a dozen of files at once from the client, several times in a row, I'm starting to receive the following errors on client, and HDFS browse function. HDFS Client: Could not get block locations. Aborting... HDFS browse: Too many open files I can increase the maximum number of files that can opened, as I have it set to the default 1024, but would like to first solve the problem, as larger value just means it would run out of files again later on. So my questions are: 1) Does the HDFS datanode keeps any files opened, even after the HDFS client have already closed them? 2) Is it possible to find out, who keeps the opened files - datanode or client (so I could pin-point the source of the problem). Thanks in advance!
Re: Too many open files error, which gets resolved after some time
Ok, seems this issue is already patched in the Hadoop distro I'm using (Cloudera). Any idea if I still should call GC manually/periodically to clean out all the stale pipes / epolls? 2009/6/22 Steve Loughran ste...@apache.org Stas Oskin wrote: Hi. So what would be the recommended approach to pre-0.20.x series? To insure each file is used only by one thread, and then it safe to close the handle in that thread? Regards. good question -I'm not sure. For anythiong you get with FileSystem.get(), its now dangerous to close, so try just setting the reference to null and hoping that GC will do the finalize() when needed
Re: Too many open files error, which gets resolved after some time
Hi. After tracing some more with the lsof utility, and I managed to stop the growth on the DataNode process, but still have issues with my DFS client. It seems that my DFS client opens hundreds of pipes and eventpolls. Here is a small part of the lsof output: java10508 root 387w FIFO0,6 6142565 pipe java10508 root 388r FIFO0,6 6142565 pipe java10508 root 389u 0,100 6142566 eventpoll java10508 root 390u FIFO0,6 6135311 pipe java10508 root 391r FIFO0,6 6135311 pipe java10508 root 392u 0,100 6135312 eventpoll java10508 root 393r FIFO0,6 6148234 pipe java10508 root 394w FIFO0,6 6142570 pipe java10508 root 395r FIFO0,6 6135857 pipe java10508 root 396r FIFO0,6 6142570 pipe java10508 root 397r 0,100 6142571 eventpoll java10508 root 398u FIFO0,6 6135319 pipe java10508 root 399w FIFO0,6 6135319 pipe I'm using FSDataInputStream and FSDataOutputStream, so this might be related to pipes? So, my questions are: 1) What happens these pipes/epolls to appear? 2) More important, how I can prevent their accumation and growth? Thanks in advance! 2009/6/21 Stas Oskin stas.os...@gmail.com Hi. I have HDFS client and HDFS datanode running on same machine. When I'm trying to access a dozen of files at once from the client, several times in a row, I'm starting to receive the following errors on client, and HDFS browse function. HDFS Client: Could not get block locations. Aborting... HDFS browse: Too many open files I can increase the maximum number of files that can opened, as I have it set to the default 1024, but would like to first solve the problem, as larger value just means it would run out of files again later on. So my questions are: 1) Does the HDFS datanode keeps any files opened, even after the HDFS client have already closed them? 2) Is it possible to find out, who keeps the opened files - datanode or client (so I could pin-point the source of the problem). Thanks in advance!
Re: InputStream.open() efficiency
Hi. Thanks for the advice. Regards. 2009/5/26 Raghu Angadi rang...@yahoo-inc.com Stas Oskin wrote: Hi. Thanks for the answer. Would up to 5 minute of handlers cause any issues? 5 min should not cause any issues.. And same about writing? writing is not affected by the couple of issues I mentioned. Writing over a long time should work as well as writing over shorter time. Raghu. Regards. 2009/5/26 Raghu Angadi rang...@yahoo-inc.com 'in.seek(); in.read()' is certainly better than, 'in = fs.open(); in.seek(); in.read()' The difference is is exactly one open() call. So you would save an RPC to NameNode. There are couple of issues that affect apps that keep the handlers open very long time (many hours to days).. but those will be fixed soon. Raghu. Stas Oskin wrote: Hi. I'm looking to find out, how the InputStream.open() + skip(), compares to keeping a handle of InputStream() and just seeking the position. Has anyone compared these approaches, and can advice on their speed? Regards.
Re: Specifying NameNode externally to hadoop-site.xml
Hi. Thanks for the tip. Regards. 2009/5/26 Aaron Kimball aa...@cloudera.com Same way. Configuration conf = new Configuration(); conf.set(fs.default.name, hdfs://foo); FileSystem fs = FileSystem.get(conf); - Aaron On Mon, May 25, 2009 at 1:02 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. And if I don't use jobs but only DFS for now? Regards. 2009/5/25 jason hadoop jason.had...@gmail.com conf.set(fs.default.name, hdfs://host:port); where conf is the JobConf object of your job, before you submit it. On Mon, May 25, 2009 at 10:16 AM, Stas Oskin stas.os...@gmail.com wrote: Hi. Thanks for the tip, but is it possible to set this in dynamic way via code? Thanks. 2009/5/25 jason hadoop jason.had...@gmail.com if you launch your jobs via bin/hadoop jar jar_file [main class] [options] you can simply specify -fs hdfs://host:port before the jar_file On Sun, May 24, 2009 at 3:02 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. I'm looking to move the Hadoop NameNode URL outside the hadoop-site.xml file, so I could set it at the run-time. Any idea how to do it? Or perhaps there is another configuration that can be applied to the FileSystem object? Regards. -- Alpha Chapters of my book on Hadoop are available http://www.apress.com/book/view/9781430219422 www.prohadoopbook.com a community for Hadoop Professionals -- Alpha Chapters of my book on Hadoop are available http://www.apress.com/book/view/9781430219422 www.prohadoopbook.com a community for Hadoop Professionals
When directly writing to HDFS, the data is moved only on file close
Hi.
I'm trying to continuously write data to HDFS via OutputStream(), and want
to be able to read it at the same time from another client.
Problem is, that after the file is created on HDFS with size of 0, it stays
that way, and only fills up when I close the OutputStream().
Here is a simple code sample illustrating this issue:
try {
FSDataOutputStream out=fileSystem.create(new
Path(/test/test.bin)); // Here the file created with 0 size
for(int i=0;i1000;i++)
{
out.write(1); // Still stays 0
out.flush(); // Even when I flush it out???
}
Thread.currentThread().sleep(1);
out.close(); //Only here the file is updated
} catch (Exception e) {
e.printStackTrace();
}
So, two questions here:
1) How it's possible to write the files directly to HDFS, and have them
update there immedaitely?
2) Just for information, in this case, where the file content stays all the
time - on server local disk, in memory, etc...?
Thanks in advance.
InputStream.open() efficiency
Hi. I'm looking to find out, how the InputStream.open() + skip(), compares to keeping a handle of InputStream() and just seeking the position. Has anyone compared these approaches, and can advice on their speed? Regards.
Re: When directly writing to HDFS, the data is moved only on file close
Hi.
Does it means there is no way to access the data being written to HDFS,
while it's written?
Where it's stored then via the writing - on cluster or on local disks?
Thanks.
2009/5/26 Tom White t...@cloudera.com
This feature is not available yet, and is still under active
discussion. (The current version of HDFS will make the previous block
available to readers.) Michael Stack gave a good summary on the HBase
dev list:
http://mail-archives.apache.org/mod_mbox/hadoop-hbase-dev/200905.mbox/%3c7c962aed0905231601g533088ebj4a7a068505ba3...@mail.gmail.com%3e
Tom
On Tue, May 26, 2009 at 12:08 PM, Stas Oskin stas.os...@gmail.com wrote:
Hi.
I'm trying to continuously write data to HDFS via OutputStream(), and
want
to be able to read it at the same time from another client.
Problem is, that after the file is created on HDFS with size of 0, it
stays
that way, and only fills up when I close the OutputStream().
Here is a simple code sample illustrating this issue:
try {
FSDataOutputStream out=fileSystem.create(new
Path(/test/test.bin)); // Here the file created with 0 size
for(int i=0;i1000;i++)
{
out.write(1); // Still stays 0
out.flush(); // Even when I flush it out???
}
Thread.currentThread().sleep(1);
out.close(); //Only here the file is updated
} catch (Exception e) {
e.printStackTrace();
}
So, two questions here:
1) How it's possible to write the files directly to HDFS, and have them
update there immedaitely?
2) Just for information, in this case, where the file content stays all
the
time - on server local disk, in memory, etc...?
Thanks in advance.
Re: When directly writing to HDFS, the data is moved only on file close
Hi.
You probably referring to the following paragraph?
After some back and forth over a set of slides presented by Sanjay on
work being done by Hairong as part of HADOOP-5744, Revising append,
the room settled on API3 from the list of options below as the
priority feature needed by HADOOP 0.21.0. Readers must be able to
read up to the last writer 'successful' flush. Its not important that
the file length is 'inexact'.
If I'm understand correctly, this, means the data actually gets written to
cluster - but it's not visible until the block is closed.
Work is ongoing to allow in version 0.21 to make the file visible on
flush().
Am I correct up to here?
Regards.
2009/5/26 Tom White t...@cloudera.com
This feature is not available yet, and is still under active
discussion. (The current version of HDFS will make the previous block
available to readers.) Michael Stack gave a good summary on the HBase
dev list:
http://mail-archives.apache.org/mod_mbox/hadoop-hbase-dev/200905.mbox/%3c7c962aed0905231601g533088ebj4a7a068505ba3...@mail.gmail.com%3e
Tom
On Tue, May 26, 2009 at 12:08 PM, Stas Oskin stas.os...@gmail.com
wrote:
Hi.
I'm trying to continuously write data to HDFS via OutputStream(), and
want
to be able to read it at the same time from another client.
Problem is, that after the file is created on HDFS with size of 0, it
stays
that way, and only fills up when I close the OutputStream().
Here is a simple code sample illustrating this issue:
try {
FSDataOutputStream out=fileSystem.create(new
Path(/test/test.bin)); // Here the file created with 0 size
for(int i=0;i1000;i++)
{
out.write(1); // Still stays 0
out.flush(); // Even when I flush it out???
}
Thread.currentThread().sleep(1);
out.close(); //Only here the file is updated
} catch (Exception e) {
e.printStackTrace();
}
So, two questions here:
1) How it's possible to write the files directly to HDFS, and have them
update there immedaitely?
2) Just for information, in this case, where the file content stays all
the
time - on server local disk, in memory, etc...?
Thanks in advance.
Re: InputStream.open() efficiency
Hi. Thanks for the answer. Would up to 5 minute of handlers cause any issues? And same about writing? Regards. 2009/5/26 Raghu Angadi rang...@yahoo-inc.com 'in.seek(); in.read()' is certainly better than, 'in = fs.open(); in.seek(); in.read()' The difference is is exactly one open() call. So you would save an RPC to NameNode. There are couple of issues that affect apps that keep the handlers open very long time (many hours to days).. but those will be fixed soon. Raghu. Stas Oskin wrote: Hi. I'm looking to find out, how the InputStream.open() + skip(), compares to keeping a handle of InputStream() and just seeking the position. Has anyone compared these approaches, and can advice on their speed? Regards.
Blocks amount is stuck in statistics
Hi. I just did an erase of large test folder with about 20,000 blocks, and created a new one. I copied about 128 blocks, and fsck reflects it correctly, but NN statistics still shows the old number. It does shows the currently used space correctly. Any idea if this a known issue and was fixed? Also, does it has any influence over the operations? I'm using Hadoop 0.18.3. Thanks.
Re: Blocks amount is stuck in statistics
Hi. Ok, was too eager to report :). It got sorted out after some time. Regards. 2009/5/25 Stas Oskin stas.os...@gmail.com Hi. I just did an erase of large test folder with about 20,000 blocks, and created a new one. I copied about 128 blocks, and fsck reflects it correctly, but NN statistics still shows the old number. It does shows the currently used space correctly. Any idea if this a known issue and was fixed? Also, does it has any influence over the operations? I'm using Hadoop 0.18.3. Thanks.
Re: RandomAccessFile with HDFS
Hi. I understand that the append files functionality is going to provide Random Access support? Any idea which version this expected to be? Regards. 2009/5/25 Tom White t...@cloudera.com RandomAccessFile isn't supported directly, but you can seek when reading from files in HDFS (see FSDataInputStream's seek() method). Writing at an arbitrary offset in an HDFS file is not supported however. Cheers, Tom On Sun, May 24, 2009 at 1:33 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. Any idea if RandomAccessFile is going to be supported in HDFS? Regards.
LeaseExpiredException Exception
Hi. I have a process that writes to file on DFS from time to time, using OutputStream. After some time of writing, I'm starting getting the exception below, and the write fails. The DFSClient retries several times, and then fails. Copying the file from local disk to DFS via CopyLocalFile() works fine. Can anyone advice on the matter? I'm using Hadoop 0.18.3. Thanks in advance. 09/05/25 15:35:35 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs.LeaseExpiredException: No lease on /test/test.bin File does not exist. Holder DFSClient_-951664265 does not have any open files. at org.apache.hadoop.dfs.FSNamesystem.checkLease(FSNamesystem.java:1172) at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1103 ) at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25 ) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:890) at org.apache.hadoop.ipc.Client.call(Client.java:716) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25 ) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82 ) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59 ) at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2450 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2333 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1745 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1922 )
Re: Specifying NameNode externally to hadoop-site.xml
Hi. Thanks for the tip, but is it possible to set this in dynamic way via code? Thanks. 2009/5/25 jason hadoop jason.had...@gmail.com if you launch your jobs via bin/hadoop jar jar_file [main class] [options] you can simply specify -fs hdfs://host:port before the jar_file On Sun, May 24, 2009 at 3:02 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. I'm looking to move the Hadoop NameNode URL outside the hadoop-site.xml file, so I could set it at the run-time. Any idea how to do it? Or perhaps there is another configuration that can be applied to the FileSystem object? Regards. -- Alpha Chapters of my book on Hadoop are available http://www.apress.com/book/view/9781430219422 www.prohadoopbook.com a community for Hadoop Professionals
Re: Specifying NameNode externally to hadoop-site.xml
Hi. And if I don't use jobs but only DFS for now? Regards. 2009/5/25 jason hadoop jason.had...@gmail.com conf.set(fs.default.name, hdfs://host:port); where conf is the JobConf object of your job, before you submit it. On Mon, May 25, 2009 at 10:16 AM, Stas Oskin stas.os...@gmail.com wrote: Hi. Thanks for the tip, but is it possible to set this in dynamic way via code? Thanks. 2009/5/25 jason hadoop jason.had...@gmail.com if you launch your jobs via bin/hadoop jar jar_file [main class] [options] you can simply specify -fs hdfs://host:port before the jar_file On Sun, May 24, 2009 at 3:02 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. I'm looking to move the Hadoop NameNode URL outside the hadoop-site.xml file, so I could set it at the run-time. Any idea how to do it? Or perhaps there is another configuration that can be applied to the FileSystem object? Regards. -- Alpha Chapters of my book on Hadoop are available http://www.apress.com/book/view/9781430219422 www.prohadoopbook.com a community for Hadoop Professionals -- Alpha Chapters of my book on Hadoop are available http://www.apress.com/book/view/9781430219422 www.prohadoopbook.com a community for Hadoop Professionals
RandomAccessFile with HDFS
Hi. Any idea if RandomAccessFile is going to be supported in HDFS? Regards.
Specifying NameNode externally to hadoop-site.xml
Hi. I'm looking to move the Hadoop NameNode URL outside the hadoop-site.xml file, so I could set it at the run-time. Any idea how to do it? Or perhaps there is another configuration that can be applied to the FileSystem object? Regards.
Re: Shutdown in progress exception
After you've performed your application shutdown actions you should call FileSystem's closeAll() method. Ahh, thanks for the tip. Regards.
Could only be replicated to 0 nodes, instead of 1
Hi. I'm testing Hadoop in our lab, and started getting the following message when trying to copy a file: Could only be replicated to 0 nodes, instead of 1 I have the following setup: * 3 machines, 2 of them with only 80GB of space, and 1 with 1.5GB * Two clients are copying files all the time (one of them is the 1.5GB machine) * The replication is set on 2 * I let the space on 2 smaller machines to end, to test the behavior Now, one of the clients (the one located on 1.5GB) works fine, and the other one - the external, unable to copy and displays the error + the exception below Any idea if this expected on my scenario? Or how it can be solved? Thanks in advance. 09/05/21 10:51:03 WARN dfs.DFSClient: NotReplicatedYetException sleeping /test/test.bin retries left 1 09/05/21 10:51:06 WARN dfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /test/test.bin could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1123 ) at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25 ) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:890) at org.apache.hadoop.ipc.Client.call(Client.java:716) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25 ) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82 ) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59 ) at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2450 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2333 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1745 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1922 ) 09/05/21 10:51:06 WARN dfs.DFSClient: Error Recovery for block null bad datanode[0] java.io.IOException: Could not get block locations. Aborting... at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2153 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1745 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1899 )
Re: Could only be replicated to 0 nodes, instead of 1
Hi. i)Choose a right version ( Hadoop- 0.18 is good) I'm using 0.18.3. ii)replication should be 3 as ur having 3 modes.( Indirectly see to it that ur configuration is correct !!) Actually I'm testing 2x replication on any number of DN's, to see how reliable is it. Hey even i am just suggesting this as i am also a new to hadoop Ashish Pareek On Thu, May 21, 2009 at 2:41 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. I'm testing Hadoop in our lab, and started getting the following message when trying to copy a file: Could only be replicated to 0 nodes, instead of 1 I have the following setup: * 3 machines, 2 of them with only 80GB of space, and 1 with 1.5GB * Two clients are copying files all the time (one of them is the 1.5GB machine) * The replication is set on 2 * I let the space on 2 smaller machines to end, to test the behavior Now, one of the clients (the one located on 1.5GB) works fine, and the other one - the external, unable to copy and displays the error + the exception below Any idea if this expected on my scenario? Or how it can be solved? Thanks in advance. 09/05/21 10:51:03 WARN dfs.DFSClient: NotReplicatedYetException sleeping /test/test.bin retries left 1 09/05/21 10:51:06 WARN dfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /test/test.bin could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1123 ) at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25 ) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:890) at org.apache.hadoop.ipc.Client.call(Client.java:716) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25 ) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82 ) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59 ) at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2450 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2333 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1745 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1922 ) 09/05/21 10:51:06 WARN dfs.DFSClient: Error Recovery for block null bad datanode[0] java.io.IOException: Could not get block locations. Aborting... at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2153 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1745 ) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1899 )
Re: Could only be replicated to 0 nodes, instead of 1
Hi. I think you should file a jira on this. Most likely this is what is happening : Will do - this goes to DFS section, correct? * two out of 3 dns can not take anymore blocks. * While picking nodes for a new block, NN mostly skips the third dn as well since '# active writes' on it is larger than '2 * avg'. * Even if there is one other block is being written on the 3rd, it is still greater than (2 * 1/3). Frankly I'm not so familiar with Hadoop inner workings to understand this completely, but from what I digest, NN doesn't like the 3rd DN because there is too many blocks on it, compared to other servers? To test this, if you write just one block to an idle cluster it should succeed. What exactly is idle cluster? Something that nothing is being written to (including the 3rd DN)? Writing from the client on the 3rd dn succeeds since local node is always favored. Makes sense. This particular problem is not that severe on a large cluster but HDFS should do the sensible thing. Yes, I agree that this is a non-standard situation, but IMHO the best way of action would be write anyway, but throw a warning. There is one already appearing when there is not enough space for replication, and it explains quite well the matter. So similar one would be great.
Re: Could only be replicated to 0 nodes, instead of 1
Hi. If this analysis is right, I would add it can happen even on large clusters! I've seen this error at our cluster when we're very full (97%) and very few nodes have any empty space. This usually happens because we have two very large nodes (10x bigger than the rest of the cluster), and HDFS tends to distribute writes randomly -- meaning the smaller nodes fill up quickly, until the balancer can catch up. A bit of topic, do you ran the balancer manually? Or you have some scheduler that does it?
Re: Could only be replicated to 0 nodes, instead of 1
The real trick has been to make sure the balancer doesn't get stuck -- a Nagios plugin makes sure that the stdout has been printed to in the last hour or so, otherwise it kills the running balancer. Stuck balancers have been an issue in the past. Thanks for the advice.
Re: Could only be replicated to 0 nodes, instead of 1
I think you should file a jira on this. Most likely this is what is happening : Here it is - hope it's ok: https://issues.apache.org/jira/browse/HADOOP-5886
Re: Shutdown in progress exception
Hi. 2009/5/20 Tom White t...@cloudera.com Looks like you are trying to copy file to HDFS in a shutdown hook. Since you can't control the order in which shutdown hooks run, this is won't work. There is a patch to allow Hadoop's FileSystem shutdown hook to be disabled so it doesn't close filesystems on exit. See https://issues.apache.org/jira/browse/HADOOP-4829. Thanks! Any downside to this? Also, I see that the patch doesn't work against tree- but the 0.18.3 is supported? Regards.
Re: Shutdown in progress exception
You should only use this if you plan on manually closing FileSystems yourself from within your own shutdown hook. It's somewhat of an advanced feature, and I wouldn't recommend using this patch unless you fully understand the ramifications of modifying the shutdown sequence. Standard dfs.close() would do the trick, no? Just uploaded a patch based on branch 18 for you to that JIRA. Thanks a lot!
Shutdown in progress exception
Hi. I have an issue where my application, when shutting down (at ShutdownHook level), is unable to copy files to HDFS. Each copy throws the following exception: java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:39) at java.lang.Runtime.addShutdownHook(Runtime.java:192) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1353) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213) at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:189) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1185) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1161) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1133) at app.util.FileUtils.copyToCluster(FileUtils.java:392) I read some reports, including this one ( https://issues.apache.org/jira/browse/HADOOP-3818), but there was no definite answer how to do it. Is there any good solution to this issue? Thanks in advance.
Could not complete file info?
Hi. I'm getting a very strange message, marked as INFO: 09/05/14 02:16:12 INFO dfs.DFSClient: Could not complete file /test/15334.bin retrying What does it mean, and is there any reason for concern? Thanks.
Re: Namenode failed to start with FSNamesystem initialization failed error
Hi. Yes, this was probably it. The strangest part, that the HDFS somehow worked even with all files empty in the NN directory. Go figure... Regards. 2009/5/5 Raghu Angadi rang...@yahoo-inc.com the image is stored in two files : fsimage and edits (under namenode-directory/current/). Stas Oskin wrote: Well, it definitely caused the SecondaryNameNode to crash, and also seems to have triggered some strange issues today as well. By the way, how the image file is named?
Re: Namenode failed to start with FSNamesystem initialization failed error
Hi. This quite worry-some issue. Can anyone advice on this? I'm really concerned it could appear in production, and cause a huge data loss. Is there any way to recover from this? Regards. 2009/5/5 Tamir Kamara tamirkam...@gmail.com I didn't have a space problem which led to it (I think). The corruption started after I bounced the cluster. At the time, I tried to investigate what led to the corruption but didn't find anything useful in the logs besides this line: saveLeases found path /tmp/temp623789763/tmp659456056/_temporary_attempt_200904211331_0010_r_02_0/part-2 but no matching entry in namespace I also tried to recover from the secondary name node files but the corruption my too wide-spread and I had to format. Tamir On Mon, May 4, 2009 at 4:48 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. Same conditions - where the space has run out and the fs got corrupted? Or it got corrupted by itself (which is even more worrying)? Regards. 2009/5/4 Tamir Kamara tamirkam...@gmail.com I had the same problem a couple of weeks ago with 0.19.1. Had to reformat the cluster too... On Mon, May 4, 2009 at 3:50 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. After rebooting the NameNode server, I found out the NameNode doesn't start anymore. The logs contained this error: FSNamesystem initialization failed I suspected filesystem corruption, so I tried to recover from SecondaryNameNode. Problem is, it was completely empty! I had an issue that might have caused this - the root mount has run out of space. But, both the NameNode and the SecondaryNameNode directories were on another mount point with plenty of space there - so it's very strange that they were impacted in any way. Perhaps the logs, which were located on root mount and as a result, could not be written, have caused this? To get back HDFS running, i had to format the HDFS (including manually erasing the files from DataNodes). While this reasonable in test environment - production-wise it would be very bad. Any idea why it happened, and what can be done to prevent it in the future? I'm using the stable 0.18.3 version of Hadoop. Thanks in advance!
Re: Namenode failed to start with FSNamesystem initialization failed error
Hi Raghu. The only lead I have, is that my root mount has filled-up completely. This in itself should not have caused the metadata corruption, as it has been stored on another mount point, which had plenty of space. But perhaps the fact that NameNode/SecNameNode didn't have enough space for logs has caused this? Unfortunately I was pressed in time to get the cluster up and running, and didn't preserve the logs or the image. If this happens again - I will surely do so. Regards. 2009/5/5 Raghu Angadi rang...@yahoo-inc.com Stas, This is indeed a serious issue. Did you happen to store the the corrupt image? Can this be reproduced using the image? Usually you can recover manually from a corrupt or truncated image. But more importantly we want to find how it got in to this state. Raghu. Stas Oskin wrote: Hi. This quite worry-some issue. Can anyone advice on this? I'm really concerned it could appear in production, and cause a huge data loss. Is there any way to recover from this? Regards. 2009/5/5 Tamir Kamara tamirkam...@gmail.com I didn't have a space problem which led to it (I think). The corruption started after I bounced the cluster. At the time, I tried to investigate what led to the corruption but didn't find anything useful in the logs besides this line: saveLeases found path /tmp/temp623789763/tmp659456056/_temporary_attempt_200904211331_0010_r_02_0/part-2 but no matching entry in namespace I also tried to recover from the secondary name node files but the corruption my too wide-spread and I had to format. Tamir On Mon, May 4, 2009 at 4:48 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. Same conditions - where the space has run out and the fs got corrupted? Or it got corrupted by itself (which is even more worrying)? Regards. 2009/5/4 Tamir Kamara tamirkam...@gmail.com I had the same problem a couple of weeks ago with 0.19.1. Had to reformat the cluster too... On Mon, May 4, 2009 at 3:50 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. After rebooting the NameNode server, I found out the NameNode doesn't start anymore. The logs contained this error: FSNamesystem initialization failed I suspected filesystem corruption, so I tried to recover from SecondaryNameNode. Problem is, it was completely empty! I had an issue that might have caused this - the root mount has run out of space. But, both the NameNode and the SecondaryNameNode directories were on another mount point with plenty of space there - so it's very strange that they were impacted in any way. Perhaps the logs, which were located on root mount and as a result, could not be written, have caused this? To get back HDFS running, i had to format the HDFS (including manually erasing the files from DataNodes). While this reasonable in test environment - production-wise it would be very bad. Any idea why it happened, and what can be done to prevent it in the future? I'm using the stable 0.18.3 version of Hadoop. Thanks in advance!
Re: Namenode failed to start with FSNamesystem initialization failed error
Actually, we discovered today an annoying bug in our test-app, which might have moved some of the HDFS files to the cluster, including the metadata files. I presume it could be the possible reason for such behavior? :) 2009/5/5 Stas Oskin stas.os...@gmail.com Hi Raghu. The only lead I have, is that my root mount has filled-up completely. This in itself should not have caused the metadata corruption, as it has been stored on another mount point, which had plenty of space. But perhaps the fact that NameNode/SecNameNode didn't have enough space for logs has caused this? Unfortunately I was pressed in time to get the cluster up and running, and didn't preserve the logs or the image. If this happens again - I will surely do so. Regards. 2009/5/5 Raghu Angadi rang...@yahoo-inc.com Stas, This is indeed a serious issue. Did you happen to store the the corrupt image? Can this be reproduced using the image? Usually you can recover manually from a corrupt or truncated image. But more importantly we want to find how it got in to this state. Raghu. Stas Oskin wrote: Hi. This quite worry-some issue. Can anyone advice on this? I'm really concerned it could appear in production, and cause a huge data loss. Is there any way to recover from this? Regards. 2009/5/5 Tamir Kamara tamirkam...@gmail.com I didn't have a space problem which led to it (I think). The corruption started after I bounced the cluster. At the time, I tried to investigate what led to the corruption but didn't find anything useful in the logs besides this line: saveLeases found path /tmp/temp623789763/tmp659456056/_temporary_attempt_200904211331_0010_r_02_0/part-2 but no matching entry in namespace I also tried to recover from the secondary name node files but the corruption my too wide-spread and I had to format. Tamir On Mon, May 4, 2009 at 4:48 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. Same conditions - where the space has run out and the fs got corrupted? Or it got corrupted by itself (which is even more worrying)? Regards. 2009/5/4 Tamir Kamara tamirkam...@gmail.com I had the same problem a couple of weeks ago with 0.19.1. Had to reformat the cluster too... On Mon, May 4, 2009 at 3:50 PM, Stas Oskin stas.os...@gmail.com wrote: Hi. After rebooting the NameNode server, I found out the NameNode doesn't start anymore. The logs contained this error: FSNamesystem initialization failed I suspected filesystem corruption, so I tried to recover from SecondaryNameNode. Problem is, it was completely empty! I had an issue that might have caused this - the root mount has run out of space. But, both the NameNode and the SecondaryNameNode directories were on another mount point with plenty of space there - so it's very strange that they were impacted in any way. Perhaps the logs, which were located on root mount and as a result, could not be written, have caused this? To get back HDFS running, i had to format the HDFS (including manually erasing the files from DataNodes). While this reasonable in test environment - production-wise it would be very bad. Any idea why it happened, and what can be done to prevent it in the future? I'm using the stable 0.18.3 version of Hadoop. Thanks in advance!
Re: Namenode failed to start with FSNamesystem initialization failed error
Hi. 2009/5/5 Raghu Angadi rang...@yahoo-inc.com Stas Oskin wrote: Actually, we discovered today an annoying bug in our test-app, which might have moved some of the HDFS files to the cluster, including the metadata files. oops! presumably it could have removed the image file itself. I presume it could be the possible reason for such behavior? :) certainly. It could lead to many different failures. If you had stack trace of the exception, it would be more clear what the error was this time. Raghu. Well, it definitely caused the SecondaryNameNode to crash, and also seems to have triggered some strange issues today as well. By the way, how the image file is named? Regards.
Namenode failed to start with FSNamesystem initialization failed error
Hi. After rebooting the NameNode server, I found out the NameNode doesn't start anymore. The logs contained this error: FSNamesystem initialization failed I suspected filesystem corruption, so I tried to recover from SecondaryNameNode. Problem is, it was completely empty! I had an issue that might have caused this - the root mount has run out of space. But, both the NameNode and the SecondaryNameNode directories were on another mount point with plenty of space there - so it's very strange that they were impacted in any way. Perhaps the logs, which were located on root mount and as a result, could not be written, have caused this? To get back HDFS running, i had to format the HDFS (including manually erasing the files from DataNodes). While this reasonable in test environment - production-wise it would be very bad. Any idea why it happened, and what can be done to prevent it in the future? I'm using the stable 0.18.3 version of Hadoop. Thanks in advance!
Re: Getting free and used space
Hi. Actually, I'm trying to use the getDiskStatus() function, but it doesn't seem to work so well in 0.18.3. Can someone advice of a reliable way to get the HDFS overall free and used space? Same function that reports the space to the NameNode web panel? Thanks. 2009/5/2 Edward Capriolo edlinuxg...@gmail.com You can also pull these variables from the name node, datanode with JMX. I am doing this to graph them with cacti. Both the JMX READ/WRITE and READ user can access this variable. On Tue, Apr 28, 2009 at 8:29 AM, Stas Oskin stas.os...@gmail.com wrote: Hi. Any idea if the getDiskStatus() function requires superuser rights? Or it can work for any user? Thanks. 2009/4/9 Aaron Kimball aa...@cloudera.com You can insert this propery into the jobconf, or specify it on the command line e.g.: -D hadoop.job.ugi=username,group,group,group. - Aaron On Wed, Apr 8, 2009 at 7:04 AM, Brian Bockelman bbock...@cse.unl.edu wrote: Hey Stas, What we do locally is apply the latest patch for this issue: https://issues.apache.org/jira/browse/HADOOP-4368 This makes getUsed (actually, it switches to FileSystem.getStatus) not a privileged action. As far as specifying the user ... gee, I can't think of it off the top of my head. It's a variable you can insert into the JobConf, but I'd have to poke around google or the code to remember which one (I try to not override it if possible). Brian On Apr 8, 2009, at 8:51 AM, Stas Oskin wrote: Hi. Thanks for the explanation. Now for the easier part - how do I specify the user when connecting? :) Is it a config file level, or run-time level setting? Regards. 2009/4/8 Brian Bockelman bbock...@cse.unl.edu Hey Stas, Did you try this as a privileged user? There might be some permission errors... in most of the released versions, getUsed() is only available to the Hadoop superuser. It may be that the exception isn't propagating correctly. Brian On Apr 8, 2009, at 3:13 AM, Stas Oskin wrote: Hi. I'm trying to use the API to get the overall used and free spaces. I tried this function getUsed(), but it always returns 0. Any idea? Thanks.
Re: Blocks replication in downtime even
Hi. I think one needs to run balancer in order to clean out the redundant blocks. Can anyone confirm this? Regards. 2009/4/28 Piotr piotr.prac...@gmail.com Hi What happens when the node rejoins then ? - The replication level of several blocks increases ? - The old replicas are removed in favor of new replicas ? (Or the new ones) regards Piotr 2009/4/27 Stas Oskin stas.os...@gmail.com Thanks. 2009/4/27 Koji Noguchi knogu...@yahoo-inc.com http://hadoop.apache.org/core/docs/current/hdfs_design.html#Data+Disk+Fa ilure%2C+Heartbeats+and+Re-Replication http://hadoop.apache.org/core/docs/current/hdfs_design.html#Data+Disk+Fa%0Ailure%2C+Heartbeats+and+Re-Replication hope this helps. Koji -Original Message- From: Stas Oskin [mailto:stas.os...@gmail.com] Sent: Monday, April 27, 2009 4:11 AM To: core-user@hadoop.apache.org Subject: Blocks replication in downtime even Hi. I have a question: If I have N of DataNodes, and one or several of the nodes have become unavailable, would HDFS re-synchronize the blocks automatically, according to replication level set? And if yes, when? As soon as the offline node was detected, or only on file access? Regards.
Re: Getting free and used space
Hi. Any idea if the getDiskStatus() function requires superuser rights? Or it can work for any user? Thanks. 2009/4/9 Aaron Kimball aa...@cloudera.com You can insert this propery into the jobconf, or specify it on the command line e.g.: -D hadoop.job.ugi=username,group,group,group. - Aaron On Wed, Apr 8, 2009 at 7:04 AM, Brian Bockelman bbock...@cse.unl.edu wrote: Hey Stas, What we do locally is apply the latest patch for this issue: https://issues.apache.org/jira/browse/HADOOP-4368 This makes getUsed (actually, it switches to FileSystem.getStatus) not a privileged action. As far as specifying the user ... gee, I can't think of it off the top of my head. It's a variable you can insert into the JobConf, but I'd have to poke around google or the code to remember which one (I try to not override it if possible). Brian On Apr 8, 2009, at 8:51 AM, Stas Oskin wrote: Hi. Thanks for the explanation. Now for the easier part - how do I specify the user when connecting? :) Is it a config file level, or run-time level setting? Regards. 2009/4/8 Brian Bockelman bbock...@cse.unl.edu Hey Stas, Did you try this as a privileged user? There might be some permission errors... in most of the released versions, getUsed() is only available to the Hadoop superuser. It may be that the exception isn't propagating correctly. Brian On Apr 8, 2009, at 3:13 AM, Stas Oskin wrote: Hi. I'm trying to use the API to get the overall used and free spaces. I tried this function getUsed(), but it always returns 0. Any idea? Thanks.
Blocks replication in downtime even
Hi. I have a question: If I have N of DataNodes, and one or several of the nodes have become unavailable, would HDFS re-synchronize the blocks automatically, according to replication level set? And if yes, when? As soon as the offline node was detected, or only on file access? Regards.
Re: Blocks replication in downtime even
Thanks. 2009/4/27 Koji Noguchi knogu...@yahoo-inc.com http://hadoop.apache.org/core/docs/current/hdfs_design.html#Data+Disk+Fa ilure%2C+Heartbeats+and+Re-Replicationhttp://hadoop.apache.org/core/docs/current/hdfs_design.html#Data+Disk+Fa%0Ailure%2C+Heartbeats+and+Re-Replication hope this helps. Koji -Original Message- From: Stas Oskin [mailto:stas.os...@gmail.com] Sent: Monday, April 27, 2009 4:11 AM To: core-user@hadoop.apache.org Subject: Blocks replication in downtime even Hi. I have a question: If I have N of DataNodes, and one or several of the nodes have become unavailable, would HDFS re-synchronize the blocks automatically, according to replication level set? And if yes, when? As soon as the offline node was detected, or only on file access? Regards.
Re: No route to host prevents from storing files to HDFS
Hi. Shouldn't you be testing connecting _from_ the datanode? The error you posted is while this DN is trying connect to another DN. You might be into something here indeed: 1) Telnet to 192.168.253.20 8020 / 192.168.253.20 50010 works 2) Telnet to localhost 8020 / localhost 50010 doesn't work 3) Telnet to 127.0.0.1 8020 / 127.0.0.1 50010 doesn't work In the 2 last cases I get: Trying 127.0.0.1... telnet: connect to address 127.0.0.1: Connection refused telnet: Unable to connect to remote host: Connection refused Could it be related? Regards.
Re: No route to host prevents from storing files to HDFS
Hi. I have one question, is the ip address consistent, I think in one of the thread mails, it was stated that the ip address sometimes changes. Same static IP's for all servers. By the way, I have the fs.default.name defined in IP address could it be somehow related? I read that there were some issues with this, but it ran fine for me - that it, until the power crash. Regards.
Re: No route to host prevents from storing files to HDFS
Hi. Maybe, but there will still be at least one virtual network adapter on the host. Try turning them off. Nope, still throws No route to host exceptions. I have another IP address defined on this machine - 192.168.253.21, for the same network adapter. Any idea if it has impact? The fs.default.name is: hdfs://192.168.253.20:8020 what happens if you switch to hostnames over IP addresses? Actually, I never tried this, but point is that the HDFS worked just fine with this before. Regards.
Re: No route to host prevents from storing files to HDFS
Hi. 2009/4/23 Matt Massie m...@cloudera.com Just for clarity: are you using any type of virtualization (e.g. vmware, xen) or just running the DataNode java process on the same machine? What is fs.default.name set to in your hadoop-site.xml? This machine has OpenVZ installed indeed, but all the applications run withing the host node, meaning all Java processes are running withing same machine. The fs.default.name is: hdfs://192.168.253.20:8020 Thanks.
Re: No route to host prevents from storing files to HDFS
Just to clarify one point - the iptables were running on 2nd DataNode which I didn't check, as I was sure the problem is in the NameNode/DataNode, and on NameNode/DataNode. But I can't understand what and when launched them, as I checked multiple times and nothing was running before. Moreover, they were disabled on start-up, so they shouldn't come up in the first place. Regards. 2009/4/23 Stas Oskin stas.os...@gmail.com Hi. Also iptables -L for each machine as an afterthought - just for paranoia's sake Well, I started preparing all the information you requested, but when I got to this stage - I found out there were INDEED iptables running on 2 servers from 3. The strangest thing is that I don't recall enabling them at all. Perhaps some 3rd party software have enabled them? In any case, all seems to be working now. Thanks for everybody that helped - I will be sure to check iptables on all the cluster machines from now on :). Regards.
Re: No route to host prevents from storing files to HDFS
Hi. 2009/4/22 jason hadoop jason.had...@gmail.com Most likely that machine is affected by some firewall somewhere that prevents traffic on port 50075. The no route to host is a strong indicator, particularly if the Datanote registered with the namenode. Yes, this was my first thought as well. But there is no firewall, and the port can be connected via netcat from any other machine. Any other idea? Thanks.
Re: No route to host prevents from storing files to HDFS
Hi. There is some mismatch here.. what is the expected ip address of this machine (or does it have multiple interfaces and properly routed)? Looking at the Receiving Block message DN thinks its address is 192.168.253.20 but NN thinks it is 253.32 (and client is able to connect using 253.32). If you want to find the destination ip that this DN is unable to connect to, you can check client's log for this block number. Hmm, .253.32 is the client workstation (has only our test application with core-hadoop.jar + configs). The expected address of the DataNode should be 192.168.253.20. According to what I seen, the problem is in DataNode itself - it just throws the Datanoderegistration every so often: 2009-04-23 00:05:05,961 INFO org.apache.hadoop.dfs.DataNode: Receiving block blk_7209884038924026671_8033 src: /192.168.253.32:42932 dest: /192.168.253.32:50010 2009-04-23 00:05:05,962 INFO org.apache.hadoop.dfs.DataNode: writeBlock blk_7209884038924026671_8033 received exception java.net.NoR outeToHostException: No route to host 2009-04-23 00:05:05,962 ERROR org.apache.hadoop.dfs.DataNode: DatanodeRegistration(192.168.253.20:50010, storageID=DS-1790181121-127 .0.0.1-50010-1239123237447, infoPort=50075, ipcPort=50020):DataXceiver: java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:402) at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1255) at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1092) at java.lang.Thread.run(Thread.java:619) Regards.
Re: No route to host prevents from storing files to HDFS
Hi. The way to diagnose this explicitly is: 1) on the server machine that should be accepting connections on the port, telnet localhost PORT, and telnet IP PORT you should get a connection, if not then the server is not binding the port. 2) on the remote machine verify that you can communicate to the server machine via normal tools such as ssh and or ping and or traceroute, using the IP address from the error message in your log file 3) on the remote machine run telnet IP PORT. if (1) and (2) succeeded and (3) does not, then there is something blocking packets for the port range in question. If (3) does succeed then there is some probably interesting problem. Tried in step 3 to telnet both the 50010 and the 8010 ports of the problematic datanode - both worked. I agree there is indeed an interesting problem :). Question is how it can be solved. Thanks.
Re: No route to host prevents from storing files to HDFS
Hi. Is it possible to paste the output from the following command on both your DataNode and NameNode? % route -v -n Sure, here it is: Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 192.168.253.0 0.0.0.0 255.255.255.0 U 0 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 00 eth0 0.0.0.0 192.168.253.1 0.0.0.0 UG0 00 eth0 As you might recall, the problematic data node runs in same server as the NameNode. Regards.
No route to host prevents from storing files to HDFS
Hi. I have quite a strange issue, where one of the datanodes that I have, rejects any blocks with error messages. I looked in the datanode logs, and found the following error: 2009-04-21 16:59:19,092 ERROR org.apache.hadoop.dfs.DataNode: DatanodeRegistration(192.168.253.20:50010, storageID=DS-1790181121-127.0.0.1-50010-1239123237447, infoPort=50075, ipcPort=50020):DataXceiver: java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:402) at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1255) at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1092) at java.lang.Thread.run(Thread.java:619) 2009-04-21 16:59:31,047 INFO org.apache.hadoop.dfs.DataNode: Receiving block blk_1882546734729403703_7805 src: /192.168.253.32:54917 dest: / 192.168.253.32:50010 2009-04-21 16:59:31,048 INFO org.apache.hadoop.dfs.DataNode: writeBlock blk_1882546734729403703_7805 received exception java.net.NoRouteToHostException: No route to host Several facts: 1) I use the stable 0.18.3 2) It worked before correctly, before I had an overall power crash which brought down all the machines. 3) This datanode is located on same machine as the NameNode and the SecondaryNameNode. 4) I can ping the machine from itself - no error messages. Any idea what should be done to resolve it? Thanks in advance.
Re: No route to host prevents from storing files to HDFS
Hi again. Other tools, like balancer, or the web browsing from namenode, don't work as well. This because other nodes complain about not reaching the offending node as well. I even tried netcat'ing the IP/port from another node - and it successfully connected. Any advice on this No route to host error? 2009/4/21 Stas Oskin stas.os...@gmail.com Hi. I have quite a strange issue, where one of the datanodes that I have, rejects any blocks with error messages. I looked in the datanode logs, and found the following error: 2009-04-21 16:59:19,092 ERROR org.apache.hadoop.dfs.DataNode: DatanodeRegistration(192.168.253.20:50010, storageID=DS-1790181121-127.0.0.1-50010-1239123237447, infoPort=50075, ipcPort=50020):DataXceiver: java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:402) at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1255) at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1092) at java.lang.Thread.run(Thread.java:619) 2009-04-21 16:59:31,047 INFO org.apache.hadoop.dfs.DataNode: Receiving block blk_1882546734729403703_7805 src: /192.168.253.32:54917 dest: / 192.168.253.32:50010 2009-04-21 16:59:31,048 INFO org.apache.hadoop.dfs.DataNode: writeBlock blk_1882546734729403703_7805 received exception java.net.NoRouteToHostException: No route to host Several facts: 1) I use the stable 0.18.3 2) It worked before correctly, before I had an overall power crash which brought down all the machines. 3) This datanode is located on same machine as the NameNode and the SecondaryNameNode. 4) I can ping the machine from itself - no error messages. Any idea what should be done to resolve it? Thanks in advance.
HDFS and web server
Hi. Has any succeed running web-server from HDFS? I mean, to serve websites and application directly from HDFS, perhaps via FUSE/WebDav? Regards.
Re: HDFS and web server
Hi. 2009/4/14 Michael Bieniosek micb...@microsoft.com webdav server - https://issues.apache.org/jira/browse/HADOOP-496 There's a fuse issue somewhere too, but I never managed to get it working. As far as serving websites directly from HDFS goes, I would say you'd probably have better luck writing a pure-java webserver that served directly from the java HDFS api. I've had good experiences serving (internal tools) with jetty -- http://docs.codehaus.org/display/JETTY/Embedding+Jetty -Michael Thanks for the advice, but I need PHP support, which Java can't probably do much about. Exposing HDFS via FUSE would be ideal, but from what I heard, it's not considered very stable yet. Might not be entirely related to this list, but what other measures can be used to reliably replicate the content over several servers, and get it ready for HTTP serving? Regards.
Running balancer throws exceptions
Hi. When I run hadoop balancer, I get the following error: 09/04/12 10:28:46 INFO dfs.Balancer: Will move 3.02 GBbytes in this iteration Apr 12, 2009 10:28:46 AM 0 0 KB19.02 GB3.02 GB 09/04/12 10:28:46 INFO dfs.Balancer: Decided to move block -4854334996783373210 with a length of 0.02 KB bytes from 192.168.253.42:50010 to 192.168.253.20:50010 using proxy source 192.168.253.42:50010 09/04/12 10:28:46 INFO dfs.Balancer: Starting moving -4854334996783373210 from 192.168.253.42:50010 to 192.168.253.20:50010 09/04/12 10:28:46 WARN dfs.Balancer: Error moving block -4854334996783373210 from 192.168.253.42:50010 to 192.168.253.20:50010 through 192.168.253.42:50010: Moving block -4854334996783373210 from 192.168.253.42:50010 to 192.168.253.20:50010 through 192.168.253.42:50010 failed java.io.IOException: Moving block -4854334996783373210 from 192.168.253.42:50010 to 192.168.253.20:50010 through 192.168.253.42:50010 failed at org.apache.hadoop.dfs.Balancer$PendingBlockMove.receiveResponse(Balancer.java:366) at org.apache.hadoop.dfs.Balancer$PendingBlockMove.dispatch(Balancer.java:316) at org.apache.hadoop.dfs.Balancer$PendingBlockMove.access$1700(Balancer.java:224) at org.apache.hadoop.dfs.Balancer$PendingBlockMove$1.run(Balancer.java:388) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) I'm using 0.18.3. Any idea? Regards.
Re-balancing blocks
Hi. Should I call the block re-balancer every time a new DataNode is added or removed? Or what is the recommended procedure of re-balancing the blocks for better fault tolerance? Regards.
NameNode resilency
Hi. I wonder, what Hadoop community uses in order to make NameNode resilient to failures? I mean, what High-Availability measures are taken to keep the HDFS available even in case of NameNode failure? So far I read a possible solution using DRBD, and another one using carp. Both of them had the downside of keeping a passive machine aside taking the IP of the NameNode. Perhaps there is a way to keep only a passive NameNode service on another machine (which does other tasks), taking the IP only when the main has failed? That of course until the human operator restores the main node to action.\ Regards.
Re: NameNode resilency
Hi. Any tutorial about using Zookeeper with NameNode? Thanks! 2009/4/12 Billy Pearson sa...@pearsonwholesale.com Not 100% sure but I thank they plan on using zookeeper to help with namenode fail over but that may have changed. Billy Stas Oskin stas.os...@gmail.com wrote in message news:77938bc20904110243u7a2baa6dw6d710e4e51ae0...@mail.gmail.com... Hi. I wonder, what Hadoop community uses in order to make NameNode resilient to failures? I mean, what High-Availability measures are taken to keep the HDFS available even in case of NameNode failure? So far I read a possible solution using DRBD, and another one using carp. Both of them had the downside of keeping a passive machine aside taking the IP of the NameNode. Perhaps there is a way to keep only a passive NameNode service on another machine (which does other tasks), taking the IP only when the main has failed? That of course until the human operator restores the main node to action.\ Regards.
Re: HDFS read/write speeds, and read optimization
Hi. Hypertable (a BigTable implementation) has a good KFS vs. HDFS breakdown: http://code.google.com/p/hypertable/wiki/KFSvsHDFS From this comparison it seems KFS is quite faster then HDFS for small data transfers (for SQL commands). Any idea if same holds true for small-medium (20Mb - 150 MB) files? 2) If the chunk server is located on same host as the client, is there any optimization in read operations? For example, Kosmos FS describe the following functionality: Localhost optimization: One copy of data is placed on the chunkserver on the same host as the client doing the write Helps reduce network traffic In Hadoop-speak, we're interested in DataNodes (storage nodes) and TaskTrackers (compute nodes). In terms of MapReduce, Hadoop does try and schedule tasks such that the data being processed by a given task on a given machine is also on that machine. As for loading data onto a DataNode, loading data from a DataNode will put a replica on that node. However, if you're loading data from, say, your local machine, Hadoop will choose a DataNode at random. Ah, so if DataNode will store file to HDFS, it would try to place a replica on this same DataNode as well? And then if this DataNode would try to read the file. HDFS would try to read it first from itself first? Regards.
Re: HDFS read/write speeds, and read optimization
Hi. Depends. What hardware? How much hardware? Is the cluster under load? What does your I/O load look like? As a rule of thumb, you'll probably expect very close to hardware speed. Standard Xeon dual cpu, quad core servers, 4 GB RAM. The DataNodes also do some processing, with usual loads about ~4 (from 8 recommended). The IO load is linear, there are almost no write or read peaks. By close to hardware speed, you mean results very near the results I get via iozone? Thanks.
Does the HDFS client read the data from NameNode, or from DataNode directly?
Hi. I wanted to verify a point about HDFS client operations: When asking for file, is the all communication done through the NameNode? Or after being pointed to correct DataNode, does the HDFS works directly against it? Also, NameNode provides a URL named streamFile which allows any HTTP client to get the stored files. Any idea how it's operations compare in terms of speed to client HDFS access? Regards.
Thin version of Hadoop jar for client
Hi. Is there any thin version of Hadoop jar, specifically for Java client? Regards.
Re: HDFS read/write speeds, and read optimization
Thanks for sharing. For comparison, on a 1400 node cluster, I can checksum 100 TB in around 10 minutes, which means I'm seeing read averages of roughly 166 GB/sec. For writes with replication of 3, I see roughly 40-50 minutes to write 100TB, so roughly 33 GB/sec average. Of course the peaks are much higher. Each node has 4 SATA disks, dual quad core, and 8 GB of ram. From your experience, how RAM hungry HDFS is? Meaning, additional 4GB or ram (to make it 8GB aas in your case), really change anything? Regards.
Two degrees of replications reliability
Hi. I know that there were some hard to find bugs with replication set to 2, which caused data loss to HDFS users. Was there any progress with these issues, and if there any fixes which were introduced? Regards.
Re: Does the HDFS client read the data from NameNode, or from DataNode directly?
Thanks, this is what I thought. Regards. 2009/4/10 Alex Loddengaard a...@cloudera.com Data is streamed directly from the data nodes themselves. The name node is only queried for block locations and other meta data. Alex On Fri, Apr 10, 2009 at 8:33 AM, Stas Oskin stas.os...@gmail.com wrote: Hi. I wanted to verify a point about HDFS client operations: When asking for file, is the all communication done through the NameNode? Or after being pointed to correct DataNode, does the HDFS works directly against it? Also, NameNode provides a URL named streamFile which allows any HTTP client to get the stored files. Any idea how it's operations compare in terms of speed to client HDFS access? Regards.
Re: HDFS read/write speeds, and read optimization
Hi. Depends on what kind of I/O you do - are you going to be using MapReduce and co-locating jobs and data? If so, it's possible to get close to those speeds if you are I/O bound in your job and read right through each chunk. If you have multiple disks mounted individually, you'll need the number of streams equal to the number of disks. If you're going to do I/O that's not through MapReduce, you'll probably be bound by the network interface. Btw, this what I wanted to ask as well: Is it more efficient to unify the disks into one volume (RAID or LVM), and then present them as a single space? Or it's better to specify each disk separately? Reliability-wise, the latter sounds more correct, as a single/several (up to 3) disks going down won't take the whole node with them. But perhaps there is a performance penalty?
Re: Two degrees of replications reliability
2009/4/10 Brian Bockelman bbock...@cse.unl.edu Most of the issues were resolved in 0.19.1 -- I think 0.20.0 is going to be even better. We run about 300TB @ 2 replicas, and haven't had file loss that was Hadoop's fault since about January. Brian And you running 0.19.1? Regards.
Re: Two degrees of replications reliability
Actually, now I remember that you posted some time ago about your University loosing about 300 files. So since then the situation has improved I presume? 2009/4/10 Stas Oskin stas.os...@gmail.com 2009/4/10 Brian Bockelman bbock...@cse.unl.edu Most of the issues were resolved in 0.19.1 -- I think 0.20.0 is going to be even better. We run about 300TB @ 2 replicas, and haven't had file loss that was Hadoop's fault since about January. Brian And you running 0.19.1? Regards.
Re: Does the HDFS client read the data from NameNode, or from DataNode directly?
Hi. What happens here is that the NameNode redirects you to a smartly (a data node that has some of the file's first 5 blocks, I think) chosen DataNode, and that DataNode proxies the file for you. Specifically, the assembling of a full file from multiple nodes is happening on that DataNode. If you were using a DFSClient, it would assemble the file from blocks at the client, and talk to many data nodes. I see, thanks for the explanation.
Listing the files in HDFS directory
Hi. I must be completely missing it in API, but what is the function to list the directory contents? Thanks!
Abandoning block messages
Hi. I'm testing HDFS fault-tolerance, and randomly disconnecting and powering off the chunk nodes to see how HDFS withstands it. After I have disconnected a chunk node, the logs have started to fill with the following errors: 09/04/11 02:25:12 INFO dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.253.41:50010 09/04/11 02:25:12 INFO dfs.DFSClient: Abandoning block blk_-656235730036158252_1379 09/04/11 02:25:12 INFO dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.253.41:50010 09/04/11 02:25:12 INFO dfs.DFSClient: Abandoning block blk_4525038275659790960_1379 09/04/11 02:25:12 INFO dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.253.41:50010 09/04/11 02:25:12 INFO dfs.DFSClient: Abandoning block blk_4496668387968639765_1379 09/04/11 02:25:12 INFO dfs.DFSClient: Waiting to find target node: 192.168.253.20:50010 09/04/11 02:25:12 INFO dfs.DFSClient: Waiting to find target node: 192.168.253.20:50010 09/04/11 02:25:12 INFO dfs.DFSClient: Waiting to find target node: 192.168.253.20:50010 09/04/11 02:25:21 INFO dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.253.41:50010 09/04/11 02:25:21 INFO dfs.DFSClient: Abandoning block blk_7909869387543801936_1379 09/04/11 02:25:21 INFO dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.253.41:50010 09/04/11 02:25:21 INFO dfs.DFSClient: Abandoning block blk_3548805008548426073_1368 09/04/11 02:25:21 INFO dfs.DFSClient: Waiting to find target node: 192.168.253.20:50010 09/04/11 02:25:25 INFO dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.253.41:50010 09/04/11 02:25:25 INFO dfs.DFSClient: Abandoning block blk_6143739150500056332_1379 09/04/11 02:25:25 INFO dfs.DFSClient: Waiting to find target node: 192.168.253.20:50010 09/04/11 02:25:30 INFO dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.253.41:50010 09/04/11 02:25:30 INFO dfs.DFSClient: Abandoning block blk_5626187863612769755_1379 Any idea what it means, and should I be concerned about it. Regards.
Re: Listing the files in HDFS directory
Hi. Just wanted to tell that FileStatus did the trick. Regards. 2009/4/11 Stas Oskin stas.os...@gmail.com Hi. I must be completely missing it in API, but what is the function to list the directory contents? Thanks!
HDFS read/write speeds, and read optimization
Hi. I have 2 questions about HDFS performance: 1) How fast are the read and write operations over network, in Mbps per second? 2) If the chunk server is located on same host as the client, is there any optimization in read operations? For example, Kosmos FS describe the following functionality: Localhost optimization: One copy of data is placed on the chunkserver on the same host as the client doing the write Helps reduce network traffic Regards.
Getting free and used space
Hi. I'm trying to use the API to get the overall used and free spaces. I tried this function getUsed(), but it always returns 0. Any idea? Thanks.
Re: Getting free and used space
Hi. Thanks for the explanation. Now for the easier part - how do I specify the user when connecting? :) Is it a config file level, or run-time level setting? Regards. 2009/4/8 Brian Bockelman bbock...@cse.unl.edu Hey Stas, Did you try this as a privileged user? There might be some permission errors... in most of the released versions, getUsed() is only available to the Hadoop superuser. It may be that the exception isn't propagating correctly. Brian On Apr 8, 2009, at 3:13 AM, Stas Oskin wrote: Hi. I'm trying to use the API to get the overall used and free spaces. I tried this function getUsed(), but it always returns 0. Any idea? Thanks.
Re: Hadoop for real time
Hi Ted. Thanks for sharing some of inner workings of Veoh, which btw I'm a frequent user of (or at least when time permits :) ). I indeed recall reading somewhere that Veoh used a heavily modified version of MogileFS, but have switched since as it wasn't ready enough for Veoh needs. If not Hadoop, are there any other available solutions that can assist in distributing the processing of real-time video data? Or the old way of separate application severs is the only way to go? Regards. 2008/10/20 Ted Dunning [EMAIL PROTECTED] Hadoop may not be quite what you want for this. You could definitely use Hadopo for storage and streaming. You can also do various kinds of processing on hadoop. But because Hadoop is primarily intended for batch style operations, there is a bit of an assumption that some administrative tasks will take down the cluster. That may be a problem (video serving tends to have a web audience that isn't very tolerant of downtime). At Veoh, we used a simpler, but simpler system for serving videos that was originally based on Mogile. The basic idea is that there is a database that contains name to URL mappings. The URL's point to storage boxes that have a bunch of disks that are served out to the net via LightHttpd. A management machine runs occasionally to make sure that files are replicated according to policy. The database is made redundant via conventional mechanisms. Requests for files can be proxied a farm of front end machines that query the database for locations or you can use redirects directly to the content. How you do it depends on network topology and your sensitivity about divulging internal details. Redirects can give higher peak read speed since you are going direct. Proxying avoids a network round trip for the redirect. At Veoh, this system fed the content delivery networks as a caching layer which meant that the traffic was essentially uniform random access. This system handled a huge number of files (10^9 or so) very easily and has essentially never had customer visible downtime. Extension with new files systems is trivial (just tell the manager box and it starts using them). This arrangement lacks most of the things that make Hadoop really good for what it does. But, in return, it is incredibly simple. It isn't very suitable for map-reduce or other high bandwidth processing tasks. It doesn't allow computation to go to the data. It doesn't allow large files to be read in parallel from many machines. On the other hand, it handles way more files than Hadoop does and it handles gobs of tiny files pretty well. Video is also kind of a write-once medium in many cases and video files aren't real splittable for map-reduce purposes. That might mean that you could get away with a mogile-ish system. On Tue, Oct 14, 2008 at 1:29 PM, Stas Oskin [EMAIL PROTECTED] wrote: Hi. Video storage, processing and streaming. Regards. 2008/9/25 Edward J. Yoon [EMAIL PROTECTED] What kind of the real-time app? On Wed, Sep 24, 2008 at 4:50 AM, Stas Oskin [EMAIL PROTECTED] wrote: Hi. Is it possible to use Hadoop for real-time app, in video processing field? Regards. -- Best regards, Edward J. Yoon [EMAIL PROTECTED] http://blog.udanax.org -- ted
Re: Hadoop for real time
Hi. Video storage, processing and streaming. Regards. 2008/9/25 Edward J. Yoon [EMAIL PROTECTED] What kind of the real-time app? On Wed, Sep 24, 2008 at 4:50 AM, Stas Oskin [EMAIL PROTECTED] wrote: Hi. Is it possible to use Hadoop for real-time app, in video processing field? Regards. -- Best regards, Edward J. Yoon [EMAIL PROTECTED] http://blog.udanax.org
Hadoop for real time
Hi. Is it possible to use Hadoop for real-time app, in video processing field? Regards.
