Hello, Doug!
I have fixed my problem. As I suppose, problem was with network
interfaces. Datanode take internet(external) address, instead local.
I believe that it can be configured in virtual machine, but I can't
find where it.
I think that many peoples have several IP per server, but need to use
specific IP for NDFS. My solution for this situation is below.
Changes at NDFS/DataNode.java:
--------------------------------------------------------------------
public DataNode(NutchConf conf, String datadir) throws IOException {
this(/*InetAddress.getLocalHost().getHostName(),*/
conf.get(InetAddress.getLocalHost().getHostName()+".realip",InetAddress.getLocalHost().getHostName()),
new File(datadir),
createSocketAddr(conf.get("fs.default.name", "local")));
--------------------------------------------------------------------
And now we can define hostname or IP for ndfs at nutch-site.xml like this:
<property>
<name>your.hostname.here.realip</name>
<value>192.168.0.24</value>
</property>
P.S. I've used nutch-mapred release.
Tuesday, August 30, 2005, 10:51:10 PM, you wrote:
Doug Cutting> It sounds like you're using a nightly
Doug Cutting> build of trunk. The NDFS code in
Doug Cutting> trunk is old. The NDFS code is currently
Doug Cutting> maintained in a branch named
Doug Cutting> "mapred". Please check out the mapred branch and retry.
Doug Cutting> svn co
Doug Cutting> https://svn.apache.org/repos/asf/lucene/nutch/branches/mapred/
Doug Cutting> Doug
Doug Cutting> Egor Chernodarov wrote:
>> Hello!
>>
>> I want to test NDFS on my nutch installation, but I have some problem.
>> I have started from wiki, where is quick demo for NDFS:
>> http://wiki.apache.org/nutch/NutchDistributedFileSystem
>>
>> On "$ nutch ndfs -put local_file /test/testfile"(or ./nutch admin db
>> -create and etc.) I always have exception "Could not obtain new output
>> block":
>> =======================================================================
>> 050830 061956 Waiting to find target node
>> Exception in thread "main" java.io.IOException: Could not obtain new
>> output block for file /test/testfile
>> at
>> org.apache.nutch.ndfs.NDFSClient$NameNodeCaller.getNewOutputBlock(NDFSClient.java:921)
>> at
>> org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.nextBlockOutputStream(NDFSClient.java:616)
>> at
>> org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.<init>(NDFSClient.java:597)
>> at
>> org.apache.nutch.ndfs.NDFSClient.create(NDFSClient.java:85)
>> at
>> org.apache.nutch.fs.NDFSFileSystem.create(NDFSFileSystem.java:76)
>> at
>> org.apache.nutch.fs.NDFSFileSystem.create(NDFSFileSystem.java:71)
>> at
>> org.apache.nutch.io.SequenceFile$Writer.<init>(SequenceFile.java:80)
>> at
>> org.apache.nutch.io.MapFile$Writer.<init>(MapFile.java:94)
>> at
>> org.apache.nutch.db.WebDBWriter.<init>(WebDBWriter.java:1507)
>> at
>> org.apache.nutch.db.WebDBWriter.createWebDB(WebDBWriter.java:1438)
>> at
>> org.apache.nutch.tools.WebDBAdminTool.main(WebDBAdminTool.java:172)
>> =======================================================================
>>
>> On namenode I see something like this:
>> =======================================================================
>> 050830 061445 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061447 Renewed lease [Lease. Holder:
>> NDFSClient_-1094164187, heldlocks: 1, pendingcreates: 1]
>> 050830 061448 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061451 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061454 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061455 Renewed lease [Lease. Holder:
>> NDFSClient_-1094164187, heldlocks: 1, pendingcreates: 1]
>> 050830 061457 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061500 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061503 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061503 Renewed lease [Lease. Holder:
>> NDFSClient_-1094164187, heldlocks: 1, pendingcreates: 1]
>> =======================================================================
>>
>> But if I run datanode and namenode on the same server - all's ok!
>>
>> On "$ nutch ndfs -report" I see list of my datanodes, but these
>> datanodes defined by external hostname. I think that namenode tries
>> to connect to datanodes by this NOT LOCAL hostnames. It is impossible
>> because firewall not allow incoming connection from external network
>> interfaces to this port(7000).
>>
>> It's right? The error can be generated in this case?
>>
>> So, can you tell me, please, what I can make to define namenode for use
>> local interfaces for data transfer? I can't reconfigure firewall..
>>
>> Red hat ES3.0, nutch-2005-08-25 (>nutch-0.7).
>> $ java -version
>> java version "1.4.2-01"
>> Java(TM) 2 Runtime Environment, Standard Edition (build Blackdown-1.4.2-01)
>> Java HotSpot(TM) 64-Bit Server VM (build Blackdown-1.4.2-01, mixed mode)
>>
>>
>> Thanks for your time!
>>
>>
--
Best regards,
Chernodarov Egor
