Many thanks Andrzej,
I have rebuilt, and re-deployed 0.8-dev (-r 374745), and the
distributed search
is working.
I don't know if it's critical, but "ant test" on my patched
distribution freezes
at during test-core:
test-core:
[delete] Deleting directory
/usr/local/src/nutch-374745/nutch/build/test/data
[mkdir] Created dir: /usr/local/src/nutch-374745/nutch/build/test/data
[junit] Running org.apache.nutch.analysis.TestQueryParser
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.617 sec
[junit] Running org.apache.nutch.fs.TestNutchFileSystem
....
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.275 sec
[junit] Running org.apache.nutch.ndfs.TestNDFS
Also, since HADOOP versions of nutch address the problem I had here I guess no
further patches will be released for pre-hadoop versions. So I guess it would
be helpful to "publish" the fix here?:
Index: src/java/org/apache/nutch/searcher/Query.java
===================================================================
--- src/java/org/apache/nutch/searcher/Query.java (revision 374745)
+++ src/java/org/apache/nutch/searcher/Query.java (working copy)
@@ -27,12 +27,13 @@
import org.apache.nutch.util.LogFormatter;
import org.apache.nutch.util.NutchConf;
+import org.apache.nutch.util.NutchConfigurable;
import org.apache.nutch.analysis.NutchAnalysis;
import org.apache.nutch.io.Writable;
/** A Nutch query. */
-public final class Query implements Writable, Cloneable {
+public final class Query implements Writable, Cloneable, NutchConfigurable {
public static final Logger LOG =
LogFormatter.getLogger("org.apache.nutch.searcher.Query");
@@ -282,11 +283,22 @@
private NutchConf nutchConf;
private static final Clause[] CLAUSES_PROTO = new Clause[0];
+
+ public Query() {
+ }
public Query(NutchConf nutchConf) {
this.nutchConf = nutchConf;
}
+ public void setConf(NutchConf conf) {
+ this.nutchConf = conf;
+ }
+
+ public NutchConf getConf() {
+ return nutchConf;
+ }
+
/** Return all clauses. */
public Clause[] getClauses() {
return (Clause[])clauses.toArray(CLAUSES_PROTO);
Index: src/java/org/apache/nutch/searcher/DistributedSearch.java
===================================================================
--- src/java/org/apache/nutch/searcher/DistributedSearch.java (revision
374745)
+++ src/java/org/apache/nutch/searcher/DistributedSearch.java (working copy)
@@ -115,10 +115,10 @@
/** Construct a client talking to the named servers. */
public Client(InetSocketAddress[] addresses, NutchConf nutchConf) throws
IOException {
this.defaultAddresses = addresses;
+ this.nutchConf = nutchConf;
updateSegments();
setDaemon(true);
start();
- this.nutchConf = nutchConf;
}
private static final Method GET_SEGMENTS;
@@ -151,6 +151,8 @@
int liveServers=0;
int liveSegments=0;
Vector liveAddresses=new Vector();
+ System.out.println("defaultAddresses=" + defaultAddresses);
+ System.out.println("defaultAddresses.length=" +
defaultAddresses.length);
// build segmentToAddress map
Object[][] params = new Object[defaultAddresses.length][0];
Index: src/java/org/apache/nutch/ipc/Server.java
===================================================================
--- src/java/org/apache/nutch/ipc/Server.java (revision 374745)
+++ src/java/org/apache/nutch/ipc/Server.java (working copy)
@@ -34,6 +34,7 @@
import org.apache.nutch.util.LogFormatter;
import org.apache.nutch.util.NutchConf;
+import org.apache.nutch.util.NutchConfigurable;
import org.apache.nutch.io.Writable;
import org.apache.nutch.io.UTF8;
@@ -223,6 +224,8 @@
LOG.info(getName() + ": exiting");
}
}
+
+ private NutchConf conf;
/** Constructs a server listening on the named port. Parameters passed must
* be of the named class. The <code>handlerCount</handlerCount> determines
@@ -234,6 +237,7 @@
this.handlerCount = handlerCount;
this.maxQueuedCalls = handlerCount;
this.timeout = nutchConf.getInt("ipc.client.timeout",10000);
+ this.conf = nutchConf;
}
/** Sets the timeout used for network i/o. */
@@ -280,6 +284,9 @@
Writable param; // construct param
try {
param = (Writable)paramClass.newInstance();
+ if (param instanceof NutchConfigurable) {
+ ((NutchConfigurable)param).setConf(conf);
+ }
} catch (InstantiationException e) {
throw new RuntimeException(e.toString());
} catch (IllegalAccessException e) {
Many thanks, again, Adrzej
Monu Ogbe
Quoting Andrzej Bialecki <[EMAIL PROTECTED]>:
Gal Nitzan wrote:
Make sure the following exists:
1. make sure your tomcat/webapps/ROOT/WEB-INF/classes/hadoop-site.xml
fs.default.name value is local
2. make sure the machine name in your /hosts/search-servers.txt is
registered in your /etc/hosts or use the IP.
3. Make sure tomcat/webapps/ROOT/WEB-INF/classes/nutch-site.xml searcher.dir
value is set to /hosts
4. Make sure the Nutch user has access to /hosts On your search
server machine tail the log. the connection from tomcat
should appear.
FYI, I helped Monu to solve the issue. It turns out that the
particular revision he is using was missing some later fixes,
specifically NUTCH-217 and related fixes to Hadoop. After applying
these patches it started working. Please be aware that if you use the
latest pre-Hadoop version you will face the same issue, i.e. it won't
work without patching.
....
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
