https://gist.github.com/bretthoerner/0dc6bfdbf45a18328d4b
On Thu, Apr 17, 2014 at 11:31 AM, Mark Miller <markrmil...@gmail.com> wrote: > Odd - might be helpful if you can share your sorlconfig.xml being used. > > -- > Mark Miller > about.me/markrmiller > > On April 17, 2014 at 12:18:37 PM, Brett Hoerner (br...@bretthoerner.com) > wrote: > > I'm doing HDFS input and output in my job, with the following: > > hadoop jar /mnt/faas-solr.jar \ > -D mapreduce.job.map.class=com.massrel.faassolr.SolrMapper \ > --update-conflict-resolver com.massrel.faassolr.SolrConflictResolver > \ > --morphline-file /mnt/morphline-ignore.conf \ > --zk-host $ZKHOST \ > --output-dir hdfs://$MASTERIP:9000/output/ \ > --collection $COLLECTION \ > --go-live \ > --verbose \ > hdfs://$MASTERIP:9000/input/ > > Index creation works, > > $ hadoop fs -ls -R hdfs://$MASTERIP:9000/output/results/part-00000 > drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data > drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index > -rwxr-xr-x 1 hadoop supergroup 61 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/_0.fdt > -rwxr-xr-x 1 hadoop supergroup 45 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/_0.fdx > -rwxr-xr-x 1 hadoop supergroup 1681 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/_0.fnm > -rwxr-xr-x 1 hadoop supergroup 396 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/_0.si > -rwxr-xr-x 1 hadoop supergroup 67 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene41_0.doc > -rwxr-xr-x 1 hadoop supergroup 37 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene41_0.pos > -rwxr-xr-x 1 hadoop supergroup 508 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene41_0.tim > -rwxr-xr-x 1 hadoop supergroup 305 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene41_0.tip > -rwxr-xr-x 1 hadoop supergroup 120 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene45_0.dvd > -rwxr-xr-x 1 hadoop supergroup 351 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/_0_Lucene45_0.dvm > -rwxr-xr-x 1 hadoop supergroup 45 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/segments_1 > -rwxr-xr-x 1 hadoop supergroup 110 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/index/segments_2 > drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs:// > 10.98.33.114:9000/output/results/part-00000/data/tlog > -rw-r--r-- 1 hadoop supergroup 333 2014-04-17 16:00 hdfs:// > > 10.98.33.114:9000/output/results/part-00000/data/tlog/tlog.0000000000000000000 > > But the go-live step fails, it's trying to use the HDFS path as the remote > index path? > > 14/04/17 16:00:31 INFO hadoop.GoLive: Live merging of output shards into > Solr cluster... > 14/04/17 16:00:31 INFO hadoop.GoLive: Live merge hdfs:// > 10.98.33.114:9000/output/results/part-00000 into > http://discover8-test-1d.i.massrel.com:8983/solr > 14/04/17 16:00:31 ERROR hadoop.GoLive: Error sending live merge command > java.util.concurrent.ExecutionException: > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: > directory '/mnt/solr_8983/home/hdfs:/ > 10.98.33.114:9000/output/results/part-00000/data/index' does not exist > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at org.apache.solr.hadoop.GoLive.goLive(GoLive.java:126) > at > > org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:867) > at > > org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:609) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > > org.apache.solr.hadoop.MapReduceIndexerTool.main(MapReduceIndexerTool.java:596) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: > directory '/mnt/solr_8983/home/hdfs:/ > 10.98.33.114:9000/output/results/part-00000/data/index' does not exist > at > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495) > at > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199) > at > > org.apache.solr.client.solrj.request.CoreAdminRequest.process(CoreAdminRequest.java:493) > at org.apache.solr.hadoop.GoLive$1.call(GoLive.java:100) > at org.apache.solr.hadoop.GoLive$1.call(GoLive.java:89) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 14/04/17 16:00:31 INFO hadoop.GoLive: Live merging of index shards into > Solr cluster took 2.31269488E8 secs > 14/04/17 16:00:31 INFO hadoop.GoLive: Live merging failed > > I'm digging into the code now, but wanted to send this out as a sanity > check. > > Thanks, > Brett >
