Hey Stuart: Would the Mapper and Reducer talk to different clusters?
Do you need to go to a reduce? Can you write to the output from the mapper? (See http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#sink). Can you use http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapreduce/Reducer.html#setup(org.apache.hadoop.mapreduce.Reducer.Context) in TableReducer to set output? You want to do this programmatically rather than via configuration files, adding hbase-site.xml to your job? St.Ack On Wed, Mar 30, 2011 at 8:34 AM, Stuart Scott <[email protected]> wrote: > Hi, > > > > I have a map/reduce job that reads from a Hbase Table and writes to > another. > > Does anyone know how to programmatically set the Zoo Keeper address for > a Reducer? > > I can create a job as below, set the IP address using .set... it works > fine for the Maper. The reducer defaults to localhost. We have a couple > of clusters and wanted to be able to connect to them at different times. > > > > HBaseConfiguration conf = new HBaseConfiguration(); > > conf.set("hbase.master", "44.128.161.86:60000"); > > conf.set("hbase.zookeeper.quorum","44.128.161.86"); > > > > ... > > > > Job job = new Job(conf, "MyJob_Job"); > > job.setJarByClass(MyClass.class); > > Scan scan = new Scan(); > > String columns = "A:B"; > > > scan.addColumns(columns); > > scan.setStartRow(Bytes.toBytes(GetPaddedLongKey(1))); > > scan.setStopRow(Bytes.toBytes(GetPaddedLongKey(100+1))); > > TableMapReduceUtil.initTableMapperJob("MPI", scan, > Mapper2.class, ImmutableBytesWritable.class, IntWritable.class, job); > > > > TableMapReduceUtil.initTableReducerJob("MyClass", > Reducer1.class, job); > > > > job.waitForCompletion(true); > > > > Runs the mapper without problem, then fails.. > > > > 11/03/30 16:19:50 INFO mapred.LocalJobRunner: > > 11/03/30 16:19:50 INFO zookeeper.ZooKeeperWrapper: Reconnecting to > zookeeper > > 11/03/30 16:19:50 INFO zookeeper.ZooKeeper: Initiating client > connection, connectString=localhost:21810 sessionTimeout=60000 > watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@ae97c4 > > 11/03/30 16:19:50 INFO zookeeper.ClientCnxn: Opening socket connection > to server localhost/127.0.0.1:21810 > > > > Closing socket connection and attempting reconnect > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > > 2011-03-30 16:30:14,637 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server localhost/127.0.0.1:21810 > > 2011-03-30 16:30:14,638 WARN org.apache.zookeeper.ClientCnxn: Session > 0x0 for server null, unexpected error, closing socket connection and > attempting reconnect > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > > > > > > > Any help would be really appreciated. > > > > Stuart > > > > > >
