Please see https://issues.apache.org/jira/browse/HBASE-10906
Cheers On Jun 25, 2014, at 2:12 AM, Li Li <[email protected]> wrote: > there is not any DNS server for me . you mean find name by ip? if no > DNS server, will it work correctly? > > On Wed, Jun 25, 2014 at 5:08 PM, Samir Ahmic <[email protected]> wrote: >> Do you use DNS server for name resolution ? Did you setup reverse DNS zone >> in your cluster ? I have seen this errors before when there is no reverse >> DNS setup. I believe that TableInputFormatBase class requires revers DNS >> name resolution. >> >> Regards >> Samir >> >> >> On Wed, Jun 25, 2014 at 10:57 AM, Li Li <[email protected]> wrote: >> >>> I have many map reduce jobs using hbase table as input. Others are all >>> correct. >>> This one is a little bit difference because it use both hdfs and hbase >>> as input source. >>> btw, even there are errors, the job can run successfully. >>> >>> My codes: >>> >>> 1. Hbase Table Mapper, mapper output key is Text and value is my custom >>> writable >>> public class UrlDedupHbaseMapper extends TableMapper<Text, >>> ExtractResultWritable> { >>> >>> @Override >>> public void map(NullWritable key, ExtractResultWritable value, Context >>> context) >>> throws IOException, InterruptedException { >>> >>> } >>> >>> } >>> >>> 2. Hdfs Mapper, which read from sequence files, input key is >>> NullWritable, value is my custom writable. Mapper output key is Text >>> and value is my custome writable >>> >>> public class UrlDedupHbaseMapper extends TableMapper<Text, >>> ExtractResultWritable> { >>> public void map(ImmutableBytesWritable key, Result value, Context >>> context) >>> throws IOException, InterruptedException { >>> } >>> } >>> >>> 3. Reducer >>> public class UrlDedupReducer extends >>> Reducer<Text, ExtractResultWritable, NullWritable, ExtractResultWritable> { >>> >>> @Override >>> public void reduce(Text key, Iterable<ExtractResultWritable> >>> values, Context context) >>> throws IOException, InterruptedException { >>> } >>> } >>> >>> 4. driver class >>> >>> Configuration conf=getConf(); >>> Configuration myConf = HBaseConfiguration.create(conf); >>> myConf.set("hbase.zookeeper.quorum", zkQuorum); >>> myConf.set("hbase.zookeeper.property.clientPort", zkPort); >>> myConf.set("mapred.child.java.opts", "-Xmx1g"); >>> myConf.set("mapred.output.compress", "true"); >>> Job job = new Job(myConf); >>> job.setJobName(UrlDedup.class.getSimpleName()); >>> >>> job.setJarByClass(UrlDedup.class); >>> ArrayList<Path> fileList=new ArrayList<Path>(); >>> FileSystem fs = FileSystem.get(conf); >>> Path inFile = new Path(args[0]); >>> FileStatus[] status = fs.listStatus(inFile); >>> for(FileStatus file:status){ >>> fileList.add(file.getPath()); >>> } >>> >>> for(Path path:fileList){ >>> MultipleInputs.addInputPath(job, path, >>> SequenceFileInputFormat.class,UrlDedupHdfsMapper.class); >>> } >>> Scan urldbScan=new Scan(); >>> urldbScan.setCaching(hbaseBlockCache); >>> urldbScan.setCacheBlocks(false); >>> urldbScan.addFamily(HbaseTools.CF_BT); >>> >>> job.setInputFormatClass(SequenceFileInputFormat.class); >>> TableMapReduceUtil.initTableMapperJob(HbaseTools.TB_URL_DB_BT, >>> urldbScan, >>> UrlDedupHbaseMapper.class, NullWritable.class, >>> ExtractResultWritable.class, job); >>> MultipleInputs.addInputPath(job, new Path("/null/path"), >>> TableInputFormat.class, >>> UrlDedupHbaseMapper.class); >>> job.setOutputFormatClass(SequenceFileOutputFormat.class); >>> FileOutputFormat.setOutputPath(job, new Path(args[1])); >>> job.setReducerClass(UrlDedupReducer.class); >>> >>> job.setOutputKeyClass(NullWritable.class); >>> job.setOutputValueClass(ExtractResultWritable.class); >>> job.setMapOutputKeyClass(Text.class); >>> job.setMapOutputValueClass(ExtractResultWritable.class); >>> >>> On Wed, Jun 25, 2014 at 4:49 PM, Ted Yu <[email protected]> wrote: >>>> Can you ping vc141 from this machine ? >>>> >>>> Cheers >>>> >>>> On Jun 25, 2014, at 1:29 AM, Li Li <[email protected]> wrote: >>>> >>>>> I have a map reduce job using hbase table as input. when the job >>>>> starts, it says: >>>>> >>>>> ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase >>>>> Cannot resolve the host name for vc141/172.16.10.141 because of >>>>> javax.naming.CommunicationException: DNS error [Root exception is >>>>> java.net.PortUnreachableException: ICMP Port Unreachable]; remaining >>>>> name '141.10.16.172.in-addr.arpa' >>>>> >>>>> the /etc/hosts file: >>>>> 172.16.10.137 vc137 >>>>> 172.16.10.138 vc138 >>>>> 172.16.10.139 vc139 >>>>> 172.16.10.140 vc140 >>>>> 172.16.10.141 vc141 >>>>> 172.16.10.142 vc142 >>>>> 172.16.10.143 vc143 >>>>> 172.16.10.144 vc144 >>>>> 172.16.10.145 vc145 >>>>> 172.16.10.146 vc146 >>>>> 172.16.10.147 vc147 >>>>> 172.16.10.148 vc148 >>>>> 172.16.10.149 vc149 >>>
