Yes. From IP address to name that is reverse DNS lookup. Yes mapreduce will work but you will see that error in logs.
Regards On Jun 25, 2014 11:12 AM, "Li Li" <[email protected]> wrote: > there is not any DNS server for me . you mean find name by ip? if no > DNS server, will it work correctly? > > On Wed, Jun 25, 2014 at 5:08 PM, Samir Ahmic <[email protected]> > wrote: > > Do you use DNS server for name resolution ? Did you setup reverse DNS > zone > > in your cluster ? I have seen this errors before when there is no reverse > > DNS setup. I believe that TableInputFormatBase class requires revers DNS > > name resolution. > > > > Regards > > Samir > > > > > > On Wed, Jun 25, 2014 at 10:57 AM, Li Li <[email protected]> wrote: > > > >> I have many map reduce jobs using hbase table as input. Others are all > >> correct. > >> This one is a little bit difference because it use both hdfs and hbase > >> as input source. > >> btw, even there are errors, the job can run successfully. > >> > >> My codes: > >> > >> 1. Hbase Table Mapper, mapper output key is Text and value is my custom > >> writable > >> public class UrlDedupHbaseMapper extends TableMapper<Text, > >> ExtractResultWritable> { > >> > >> @Override > >> public void map(NullWritable key, ExtractResultWritable value, Context > >> context) > >> throws IOException, InterruptedException { > >> > >> } > >> > >> } > >> > >> 2. Hdfs Mapper, which read from sequence files, input key is > >> NullWritable, value is my custom writable. Mapper output key is Text > >> and value is my custome writable > >> > >> public class UrlDedupHbaseMapper extends TableMapper<Text, > >> ExtractResultWritable> { > >> public void map(ImmutableBytesWritable key, Result value, Context > >> context) > >> throws IOException, InterruptedException { > >> } > >> } > >> > >> 3. Reducer > >> public class UrlDedupReducer extends > >> Reducer<Text, ExtractResultWritable, NullWritable, > ExtractResultWritable> { > >> > >> @Override > >> public void reduce(Text key, Iterable<ExtractResultWritable> > >> values, Context context) > >> throws IOException, InterruptedException { > >> } > >> } > >> > >> 4. driver class > >> > >> Configuration conf=getConf(); > >> Configuration myConf = HBaseConfiguration.create(conf); > >> myConf.set("hbase.zookeeper.quorum", zkQuorum); > >> myConf.set("hbase.zookeeper.property.clientPort", zkPort); > >> myConf.set("mapred.child.java.opts", "-Xmx1g"); > >> myConf.set("mapred.output.compress", "true"); > >> Job job = new Job(myConf); > >> job.setJobName(UrlDedup.class.getSimpleName()); > >> > >> job.setJarByClass(UrlDedup.class); > >> ArrayList<Path> fileList=new ArrayList<Path>(); > >> FileSystem fs = FileSystem.get(conf); > >> Path inFile = new Path(args[0]); > >> FileStatus[] status = fs.listStatus(inFile); > >> for(FileStatus file:status){ > >> fileList.add(file.getPath()); > >> } > >> > >> for(Path path:fileList){ > >> MultipleInputs.addInputPath(job, path, > >> > SequenceFileInputFormat.class,UrlDedupHdfsMapper.class); > >> } > >> Scan urldbScan=new Scan(); > >> urldbScan.setCaching(hbaseBlockCache); > >> urldbScan.setCacheBlocks(false); > >> urldbScan.addFamily(HbaseTools.CF_BT); > >> > >> job.setInputFormatClass(SequenceFileInputFormat.class); > >> TableMapReduceUtil.initTableMapperJob(HbaseTools.TB_URL_DB_BT, > >> urldbScan, > >> UrlDedupHbaseMapper.class, NullWritable.class, > >> ExtractResultWritable.class, job); > >> MultipleInputs.addInputPath(job, new Path("/null/path"), > >> TableInputFormat.class, > >> UrlDedupHbaseMapper.class); > >> job.setOutputFormatClass(SequenceFileOutputFormat.class); > >> FileOutputFormat.setOutputPath(job, new Path(args[1])); > >> job.setReducerClass(UrlDedupReducer.class); > >> > >> job.setOutputKeyClass(NullWritable.class); > >> job.setOutputValueClass(ExtractResultWritable.class); > >> job.setMapOutputKeyClass(Text.class); > >> job.setMapOutputValueClass(ExtractResultWritable.class); > >> > >> On Wed, Jun 25, 2014 at 4:49 PM, Ted Yu <[email protected]> wrote: > >> > Can you ping vc141 from this machine ? > >> > > >> > Cheers > >> > > >> > On Jun 25, 2014, at 1:29 AM, Li Li <[email protected]> wrote: > >> > > >> >> I have a map reduce job using hbase table as input. when the job > >> >> starts, it says: > >> >> > >> >> ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase > >> >> Cannot resolve the host name for vc141/172.16.10.141 because of > >> >> javax.naming.CommunicationException: DNS error [Root exception is > >> >> java.net.PortUnreachableException: ICMP Port Unreachable]; remaining > >> >> name '141.10.16.172.in-addr.arpa' > >> >> > >> >> the /etc/hosts file: > >> >> 172.16.10.137 vc137 > >> >> 172.16.10.138 vc138 > >> >> 172.16.10.139 vc139 > >> >> 172.16.10.140 vc140 > >> >> 172.16.10.141 vc141 > >> >> 172.16.10.142 vc142 > >> >> 172.16.10.143 vc143 > >> >> 172.16.10.144 vc144 > >> >> 172.16.10.145 vc145 > >> >> 172.16.10.146 vc146 > >> >> 172.16.10.147 vc147 > >> >> 172.16.10.148 vc148 > >> >> 172.16.10.149 vc149 > >> >
