Are you sure your hdfs url is correct. I believe with both open source Apache Hadoop and cdh, the name node port is 8020 not 9000? Sent from my iPhone
> On Jun 22, 2015, at 8:52 PM, Ajay Chander <[email protected]> wrote: > > Thank you Taylor! And Justin! > > I have tried doing the same with open source Apache Hadoop and it's the same > Problem with it too. In specified this ... > HdfsBolt().withFsUrl(hdfs://namenodepath:9000) > > This is where my storm application is going and talking to hdfs. On the ( > open source Apache Hadoop ) hdfs side particularly on each node in > hdfs-site.xml in enabled dfs.client.use.datanode.hostname and > dfs.datanode.use.datanode.hostname to true expecting that the communication > would happen through hostnames. > > On my storm side, how do I make my topology aware of those configuration > changes?? How do I pass those two properties on my storm client side to my > HdfsBolt? > > Basically what I did was, after I built the topology with builder I used > addConfigurations() to add those two properties namely dfs.client.use..... > > public StormTopology build() { > LOGGER.info("Building topology"); > Map<String, Object> clientHdfsConfig = newHashMap<String, Object>(); > clientHdfsConfig.put("dfs.client.use.datanode.hostname", true); > clientHdfsConfig.put("dfs.datanode.use.datanode.hostname", true); > TopologyBuilder builder = newTopologyBuilder(); > builder.setSpout(this.realtimeConfiguration.getKafkaSpoutId(), > this.kafkaSpout); > builder.setBolt(MESSAGE_KEY_ASSIGNER_BOLT_ID, > new MessageKeyAssignerBolt()).shuffleGrouping( > this.realtimeConfiguration.getKafkaSpoutId()); > builder.setBolt(this.realtimeConfiguration.getHdfsBoltId(), > this.hdfsBolt).shuffleGrouping(MESSAGE_KEY_ASSIGNER_BOLT_ID) > .addConfigurations(clientHdfsConfig); > > StormTopology stormTopology = builder.createTopology(); > LOGGER.info("Successfully built topology"); > return stormTopology; > } > > I did it assuming that the storm will now be aware of those two propertiy > changes on Hadoop side. But it doesn't resolve the problem. > > Below are the dependencies from my Pom.xml : > > <dependency> > <groupId>org.apache.storm</groupId> > <artifactId>storm-core</artifactId> > <version>${storm.version}</version> > <scope>provided</scope> > </dependency> > > <dependency> > <groupId>org.apache.storm</groupId> > <artifactId>storm-hdfs</artifactId> > <version>0.9.3</version> > <exclusions> > <exclusion> > <groupId>org.slf4j</groupId> > <artifactId>slf4j-log4j12</artifactId> > </exclusion> > </exclusions> > </dependency> > > <dependency> > <groupId>org.apache.hadoop</groupId> > <artifactId>hadoop-client</artifactId> > <version>${hadoop.version}</version> > <exclusions> > <exclusion> > <groupId>org.slf4j</groupId> > <artifactId>slf4j-log4j12</artifactId> > </exclusion> > </exclusions> > </dependency> > > <dependency> > <groupId>org.apache.hadoop</groupId> > <artifactId>hadoop-hdfs</artifactId> > <version>2.6.0-cdh5.4.2</version> > <exclusions> > <exclusion> > <groupId>org.slf4j</groupId> > <artifactId>slf4j-log4j12</artifactId> > </exclusion> > </exclusions> > </dependency> > > Any pointers are highly appreciated. > > Thank you, > Ajay > > >> On Monday, June 22, 2015, P. Taylor Goetz <[email protected]> wrote: >> There might be others here that are using Storm with CDH, but since Cloudera >> Manager is closed source/proprietary, you may well be better off asking the >> question on a Cloudera forum for specific details. >> >> That being said, the community here is likely willing to help, but you’ll >> need to get past the Cloudera-specific pieces and down to the open source >> parts. If you know the HDFS URL for the cluster, you should be able to >> connect. >> >> What version of Storm are you using? When you say “trying to write to >> hadoop” do you mean HDFS, or something else? Are you using the storm-hdfs >> component that ships with Apache Storm? >> >> The more details you can provide the better the community will be able to >> help you. >> >> -Taylor >> >> On Jun 22, 2015, at 8:06 PM, Ajay Chander <[email protected]> wrote: >> >> > Hi Everyone, >> > >> > I am trying to write the data into hadoop from my storm topology. For this >> > communication to happen through hostnames, I have enabled couple of >> > properties namely "dfs.client.use.datanode.hostname"=true and >> > "dfs.datanode.use.datanode.hostname" = true in my cloudera manager. Now >> > how do I make my storm aware of those two properties. When my storm >> > topology is running by default it takes those properties as false. Now how >> > do I override those properties ("dfs.client.use.datanode.hostname"=true >> > and "dfs.datanode.use.datanode.hostname" = true) which are specific to my >> > hadoop in my hdfsbolt. ?? >> > >> > Any help is highly appreciated. >> > >> > Thank you, >> > Ajay
