Thank you Taylor! And Justin!
I have tried doing the same with open source Apache Hadoop and it's the
same Problem with it too. In specified this ...
HdfsBolt().withFsUrl(hdfs://namenodepath:9000)
This is where my storm application is going and talking to hdfs. On the
( open source Apache Hadoop ) hdfs side particularly on each node in
hdfs-site.xml in enabled dfs.client.use.datanode.hostname and
dfs.datanode.use.datanode.hostname to true expecting that the communication
would happen through hostnames.
On my storm side, how do I make my topology aware of those configuration
changes?? How do I pass those two properties on my storm client side to my
HdfsBolt?
Basically what I did was, after I built the topology with builder I
used addConfigurations() to add those two properties namely
dfs.client.use.....
public StormTopology build() {
LOGGER.info("Building topology");
Map<String, Object> clientHdfsConfig = newHashMap<String, Object>();
clientHdfsConfig.put("dfs.client.use.datanode.hostname", true);
clientHdfsConfig.put("dfs.datanode.use.datanode.hostname", true);
TopologyBuilder builder = newTopologyBuilder();
builder.setSpout(this.realtimeConfiguration.getKafkaSpoutId(),
this.kafkaSpout);
builder.setBolt(MESSAGE_KEY_ASSIGNER_BOLT_ID,
new MessageKeyAssignerBolt()).shuffleGrouping(
this.realtimeConfiguration.getKafkaSpoutId());
builder.setBolt(this.realtimeConfiguration.getHdfsBoltId(),
this.hdfsBolt).shuffleGrouping(MESSAGE_KEY_ASSIGNER_BOLT_ID)
.addConfigurations(clientHdfsConfig);
StormTopology stormTopology = builder.createTopology();
LOGGER.info("Successfully built topology");
return stormTopology;
}
I did it assuming that the storm will now be aware of those two propertiy
changes on Hadoop side. But it doesn't resolve the problem.
Below are the dependencies from my Pom.xml :
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>${storm.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-hdfs</artifactId>
<version>0.9.3</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.6.0-cdh5.4.2</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
Any pointers are highly appreciated.
Thank you,
Ajay
On Monday, June 22, 2015, P. Taylor Goetz <[email protected]> wrote:
> There might be others here that are using Storm with CDH, but since
> Cloudera Manager is closed source/proprietary, you may well be better off
> asking the question on a Cloudera forum for specific details.
>
> That being said, the community here is likely willing to help, but you’ll
> need to get past the Cloudera-specific pieces and down to the open source
> parts. If you know the HDFS URL for the cluster, you should be able to
> connect.
>
> What version of Storm are you using? When you say “trying to write to
> hadoop” do you mean HDFS, or something else? Are you using the storm-hdfs
> component that ships with Apache Storm?
>
> The more details you can provide the better the community will be able to
> help you.
>
> -Taylor
>
> On Jun 22, 2015, at 8:06 PM, Ajay Chander <[email protected]> wrote:
>
> > Hi Everyone,
> >
> > I am trying to write the data into hadoop from my storm topology. For
> this communication to happen through hostnames, I have enabled couple of
> properties namely "dfs.client.use.datanode.hostname"=true and
> "dfs.datanode.use.datanode.hostname" = true in my cloudera manager. Now how
> do I make my storm aware of those two properties. When my storm topology is
> running by default it takes those properties as false. Now how do I
> override those properties ("dfs.client.use.datanode.hostname"=true and
> "dfs.datanode.use.datanode.hostname" = true) which are specific to my
> hadoop in my hdfsbolt. ??
> >
> > Any help is highly appreciated.
> >
> > Thank you,
> > Ajay
>
>
