Hello,
I would like to start topologies in LocalCluster mode for testing purposes
using my own settings for the Zookeeper server started by the LocalCluster
mode.
For example, I would like Zookeeper to write in e:\tmp\exclam\stormlocaldir
and use port 123456.
Storm documentation isn't quite verbose about how to set LocalCluster mode
options, unless I missed the appropriate page (in which please, please let
me know at which URL I can find how to pass options to LocalCluster mode).
I tried to modify the main() method of ExclamationTopology classic Storm
sampe to fix Zookeeer properties, like this:
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("word", new TestWordSpout(), 10);
builder.setBolt("exclaim1", new ExclamationBolt(),
3).shuffleGrouping("word");
builder.setBolt("exclaim2", new ExclamationBolt(),
2).shuffleGrouping("exclaim1");
Config conf = new Config();
conf.setDebug(true);
conf.put(Config.STORM_LOCAL_DIR,
"E:\\tmp\\exclam\\stormlocaldir\\");
conf.put(Config.STORM_ZOOKEEPER_ROOT,
"E:\\tmp\\exclam\\stormlocaldir\\");
conf.put(Config.STORM_ZOOKEEPER_PORT,12346);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("test", conf, builder.createTopology());
Utils.sleep(120000);
cluster.killTopology("test");
cluster.shutdown();
}
}
But to my despair, all data is written into subdirectories
C:\Users\ave\AppData\Local\Temp\<some UUID> instead of to
E:\tmp\exclam\stormlocaldir\ subdirectories.
Also, during the program's execution, Zookeeper listens on port 2000
instead of listening on port 123456 (I checked this: it's not just a wrong
trace).
I was able to partially by-pass my problem with data directory by setting
Java's system property which gives the temporary directory path, but this
is a very ugly by-pass:
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("word", new TestWordSpout(), 10);
builder.setBolt("exclaim1", new ExclamationBolt(),
3).shuffleGrouping("word");
builder.setBolt("exclaim2", new ExclamationBolt(),
2).shuffleGrouping("exclaim1");
Config conf = new Config();
conf.setDebug(true);
System.setProperty("java.io.tmpdir",
"E:\\tmp\\exclam\\stormlocaldir\\");
conf.put(Config.STORM_ZOOKEEPER_PORT,12346);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("test", conf, builder.createTopology());
Utils.sleep(120000);
cluster.killTopology("test");
cluster.shutdown();
}
But I found no by-pass to set Zookeeper's port.
I am aware that another by-pass could be to use my own Zookeeper server and
then use the un-documented LocalCluster(host,port) constructor to start a
LocalCluster connect to this Zookeeper server, but frankly... isn't there a
simpler way to setup properties for the Zookeeper server started by default
constructor of LocalCluster ?
Best regards,
Alexandre Vermeerbergen