Hello,
I am trying to create my cluster of storm and I copy the apache-storm-0.9.2
directory in the /opt folder (my machines run Ubuntu 14.04). I have created
a storm-users user group and I have added my username to this group. When I
perform an ls -l to my /opt folder I get the following:
katsip@mini01:/opt$ ls -al
total 12
drwxr-xr-x 3 root root 4096 Oct 9 11:07 .
drwxr-xr-x 27 root root 4096 Oct 9 06:50 ..
drwxr-xr-x 9 katsip storm-users 4096 Oct 9 13:16
apache-storm-0.9.2-incubating
I have three nodes in my cluster: mini01, mini02, mini03 and mini01 will
have the nimbus, the zookeeper and he is going be the master node. The
storm.yaml file in the mini01 has the following text:
katsip@mini01:/opt/apache-storm-0.9.2-incubating/conf$ cat storm.yaml
java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib"
storm.home: "/opt/apache-storm-0.9.2-incubating"
storm.local.dir: "/opt/apache-storm-0.9.2-incubating"
storm.zookeeper.servers:
- "mini01"
storm.zookeeper.port: 2181
storm.zookeeper.root: "/storm"
storm.zookeeper.session.timeout: 20000
storm.zookeeper.connection.timeout: 15000
storm.zookeeper.retry.times: 5
storm.zookeeper.retry.interval: 1000
storm.zookeeper.retry.intervalceiling.millis: 30000
storm.cluster.mode: "distributed" # can be distributed or local
storm.local.mode.zmq: false
storm.thrift.transport: "backtype.storm.security.auth.SimpleTransportPlugin"
storm.messaging.transport: "backtype.storm.messaging.netty.Context"
storm.scheduler.meta:
name: "mini01"
nimbus.host: "localhost"
nimbus.thrift.port: 6627
nimbus.thrift.max_buffer_size: 1048576
nimbus.childopts: "-Xmx1024m"
nimbus.task.timeout.secs: 30
nimbus.supervisor.timeout.secs: 60
nimbus.monitor.freq.secs: 10
nimbus.cleanup.inbox.freq.secs: 600
nimbus.inbox.jar.expiration.secs: 3600
nimbus.task.launch.secs: 120
nimbus.reassign: true
nimbus.file.copy.expiration.secs: 600
nimbus.topology.validator: "backtype.storm.nimbus.DefaultTopologyValidator"
### ui.* configs are for the master
ui.port: 8080
ui.childopts: "-Xmx768m"
logviewer.port: 8000
logviewer.childopts: "-Xmx128m"
logviewer.appender.name: "A1"
drpc.port: 3772
drpc.worker.threads: 64
drpc.queue.size: 128
drpc.invocations.port: 3773
drpc.request.timeout.secs: 600
drpc.childopts: "-Xmx768m"
transactional.zookeeper.root: "/transactional"
transactional.zookeeper.servers: null
transactional.zookeeper.port: null
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
supervisor.childopts: "-Xmx256m"
supervisor.worker.start.timeout.secs: 120
supervisor.worker.timeout.secs: 30
supervisor.monitor.frequency.secs: 3
supervisor.heartbeat.frequency.secs: 5
supervisor.enable: true
### worker.* configs are for task workers
worker.childopts: "-Xmx768m"
worker.heartbeat.frequency.secs: 1
task.heartbeat.frequency.secs: 3
task.refresh.poll.secs: 10
zmq.threads: 1
zmq.linger.millis: 5000
zmq.hwm: 0
storm.messaging.netty.server_worker_threads: 1
storm.messaging.netty.client_worker_threads: 1
storm.messaging.netty.buffer_size: 5242880 #5MB buffer
storm.messaging.netty.max_retries: 30
storm.messaging.netty.max_wait_ms: 1000
storm.messaging.netty.min_wait_ms: 100
### topology.* configs are for specific executing storms
topology.enable.message.timeouts: true
topology.debug: false
topology.optimize: true
topology.workers: 1
topology.acker.executors: null
topology.tasks: null
# maximum amount of time a message has to complete before it's considered
failed
topology.message.timeout.secs: 30
topology.skip.missing.kryo.registrations: false
topology.max.task.parallelism: null
topology.max.spout.pending: null
topology.state.synchronization.timeout.secs: 60
topology.stats.sample.rate: 0.05
topology.builtin.metrics.bucket.size.secs: 60
topology.fall.back.on.java.serialization: true
topology.worker.childopts: null
topology.executor.receive.buffer.size: 1024 #batched
topology.executor.send.buffer.size: 1024 #individual messages
topology.receiver.buffer.size: 8 # setting it too high causes a lot of
problems (heartbeat thread gets starved, throughput plummets)
topology.transfer.buffer.size: 1024 # batched
topology.tick.tuple.freq.secs: null
topology.worker.shared.thread.pool.size: 4
topology.disruptor.wait.strategy: "com.lmax.disruptor.BlockingWaitStrategy"
topology.spout.wait.strategy: "backtype.storm.spout.SleepSpoutWaitStrategy"
topology.sleep.spout.wait.strategy.time.ms: 1
topology.error.throttle.interval.secs: 10
topology.max.error.report.per.interval: 5
topology.kryo.factory: "backtype.storm.serialization.DefaultKryoFactory"
topology.tuple.serializer:
"backtype.storm.serialization.types.ListDelegateSerializer"
topology.trident.batch.emit.interval.millis: 500
dev.zookeeper.path: "/tmp/dev-storm-zookeeper"
and the storm.yaml for mini02 and mini03 are the following:
storm.home: "/opt/apache-storm-0.9.2-incubating"
########### These MUST be filled in for a storm configuration
# storm.zookeeper.servers:
# - "server1"
# - "server2"
#
nimbus.host: "mini01.ip" (here is the ip of the mini01 server)
supervisor.scheduler.eth:
name: "mini02"
supervisor.scheduler.meta:
name: "mini02"
In the storm_env.ini I just set the JAVA_HOME environment variable. In the
logback/cluster.xml I define the following (it is the same for all
machines):
<configuration scan="true" scanPeriod="60 seconds">
<appender name="A1"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${storm.home}/logs/${logfile.name}</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>${storm.home}/logs/${logfile.name
}.%i</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>9</maxIndex>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>100MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n</pattern>
</encoder>
</appender>
<appender name="ACCESS"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${storm.home}/logs/access.log</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>${storm.home}/logs/access.log.%i</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>9</maxIndex>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>100MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n</pattern>
</encoder>
</appender>
<appender name="METRICS"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${storm.home}/logs/metrics.log</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>${storm.home}/logs/metrics.log.%i</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>9</maxIndex>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>2MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%d %-8r %m%n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="A1"/>
</root>
<logger name="backtype.storm.security.auth.authorizer" additivity="false">
<level value="INFO" />
<appender-ref ref="ACCESS" />
</logger>
<logger name="backtype.storm.metric.LoggingMetricsConsumer"
additivity="false" >
<level value="INFO"/>
<appender-ref ref="METRICS"/>
</logger>
</configuration>
When I run the ExclamationTopology from the storm-starter project it
finishes successfully. However, I can not see any log files created. What
am I doing wrong? I execute the ExclamationTopology as the user katsip but
no logs directory is created in the {storm.home} directory. Any indications
on what I am doing wrong?
Thank you,
Nick
P.S. Sorry for the long post
