To separate out the worker logs per topology, you need :
* create separate cluster.xml files per application.
** currently, cluster.xml is hardcoded to the storm worker application
** cluster.xml is found in ${STORM_HOME}/logback
** deploy the custom xml files to all of the storm nodes
** change the A1 file name from ${logfile.name} to ${logfile.name}_${
storm.id}
* alter the python file, storm. pass the custom cluster_{program_name} to
the storm application. search the file for "cluster.xml"
The downside is that logback's log rotation as a means of keeping your disk
from overfilling is no longer practical. This becomes more and more true
whenever you are loading and killing topologies on a regular basis.
Basically, your disk will become full of logs. You will need to implement
some other method of log cleanup.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <[email protected]>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Wed, May 13, 2015 at 1:06 PM, <[email protected]> wrote:
> *Dell - Internal Use - Confidential *
>
> what is every one using to process the storm logs from all over the
> cluster for troubleshooting and debugging purposes.
>
>
>
> The accessing storm logs via the storm UI or tailing worker logs is
> getting to be pretty painful.
>
>
>
> I guess this could be said of a lot of other Hadoop components like Kafka,
> ZooKeeper, HBase and such.
>
>
>
> P.S: Did anyone get logback working properly to separate out topology
> level logs from worker-xxxx.log if so what are the gotchas you ran into.
>
