I do not at this time have suggestions. Thank you for your time!
+++++++++++++++++++++ Jeff Maass <[email protected]> linkedin.com/in/jeffmaass stackoverflow.com/users/373418/maassql +++++++++++++++++++++ On Thu, May 14, 2015 at 9:14 PM, <[email protected]> wrote: > *Dell - Internal Use - Confidential * > > Thanks Jeffery, great to see enthusiastic community around storm. > > > > Do you have any suggestions on the log retention and indexing mechanisms > (logstash etc?) > > > > *From:* Jeffery Maass [mailto:[email protected]] > *Sent:* Thursday, May 14, 2015 4:42 PM > *To:* [email protected] > *Subject:* Re: storm topology logsaggregate and processing them > > > > To separate out the worker logs per topology, you need : > > * create separate cluster.xml files per application. > ** currently, cluster.xml is hardcoded to the storm worker application > > ** cluster.xml is found in ${STORM_HOME}/logback > > ** deploy the custom xml files to all of the storm nodes > > ** change the A1 file name from ${logfile.name} to ${logfile.name}_${ > storm.id} > > * alter the python file, storm. pass the custom cluster_{program_name} to > the storm application. search the file for "cluster.xml" > > > > The downside is that logback's log rotation as a means of keeping your > disk from overfilling is no longer practical. This becomes more and more > true whenever you are loading and killing topologies on a regular basis. > Basically, your disk will become full of logs. You will need to implement > some other method of log cleanup. > > > > > Thank you for your time! > > +++++++++++++++++++++ > Jeff Maass <[email protected]> > linkedin.com/in/jeffmaass > stackoverflow.com/users/373418/maassql > +++++++++++++++++++++ > > > > On Wed, May 13, 2015 at 1:06 PM, <[email protected]> wrote: > > *Dell - Internal Use - Confidential * > > what is every one using to process the storm logs from all over the > cluster for troubleshooting and debugging purposes. > > > > The accessing storm logs via the storm UI or tailing worker logs is > getting to be pretty painful. > > > > I guess this could be said of a lot of other Hadoop components like Kafka, > ZooKeeper, HBase and such. > > > > P.S: Did anyone get logback working properly to separate out topology > level logs from worker-xxxx.log if so what are the gotchas you ran into. > > >
