Hi Shashank, previously I used Fluentd with some plugins to tail log lines, parse, transform, and store to ES. I think you can do it similar with Logstash.
There're some frameworks like Apache NiFi to help ETL and I guess it can achieve what you'd like to do. Thanks, Jungtaek Lim (HeartSaVioR) 2017년 3월 31일 (금) 오후 4:46, Shashank Prasad <[email protected]>님이 작성: > Hi folks, > > Storm is a great tool but the logs are all over the place. As you increase > your workers, your log files will increase as well and there is no single > file it logs to. > > This makes it very hard to troubleshoot since you have to tail multiple > logs. > > Ideally, i would like to ship all the logs for a topology to a centralized > log server where i could use something like Kibana and filter the logs on > what i am searching for. > > Anyone has any suggestion on how to achieve this or has a use case of how > you currently doing it. > > Thanks a lot for your time! > > -shashank >
