Hi all, Chukwa has been a fun project to work. However, Apache is about building a community where code can thrive to have long lasting Software that everyone would use. This brings us to a cross road of contribute and thrive or retire dead project. We are at the cross road again, that we can either united and build a monitoring and log collection system that people would want to use, or we can retire Chukwa.
The recent rise of ELK stack, or collectd/Grafana stack are providing log collection and metrics in their own limited approach. There is so much more potential for Chukwa because Chukwa uses HDFS to keep data more resilient than the traditional implementations. Spark on HBase provides a great window for Chukwa to transport log and metrics to HBase, and Chukwa can provide the insights of the metrics and logs by cross referencing data from different data sources. For Chukwa to provide value to our users, it needs the following features: 1. Ease of Deployment. (Integrate with Ambari for deployment) 2. Better integration with metrics coming from Ambari Metrics. It might be possible to replace Ambari Metrics collector with a more robust and distributed version as Chukwa Agent adaptors. 3. Spark scripts on top of Spark on HBase to compute aggregate and analyze data. YARN provides a really difficult user model to find user application logs. Chukwa provided search capability give user the power to search for application logs, and drill down to the actual container log. However, not many people know about these features. It will require Chukwa developers to show other the way. This can not be done by one person. It will take a community to implement missing features from above points to bring Chukwa to be mainstream. I am not sure if anyone is still interested in building out those features, or we should retire Chukwa for good. I like to offer people to committer access if they can implement one of the features above. I like to hear from the community to understand where our trajectory is going to land. This will help Apache to decide the direction of the project. Thank you all for your input. regards, Eric
