I totally agree and I would be happy to help. For setup specifically, should we prioritize a certain OS? To date the focus has mostly been macOS.
In order to help my personal development process, I scripted the install process <https://github.com/JonZeolla/lab-SecurityDataAnalysis/blob/master/setup/setup.sh> for a fresh CentOS 6.8 box. You should be able to run that script with some basic arguments and get Metron up and running. It also has some nice features for testing new PRs or branches, but you can safely ignore those if you just want to get things up and running. In response to this email I also wrote a simple Mac installation script <https://gist.github.com/JonZeolla/4abcbfee96437b948ab46b6782f00caa#file-metron-mac-setup-sh> that should get things up and running. On my local machine it looked like it was working - if you give it a shot, please let me know if it worked for you. A huge focus of the project has been to make things more simple to run (and believe me, it is many many times easier than it used to be in the OpenSOC days), in fact, there are numerous efforts being undertaken right now to continue to make this easier. However you are definitely right, it is still too complicated, and our documentation is severely lacking. What do you think about the following regarding tutorials/videos: - An Introduction to Hadoop (focused on the aspects that Metron uses) - A step by step explanation of data flow in Metron (Parsers, Enrichment, Indexing). - Data Ingestion (getting data into the cluster) - Data Retrieval and Visualization - Cluster Troubleshooting Is there anything missing? Thanks, Jon On Sun, Mar 5, 2017, 4:39 AM Khurram Ahmed <khurramah...@gmail.com> wrote: I agree. The barrier to entry for new contributors is prohibitive due to lack of accurate documentation for setting up dev environment. On Mar 5, 2017 2:23 PM, "moshe jarusalem" <tuu...@gmail.com> wrote: Hi All, I have been trying to get involved with the metron project for a while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to set up a running environment and understand the picture how it works. There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I assume most of you guys are expert at these technologies so you solve your problems. I have also browsed the user's list and there are many people struggling to set up a proper environment. Would you guys make a document describing step by step setting up a development environment with a specific release (0.3.1 as its the latest) ? Can you describe each technology shortly and give references and most importantly metron's usage of them? Indeed a video recording or a webinar would be much more useful so that there can be some discussions. Regards, Moshe -- Jon Sent from my mobile device
