[Sorry about the stupid COMMERCIAL: tag being added - I'm trying to fix with our email folks]
Nice. I was just looking around at the geotagging enrichment adapter. The city data is split between the two files of 70Mb and 40Mb sizes. It seems like the data is small enough to just load it all into memory. This would eliminate two SQL queries for every event. Bryan On 12/9/15 2:14 PM, "Mark Bittmann" <[email protected]> wrote: >Hi Bryan, > >For automation, B23 is planning to contribute Ansible scripts to deploy >the Metron stack. The playbooks use Ambari blueprints for the hadoop >ecosystem. We also install Elasticsearch, configure the legacy OpenSOC UI >(based on Kibana/nodejs), create directories in hdfs, populate a MySQL >database for geotagging. We template the OpenSOC_Config files so that we >can use variable injection for the different services: Zookeeper, Hbase, >Elasticsearch, MySQL, etc. Everyone's deployment might be slightly >different, but I think this will be a really good start. > >I'm in the process of decoupling the scripts from our internal tooling - >I should be able to make available the Ambari stuff later this week. Once >we merge the disparate forks of the Cisco codebase, there will be some >work to bring the playbooks up to date (i.e., use of Storm Flux), but not >a ton. > >Mark > > > >On 12/9/15, 2:48 PM, "Bryan Taylor" <[email protected]> wrote: > >> >>Seems like a good list. I'm probably not your UI guy, but I'll be happy >>to >>help with any of the others. Test coverage is probably a good place for >>me >>to start learning my way around. Do we have CI set up? I see a travis >>file >>in the code. >> >>Regarding automation deployments, are we targeting Ambari or something >>else? >> >>On the hadoop component compatibility, I see from the opens-streaming pom >>that we are using Storm-0.9.2, Kafka 0.8.0, Hadoop 2.2.0, and HBase >>0.98.0-hadoop2. These are all several iterations old. How aggressive do >>we >>want to be, generally, with tracking new releases? Are Hive, Flume, and >>Spark also going to up rev for us? >> >>Bryan >> >>On 12/9/15 12:22 PM, "James Sirota" <[email protected]> wrote: >> >>>Hi Brian, >>> >>>Welcome. Glad to have you contribute. There will be changes to the >>>code >>>base that the community will contribute back shortly. We are waiting >>>for >>>the Jira to be setup so the backlog can be created and voted on. I >>>think >>>the overall feeling is that we need to make the code base compatible >>>with >>>the latest version of HDP, automate deployments, increase test coverage, >>>and start working on a new UI. There may be more significant >>>architectural changes to the code base, but we need to get the essential >>>items knocked out before we go there. >>> >>>Thanks, >>>James >>> >>> >>> >>> >>>On 12/9/15, 11:52 AM, "Bryan Taylor" <[email protected]> wrote: >>> >>>>Hi Folks, >>>> >>>>I just joined the list and thought I'd say "hi". I work at Rackspace >>>>and >>>>will be joining Andrew Hartnett's team there and hacking on metron. >>>>This >>>>is my first ASF project and I'm looking forward to being part of this >>>>community. >>>> >>>>I'm curious what the development vision is for metron. What do people >>>>like and not like about the codebase? I gather this code transitioned >>>>from a Cisco internal project and is incubating now at the ASF. Are >>>>there any code changes that need to be made to support incubation? >>>> >>>>Bryan >>
