After a visual review that is about what I was thinking. Like you mentioned it could use some work - I added some comments to the JIRA hitting on some suggested improvements.
Jon On Thu, Jan 5, 2017 at 12:03 PM Tyler Moore <[email protected]> wrote: > I attached a sample function to the Jira, just calls a script from on the > server by passing in the path. Certainly could use some work, but I wanted > to make sure this was the right idea and get a consensus, let me know your > thoughts. > > My thought process for this is that if we only allow script execution from > a shell script that must be on the storm host, then the user is in a sense > authenticated by their own user-defined access restrictions for that > system. > > > Regards, > > Tyler Moore > Software Engineer > Phone: 248-909-2769 <(248)%20909-2769> > Email: [email protected] > > > On Tue, Jan 3, 2017 at 3:47 PM, Carolyn Duby <[email protected]> > wrote: > > > Also please consider the security of the scripts and script injection > > attacks. For example, we should probably restrict file access. > > > > Thanks > > Carolyn > > > > > > > > On 1/3/17, 3:25 PM, "Otto Fowler" <[email protected]> wrote: > > > > >A script bolt would still allow them to write the script the way they > want > > >to, but would avoid having to write all the scaffolding. > > >The matter then would be how to integrate that script bolt into the > > >topologies. > > > > > > > > >On January 3, 2017 at 15:17:59, [email protected] ([email protected]) > > wrote: > > > > > >Right, that definitely is more efficient, but part of the point here is > to > > >lower the barrier of entry to using Metron. > > > > > >It makes Metron's triage abilities more flexible and allows a user to > > reuse > > >existing code quickly and easily. Having this available for PoC, > > >prototyping, and low volume environments or situations (only when threat > > >score is 100, for instance) is important, as it lowers the barrier to > > entry > > >of migrating a company to a Metron environment. > > > > > >I see this as a tradeoff where I would prioritize ease of use over > > >efficiency. There's nothing wrong with making both options available, > at > > >some point, and making their different use cases clear. > > > > > >Jon > > > > > >On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <[email protected]> wrote: > > > > > >Well, yes :-) > > >And clearly it should always be more efficient to write a custom bolt in > > >Java than to invoke a script and manage it. > > > > > >--Matt > > > > > >From: Otto Fowler <[email protected]> > > >Date: Tuesday, January 3, 2017 at 7:08 AM > > >To: "[email protected]" <[email protected] > >, > > >Matt Foley <[email protected]> > > >Subject: Re: Custom Storm Topologies > > > > > >Wouldn’t that be a bolt? > > > > > > > > >On January 2, 2017 at 14:39:34, Matt Foley ([email protected]) wrote: > > >Should we consider a script calling capability that can launch a > streaming > > >script and keep it alive and fed, long-term, rather than launching the > > >script anew every time the Stellar function is invoked? I’m thinking two > > >basic rules: Write a line, read a line; and always have a timeout. Prob > > >need a UID of some sort for a cache of running process objects. > > > > > >--Matt > > > > > >On 1/2/17, 8:50 AM, "Carolyn Duby" <[email protected]> wrote: > > > > > > > > >Inserting a script inline is ok for low throughput and prototyping but > > once > > >you get higher throughput (millions of events per second), it’s probably > > >going to be a bottleneck. > > > > > > > > >For Metron-571 you might want to consider a java based extension plugin > > >similar to Eclipse plugins. > > > > > >Thanks > > >Carolyn > > > > > >On 12/31/16, 5:22 PM, "Tyler Moore" <[email protected]> wrote: > > > > > >>Thanks Jon, > > >> > > >>I'll look over the tutorial and put something together for the > SHELL_EXEC > > >>stellar function. > > >>I don't believe I have permissions to assign in Jira if you want to > > assign > > >>to me my username is devopsec. > > >>I'll post back details and we can review security issues > > >> > > >>Regards, > > >> > > >>Tyler Moore > > >>Software Engineer > > >>Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769> > > >>Email: [email protected] > > >> > > >> > > >>On Sat, Dec 31, 2016 at 9:46 AM, [email protected] <[email protected]> > > wrote: > > >> > > >>> Casey did a tutorial on how to add your own Stellar function here > > >>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an > > existing > > >>> function that does this (current functions are listed here > > >>> <https://github.com/apache/incubator-metron/tree/master/ > > >>> metron-platform/metron-common#stellar-core-functions>). > > >>> I noticed that some of the Stellar function documentation was a bit > > dated > > >>> so I've opened a PR to update it here > > >>> <https://github.com/apache/incubator-metron/pull/407>. > > >>> > > >>> As this is something I need as well, I'd be happy to assist you > where I > > >>> can. Perhaps you want to self-assign METRON-571 > > >>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some > > >>> security concerns with a SHELL_EXEC function because it could result > in > > >RCE > > >>> - if that's the route you go I could probably help with a thorough > > secure > > >>> code review. > > >>> > > >>> Jon > > >>> > > >>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <[email protected]> > > >wrote: > > >>> > > >>> Thank you everyone for your suggestions, > > >>> > > >>> I believe that kicking off the function via stellar would be the > > optimal > > >>> solution. If anyone has an example of calling external code via > stellar > > >>> that would be very helpful. Thanks! > > >>> > > >>> Regards, > > >>> > > >>> Tyler Moore > > >>> IT Specialist > > >>> [email protected] > > >>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769> > <(248)%20909-2769> > > >>> > > >>> > On Dec 30, 2016, at 17:54, Otto Fowler <[email protected]> > > wrote: > > >>> > > > >>> > They are all extension points. > > >>> > > > >>> >> On December 30, 2016 at 16:34:58, [email protected] ( > > [email protected]) > > >>> wrote: > > >>> >> > > >>> >> Right but unless I'm missing something, both of those options are > > more > > >>> >> rigid and the MaaS service would have an unnecessary delay as > > opposed > > >to > > >>> >> doing it entirely in Stellar. Unless there's a reason to do > > otherwise > > >>> that > > >>> >> I'm missing, I would think doing this in Stellar gives you a more > > >timely > > >>> >> and (re)configurable end result. > > >>> >> > > >>> >> Jon > > >>> >> > > >>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <[email protected] > > > > >>> wrote: > > >>> >>> > > >>> >>> I think there are a couple of things you can do here. There way > to > > >get > > >>> >>> something else into the split is to have another adapter to split > > to, > > >>> which > > >>> >>> is what I think you mean. You can also integrate with MaaS and > > create > > >>> a > > >>> >>> service that you can call via STELLAR. > > >>> >>> > > >>> >>> > > >>> >>> > > >>> >>> On December 30, 2016 at 15:08:48, Otto Fowler ( > > >[email protected] > > >>> ) > > >>> >>> wrote: > > >>> >>> > > >>> >>> Or a Maas service? > > >>> >>> > > >>> >>> > > >>> >>> On December 30, 2016 at 13:52:06, [email protected] ( > > [email protected]) > > >>> >>> wrote: > > >>> >>> > > >>> >>> Depending on the details it sounds like a much simpler solution > > would > > >>> be > > >>> >>> to > > >>> >>> handle this in a Stellar function. > > >>> >>> > > >>> >>> Jon > > >>> >>> > > >>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <[email protected]> > > >wrote: > > >>> >>>> > > >>> >>>> Happy Holidays Metron Devs! > > >>> >>>> > > >>> >>>> Could anyone lend me some guidance on customizing the storm > > >topologies > > >>> >>> in > > >>> >>>> metron? What I am am trying to accomplish: > > >>> >>>> > > >>> >>>> 1) Add a method to the threat intel joiner bolt that sends an > http > > >>> post > > >>> >>>> with the score of the threat to a remote rest api. This will > > >>> >>> conditionally > > >>> >>>> trigger notifications based on user settings in another database > > >(the > > >>> >>>> backend processing logic is on another platform). > > >>> >>>> The score should be available within the JSONObject but I am not > > an > > >>> >>> expert > > >>> >>>> with storm and I am not completely understanding what conditions > > >>> >>> constitute > > >>> >>>> when the threat feed is considered an "alert" in metron. Please > > >>> clarify. > > >>> >>>> > > >>> >>>> 2) How would I add an external dependency, my http rest java > > class, > > >to > > >>> >>> the > > >>> >>>> metron maven build process? More specifically, if I was adding a > > >>> custom > > >>> >>>> class that needed accessed by a bolt in storm, how would I add > > this > > >in > > >>> >>>> maven as a dependency. I have limited experience with maven but, > > my > > >>> >>>> understanding is that I would add it to the pom.xml and > > recompile. > > >>> >>>> Although, the metron quick dev platform is built on a vm, would > I > > >need > > >>> >>> to > > >>> >>>> account for this? Please advise. > > >>> >>>> > > >>> >>>> Regards, > > >>> >>>> > > >>> >>>> Tyler Moore > > >>> >>>> > > >>> >>>> > > >>> >>>> Software Engineer > > >>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769> > <(248)%20909-2769> > > >>> >>>> Email: [email protected] > > >>> >>>> > > >>> >>> -- > > >>> >>> > > >>> >>> Jon > > >>> >>> > > >>> >>> Sent from my mobile device > > >>> >>> > > >>> >>> -- > > >>> >> > > >>> >> Jon > > >>> >> > > >>> >> Sent from my mobile device > > >>> >> > > >>> > > >>> -- > > >>> > > >>> Jon > > >>> > > >>> Sent from my mobile device > > >>> > > > > > > > > > > > > > > > > > >-- > > > > > >Jon > > > > > >Sent from my mobile device > > > -- Jon Sent from my mobile device
