Yeah, what otto said :) I'd just add one thing, stellar really requires nothing more than:
1. existing inside of a JVM environment. We use it inside of storm and mapreduce, but it could be used inside of spark or whatever 2. Have a VariableResolver implementation which could map your data to variable -> value pairs. The default one that we use in metron is wrapping Map<String, Object>'s, so the implementation is pretty trivial, but we also have a VariableResolver in the pcap work that will pull fields from the header of raw packets and expose them to stellar. All this to say that normalizing your data to work with stellar is as simple as creating a VariableResolver which can take your raw data format and allow stellar to query it for variables. We worked fairly hard to move stellar core into a position where it is decoupled from Metron, so it shouldn't be too difficult to repurpose it. On Thu, Jan 18, 2018 at 7:58 AM, Otto Fowler <[email protected]> wrote: > Please comment on the jira. We can come up with what would be a good > example program, obviously massively commented to show this. > Down the line, we could even have archetypes for different application > types… but that is just me thinking down the line ;) > > > On January 18, 2018 at 07:57:17, Otto Fowler ([email protected]) > wrote: > > I would also say that you should look at METRON–876 > <https://issues.apache.org/jira/browse/METRON-876>. > > This is the umbrella jira for the effort to separate stellar into a more > independent module. > > > > > On January 18, 2018 at 07:54:38, Otto Fowler ([email protected]) > wrote: > > I have created METRON–1409 > <https://issues.apache.org/jira/browse/METRON-1409> > > There are several ways to look at hosting stellar to get examples: > > - The unit tests > - The shell > - The storm bolts and transformer classes > > From a high level, to host stellar you need to: > > - Include stellar-common in you pom > - Create a Context > - Initialize the function resolver > - Create the StellarProcessor > - Create a variable resolver > > Then you set everything up, set the vars for the call in the variable > resolver, and have the processor execute a statement. > > The issue right now, and the reason we need METRON–1409 is that each of > the things above are *so* integrated into the flow of the host, that it > is not obvious what is going on. > > The tests are pretty straight forward, but don’t show the context init > very well. > > I would suggest that you start with the unit tests, as they are the most > concise. Look through them, debug through them etc. > > Then move onto the shell. > > I would look at the bolts/transformers last ( although they are the most > analogous to what I think you want to do ). > > > > > > > On January 17, 2018 at 17:34:45, Ian Abreu ([email protected]) wrote: > > Hey all, > > > > We’ve come across the design decision where we’d like to use Metron > tooling as a framework to build our SIEM around. This being the case, > stellar is something that we’d like to use, but we’ve currently got > different enrichment and normalization layers. > > > > So my question is this: Has anyone, or could anyone point me to a resource > that’d help to normalize our data in such a way that Stellar could be used > downstream from our data manipulation/normalization layer? > > > > Cheers, > > Z0r0 > >
