Just FYI, I put a PR up for this feature: https://github.com/apache/metron/pull/1250. I believe I've addressed everyone's questions either in the PR description or the code itself. I'm think we can continue the discussion there.
On Fri, Oct 19, 2018 at 12:02 PM Otto Fowler <ottobackwa...@gmail.com> wrote: > I believe the issue of introducing and supporting higher latency > enrichments is a systemic one, and should be solved as such, > with the rest and other higher latency enrichments build on top of that > framework. > > > > > On October 19, 2018 at 12:22:28, Ryan Merriman (merrim...@gmail.com) > wrote: > > Thanks Casey, good questions. > > As far as the verbs go, just thinking we might want to support calls other > than GET at some point. For the use case stated (enriching messages from > 3rd party services) GET is all we need. Probably a moot point anyways > since every http library will support the different HTTP verbs. > > Agreed on the caching. I will defer to those that are more familiar with > the Stellar internals on what the correct approach is. > > I was thinking the same thing with regards to the client libraries. Apache > HttpComponents is probably the safest choice but OkHttp looks nice and > could reduce effort and complexity as long as it meets our requirements. > > On Fri, Oct 19, 2018 at 10:58 AM Casey Stella <ceste...@gmail.com> wrote: > > > I think it makes a lot of sense. A couple of questions: > > > > - What actions do you see the REST verbs corresponding to? I would > > understand GET (which is in effect "evaluate an expression", right?), > > but > > I'm not sure about the others. > > - We should probably be careful about caching stellar expressions. Not > > all stellar expressions are deterministic (e.g. PROFILE_GET may not be > > as > > the lookback window is bound to current time). Ultimately, I think we > > should probably bake whether a function is deterministic into stellar so > > that *stellar* can cache where appropriate (e.g. if every part of an > > expression is deterministic, then pull from cache otherwise recompute). > > All of this to say, if you're going to make it configurable, IMO we > > should > > make it a configuration that the user passes in at request time so they > > have the control over whether the expression is safe to cache or > > otherwise. > > > > Without more compelling reasons to not do so, I'd suggest we use HTTP > > Components as it's another apache project and under active > > development/support. I'd also be ok with OkHttp if it's actively > > maintained. > > > > On Fri, Oct 19, 2018 at 11:46 AM Ryan Merriman <merrim...@gmail.com> > > wrote: > > > > > I want to open up discussion around adding a Stellar REST client > > function. > > > There are services available to enrich security telemetry and they are > > > commonly exposed through a REST interface. The primary purpose of this > > > discuss thread to collect requirements from the community and agree on > a > > > general architectural approach. > > > > > > At a minimum I see a Stellar REST client supporting: > > > > > > - Common HTTP verbs including GET, POST, DELETE, etc > > > - Option to provide headers and request parameters as needed > > > - Support for basic authentication > > > - Proper request and error handling (we can discuss further how this > > > should work) > > > - SSL support > > > - Option to use a proxy server (including authentication) > > > - JSON format > > > > > > In addition to these functional requirements, I would also propose we > > > include these performance requirements: > > > > > > - Provide a configurable caching layer > > > - Provide a mechanism for pooling connections > > > - Provide clear documentation and guidance on how to properly use this > > > feature since there is a significant risk of introducing latency > > issues > > > > > > What else would you like to see included? > > > > > > I think the primary architectural decision we need to make (based on > the > > > agreed upon requirements of course) is an appropriate Java HTTP/REST > > client > > > library. Ideally we choose a library that supports everything we need > > > OOTB. I think the majority of the work for this feature will involve > > > wrapping this library in a Stellar function and exposing the > > configuration > > > knobs through Metron's configuration interface (Ambari, Zookeeper, > > etc). I > > > have done some very light research and here is my initial list: > > > > > > - Apache HttpComponents - https://hc.apache.org/ > > > - Has support for all of the features listed above as far as I can > > tell > > > - Doesn't introduce a large number of new dependencies (am I wrong > > > here?) > > > - Is sort of included already (we will need to upgrade from > > > httpclient) > > > - Lower level > > > - Google HTTP Client Library for Java - > > > > > > > > > https://developers.google.com/api-client-library/java/google-http-java-client/ > > > - Higher level API with pluggable components > > > - Introduces dependencies (we've had issues with Guava in the past) > > > - Netflix Ribbon - https://github.com/Netflix/ribbon > > > - Has a lot of nice features that may be useful in the future > > > - Introduces dependencies (including guava) > > > - Hasn't been committed to in the last 5-6 months > > > - Unirest - https://github.com/Kong/unirest-java > > > - Lightweight API built on top of HttpComponents > > > - Pluggable serialization library (jackson is an issue for us so > > this > > > is nice) > > > - Also has not received a commit in a while > > > - OkHttp - http://square.github.io/okhttp/ > > > - Good documentation and looks easy to use > > > - Actively maintained > > > > > > Obviously we have a lot of choices. I think it comes down to balancing > > the > > > tradeoff between ease of use (HttpComponents will likely require the > most > > > work since it is lower level) and capability. Introducing additional > > > dependencies is something we should also be mindful of because our > > shading > > > practices. > > > > > > This should get us started. Let me know what you think! > > > > > > >
