This discussion is an attempt to clarify some questions and discuss design decisions related to METRON-1022.
The primary purpose of METRON-1022 is to provide a foundation for building Metron-specific Elasticsearch (or other search engine implementations) functions in our REST application. This translates into 3 features provided by METRON-1022: a common approach to setting up a TransportClient, a search abstraction layer and a simple Elasticsearch implementation consisting of a single function. I believe the setup part is fairly straightforward and doesn't require a detailed discussion. Please chime in if I'm wrong there. The first order of business is to all agree on an architectural approach. How and where should we query Elasticsearch? METRON-1022 duplicates some functionality in METRON-990 but is architecturally different. Instead of the client-side code interacting directly with Elasticsearch through it's REST api, this PR interacts with Elasticsearch through the Java api in a Metron REST service. I believe there are a couple of advantages to doing it this way: - A Metron-specific search service in REST can be reused by other UIs and clients. It would be possible to make an angular service reusable but that would take some work and it would only be reusable in javascript as an imported library and not as flexible as a service available over http. - Metron provides an integration testing framework for Java-based classes. METRON-1022 leverages this without much additional effort. It would take a lot more work to enable javascript modules to use this. - In my experience, the Metron community is much more comfortable with developing and reviewing features written in Java as opposed to javascript. I think that is important for foundational pieces like this. Some arguments to consider for keeping Elasticsearch functions in a javascript service: - More efficient since there is no proxy in the middle (Metron REST being the proxy) - Eliminates the task of resolving version conflicts that comes with adding the Elasticsearch dependency to a Maven module although there are ways to make this easier The second topic to discuss is the search abstraction. This has been requested several times and I think there is consensus that we need it. METRON-1022 attempts to do this by: - creating model classes that represent search requests/responses - creating a search interface that accepts these model classes as input and return parameters - creating a controller that exposes this interface over REST - using Spring's IOC framework to select the correct implementation An implementation of a search function was included in METRON-1022 as an example. ElasticsearchServiceImpl implements SearchService and is selected as the implementation by default. This could have been a separate PR but I felt having it in context would help reviewers understand the design pattern. How does this relate to METRON-990? Currently they overlap with METRON-1022 offering a subset of the functionality in the METRON-990 Elasticsearch service. The idea is to first ensure METRON-990 and METRON-1022 both conform to the same search abstraction (which has been discussed in METRON-990 feedback). The next step would be to replace the search service in METRON-990 to one that queries the Metron REST service instead. Ideally this only involves changing one class since the abstraction is used in all the other components of METRON-990 and is trivial since the complexity is now in Metron REST and not javascript. Next other services (getting index metadata for example) would be converted using the same process in incremental PRs. Then, moving forward, all Elasticsearch interactions would instead be developed as Metron REST endpoints using the foundation established in METRON-1022. This is a lot to digest so I'm happy to more detail as needed. Interested to hear others' thoughts and reactions. Ryan
