GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/911
METRON-1419: Create a SolrDao
## Contributor Comments
This PR is an initial attempt at creating a SolrDao that implements the
IndexDao interface, is functionally equivalent to ElasticsearchDao and passes
all tests in SearchIntegrationTest and UpdateIntegrationTest.
A high level summary of the changes include:
- Upgraded the Solr client version to 6.6.0
- Updated the SolrComponent to work with Solr 6.6 and added a couple
convenience methods similar to ElasticsearchComponent
- Added a new SolrDao implementation with all IndexDao methods implemented
- Refactored the SearchIntegrationTest to work for both Solr and
Elasticsearch and added an Solr implementation (more detail below)
- Created an abstract UpdateIntegrationTest and added a Solr implementation
- Added Solr schemas for test data sets
- Added new tests to SearchIntegrationTest including filtering on fields
with different types, faceting on fields with different types, and faceting on
fields with missing types.
- Broke the IndexDao down in the SolrDao to smaller, easier to understand
classes. The ElasticsearchDao class has become very large so I attempted to
make the SolrDao more readable.
There were a couple areas where Elasticsearch and Solr behave slightly
different. I attempted to accommodated for that through the SolrDao
implementation, by adjusting existing tests, and by splitting out specific
tests:
- Column metadata is different between the 2 search engines so each
implementation has their own tests
- There are cases where the clients will return different types in search
results. I am handling this in SearchIntegrationTest by first converting the
types to strings and then comparing (other ideas?). For example, the ES client
returns an Integer for timestamp while Solr returns a Long.
- There are cases where ES throws an error under certain conditions while
Solr does not (and vice-versa). These were moved to either ES or Solr
SearchIntegrationTest implementations.
- There is no support in Solr for sorting group results so I am sorting
them client-side instead.
At this point the scope is limited to tests passing. There are other PRs
in progress that are needed before automated testing with full dev can be done.
I am still actively working on manually testing in full dev and adding
documentation but this should get us started.
This PR is intended to be merged into the upgrade-solr feature branch but I
have it set to master temporarily so review is easier. We will need to merge
in master to the feature branch to get rid of the extra commits since this PR
is up to date with master.
I'm expecting a lengthy review and would request multiple reviewers and +1s.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron solr-dao
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/911.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #911
----
commit c86421739f230cfc23b2a5092af53a32348dbaef
Author: merrimanr <merrimanr@...>
Date: 2018-01-24T16:35:33Z
initial commit
commit 76455986a14d12ed720a83c091616a1904ffb2b1
Author: merrimanr <merrimanr@...>
Date: 2018-01-24T19:07:46Z
initial commit
commit f996ee1a3069116cf1063e70c7d4a2b562da9b97
Author: merrimanr <merrimanr@...>
Date: 2018-01-24T19:16:15Z
Merge branch 'METRON-1429' into solr-dao
# Conflicts:
#
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
#
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
commit a29ed14c39a79ce5918212baa9b075706319e4de
Author: merrimanr <merrimanr@...>
Date: 2018-01-24T20:31:11Z
added back metaalert test data
commit 92cc5640e7cacf4cc3b4941c3bc7d559aa7b8380
Author: merrimanr <merrimanr@...>
Date: 2018-01-24T20:31:49Z
Merge branch 'METRON-1429' into solr-dao
commit 3a17b4d9d18efe4431197521124b13cb0dc4f9e3
Author: merrimanr <merrimanr@...>
Date: 2018-01-24T22:32:52Z
resolved merge conflicts
commit 5c6293dd5302cddf3f212a982fed8c2d0ec2e391
Author: merrimanr <merrimanr@...>
Date: 2018-01-24T22:33:53Z
Merge remote-tracking branch 'mirror/master' into METRON-1429
commit bb8efc16f60147ed8198604923c391c360376859
Author: merrimanr <merrimanr@...>
Date: 2018-01-24T22:49:36Z
Merge branch 'METRON-1429' into solr-dao
commit 016403323cda9959c8edcdbe0e6a6e8d7cda9340
Author: merrimanr <merrimanr@...>
Date: 2018-01-25T16:37:10Z
Merge remote-tracking branch 'mirror/master' into solr-dao
# Conflicts:
#
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
#
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
commit d4c99e1521f7c638ab81d388b0cb8c55cc548561
Author: merrimanr <merrimanr@...>
Date: 2018-01-26T22:21:38Z
dao refactor and group implementation
commit 8fc77dff00a695f09ea6777aada9e23f7e89f6ee
Author: merrimanr <merrimanr@...>
Date: 2018-01-26T22:25:15Z
Merge remote-tracking branch 'mirror/master' into solr-dao
# Conflicts:
#
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
#
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
commit 32d8867b9e5c18c4982213ff08250b2eea7bf8a4
Author: merrimanr <merrimanr@...>
Date: 2018-01-26T22:30:22Z
resolved merge conflicts
----
---
