[jira] [Updated] (METRON-2166) FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2>
[
https://issues.apache.org/jira/browse/METRON-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic updated METRON-2166:
--
Fix Version/s: Next + 1
> FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2>
> --
>
> Key: METRON-2166
> URL: https://issues.apache.org/jira/browse/METRON-2166
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Michael Miklavcic
>Priority: Major
> Fix For: Next + 1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This seems to be an intermittent test failure. For example, see
> [https://api.travis-ci.org/v3/job/551324929/log.txt]
> {code:java}
> ---
> T E S T S
> ---
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.298 sec -
> in org.apache.metron.enrichment.converter.EnrichmentConverterTest
> Running org.apache.metron.enrichment.stellar.ObjectGetTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.95 sec - in
> org.apache.metron.pcap.PcapHelperTest
> Running org.apache.metron.pcap.mr.PcapJobTest
> Running org.apache.metron.profiler.client.HBaseProfilerClientTest
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.33 sec - in
> org.apache.metron.pcap.mr.PcapJobTest
> Running org.apache.metron.pcap.mr.OutputDirFormatterTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec -
> in org.apache.metron.pcap.mr.OutputDirFormatterTest
> Running org.apache.metron.pcap.mr.FileFilterUtilTest
> Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.114 sec <<<
> FAILURE! - in org.apache.metron.pcap.mr.FileFilterUtilTest
> test_getPaths_leftEdge(org.apache.metron.pcap.mr.FileFilterUtilTest) Time
> elapsed: 0.026 sec <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.metron.pcap.mr.FileFilterUtilTest.test_getPaths_leftEdge(FileFilterUtilTest.java:116)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Running org.apache.metron.pcap.PcapPackerComparatorTest
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 sec -
> in org.apache.metron.pcap.PcapPackerComparatorTest
> Running org.apache.metron.pcap.pattern.ByteArrayMatchingUtilTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time
[GitHub] [metron] mmiklavc commented on issue #1448: METRON-2164 Remove the Split-Join Enrichment Topology
mmiklavc commented on issue #1448: METRON-2164 Remove the Split-Join Enrichment Topology URL: https://github.com/apache/metron/pull/1448#issuecomment-506897524 @nickwallen merge with master and your build should pass now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] asfgit closed pull request #1452: METRON-2166 FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2>
asfgit closed pull request #1452: METRON-2166 FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2> URL: https://github.com/apache/metron/pull/1452 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] merrimanr commented on issue #1452: METRON-2166 FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2>
merrimanr commented on issue #1452: METRON-2166 FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2> URL: https://github.com/apache/metron/pull/1452#issuecomment-506857929 Looks good to me assuming tests pass. +1 by inspection. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc opened a new pull request #1452: METRON-2166 FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2>
mmiklavc opened a new pull request #1452: METRON-2166
FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2>
URL: https://github.com/apache/metron/pull/1452
## Contributor Comments
https://issues.apache.org/jira/browse/METRON-2166
This is a test fix for a test condition that was using
System.currentTimeMillis(). I also refactored a few of the tests for some
additional clarity on what values are being compared and why.
**Note**: this fix is required to fix a broken master build, so we should
merge this in asap instead of waiting the typical 24 hour+ cycle.
## Pull Request Checklist
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be
created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- n/a Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- n/a Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- n/a If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- n/a Have you verified the basic functionality of the build by building and
running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- n/a Have you ensured that format looks appropriate for the output in which
it is rendered by building and verifying the site-book? If not then run the
following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
- n/a Have you ensured that any documentation diagrams have been updated,
along with their source files, using [draw.io](https://www.draw.io/)? See
[Metron Development
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
for instructions.
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for
your personal repository such that your branches are built there before
submitting a pull request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (METRON-2166) FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2>
[
https://issues.apache.org/jira/browse/METRON-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875177#comment-16875177
]
Michael Miklavcic commented on METRON-2166:
---
Well, this is embarrassing. This was 100% me right here -
[https://github.com/apache/metron/blob/master/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/mr/FileFilterUtilTest.java#L115].
Must have had some really good IPAs while writing that :)
> FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2>
> --
>
> Key: METRON-2166
> URL: https://issues.apache.org/jira/browse/METRON-2166
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Michael Miklavcic
>Priority: Major
>
> This seems to be an intermittent test failure. For example, see
> [https://api.travis-ci.org/v3/job/551324929/log.txt]
> {code:java}
> ---
> T E S T S
> ---
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.298 sec -
> in org.apache.metron.enrichment.converter.EnrichmentConverterTest
> Running org.apache.metron.enrichment.stellar.ObjectGetTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.95 sec - in
> org.apache.metron.pcap.PcapHelperTest
> Running org.apache.metron.pcap.mr.PcapJobTest
> Running org.apache.metron.profiler.client.HBaseProfilerClientTest
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.33 sec - in
> org.apache.metron.pcap.mr.PcapJobTest
> Running org.apache.metron.pcap.mr.OutputDirFormatterTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec -
> in org.apache.metron.pcap.mr.OutputDirFormatterTest
> Running org.apache.metron.pcap.mr.FileFilterUtilTest
> Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.114 sec <<<
> FAILURE! - in org.apache.metron.pcap.mr.FileFilterUtilTest
> test_getPaths_leftEdge(org.apache.metron.pcap.mr.FileFilterUtilTest) Time
> elapsed: 0.026 sec <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.metron.pcap.mr.FileFilterUtilTest.test_getPaths_leftEdge(FileFilterUtilTest.java:116)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Running org.apache.metron.pcap.PcapPackerComparatorTest
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112
[GitHub] [metron] nickwallen commented on issue #1451: METRON-2168 Elasticsearch Updates Not Tested in Integration Test
nickwallen commented on issue #1451: METRON-2168 Elasticsearch Updates Not Tested in Integration Test URL: https://github.com/apache/metron/pull/1451#issuecomment-506827020 I added more unit tests for MultiIndexDao. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1374: METRON-2061: Solr documents with date fields cannot be updated with Dao classes
mmiklavc commented on issue #1374: METRON-2061: Solr documents with date fields cannot be updated with Dao classes URL: https://github.com/apache/metron/pull/1374#issuecomment-506818508 Looks good @merrimanr. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] merrimanr commented on issue #1374: METRON-2061: Solr documents with date fields cannot be updated with Dao classes
merrimanr commented on issue #1374: METRON-2061: Solr documents with date fields cannot be updated with Dao classes URL: https://github.com/apache/metron/pull/1374#issuecomment-506816761 Latest commit adds a test for nested paths. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1374: METRON-2061: Solr documents with date fields cannot be updated with Dao classes
mmiklavc commented on issue #1374: METRON-2061: Solr documents with date fields cannot be updated with Dao classes URL: https://github.com/apache/metron/pull/1374#issuecomment-506814266 @merrimanr your updates look pretty good. The only minor test addition I think we need here is for paths nested more than depth 1. e.g. can you add tests to validate something like "/foo/bar/baz/blah" still works? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen edited a comment on issue #1451: METRON-2168 Elasticsearch Updates Not Tested in Integration Test
nickwallen edited a comment on issue #1451: METRON-2168 Elasticsearch Updates Not Tested in Integration Test URL: https://github.com/apache/metron/pull/1451#issuecomment-506812474 One point that reviewers should consider is do we have enough test coverage for the `MultiIndexDao` now? I don't think the `MultiIndexDao` warrants an integration test. It doesn't integrate with any external systems. I think it can be tested sufficiently with unit testing. But is what we have in `MultiIndexDaoTest` enough? **update**: My answer would lean on the "no" side, but I don't think the integration tests really covered any of the existing gaps. I might spend a bit of time expanding on `MultiIndexDaoTest` as part of this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on issue #1451: METRON-2168 Elasticsearch Updates Not Tested in Integration Test
nickwallen commented on issue #1451: METRON-2168 Elasticsearch Updates Not Tested in Integration Test URL: https://github.com/apache/metron/pull/1451#issuecomment-506812474 One point that reviewers should consider is do we have enough test coverage for the `MultiIndexDao` now? I don't think the `MultiIndexDao` warrants an integration test. It doesn't integrate with any external systems. I think it can be tested sufficiently with unit testing. But is what we have in `MultiIndexDaoTest` enough? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (METRON-2168) Elasticsearch Updates Not Tested in Integration Test
[ https://issues.apache.org/jira/browse/METRON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875070#comment-16875070 ] Michael Miklavcic commented on METRON-2168: --- Good catch and explanation, [~nickwallen]. Thanks! > Elasticsearch Updates Not Tested in Integration Test > > > Key: METRON-2168 > URL: https://issues.apache.org/jira/browse/METRON-2168 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Attachments: test-failure.log > > Time Spent: 10m > Remaining Estimate: 0h > > The `ElasticsearchUpdateIntegrationTest` is not testing that the > `ElasticsearchDao` can update and retrieve values in a manner similar to what > would occur in production. > h3. What? > Within the Elasticsearch index, the test fails to define the 'guid' field to > be of type 'keyword', instead the type is defaulted to 'text'. In a > production setting this mistake would prevent any documents from being found > by guid. Unfortunately, the test passes despite this. The test needs to > match the behavior of what a user would experience in production. > h3. Why? > These problems arise because of the way the test is setup. Instead of > directly testing an `ElasticsearchDao` as you might expect this test runs > against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an > `HBaseDao`. On retrievals the `MultIndexDao` will return the document from > whichever index responds first. > With the current test setup, the underlying `ElasticsearchDao` will never > retrieve the document that the test case is expected. In all cases where the > test passes, the document is actually being returned from the `HBaseDao` > which is actually just interacting with a mock backend. The test needs to > actually test that we can update and retrieve documents from Elasticsearch. > h3. Proof? > If you alter the test to run against just an `ElasticsearchDao` the test > will fail as follows in the attached log file. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [metron] nickwallen opened a new pull request #1451: METRON-2168 Elasticsearch Updates Not Tested in Integration Test
nickwallen opened a new pull request #1451: METRON-2168 Elasticsearch Updates Not Tested in Integration Test URL: https://github.com/apache/metron/pull/1451 The `ElasticsearchUpdateIntegrationTest` is not testing that the `ElasticsearchDao` can update and retrieve values in a manner similar to what would occur in production. ### What? Within the Elasticsearch index, the test fails to define the 'guid' field to be of type 'keyword', instead the type is defaulted to 'text'. In a production setting this mistake would prevent any documents from being found by guid. Unfortunately, the test passes despite this. The test needs to match the behavior of what a user would experience in production. ### Why? These problems arise because of the way the test is setup. Instead of directly testing an `ElasticsearchDao` as you might expect this test runs against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an `HBaseDao`. On retrievals the `MultIndexDao` will return the document from whichever index responds first. With the current test setup, the underlying `ElasticsearchDao` will never retrieve the document that the test case is expected. In all cases where the test passes, the document is actually being returned from the `HBaseDao` which is actually just interacting with a mock backend. The test needs to actually test that we can update and retrieve documents from Elasticsearch. ### Proof? If you alter the test to run against just an `ElasticsearchDao` the test will fail as follows in the attached log file. ## Testing This only changes the integration tests. Run the integration tests to validate these changes. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1443: METRON-2150: [UI] User not able to filter by multiple values of the same field on Alerts UI
mmiklavc commented on issue #1443: METRON-2150: [UI] User not able to filter by multiple values of the same field on Alerts UI URL: https://github.com/apache/metron/pull/1443#issuecomment-506806943 @anandsubbu is 2167 an issue introduced by this PR, or a pre-existing bug? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (METRON-2166) FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2>
[
https://issues.apache.org/jira/browse/METRON-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic reassigned METRON-2166:
-
Assignee: Michael Miklavcic
> FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2>
> --
>
> Key: METRON-2166
> URL: https://issues.apache.org/jira/browse/METRON-2166
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Michael Miklavcic
>Priority: Major
>
> This seems to be an intermittent test failure. For example, see
> [https://api.travis-ci.org/v3/job/551324929/log.txt]
> {code:java}
> ---
> T E S T S
> ---
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.298 sec -
> in org.apache.metron.enrichment.converter.EnrichmentConverterTest
> Running org.apache.metron.enrichment.stellar.ObjectGetTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.95 sec - in
> org.apache.metron.pcap.PcapHelperTest
> Running org.apache.metron.pcap.mr.PcapJobTest
> Running org.apache.metron.profiler.client.HBaseProfilerClientTest
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.33 sec - in
> org.apache.metron.pcap.mr.PcapJobTest
> Running org.apache.metron.pcap.mr.OutputDirFormatterTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec -
> in org.apache.metron.pcap.mr.OutputDirFormatterTest
> Running org.apache.metron.pcap.mr.FileFilterUtilTest
> Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.114 sec <<<
> FAILURE! - in org.apache.metron.pcap.mr.FileFilterUtilTest
> test_getPaths_leftEdge(org.apache.metron.pcap.mr.FileFilterUtilTest) Time
> elapsed: 0.026 sec <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.metron.pcap.mr.FileFilterUtilTest.test_getPaths_leftEdge(FileFilterUtilTest.java:116)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Running org.apache.metron.pcap.PcapPackerComparatorTest
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 sec -
> in org.apache.metron.pcap.PcapPackerComparatorTest
> Running org.apache.metron.pcap.pattern.ByteArrayMatchingUtilTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.819 sec -
> in
[GitHub] [metron] mmiklavc commented on issue #1449: METRON-2084 Add documentation notice for MacOS Mojave users for new security permissions
mmiklavc commented on issue #1449: METRON-2084 Add documentation notice for MacOS Mojave users for new security permissions URL: https://github.com/apache/metron/pull/1449#issuecomment-506803300 Looks like the unrelated test failure is the same as the one reported here - https://issues.apache.org/jira/browse/METRON-2166. I'm looking into it now so we can get master back on track. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-2168) Elasticsearch Updates Not Tested in Integration Test
[ https://issues.apache.org/jira/browse/METRON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-2168: --- Description: The `ElasticsearchUpdateIntegrationTest` is not testing that the `ElasticsearchDao` can update and retrieve values in a manner similar to what would occur in production. h3. What? Within the Elasticsearch index, the test fails to define the 'guid' field to be of type 'keyword', instead the type is defaulted to 'text'. In a production setting this mistake would prevent any documents from being found by guid. Unfortunately, the test passes despite this. The test needs to match the behavior of what a user would experience in production. h3. Why? These problems arise because of the way the test is setup. Instead of directly testing an `ElasticsearchDao` as you might expect this test runs against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an `HBaseDao`. On retrievals the `MultIndexDao` will return the document from whichever index responds first. With the current test setup, the underlying `ElasticsearchDao` will never retrieve the document that the test case is expected. In all cases where the test passes, the document is actually being returned from the `HBaseDao` which is actually just interacting with a mock backend. The test needs to actually test that we can update and retrieve documents from Elasticsearch. h3. Proof? If you alter the test to run against just an `ElasticsearchDao` the test will fail as follows in the attached log file. was: The `ElasticsearchUpdateIntegrationTest` is not testing that the `ElasticsearchDao` can update and retrieve values in a manner similar to what would occur in production. h3. What? Within the Elasticsearch index, the test fails to define the 'guid' field to be of type 'keyword', instead the type is defaulted to 'text'. In a production setting this mistake would prevent any documents from being found by guid. Unfortunately, the test passes despite this. The test needs to match the behavior of what a user would experience in production. h3. Why? These problems arise because of the way the test is setup. Instead of directly testing an `ElasticsearchDao` as you might expect this test runs against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an `HBaseDao`. On retrievals the `MultIndexDao` will return the document from whichever index responds first. With the current test setup, the underlying `ElasticsearchDao` will never retrieve the document that the test case is expected. In all cases where the test passes, the document is actually being returned from the `HBaseDao` which is actually just interacting with a mock backend. The test needs to actually test that we can update and retrieve documents from Elasticsearch. Proof? If you alter the test to run against just an `ElasticsearchDao` the test will fail as follows in the attached log file. > Elasticsearch Updates Not Tested in Integration Test > > > Key: METRON-2168 > URL: https://issues.apache.org/jira/browse/METRON-2168 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Attachments: test-failure.log > > > The `ElasticsearchUpdateIntegrationTest` is not testing that the > `ElasticsearchDao` can update and retrieve values in a manner similar to what > would occur in production. > h3. What? > Within the Elasticsearch index, the test fails to define the 'guid' field to > be of type 'keyword', instead the type is defaulted to 'text'. In a > production setting this mistake would prevent any documents from being found > by guid. Unfortunately, the test passes despite this. The test needs to > match the behavior of what a user would experience in production. > h3. Why? > These problems arise because of the way the test is setup. Instead of > directly testing an `ElasticsearchDao` as you might expect this test runs > against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an > `HBaseDao`. On retrievals the `MultIndexDao` will return the document from > whichever index responds first. > With the current test setup, the underlying `ElasticsearchDao` will never > retrieve the document that the test case is expected. In all cases where the > test passes, the document is actually being returned from the `HBaseDao` > which is actually just interacting with a mock backend. The test needs to > actually test that we can update and retrieve documents from Elasticsearch. > h3. Proof? > If you alter the test to run against just an `ElasticsearchDao` the test > will fail as follows in the attached log file. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2168) Elasticsearch Updates Not Tested in Integration Test
[ https://issues.apache.org/jira/browse/METRON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-2168: --- Description: The `ElasticsearchUpdateIntegrationTest` is not testing that the `ElasticsearchDao` can update and retrieve values in a manner similar to what would occur in production. Within the Elasticsearch index, the test fails to define the 'guid' field to be of type 'keyword', instead the type is defaulted to 'text'. In a production setting this mistake would prevent any documents from being found by guid. Unfortunately, the test passes despite this. The test needs to match the behavior of what a user would experience in production. h3. Why? These problems arise because of the way the test is setup. Instead of directly testing an `ElasticsearchDao` as you might expect this test runs against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an `HBaseDao`. On retrievals the `MultIndexDao` will return the document from whichever index responds first. With the current test setup, the underlying `ElasticsearchDao` will never retrieve the document that the test case is expected. In all cases where the test passes, the document is actually being returned from the `HBaseDao` which is actually just interacting with a mock backend. The test needs to actually test that we can update and retrieve documents from Elasticsearch. If you alter the test to run against just an `ElasticsearchDao` the test will fail as follows in the attached log file. was: The `ElasticsearchUpdateIntegrationTest` is not testing that the `ElasticsearchDao` can update and retrieve values in a manner similar to what would occur in production. Within the Elasticsearch index, the test fails to define the 'guid' field to be of type 'keyword', instead the type is defaulted to 'text'. In a production setting this mistake would prevent any documents from being found by guid. Unfortunately, the test passes despite this. The test needs to match the behavior of what a user would experience in production. h3. Why? These problems arise because of the way the test is setup. Instead of directly testing an `ElasticsearchDao` as you might expect this test runs against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an `HBaseDao`. On retrievals the `MultIndexDao` will return the document from whichever index responds first. With the current test setup, the underlying `ElasticsearchDao` will never retrieve the document that the test case is expected. In all cases where the test passes, the document is actually being returned from the `HBaseDao` which is actually just interacting with a mock backend. The test needs to actually test that we can update and retrieve documents from Elasticsearch. If you alter the test to run against just an `ElasticsearchDao` the test will fail as follows in the attached log file. > Elasticsearch Updates Not Tested in Integration Test > > > Key: METRON-2168 > URL: https://issues.apache.org/jira/browse/METRON-2168 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Attachments: test-failure.log > > > The `ElasticsearchUpdateIntegrationTest` is not testing that the > `ElasticsearchDao` can update and retrieve values in a manner similar to what > would occur in production. > Within the Elasticsearch index, the test fails to define the 'guid' field to > be of type 'keyword', instead the type is defaulted to 'text'. In a > production setting this mistake would prevent any documents from being found > by guid. Unfortunately, the test passes despite this. The test needs to > match the behavior of what a user would experience in production. > h3. Why? > These problems arise because of the way the test is setup. Instead of > directly testing an `ElasticsearchDao` as you might expect this test runs > against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an > `HBaseDao`. On retrievals the `MultIndexDao` will return the document from > whichever index responds first. > With the current test setup, the underlying `ElasticsearchDao` will never > retrieve the document that the test case is expected. In all cases where the > test passes, the document is actually being returned from the `HBaseDao` > which is actually just interacting with a mock backend. The test needs to > actually test that we can update and retrieve documents from Elasticsearch. > If you alter the test to run against just an `ElasticsearchDao` the test > will fail as follows in the attached log file. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2168) Elasticsearch Updates Not Tested in Integration Test
[ https://issues.apache.org/jira/browse/METRON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-2168: --- Description: The `ElasticsearchUpdateIntegrationTest` is not testing that the `ElasticsearchDao` can update and retrieve values in a manner similar to what would occur in production. h3. What? Within the Elasticsearch index, the test fails to define the 'guid' field to be of type 'keyword', instead the type is defaulted to 'text'. In a production setting this mistake would prevent any documents from being found by guid. Unfortunately, the test passes despite this. The test needs to match the behavior of what a user would experience in production. h3. Why? These problems arise because of the way the test is setup. Instead of directly testing an `ElasticsearchDao` as you might expect this test runs against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an `HBaseDao`. On retrievals the `MultIndexDao` will return the document from whichever index responds first. With the current test setup, the underlying `ElasticsearchDao` will never retrieve the document that the test case is expected. In all cases where the test passes, the document is actually being returned from the `HBaseDao` which is actually just interacting with a mock backend. The test needs to actually test that we can update and retrieve documents from Elasticsearch. Proof? If you alter the test to run against just an `ElasticsearchDao` the test will fail as follows in the attached log file. was: The `ElasticsearchUpdateIntegrationTest` is not testing that the `ElasticsearchDao` can update and retrieve values in a manner similar to what would occur in production. Within the Elasticsearch index, the test fails to define the 'guid' field to be of type 'keyword', instead the type is defaulted to 'text'. In a production setting this mistake would prevent any documents from being found by guid. Unfortunately, the test passes despite this. The test needs to match the behavior of what a user would experience in production. h3. Why? These problems arise because of the way the test is setup. Instead of directly testing an `ElasticsearchDao` as you might expect this test runs against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an `HBaseDao`. On retrievals the `MultIndexDao` will return the document from whichever index responds first. With the current test setup, the underlying `ElasticsearchDao` will never retrieve the document that the test case is expected. In all cases where the test passes, the document is actually being returned from the `HBaseDao` which is actually just interacting with a mock backend. The test needs to actually test that we can update and retrieve documents from Elasticsearch. If you alter the test to run against just an `ElasticsearchDao` the test will fail as follows in the attached log file. > Elasticsearch Updates Not Tested in Integration Test > > > Key: METRON-2168 > URL: https://issues.apache.org/jira/browse/METRON-2168 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Attachments: test-failure.log > > > The `ElasticsearchUpdateIntegrationTest` is not testing that the > `ElasticsearchDao` can update and retrieve values in a manner similar to what > would occur in production. > h3. What? > Within the Elasticsearch index, the test fails to define the 'guid' field to > be of type 'keyword', instead the type is defaulted to 'text'. In a > production setting this mistake would prevent any documents from being found > by guid. Unfortunately, the test passes despite this. The test needs to > match the behavior of what a user would experience in production. > h3. Why? > These problems arise because of the way the test is setup. Instead of > directly testing an `ElasticsearchDao` as you might expect this test runs > against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an > `HBaseDao`. On retrievals the `MultIndexDao` will return the document from > whichever index responds first. > With the current test setup, the underlying `ElasticsearchDao` will never > retrieve the document that the test case is expected. In all cases where the > test passes, the document is actually being returned from the `HBaseDao` > which is actually just interacting with a mock backend. The test needs to > actually test that we can update and retrieve documents from Elasticsearch. > > Proof? > If you alter the test to run against just an `ElasticsearchDao` the test > will fail as follows in the attached log file. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2168) Elasticsearch Updates Not Tested in Integration Test
[ https://issues.apache.org/jira/browse/METRON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-2168: --- Description: The `ElasticsearchUpdateIntegrationTest` is not testing that the `ElasticsearchDao` can update and retrieve values in a manner similar to what would occur in production. Within the Elasticsearch index, the test fails to define the 'guid' field to be of type 'keyword', instead the type is defaulted to 'text'. In a production setting this mistake would prevent any documents from being found by guid. Unfortunately, the test passes despite this. The test needs to match the behavior of what a user would experience in production. h3. Why? These problems arise because of the way the test is setup. Instead of directly testing an `ElasticsearchDao` as you might expect this test runs against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an `HBaseDao`. On retrievals the `MultIndexDao` will return the document from whichever index responds first. With the current test setup, the underlying `ElasticsearchDao` will never retrieve the document that the test case is expected. In all cases where the test passes, the document is actually being returned from the `HBaseDao` which is actually just interacting with a mock backend. The test needs to actually test that we can update and retrieve documents from Elasticsearch. If you alter the test to run against just an `ElasticsearchDao` the test will fail as follows in the attached log file. was: The `ElasticsearchUpdateIntegrationTest` is not testing that the `ElasticsearchDao` can update and retrieve values in a manner similar to what would occur in production. Within the Elasticsearch index, the test fails to define the 'guid' field to be of type 'keyword', instead the type is defaulted to 'text'. In a production setting this mistake would prevent any documents from being found by guid. Unfortunately, the test passes despite this. The test needs to match the behavior of what a user would experience in production. h3. Why? These problems arise because of the way the test is setup. Instead of directly testing an `ElasticsearchDao` as you might expect this test runs against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an `HBaseDao`. On retrievals the `MultIndexDao` will return the document from whichever index responds first. With the current test setup, the underlying ElasticsearchDao will never retrieve the document that the test case is expected. In all cases where the test passes, the document is actually being returned from the HBaseDao which is actually just interacting with a mock backend. The test needs to actually test that we can update and retrieve documents from Elasticsearch. If you alter the test to run against just an `ElasticsearchDao` the test will fail as follows in the attached log file. > Elasticsearch Updates Not Tested in Integration Test > > > Key: METRON-2168 > URL: https://issues.apache.org/jira/browse/METRON-2168 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Attachments: test-failure.log > > > The `ElasticsearchUpdateIntegrationTest` is not testing that the > `ElasticsearchDao` can update and retrieve values in a manner similar to what > would occur in production. > Within the Elasticsearch index, the test fails to define the 'guid' field to > be of type 'keyword', instead the type is defaulted to 'text'. In a > production setting this mistake would prevent any documents from being found > by guid. Unfortunately, the test passes despite this. The test needs to > match the behavior of what a user would experience in production. > h3. Why? > These problems arise because of the way the test is setup. Instead of > directly testing an `ElasticsearchDao` as you might expect this test runs > against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an > `HBaseDao`. On retrievals the `MultIndexDao` will return the document from > whichever index responds first. > With the current test setup, the underlying `ElasticsearchDao` will never > retrieve the document that the test case is expected. In all cases where the > test passes, the document is actually being returned from the `HBaseDao` > which is actually just interacting with a mock backend. > The test needs to actually test that we can update and retrieve documents > from Elasticsearch. > If you alter the test to run against just an `ElasticsearchDao` the test > will fail as follows in the attached log file. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-2168) Elasticsearch Updates Not Tested in Integration Test
Nick Allen created METRON-2168: -- Summary: Elasticsearch Updates Not Tested in Integration Test Key: METRON-2168 URL: https://issues.apache.org/jira/browse/METRON-2168 Project: Metron Issue Type: Bug Reporter: Nick Allen Assignee: Nick Allen Attachments: test-failure.log The `ElasticsearchUpdateIntegrationTest` is not testing that the `ElasticsearchDao` can update and retrieve values in a manner similar to what would occur in production. Within the Elasticsearch index, the test fails to define the 'guid' field to be of type 'keyword', instead the type is defaulted to 'text'. In a production setting this mistake would prevent any documents from being found by guid. Unfortunately, the test passes despite this. The test needs to match the behavior of what a user would experience in production. h3. Why? These problems arise because of the way the test is setup. Instead of directly testing an `ElasticsearchDao` as you might expect this test runs against a `MultiIndexDao` initialized with both an `ElasticseachDao` and an `HBaseDao`. On retrievals the `MultIndexDao` will return the document from whichever index responds first. With the current test setup, the underlying ElasticsearchDao will never retrieve the document that the test case is expected. In all cases where the test passes, the document is actually being returned from the HBaseDao which is actually just interacting with a mock backend. The test needs to actually test that we can update and retrieve documents from Elasticsearch. If you alter the test to run against just an `ElasticsearchDao` the test will fail as follows in the attached log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [metron] merrimanr commented on issue #1396: METRON-2034: REST endpoint for getting all parser topology status should return group name
merrimanr commented on issue #1396: METRON-2034: REST endpoint for getting all parser topology status should return group name URL: https://github.com/apache/metron/pull/1396#issuecomment-506757102 It was a Slack conversation. @ruffle1986 was testing and noticed group names were not working for the endpoint that gets status for all topologies (vs the endpoint that gets status for a single topology). He brought it to my attention and I created this fix as a result. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] merrimanr opened a new pull request #1374: METRON-2061: Solr documents with date fields cannot be updated with Dao classes
merrimanr opened a new pull request #1374: METRON-2061: Solr documents with
date fields cannot be updated with Dao classes
URL: https://github.com/apache/metron/pull/1374
## Contributor Comments
The initial purpose of this PR is to explain the cause of the problem
described in https://issues.apache.org/jira/browse/METRON-2061 and facilitate
discussion on an optimal solution. A solution is presented here but I expect
we will explore other solutions as well.
### Problem
Date type fields are a problem for our `UpdateDao` classes because of the
logic used to update documents. Currently the process is this:
1. Get the latest document using either the Elasticsearch or Solr client
API. This document is returned as a Java object.
2. Convert both the document object and patch object to a `JsonNode` object
using Jackson. This includes serializing/deserializing the objects internally.
3. Apply the patch using the `JsonPatch` API.
4. Convert the patched `JsonNode` back to a document object.
5. Index the document object into Elasticsearch or Solr.
The problem is that the date type is lost during serialization. Jackson
automatically converts it to epoch long by default:
https://fasterxml.github.io/jackson-databind/javadoc/2.7/com/fasterxml/jackson/databind/SerializationFeature.html#WRITE_DATES_AS_TIMESTAMPS.
There is also an option of converting the value to a date string. The error
happens because the original date field now contains a long when a document is
indexed.
### Possible Solutions
I think there are several ways to solve this problem and I will present some
options here. Others are welcome to propose additional solutions.
1. Add special handling for date fields - We could detect date fields and
handle the conversion. We would need to manage the Jackson date format and
ensure it lines up with the date format Elasticsearch or Solr expects.
2. Handle the patch with a custom utility (the initial solution in this PR)
- The JSON Patch specification (https://tools.ietf.org/html/rfc6902) is fairly
simple and straightforward. Instead of using a 3rd party library that requires
`JsonNode` objects and serialization/deserialization, we could apply the
patches directly to the objects.
3. Partial updates - I believe both Elasticsearch and Solr provide support
for partial updates. In this case we would only need to handle date type
fields when those specific fields are being updated.
I chose option #2 because it is simple and avoids any future document field
type issues. I believe we should move away from the `JsonPatch` API and
serialization/deserialization altogether because there is potential for other
Jackson issues and it is inefficient. This rules out option #1 for me. Option
#3 is attractive for obvious reasons but may require significant changes to our
API.
### Testing
The initial solution included here has been tested in full dev. We can use
this test script regardless of the solution we choose.
1. Spin up full dev and enable Solr using the instructions in the
metron-solr README.
2. Stop the Storm topologies. This will make it easier to isolate our test
to a single document.
3. Pick a schema and add a date type field to it:
```
```
4. Clear out the collection corresponding to the changed schema and index a
document with a `timestamp_solr` value:
```
{
"guid":"some-guid",
"source.type":"yaf",
"timestamp_solr":"2019-04-04T00:00:00Z"
}
```
I did this using the "Documents" menu item in the Solr UI at
`http://node1:8983`.
5. Attempt to patch an unrelated field:
```
curl -X PATCH --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "some-guid",
"patch": [
{"op":"add","value":"DISMISS","path":"/alert_status"}
],
"sensorType": "yaf"
}' 'http://user:password@node1:8082/api/v1/update/patch'
```
This request should succeed without error and the updated document should be
returned with the new field. Before this PR a 500 error was returned with a
message similar to:
```
Error from server at http://node1:8983/solr/yaf: Invalid Date
String:'75231198000'
```
6. This can also be tested in the Alerts UI by filtering on the guid and
changing the status in the details panel.
### Outstanding Issues
Once we land on a solution I will add comprehensive tests and proper Java
documentation.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke
[GitHub] [metron] merrimanr closed pull request #1374: METRON-2061: Solr documents with date fields cannot be updated with Dao classes
merrimanr closed pull request #1374: METRON-2061: Solr documents with date fields cannot be updated with Dao classes URL: https://github.com/apache/metron/pull/1374 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on issue #1447: METRON-2162 Ambari client exception occurred: No JSON object could be…
nickwallen commented on issue #1447: METRON-2162 Ambari client exception occurred: No JSON object could be… URL: https://github.com/apache/metron/pull/1447#issuecomment-506745387 This has been merged into the feature branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen closed pull request #1447: METRON-2162 Ambari client exception occurred: No JSON object could be…
nickwallen closed pull request #1447: METRON-2162 Ambari client exception occurred: No JSON object could be… URL: https://github.com/apache/metron/pull/1447 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] merrimanr commented on issue #1447: METRON-2161 Ambari client exception occurred: No JSON object could be…
merrimanr commented on issue #1447: METRON-2161 Ambari client exception occurred: No JSON object could be… URL: https://github.com/apache/metron/pull/1447#issuecomment-506744107 I tested this in full dev and the issue is resolved. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] anandsubbu edited a comment on issue #1443: METRON-2150: [UI] User not able to filter by multiple values of the same field on Alerts UI
anandsubbu edited a comment on issue #1443: METRON-2150: [UI] User not able to filter by multiple values of the same field on Alerts UI URL: https://github.com/apache/metron/pull/1443#issuecomment-506692482 When testing this PR, I observed one case that does not behave as expected. When a new alert is generated, the `alert_status` field is not set, but the default status is shown as 'NEW' in the UI. Attempting a filter on alerts without the field not being set does not hence yield the desired result. So the following filter: ``` alert_status:(NEW OR OPEN) ``` will not display alerts which have their default status (i.e. `NEW`) set during creation time. However, if one manually set the status of an alert to `NEW` (e.g. switch from `NEW` (default) -> `OPEN` -> `NEW`), then the query works fine. I have created [METRON-2167](https://issues.apache.org/jira/browse/METRON-2167) to track this issue separately. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] anandsubbu commented on issue #1443: METRON-2150: [UI] User not able to filter by multiple values of the same field on Alerts UI
anandsubbu commented on issue #1443: METRON-2150: [UI] User not able to filter by multiple values of the same field on Alerts UI URL: https://github.com/apache/metron/pull/1443#issuecomment-506692482 When testing this PR, I observed one case that does not behave as expected. When a new alert is generated, the `alert_status` field is not set, but the default status is shown as 'NEW'. Attempting a filter on alerts without the field not being set does not hence yield the desired result. I have created [METRON-2167](https://issues.apache.org/jira/browse/METRON-2167) to track this issue separately. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] anandsubbu edited a comment on issue #1443: METRON-2150: [UI] User not able to filter by multiple values of the same field on Alerts UI
anandsubbu edited a comment on issue #1443: METRON-2150: [UI] User not able to filter by multiple values of the same field on Alerts UI URL: https://github.com/apache/metron/pull/1443#issuecomment-506692482 When testing this PR, I observed one case that does not behave as expected. When a new alert is generated, the `alert_status` field is not set, but the default status is shown as 'NEW' in the UI. Attempting a filter on alerts without the field not being set does not hence yield the desired result. I have created [METRON-2167](https://issues.apache.org/jira/browse/METRON-2167) to track this issue separately. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (METRON-2167) Unable to filter with alert_status:NEW
Anand Subramanian created METRON-2167:
-
Summary: Unable to filter with alert_status:NEW
Key: METRON-2167
URL: https://issues.apache.org/jira/browse/METRON-2167
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
The following query does not return any results when run through
/api/v1/search/search:
{code:java}
{ "indices": [], "facetFields": [], "query": "(alert_status:NEW OR
metron_alert.alert_status:NEW)", "from": 0, "size": 25 }{code}
However when I change the query to look for any of 'OPEN' or 'ESCALATE' or
'RESOLVE', it returns results.
It is also observed that sorting on the 'Alert Status' field is broken. Here's
an example:
{code:java}
{ "indices": [], "facetFields": [], "query": "*", "from": 0, "size": 25,
"sort": [ { "field": "alert_status", "sortOrder": "asc" } ] }{code}
{{}}yields the following error...
{code:java}
{ "responseCode": 500, "message": "Failed to execute search;
error='IllegalArgumentException: Fielddata is disabled on text fields by
default. Set fielddata=true on [__anonymous_text] in order to load fielddata in
memory by uninverting the inverted index. Note that this can however use
significant memory. Alternatively use a keyword field instead.',
search='{\"from\":0,\"size\":25,\"query\":{\"constant_score\":{\"filter\":{\"bool\":{\"must\":[{\"bool\":{\"should\":[{\"query_string\":{\"query\":\"*\",\"fields\":[],\"use_dis_max\":true,\"tie_breaker\":0.0,\"default_operator\":\"or\",\"auto_generate_phrase_queries\":false,\"max_determinized_states\":1,\"enable_position_increments\":true,\"fuzziness\":\"AUTO\",\"fuzzy_prefix_length\":0,\"fuzzy_max_expansions\":50,\"phrase_slop\":0,\"escape\":false,\"split_on_whitespace\":true,\"boost\":1.0}},{\"nested\":{\"query\":{\"query_string\":{\"query\":\"*\",\"fields\":[],\"use_dis_max\":true,\"tie_breaker\":0.0,\"default_operator\":\"or\",\"auto_generate_phrase_queries\":false,\"max_determinized_states\":1,\"enable_position_increments\":true,\"fuzziness\":\"AUTO\",\"fuzzy_prefix_length\":0,\"fuzzy_max_expansions\":50,\"phrase_slop\":0,\"escape\":false,\"split_on_whitespace\":true,\"boost\":1.0}},\"path\":\"metron_alert\",\"ignore_unmapped\":false,\"score_mode\":\"none\",\"boost\":1.0}}],\"disable_coord\":false,\"adjust_pure_negative\":true,\"boost\":1.0}},{\"bool\":{\"should\":[{\"term\":{\"status\":{\"value\":\"active\",\"boost\":1.0}}},{\"bool\":{\"must_not\":[{\"exists\":{\"field\":\"status\",\"boost\":1.0}}],\"disable_coord\":false,\"adjust_pure_negative\":true,\"boost\":1.0}}],\"disable_coord\":false,\"adjust_pure_negative\":true,\"boost\":1.0}}],\"must_not\":[{\"exists\":{\"field\":\"metaalerts\",\"boost\":1.0}}],\"disable_coord\":false,\"adjust_pure_negative\":true,\"boost\":1.0}},\"boost\":1.0}},\"_source\":{\"includes\":[],\"excludes\":[]},\"sort\":[{\"alert_status\":{\"order\":\"asc\",\"missing\":\"_first\",\"unmapped_type\":\"text\"}}],\"track_scores\":true,\"aggregations\":{\"source:type_count\":{\"terms\":{\"field\":\"source:type\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_term\":\"asc\"}]}},\"ip_src_addr_count\":{\"terms\":{\"field\":\"ip_src_addr\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_term\":\"asc\"}]}},\"ip_dst_addr_count\":{\"terms\":{\"field\":\"ip_dst_addr\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_term\":\"asc\"}]}},\"enrichments:geo:ip_dst_addr:country_count\":{\"terms\":{\"field\":\"enrichments:geo:ip_dst_addr:country\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_term\":\"asc\"}]'",
"fullMessage": "IllegalArgumentException: Fielddata is disabled on text fields
by default. Set fielddata=true on [__anonymous_text] in order to load fielddata
in memory by uninverting the inverted index. Note that this can however use
significant memory. Alternatively use a keyword field instead." }
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
