[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
Doesn't necessarily have to be Docker but yes you would have to start 
something up.  That step is now outside of the integration tests.  

We could keep the current in memory code and offer a debug mode.  Is there 
a benefit to using an in-memory component when debugging?


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
So, debugging the integration tests will require having docker running in 
the background now?


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/851


---

[GitHub] metron issue #859: METRON-1345: Update EC2 README for custom Ansible tags

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/859
  
+1 Thanks!


---

[GitHub] metron issue #851: METRON-1336 Patching Can Result in Bad Configuration

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/851
  
Thanks guys.  Nice work on the meta-PRs.  I like how that turned out.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
@ottobackwards I also added a way to run the e2e tests locally.  Once your 
Docker app is up and running locally, get the $DOCKER_HOST ip address and 
replace "localhost" with that ip address in 
`metron-interface/metron-alerts/protractor.conf.js` (should be 3 places you 
have to update).  Mine looks something like this:
```
exports.config = {
  ...
  baseUrl: 'http://192.168.99.100:4201/',
  ...
  params: {
rest: {
  url: '192.168.99.100:8082'
},
elasticsearch: {
  url: '192.168.99.100:9210'
}
  }
}
```

Then you can run the e2e tests with `cd metron-interface/metron-alerts && 
npm run e2e` 


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
@justinleet latest commit includes the changes I had to make to switch a 
preexisting integration test to the new infrastructure.  I verified it works 
locally, we'll see how it goes with Travis.

It was pretty straightforward and consisted of mainly 2 steps:

1. Remove the in memory components and replace any helper methods for 
setup/teardown
2. Add a namespace to all the test assets (indices and doc types in this 
case)

I believe converting other tests will follow this pattern as well.  To 
answer @ottobackwards's original question about 
[#854](https://github.com/apache/metron/pull/854), adding namespaces will be 
required when running the tests in parallel against shared infrastructure.  
Cleaning up at the beginning/end won't be enough.

This is meant only as an example, happy to revert it and add it in a future 
PR.


---

[GitHub] metron issue #851: METRON-1336 Patching Can Result in Bad Configuration

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/851
  
+1 by inspection. Nice work Nick.


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r155347462
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -572,15 +590,22 @@ public static void 
applyConfigPatchToZookeeper(ConfigurationType configurationTy
* @param patchData a JSON patch in the format specified by RFC 6902
* @param client access to zookeeeper
*/
-  public static void applyConfigPatchToZookeeper(ConfigurationType 
configurationType,
-  Optional configName,
-  byte[] patchData, CuratorFramework client) throws Exception {
+  public static void applyConfigPatchToZookeeper(
+  ConfigurationType configurationType,
+  Optional configName,
+  byte[] patchData, CuratorFramework client) throws Exception {
+
 byte[] configData = readConfigBytesFromZookeeper(configurationType, 
configName, client);
 JsonNode source = JSONUtils.INSTANCE.readTree(configData);
 JsonNode patch = JSONUtils.INSTANCE.readTree(patchData);
 JsonNode patchedConfig = JSONUtils.INSTANCE.applyPatch(patch, source);
-writeConfigToZookeeper(configurationType, configName,
-JSONUtils.INSTANCE.toJSONPretty(patchedConfig), client);
+byte[] prettyPatchedConfig = 
JSONUtils.INSTANCE.toJSONPretty(patchedConfig);
+
+// ensure the patch produces a valid result; otherwise exception 
thrown during deserialization
--- End diff --

Heh, well our meta-PR was to address a code change that resulted in 
duplication, not just a style preference. ;)


---

[GitHub] metron issue #851: METRON-1336 Patching Can Result in Bad Configuration

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/851
  
I'm comfortable with this.  +1 by inspection


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
Sure a discussion on that topic is fine with me.  I'm referring to the 
existing integration tests.  


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
Any chance we can have a discuss on moving other integration tests over?  
You mean the regular tests right, like profiler etc?


---

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

[mmiklavc]

GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/859

METRON-1345: Update EC2 README for custom Ansible tags

https://issues.apache.org/jira/browse/METRON-1345

Simple improvement/correction to the EC2 README. This runs the command with 
the appropriate environment info and skip-tags enabled via the run.sh script 
instead of through ansible-playbook directly, which misses some things.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/metron ec2-doc-change

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/859.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #859


commit f95a4173bd71c5e8380c3e0708cb4ab7d8ddee97
Author: Michael Miklavcic 
Date:   2017-12-06T19:52:39Z

METRON-1345: Update EC2 README for custom Ansible tags




---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
@justinleet I think that is a great idea.  I didn't do it initially to keep 
the size of this PR modest but I can add it in (we can always roll it back and 
move it to another PR if we want).  I'll start with metron-elasticsearch and 
see how it goes.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
Sorry @ottobackwards, you're right the instructions are a little vague.  
First, `metron-contrib/metron-docker-e2e/compose` is the directory you should 
be in.  I should have added that at the beginning.  Not all steps require you 
be in that directory but steps 4 and 5 do.

So you can't run the tests in the same exact way now because the Alerts UI 
host is hardcoded in the e2e tests (add that to our list of things to address). 
 It is hardcoded to "localhost" which works fine in travis but won't work in 
Mac OSX because of networking limitations (Docker has to run in a vagrant 
machine).  Let me come up with a plan that will allow you to easily run the e2e 
tests locally.


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/858#discussion_r155339646
  
--- Diff: metron-interface/metron-alerts/pom.xml ---
@@ -130,4 +146,44 @@
 
 
 
+  
+
+
+  e2e
+  
+
+  
--- End diff --

Once we get closer to consensus on what this is going to look like, I 
promise I will add extensive documentation.


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/858#discussion_r155339627
  
--- Diff: metron-interface/metron-alerts/e2e/utils/e2e_util.ts ---
@@ -46,25 +48,53 @@ export function waitForStalenessOf (_element ) {
 }
 
 export function loadTestData() {
--- End diff --

Agreed


---

[GitHub] metron issue #851: METRON-1336 Patching Can Result in Bad Configuration

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/851
  
Bump.  Anything else we need on this one?


---

[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

[iraghumitra]

Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/857
  
@merrimanr renamed the variable POLLING_DEFAULT_STATE to DISABLE_POLLING 


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
@merrimanr can you add what directory you should be in when you run the 
commands?


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/858#discussion_r155288566
  
--- Diff: metron-interface/metron-alerts/pom.xml ---
@@ -130,4 +146,44 @@
 
 
 
+  
+
+
+  e2e
+  
+
+  
--- End diff --

should we document with a comment what these plugins are for?


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/858#discussion_r155288298
  
--- Diff: metron-interface/metron-alerts/e2e/utils/e2e_util.ts ---
@@ -46,25 +48,53 @@ export function waitForStalenessOf (_element ) {
 }
 
 export function loadTestData() {
--- End diff --

I think it is important that these test functions are commented and 
documented.  Not only for maintaining them, but they will possibly be examples 
for other efforts.

Some of these functions around setting up tests and tearing them down would 
themselves be useful as utility functions more general than the end to end 
possibly


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/858
  
@merrimanr Did you look into what migrating our non-e2e tests integration 
tests would involve? I think for a POC, it's important to get a sense of how 
we'd be able to unify the infrastructure of these, even if they aren't 
necessarily all unified at POC time.  If not, are you planning on investigating 
that once you resolve the current list of e2e issues?


---

[GitHub] metron issue #854: Experimental Improvements - Feedback Only - Do Not Merge

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/854
  
Casey and I investigated this previously during one of the times we were 
running into the upper time limit.

The main reason we ultimately chose not to do this was because we share the 
build queue with other Apache projects and kicking up multiple builds (at a 
period in time was backing up in the queues) seemed like it could cause 
problems for other projects.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
Wow, this is awesome.  I'm looking forward to getting into it.  First thing 
that pops to mind is how this relates to or is effected by @nickwallen's 
https://github.com/apache/metron/pull/854


---

[GitHub] metron pull request #857: METRON-1340: Improve e2e tests for metron alerts

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/857#discussion_r155261159
  
--- Diff: metron-interface/metron-alerts/src/app/utils/constants.ts ---
@@ -38,5 +38,6 @@ export let TREE_SUB_GROUP_SIZE = 5;
 export let DEFAULT_FACETS = ['source:type', 'ip_src_addr', 'ip_dst_addr', 
'host', 'enrichments:geo:ip_dst_addr:country'];
 export let DEFAULT_GROUPS = ['source:type', 'ip_src_addr', 'ip_dst_addr', 
'host', 'enrichments:geo:ip_dst_addr:country'];
 export let INDEXES =  environment.indices ? environment.indices.split(',') 
: [];
+export let POLLING_DEFAULT_STATE = environment.defaultPollingState;
--- End diff --

Would it be possible to change this to a better name?  Something like 
DISABLE_POLLING?  Setting POLLING_DEFAULT_STATE to true to stop polling doesn't 
make sense to me.


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/858

METRON-1344: Externalize the infrastructural components using integration 
tests

## Contributor Comments
This PR will add infrastructure to our Travis build that will allow the 
Alerts UI e2e test to be run.  There are several outstanding issues that still 
need to be worked out so DO NOT MERGE this yet.

This is a first pass at a potential Docker-based solution and is meant to 
be a POC.  The intention is to facilitate further discussion around the general 
approach and provide a working example that we can build off of.  

A good place to start is the .travis file.  This provides a good guide on 
how this infrastructure is spun up.  It is assumed Docker and Docker Compose 
are installed.  To use outside of travis in a local dev environment (assuming 
Mac OSX): 

1. Build Metron with `mvn clean install -DskipTests`
1. Create a Docker machine with 
`metron-contrib/metron-docker/scripts/create-docker-machine.sh`
1. Set the Docker env variables with `eval $(docker-machine env 
metron-machine)`
1. Build the base Metron image that installs Java:  `docker build 
./metron-centos/ -t "metron-centos"`
1.  Spin up the environment:  `cd metron-contrib/metron-docker-e2e/compose 
&& docker-compose up -d`

A working environment should now be available.  To verify get the Docker 
machine address with `echo $DOCKER_HOST`.  Services should be available on the 
ports specified in 
`metron-contrib/metron-docker-e2e/compose/docker-compose.yml`.  For example, 
assuming my $DOCKER_HOST is "tcp://192.168.99.100:2376", Elasticsearch should 
be available at http://192.168.99.100:9210.

At this point only a single e2e test is run due to the significant 
refactoring being done in https://github.com/apache/metron/pull/857.  

Here are my thoughts so far based on work in this PR:

- The Docker environment creation and startup adds about 3 minutes to the 
build.  This is about what I expected and hopefully we can get this time back 
as we move other tests to a reusable environment.
- I'm not convinced spinning up containers for the Alerts UI and REST are 
necessary or desired.  We may be able to cut some time off the build by running 
them directly on the Travis host instead of in Docker
- I experimented with caching the Docker images but pulling them every time 
was actually faster

There is still significant work to be done including:
- I was able to get the full suite of tests to run successfully in previous 
commits but have all but e2e tests commented out for now to make it easier to 
see what's going on in Travis.  These will need to be added back once we get 
closer to a final solution.
- Once the e2e test refactoring is done those changes will need merged in 
and tested
- The license check is commented out right now because I inadvertently 
added some new dependency versions (have no idea why).  Still need to track 
that down.
- There is an intermittent error that happens when starting up REST.  Still 
working on tracking this down.
 
Looking forward to some feedback.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 

Re: Heterogeneous indexing batch size for different Metron feeds

[Otto Fowler]

Sorry,
We flush for timeouts on every storm ‘tick’ message, not on every message.



On December 6, 2017 at 08:29:51, Otto Fowler (ottobackwa...@gmail.com)
wrote:

I have looked at it.

We maintain batch lists for each sensor which gather messages to index.
When we get a message that puts it over the batch size the messages are
flushed and written to the target.
There is also a timeout component, where the batch would be flushed based
on timeout.

While batch size checking occurs on a per sensor-message receipt basis,
each message, regardless of sensor will trigger a check of the batch
timeout for all the lists.

At least that is what I think I see.

Without understanding what the failures are for it is hard to see what the
issue is.

Do we have timing issues where all the lists are timing out all the time
causing some kind of cascading failure for example?
Does the number of sensors matter?  For example if only one sensor topology
is running with batch setup X, is everything fine?  Do failures start after
adding Nth additional sensor?

Hopefully someone else on the list may have an idea.
That code does not have any logging to speak of… well debug / trace logging
that would help here either.



On December 6, 2017 at 08:18:01, Ali Nazemian (alinazem...@gmail.com) wrote:

Everything looks normal except the high number of failed tuples. Do you
know how the indexing batch size works? Based on our observations it seems
it doesn't update the messages that are in enrichments and indexing topics.

On Thu, Dec 7, 2017 at 12:13 AM, Otto Fowler 
wrote:

> What do you see in the storm ui for the indexing topology?
>
>
> On December 6, 2017 at 07:10:17, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Both hdfs and Elasticsearch batch sizes. There is no error in the logs. It
> mpacts topology error rate and cause almost 90% error rate on indexing
> tuples.
>
> On 6 Dec. 2017 00:20, "Otto Fowler"  wrote:
>
> Where are you seeing the errors?  Screenshot?
>
>
> On December 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Which of the indexing options are you changing the batch size for?  HDFS?
> Elasticsearch?  Both?
>
> Can you give an example?
>
>
>
> On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> No specific error in the logs. I haven't enabled debug/trace, though.
>
> On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler 
> wrote:
>
>> My first thought is what are the errors when you get a high error rate?
>>
>>
>> On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>>
>> Any thoughts?
>>
>> On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian 
>> wrote:
>>
>> > Hi,
>> >
>> > We have noticed recently that no matter what batch size we use for
>> Metron
>> > indexing feeds, as long as we start using different batch size for
>> > different Metron feeds, indexing topology throughput will start dropping
>> > due to the high error rate! So I was wondering whether based on the
>> current
>> > indexing topology design, we have to choose the same batch size for all
>> the
>> > feeds or not. Otherwise, throughout will be dropped. I assume since it
>> is
>> > acceptable to use different batch sizes for different feeds, it is not
>> > expected by design.
>> >
>> > Moreover, I have noticed in practice that even if we change the batch
>> > size, it will not affect the messages that are already in enrichments or
>> > indexing topics, and it will only affect the new messages that are
>> coming
>> > to the parser. Therefore, we need to let all the messages pass the
>> indexing
>> > topology so that we can change the batch size!
>> >
>> > It would be great if we can have more details regarding the design of
>> this
>> > section so we can understand our observations are based on the design or
>> > some kind of bug.
>> >
>> > Regards,
>> > Ali
>> >
>>
>>
>>
>> --
>> A.Nazemian
>>
>>
>
>
> --
> A.Nazemian
>
>
>


--
A.Nazemian

Re: Heterogeneous indexing batch size for different Metron feeds

[Otto Fowler]

I have looked at it.

We maintain batch lists for each sensor which gather messages to index.
When we get a message that puts it over the batch size the messages are
flushed and written to the target.
There is also a timeout component, where the batch would be flushed based
on timeout.

While batch size checking occurs on a per sensor-message receipt basis,
each message, regardless of sensor will trigger a check of the batch
timeout for all the lists.

At least that is what I think I see.

Without understanding what the failures are for it is hard to see what the
issue is.

Do we have timing issues where all the lists are timing out all the time
causing some kind of cascading failure for example?
Does the number of sensors matter?  For example if only one sensor topology
is running with batch setup X, is everything fine?  Do failures start after
adding Nth additional sensor?

Hopefully someone else on the list may have an idea.
That code does not have any logging to speak of… well debug / trace logging
that would help here either.



On December 6, 2017 at 08:18:01, Ali Nazemian (alinazem...@gmail.com) wrote:

Everything looks normal except the high number of failed tuples. Do you
know how the indexing batch size works? Based on our observations it seems
it doesn't update the messages that are in enrichments and indexing topics.

On Thu, Dec 7, 2017 at 12:13 AM, Otto Fowler 
wrote:

> What do you see in the storm ui for the indexing topology?
>
>
> On December 6, 2017 at 07:10:17, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Both hdfs and Elasticsearch batch sizes. There is no error in the logs. It
> mpacts topology error rate and cause almost 90% error rate on indexing
> tuples.
>
> On 6 Dec. 2017 00:20, "Otto Fowler"  wrote:
>
> Where are you seeing the errors?  Screenshot?
>
>
> On December 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Which of the indexing options are you changing the batch size for?  HDFS?
> Elasticsearch?  Both?
>
> Can you give an example?
>
>
>
> On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> No specific error in the logs. I haven't enabled debug/trace, though.
>
> On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler 
> wrote:
>
>> My first thought is what are the errors when you get a high error rate?
>>
>>
>> On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>>
>> Any thoughts?
>>
>> On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian 
>> wrote:
>>
>> > Hi,
>> >
>> > We have noticed recently that no matter what batch size we use for
>> Metron
>> > indexing feeds, as long as we start using different batch size for
>> > different Metron feeds, indexing topology throughput will start dropping
>> > due to the high error rate! So I was wondering whether based on the
>> current
>> > indexing topology design, we have to choose the same batch size for all
>> the
>> > feeds or not. Otherwise, throughout will be dropped. I assume since it
>> is
>> > acceptable to use different batch sizes for different feeds, it is not
>> > expected by design.
>> >
>> > Moreover, I have noticed in practice that even if we change the batch
>> > size, it will not affect the messages that are already in enrichments or
>> > indexing topics, and it will only affect the new messages that are
>> coming
>> > to the parser. Therefore, we need to let all the messages pass the
>> indexing
>> > topology so that we can change the batch size!
>> >
>> > It would be great if we can have more details regarding the design of
>> this
>> > section so we can understand our observations are based on the design or
>> > some kind of bug.
>> >
>> > Regards,
>> > Ali
>> >
>>
>>
>>
>> --
>> A.Nazemian
>>
>>
>
>
> --
> A.Nazemian
>
>
>


--
A.Nazemian

Re: Heterogeneous indexing batch size for different Metron feeds

[Ali Nazemian]

Everything looks normal except the high number of failed tuples. Do you
know how the indexing batch size works? Based on our observations it seems
it doesn't update the messages that are in enrichments and indexing topics.

On Thu, Dec 7, 2017 at 12:13 AM, Otto Fowler 
wrote:

> What do you see in the storm ui for the indexing topology?
>
>
> On December 6, 2017 at 07:10:17, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Both hdfs and Elasticsearch batch sizes. There is no error in the logs. It
> mpacts topology error rate and cause almost 90% error rate on indexing
> tuples.
>
> On 6 Dec. 2017 00:20, "Otto Fowler"  wrote:
>
> Where are you seeing the errors?  Screenshot?
>
>
> On December 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Which of the indexing options are you changing the batch size for?  HDFS?
> Elasticsearch?  Both?
>
> Can you give an example?
>
>
>
> On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> No specific error in the logs. I haven't enabled debug/trace, though.
>
> On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler 
> wrote:
>
>> My first thought is what are the errors when you get a high error rate?
>>
>>
>> On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>>
>> Any thoughts?
>>
>> On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian 
>> wrote:
>>
>> > Hi,
>> >
>> > We have noticed recently that no matter what batch size we use for
>> Metron
>> > indexing feeds, as long as we start using different batch size for
>> > different Metron feeds, indexing topology throughput will start dropping
>> > due to the high error rate! So I was wondering whether based on the
>> current
>> > indexing topology design, we have to choose the same batch size for all
>> the
>> > feeds or not. Otherwise, throughout will be dropped. I assume since it
>> is
>> > acceptable to use different batch sizes for different feeds, it is not
>> > expected by design.
>> >
>> > Moreover, I have noticed in practice that even if we change the batch
>> > size, it will not affect the messages that are already in enrichments or
>> > indexing topics, and it will only affect the new messages that are
>> coming
>> > to the parser. Therefore, we need to let all the messages pass the
>> indexing
>> > topology so that we can change the batch size!
>> >
>> > It would be great if we can have more details regarding the design of
>> this
>> > section so we can understand our observations are based on the design or
>> > some kind of bug.
>> >
>> > Regards,
>> > Ali
>> >
>>
>>
>>
>> --
>> A.Nazemian
>>
>>
>
>
> --
> A.Nazemian
>
>
>


-- 
A.Nazemian

Re: Heterogeneous indexing batch size for different Metron feeds

[Otto Fowler]

What do you see in the storm ui for the indexing topology?


On December 6, 2017 at 07:10:17, Ali Nazemian (alinazem...@gmail.com) wrote:

Both hdfs and Elasticsearch batch sizes. There is no error in the logs. It
mpacts topology error rate and cause almost 90% error rate on indexing
tuples.

On 6 Dec. 2017 00:20, "Otto Fowler"  wrote:

Where are you seeing the errors?  Screenshot?


On December 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com)
wrote:

Which of the indexing options are you changing the batch size for?  HDFS?
Elasticsearch?  Both?

Can you give an example?



On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com) wrote:

No specific error in the logs. I haven't enabled debug/trace, though.

On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler 
wrote:

> My first thought is what are the errors when you get a high error rate?
>
>
> On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Any thoughts?
>
> On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian 
> wrote:
>
> > Hi,
> >
> > We have noticed recently that no matter what batch size we use for Metron
> > indexing feeds, as long as we start using different batch size for
> > different Metron feeds, indexing topology throughput will start dropping
> > due to the high error rate! So I was wondering whether based on the
> current
> > indexing topology design, we have to choose the same batch size for all
> the
> > feeds or not. Otherwise, throughout will be dropped. I assume since it is
> > acceptable to use different batch sizes for different feeds, it is not
> > expected by design.
> >
> > Moreover, I have noticed in practice that even if we change the batch
> > size, it will not affect the messages that are already in enrichments or
> > indexing topics, and it will only affect the new messages that are coming
> > to the parser. Therefore, we need to let all the messages pass the
> indexing
> > topology so that we can change the batch size!
> >
> > It would be great if we can have more details regarding the design of
> this
> > section so we can understand our observations are based on the design or
> > some kind of bug.
> >
> > Regards,
> > Ali
> >
>
>
>
> --
> A.Nazemian
>
>


--
A.Nazemian

[GitHub] metron pull request #857: METRON-1340: Improve e2e tests for metron alerts

[iraghumitra]

Github user iraghumitra commented on a diff in the pull request:

https://github.com/apache/metron/pull/857#discussion_r155219994
  
--- Diff: metron-interface/metron-alerts/src/app/utils/constants.ts ---
@@ -38,5 +38,6 @@ export let TREE_SUB_GROUP_SIZE = 5;
 export let DEFAULT_FACETS = ['source:type', 'ip_src_addr', 'ip_dst_addr', 
'host', 'enrichments:geo:ip_dst_addr:country'];
 export let DEFAULT_GROUPS = ['source:type', 'ip_src_addr', 'ip_dst_addr', 
'host', 'enrichments:geo:ip_dst_addr:country'];
 export let INDEXES =  environment.indices ? environment.indices.split(',') 
: [];
+export let POLLING_DEFAULT_STATE = environment.defaultPollingState;
--- End diff --

Added info for this in Readme and CC 


---

Re: Heterogeneous indexing batch size for different Metron feeds

[Ali Nazemian]

Both hdfs and Elasticsearch batch sizes. There is no error in the logs. It
mpacts topology error rate and cause almost 90% error rate on indexing
tuples.

On 6 Dec. 2017 00:20, "Otto Fowler"  wrote:

Where are you seeing the errors?  Screenshot?


On December 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com)
wrote:

Which of the indexing options are you changing the batch size for?  HDFS?
Elasticsearch?  Both?

Can you give an example?



On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com) wrote:

No specific error in the logs. I haven't enabled debug/trace, though.

On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler 
wrote:

> My first thought is what are the errors when you get a high error rate?
>
>
> On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Any thoughts?
>
> On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian 
> wrote:
>
> > Hi,
> >
> > We have noticed recently that no matter what batch size we use for Metron
> > indexing feeds, as long as we start using different batch size for
> > different Metron feeds, indexing topology throughput will start dropping
> > due to the high error rate! So I was wondering whether based on the
> current
> > indexing topology design, we have to choose the same batch size for all
> the
> > feeds or not. Otherwise, throughout will be dropped. I assume since it is
> > acceptable to use different batch sizes for different feeds, it is not
> > expected by design.
> >
> > Moreover, I have noticed in practice that even if we change the batch
> > size, it will not affect the messages that are already in enrichments or
> > indexing topics, and it will only affect the new messages that are coming
> > to the parser. Therefore, we need to let all the messages pass the
> indexing
> > topology so that we can change the batch size!
> >
> > It would be great if we can have more details regarding the design of
> this
> > section so we can understand our observations are based on the design or
> > some kind of bug.
> >
> > Regards,
> > Ali
> >
>
>
>
> --
> A.Nazemian
>
>


--
A.Nazemian

[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

[iraghumitra]

Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/857
  
@merrimanr 
- I removed the node version check in the ui code
- The browser.sleep is used in following areas
   - When we are using methods like waitForText, few times I see the 
test fails saying that that it got the text we were waiting to change which 
doesn't make sense. So, in this case, i am just yielding the main thread with a 
sleep of 500 milliseconds.
   - Create/Add meta-alert, these calls takes few seconds at times.
   - When we use animation to slide in the details pane, we don't have 
a class that gets added/removed for this so it makes it difficult for us to 
know when the pane is loaded. We can add/remove a class for animation by moving 
the slider to a component or directive. This might be out of scope for current 
PR.
- Refactored tests for Facets
- Looks like the meta alert creation failed can you attach the stack trace 
it might give me additional info to debug. 


---