[GitHub] incubator-metron issue #442: METRON-322 Global Batching and Flushing

Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/442 Rebased to current master -- which apparently scotches this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/468 Ok, in the aftermath of the API PR, I added support for properly setting up stellar to read dependencies from HDFS for the API. As such, I validated the API continues to work by: *

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/453 Just pushed out a commit to address recent comments. Commit includes: - unit test and javadoc for MessageGetters - error index template with the "ignore_above": 8191" setting

Re: [DISCUSS] System time vs. Event Time

Before the thought becomes obsolete, I’d like to say that I agree with Nick about the replay scenario and threat signature databases. I think a principal use case is replaying old data with new threat signatures, to detect problems that were undetectable at the time they happened. The use

[GitHub] incubator-metron pull request #449: METRON-701 Triage Metrics Produced by th...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/449 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/468 @dlyle65535 Yeah, in this case, the file was physically not in HDFS though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/468 Oh, hey, that stack trace looks familiar. I think Justin fixed this here: https://github.com/apache/incubator-metron/pull/461. -D... On Thu, Mar 2, 2017 at 2:08

Re: [DISCUSS] System time vs. Event Time

I'm just going to throw out a few of questions, that I don't have good answers to. Casey and Nick, given your familiarity with the systems involved, do you have any thoughts? - What's the smallest unit of work we can do to enable at least a useful subset of a fully featured term batch

[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API

Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/316 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

That's correct. Quick dev will be built using the MPack, but after that, Quick Dev will remove the Metron components and reinstall them using REST calls to Ambari, so pieces of the MPack that was used to create the Quick Dev image would be executed. RPMs are executed in all cases. The

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

Just to clarify, your 1 and 2, which you're working on, will give us the ability with full-dev (not quick-dev) to exercise the RPMs and management pack on the non-sensor code (i.e. the current state of the management pack). As far as I'm concerned, this is huge. This ensures we have an easy

[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/468 +1 tested this in Vagrant quick-dev. @cestella I'm good with waiting on the default dir. I was also able to run this through e2e with your latest version. I also ran into issues

[GitHub] incubator-metron pull request #469: DO NOT MERGE METRON-745: Create Error Da...

GitHub user justinleet opened a pull request: https://github.com/apache/incubator-metron/pull/469 DO NOT MERGE METRON-745: Create Error Dashboards # DO NOT MERGE ## Summary Based on Ryan's work in https://github.com/apache/incubator-metron/pull/453, I went ahead and

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/453 Hi guys, this PR is built on one fundamental assumption: kafka is always available. The source of truth for errors, therefore, is a kafka topic. In a production setting errors

[GitHub] incubator-metron issue #316: METRON-503: Metron REST API

Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/316 All integration tests are passing. I was able to start up the application against quick dev and all the endpoints look like they are working. --- If your project is set up for it, you

[GitHub] incubator-metron issue #316: METRON-503: Metron REST API

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/316 For convenience, I staged the commit in a remote branch: https://github.com/cestella/incubator-metron/tree/METRON-503_sandbox . I'd ask that @merrimanr take it for a test drive. --- If

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/453 I agree, I [recommended the same thing](https://lists.apache.org/thread.html/2a673bc97d975bc7e8e160228742c07a8cf45e43c1b1efb3fea83579@%3Cdev.metron.apache.org%3E) on the dev list and in

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/453 I tried running this up and discovered that there's at least one error that doesn't get caught. Json parsing errors, e.g. if someone gives outright badly formatted messages to indexing

[GitHub] incubator-metron issue #449: METRON-701 Triage Metrics Produced by the Profi...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/449 Yep, looks good, got my +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

Just wanted to update this thread: I've been diligently working to the plan we discussed above: *1) Refactor existing Ansible deployment to use the Ambari MPack to install metron-common, metron-enrichments and metron-parsers. * *2) Regenerate quick-dev to leverage the change.* 3) Create rpm

[GitHub] incubator-metron issue #316: METRON-503: Metron REST API

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr @jjmeyer0 I think we should make sure JJ gets credit for his contribution on this. We should squash the commits into the minimal possible each named "METRON-503: Metron REST

Re: [GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...

I did not see anything like that in the ansible logs, but I guess I can't be sure. On Thu, Mar 2, 2017 at 8:49 AM, David Lyle wrote: > I have not. Did HDFS die while quick-dev was coming up? > > -D... > > On Wed, Mar 1, 2017 at 9:15 PM, cestella

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 Well, if we don't want to put deployment things with the src, then ansible is a more flexible and easier to use tool for certain tasks too. But we will talk about it when I get it

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 No. It's not meant for that. If you want to build standalone, Maven is the supported way. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 Maybe we should add that line to the doc or create a script?? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

Re: [GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...

I have not. Did HDFS die while quick-dev was coming up? -D... On Wed, Mar 1, 2017 at 9:15 PM, cestella wrote: > Github user cestella commented on the issue: > > https://github.com/apache/incubator-metron/pull/468 > > I ran this through from the beginning to the end

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/453 I think focusing on the one specific error that we've seen is not the right way to think about this. Many different types of errors would cause unexpected looping, no? When unexpected

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 ansible-playbook -v -i "localhost," -c local playbooks/metron_build.yml now it is running and i'll see about the errors --- If your project is set up for it, you can reply

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Yeah, that seems like a good thing to try. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 Do I have to create an inventory with my local machine name? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 ansible-playbook -v playbooks/metron_build.yml Using /Users/ottofowler/src/apache/forks/incubator-metron/metron-deployment/ansible.cfg as config file [WARNING]:

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 I need to run the playbook though. It doesn't match any hosts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Sure. It's vagrant up or ./run.sh. :) I run it as part of running full dev or ec2. What error are you getting? Btw, to only build the rpms, you may can still do an

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 can you share your command line? Do you run it from /playbooks? does you -i an inventory? Did you copy or create an ansible.cfg? --- If your project is set up for it, you can

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Yes, that works on my rig. Can you tell me a bit more about what you're experiencing? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 I want to run the playbook to just build the rpm's, not deploy. So just metron_build --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 I am, for both full dev and ec2. What are you trying to do? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 How are you running the playbook? I cannot get it to execute --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your