[
https://issues.apache.org/jira/browse/METRON-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15955771#comment-15955771
]
ASF GitHub Bot commented on METRON-822:
---------------------------------------
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/509
METRON-822 Improve Fastcapa Performance
This PR contains significant improvements to the performance and
scalability of Fastcapa.
* Previously the 'distributor' framework was used. This did not scale well
and has been replaced with a burst-oriented design.
* Receive and transmission functions have been separated to allow each to
scale independently.
* Additional parameters have been added to allow the process to be tuned
easily.
* Output provides basic transparency into the current state of processing.
* If the probe is overwhelmed with more packets than can be handled, it
will continue processing the packets that it can without crashing.
* The fact that packets are being dropped is very clear.
* A great deal of documentation has been added to the README.
This change has been tested on Cisco UCS hardware with a 10G Cisco VNIC.
The probe was able to capture 1 gbps before packets started to drop.
Additional performance tuning would push this ceiling much higher, but for my
purposes, I just needed to reach 1 gbps. Additional work will proceed in the
future to find its true performance ceiling.
To test the change yourself, simply spin-up the virtualized test
environment which will deploy and validate that Fastcapa can land packets in
Kafka correctly.
```
cd metron-deployment/vagrant/fastcapa-test-environment
vagrant up
```
## Pull Request Checklist
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR?
- [ ] Does your PR title start with METRON-XXXX?
- [ ] Has your PR been rebased against the latest commit within the target
branch?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root incubating-metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-822
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/509.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #509
----
commit 34614cc9006a77826ed729f2c8a40f99bd84d4b4
Author: dlyle65535 <[email protected]>
Date: 2017-04-04T12:15:42Z
METRON-196 Deployment Fails Without Ansible 2.0.0.2 closes
apache/incubator-metron#499
commit fccb67aecca199d7aebd14b16debff4bf16f8acd
Author: KunalAggarwal <[email protected]>
Date: 2017-04-04T16:14:23Z
METRON-808: Amazon EC2 deployment fails at Expanding Volume Step
(KunalAggarwal via mmiklavc) closes apache/incubator-metron#496
commit 6410229e6a6a031f470d01ced2eb316d43bae29a
Author: Nick Allen <[email protected]>
Date: 2017-03-21T17:51:55Z
METRON-822 Improve Fastcapa Performance
----
> Improve Fastcapa Performance
> ----------------------------
>
> Key: METRON-822
> URL: https://issues.apache.org/jira/browse/METRON-822
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> Improve the performance and scalability of the Fastcapa probe.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
