GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/509
METRON-822 Improve Fastcapa Performance
This PR contains significant improvements to the performance and
scalability of Fastcapa.
* Previously the 'distributor' framework was used. This did not scale well
and has been replaced with a burst-oriented design.
* Receive and transmission functions have been separated to allow each to
scale independently.
* Additional parameters have been added to allow the process to be tuned
easily.
* Output provides basic transparency into the current state of processing.
* If the probe is overwhelmed with more packets than can be handled, it
will continue processing the packets that it can without crashing.
* The fact that packets are being dropped is very clear.
* A great deal of documentation has been added to the README.
This change has been tested on Cisco UCS hardware with a 10G Cisco VNIC.
The probe was able to capture 1 gbps before packets started to drop.
Additional performance tuning would push this ceiling much higher, but for my
purposes, I just needed to reach 1 gbps. Additional work will proceed in the
future to find its true performance ceiling.
To test the change yourself, simply spin-up the virtualized test
environment which will deploy and validate that Fastcapa can land packets in
Kafka correctly.
```
cd metron-deployment/vagrant/fastcapa-test-environment
vagrant up
```
## Pull Request Checklist
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR?
- [ ] Does your PR title start with METRON-XXXX?
- [ ] Has your PR been rebased against the latest commit within the target
branch?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root incubating-metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-822
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/509.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #509
----
commit 34614cc9006a77826ed729f2c8a40f99bd84d4b4
Author: dlyle65535 <[email protected]>
Date: 2017-04-04T12:15:42Z
METRON-196 Deployment Fails Without Ansible 2.0.0.2 closes
apache/incubator-metron#499
commit fccb67aecca199d7aebd14b16debff4bf16f8acd
Author: KunalAggarwal <[email protected]>
Date: 2017-04-04T16:14:23Z
METRON-808: Amazon EC2 deployment fails at Expanding Volume Step
(KunalAggarwal via mmiklavc) closes apache/incubator-metron#496
commit 6410229e6a6a031f470d01ced2eb316d43bae29a
Author: Nick Allen <[email protected]>
Date: 2017-03-21T17:51:55Z
METRON-822 Improve Fastcapa Performance
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
