Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
Just FYI, as part of the performance experimentation in the lab here, we
found that one major impediment to scale was the guava cache in this topology
when the size of the cache becomes non-trivial
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/947
METRON-1467: Replace guava caches in places where the keyspace might be
large
## Contributor Comments
Based on the performance tuning exercise as part of METRON-1460, guava has
difficulties
Github user wardbekker commented on the issue:
https://github.com/apache/metron/pull/946
Steps to test:
- build centos vagrant dev build box
- vagrant ssh
- sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
- sudo
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
@arunmahadevan Thanks for chiming in Arun. I would say that most of the
enrichment work is I/O bound and we try to avoid it whenever possible with a a
time-evicted LRU cache in front of the
Github user arunmahadevan commented on the issue:
https://github.com/apache/metron/pull/940
Managing threadpools within a bolt isn't fundamentally wrong, we have see
some use cases where this is done. However, we have been putting efforts to
reduce the overall number of threads
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
@ottobackwards I haven't sent an email to the storm team, but I did run the
PR past a storm committer that I know and asked his opinion prior to submitting
the PR. The general answer was something
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/940
have we thought to send a mail to the storm dev list and ask if anyone has
done this? potential issues?
---
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/940
If we integrated storm with yarn this would also be a problem, as our
resource management may be at odds with yarn's. I think?
What would be nice is if storm could manage the pool and
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
@mraliagha It's definitely a tradeoff. This is why this is as a complement
to the original split/join topology. Keep in mind, also, that this
architecture enables use-cases that the other would
