[
https://issues.apache.org/jira/browse/METRON-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15685132#comment-15685132
]
ASF GitHub Bot commented on METRON-576:
---------------------------------------
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/366
METRON-576 Stellar function resolution takes too long on running cluster
### [METRON-576](https://issues.apache.org/jira/browse/METRON-576)
When running the Stellar REPL in a cluster on AWS, function resolution
takes 50-60 seconds. The user is not able to execute any functions in the REPL
until this process completes.
The default function resolver searches the entire classpath for Stellar
functions. This includes all library dependencies. The delay is because there
are just too many classes in the classpath to search on a running cluster. As
more libraries are added as dependencies under `/usr/metron/0.3.0/lib` this
problem just gets worse.
#### Testing
Tested on Quick Dev and showed definite performance improvement. Before
limiting the search path, the search takes roughly 18-20 seconds for me. After
adding a sensible "include" like `org.apache.metron.*`, the search takes
roughly 2-3 seconds.
Have not yet deployed on AWS to compare the load time.
#### Changes
* A global property allows the user to specify packages that should be
searched for Stellar functions. `stellar.function.resolver.includes =
org.apache.metron.*` This significantly reduces the delay when waiting for the
Stellar REPL to be ready to execute a function.
* Another global property allows the user to exclude specific packages from
being searched for Stellar functions. `stellar.function.resolver.excludes =
org.apache.metron.management.*`
* By default, the entire classpath is searched, which matches the previous
behavior. I'd be open to changing default behavior if there is interest.
* All of the logic that was previously contained in
SingletonFunctionResolver is now broken out across multiple classes.
* The `BaseFunctionResolver` leverages Guava for lazy initialization
and thread-safety, instead of the read/write locking scheme as before. This is
much simpler and I believe is effectively the same.
* The `ClasspathFunctionResolver` uses the Reflections API to search
the classpath for Stellar functions as before. This functionality was enhanced
to explicitly include or exclude specific packages from function resolution.
* The `SingletonFunctionResolver` remains, but is really only a
singleton that is used as the default function resolver. Keeping this separate
made testing simpler.
* The `SimpleFunctionResolver` is a function resolver that allows
single classes to be searched for Stellar functions. This is useful when
writing unit tests for Stellar functions.
```
FunctionResolver functionResolver = new SimpleFunctionResolver()
.withClass(KafkaFunctions.KafkaGet.class)
.withClass(KafkaFunctions.KafkaPut.class)
.withClass(KafkaFunctions.KafkaProps.class)
.withClass(KafkaFunctions.KafkaTail.class);
```
* Added unit tests for the function resolvers.
* Removed the `clear()` function from the `FunctionResolver` API. This was
only needed for testing. This was originally only a work-around since it was
not easy to swap in another `FunctionResolver` at the time.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-576
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/366.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #366
----
----
> Stellar function resolution takes too long on running cluster
> -------------------------------------------------------------
>
> Key: METRON-576
> URL: https://issues.apache.org/jira/browse/METRON-576
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> When running the Stellar REPL in a cluster on AWS, function resolution takes
> 50-60 seconds. The user is not able to execute any functions in the REPL
> until this process completes.
> The default function resolver searches the classpath for Stellar functions.
> The delay may be because there are just too many classes in the classpath to
> search on a running cluster. As more libraries are added as dependencies
> under /usr/metron/<version>/lib this problem just gets worse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
