[
https://issues.apache.org/jira/browse/METRON-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15690157#comment-15690157
]
ASF GitHub Bot commented on METRON-576:
---------------------------------------
GitHub user nickwallen reopened a pull request:
https://github.com/apache/incubator-metron/pull/366
METRON-576 Stellar function resolution takes too long on running cluster
### [METRON-576](https://issues.apache.org/jira/browse/METRON-576)
When running the Stellar REPL in a cluster on AWS, function resolution
takes 50-60 seconds. The user is not able to execute any functions in the REPL
until this process completes.
The default function resolver searches the entire classpath for Stellar
functions. This includes all library dependencies. The delay is because there
are just too many classes in the classpath to search on a running cluster. As
more libraries are added as dependencies under `/usr/metron/0.3.0/lib` this
problem just gets worse.
#### Testing
Tested on Quick Dev and showed definite performance improvement. Before
limiting the search path, the search takes roughly 18-20 seconds for me. After
adding a sensible "include" like `org.apache.metron.*`, the search takes
roughly 2-3 seconds.
Have not yet deployed on AWS to compare the load time.
#### Changes
* A global property allows the user to specify packages that should be
searched for Stellar functions. `stellar.function.resolver.includes =
org.apache.metron.*` This significantly reduces the delay when waiting for the
Stellar REPL to be ready to execute a function.
* Another global property allows the user to exclude specific packages from
being searched for Stellar functions. `stellar.function.resolver.excludes =
org.apache.metron.management.*`
* By default, the entire classpath is searched, which matches the previous
behavior. I'd be open to changing default behavior if there is interest.
* All of the logic that was previously contained in
SingletonFunctionResolver is now broken out across multiple classes.
* The `BaseFunctionResolver` leverages Guava for lazy initialization
and thread-safety, instead of the read/write locking scheme as before. This is
much simpler and I believe is effectively the same.
* The `ClasspathFunctionResolver` uses the Reflections API to search
the classpath for Stellar functions as before. This functionality was enhanced
to explicitly include or exclude specific packages from function resolution.
* The `SingletonFunctionResolver` remains, but is really only a
singleton that is used as the default function resolver. Keeping this separate
made testing simpler.
* The `SimpleFunctionResolver` is a function resolver that allows
single classes to be searched for Stellar functions. This is useful when
writing unit tests for Stellar functions.
```
FunctionResolver functionResolver = new SimpleFunctionResolver()
.withClass(KafkaFunctions.KafkaGet.class)
.withClass(KafkaFunctions.KafkaPut.class)
.withClass(KafkaFunctions.KafkaProps.class)
.withClass(KafkaFunctions.KafkaTail.class);
```
* Added unit tests for the function resolvers.
* Removed the `clear()` function from the `FunctionResolver` API. This was
only needed for testing. This was originally only a work-around since it was
not easy to swap in another `FunctionResolver` at the time.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-576
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/366.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #366
----
commit 745e66c9f87cea9fcca98b422bcd8d38ee20e6d7
Author: Nick Allen <[email protected]>
Date: 2016-11-17T17:59:06Z
METRON-576 Stellar function resolution takes too long on running cluster
commit 65324fde472f7aa986051db33b2c9be90c28e1e4
Author: Nick Allen <[email protected]>
Date: 2016-11-22T15:15:22Z
METRON-576 Moved includes/excludes to a new set of Stellar properties,
rather than global
commit ec37e3dd30192e61c0f50b8b410de7b58baa0349
Author: Nick Allen <[email protected]>
Date: 2016-11-22T15:25:58Z
Corrected comments
commit 18ef71774dc1a737fe03a9635da9bb8e0f8f1fed
Author: Nick Allen <[email protected]>
Date: 2016-11-22T15:46:47Z
Slight refactoring
commit 26d8a76772d82ac477cf6228d8c0b3f66c74e6e4
Author: Nick Allen <[email protected]>
Date: 2016-11-22T16:17:02Z
Can use -p to load properties file
commit 24a5dc1e7b28c57ded3afd2ee94f40c56c828de3
Author: Nick Allen <[email protected]>
Date: 2016-11-22T16:19:12Z
Fixed comment
commit 032a3729468d517de6619b6ca9471517bac0cb5d
Author: Nick Allen <[email protected]>
Date: 2016-11-22T16:51:50Z
Assuming Map<String,Object> for Stellar properties
----
> Stellar function resolution takes too long on running cluster
> -------------------------------------------------------------
>
> Key: METRON-576
> URL: https://issues.apache.org/jira/browse/METRON-576
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> When running the Stellar REPL in a cluster on AWS, function resolution takes
> 50-60 seconds. The user is not able to execute any functions in the REPL
> until this process completes.
> The default function resolver searches the classpath for Stellar functions.
> The delay may be because there are just too many classes in the classpath to
> search on a running cluster. As more libraries are added as dependencies
> under /usr/metron/<version>/lib this problem just gets worse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
