nickwallen opened a new pull request #1328: METRON-1986 Batch Profiler Fails to Resolve Stats Stellar Functions URL: https://github.com/apache/metron/pull/1328 The Batch Profiler does not seem to see the STATS_* functions. When I attempt to run a profile using any of those functions, it fails. It is able to 'see' a limited subset of functions, but not all those that I would expect. ## Root Cause The ClassIndex library is used to create an index of all Stellar functions at compile-time. This index is packaged in the JAR as `META-INF/annotations/org.apache.metron.stellar.dsl.Stellar`. When creating a shaded jar using the Maven Shade Plugin, the index files created by ClassIndex are overwritten by default. ClassIndex provides a special transformer for Maven which merges the index files instead of overwriting them. Without using this transformer, the index files are overwritten and Stellar function resolution will not work. This is why Stellar function resolution is not working. https://github.com/atteo/classindex#making-shaded-jar ## Testing 1. Spin-up this profile in the Batch Profiler. It should write successfully to HBase. ``` { "profiles": [ { "profile": "profile-using-stats", "foreach": "ip_src_addr", "update": { "s": "STATS_ADD(s, length)" }, "result": "STATS_SUM(s)" } ], "timestampField":"timestamp" } ``` ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
