[
https://issues.apache.org/jira/browse/METRON-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707272#comment-16707272
]
ASF GitHub Bot commented on METRON-1916:
----------------------------------------
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1288
> In my mind we don't have a current state where Stellar is running but not
all the functions in the class path are loaded.
The same thing would happen today if an `Exception` is thrown in any of the
functions being loaded. If an Exception is thrown, instead of N functions
ready-to-rock, we would have N-1 functions. This just expands that behavior to
`ClassNotFoundException` which gets thrown when there is a missing dependency..
> Before we would have crashed starting up. Now we will run and crash get
an error later.
I don't see it that way. Let's look at both scenarios.
* If I am NOT using the REST function, then things just work as they
should. Yay! I don't want to worry about the missing dependency for a
function that I have no knowledge about.
* If I am using the REST function, an exception would still be thrown when
the function definition could not be found. So the user still gets an
exception when there is a problem.
(Q) Any thoughts on alternative approaches?
I prefer not to have to worry about classpath dependencies (like those
required by the REST function) that I am not using. In this example, I was
only trying to use the `STATS_*` functions, but it was the REST function that
blew things up for me. It is difficult for a user to track this down, because
they have no knowledge of the REST function and its dependencies.
(Alternative 1) An alternative approach would be to be more selective about
what functions we add to `stellar-common`. Anything added to `stellar-common`
forces a required dependency on all Stellar users. If the REST function had
been a separate project, then a user could choose to use or not use that
projects and not be burdened by the additional dependency
(Alternative 2) Could the `stellar.function.resolver.includes` and
`stellar.function.resolver.excludes` be enhanced to allow users to exclude
functions they don't want to load?
> Stellar Classpath Function Resolver Should Handle ClassNotFoundException
> ------------------------------------------------------------------------
>
> Key: METRON-1916
> URL: https://issues.apache.org/jira/browse/METRON-1916
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
> Priority: Major
>
> Stellar can be executed in an environment where not all of the dependencies
> are available for each of the functions provided in Stellar Common. When this
> happens, the classpath resolver blows-up even if you do not use the function
> that is missing dependencies.
> For example this exception is thrown when not all of the dependencies are
> available for the new REST function. I was only attempting to use
> STATS_INIT, not the REST function.
> {code:java}
> 2018-11-30 15:47:00 DEBUG BaseFunctionResolver:165 - Resolving Stellar
> function in class RestFunctions.RestGet
> 2018-11-30 15:47:00 ERROR DefaultProfileBuilder:302 - Bad 'init' expression:
> error='Unable to parse: STATS_INIT() due to:
> org/apache/http/conn/HttpClientConnectionManager', expr='STATS_INIT()',
> profile='profile-with-stats', entity='global',
> variables-available='[adapter.threatinteladapter.end.ts, bro_timestamp,
> ip_dst_port, enrichmentsplitterbolt.splitter.end.ts,
> enrichmentsplitterbolt.splitter.begin.ts,
> adapter.hostfromjsonlistadapter.end.ts, adapter.geoadapter.begin.ts, uid,
> trans_depth, protocol, original_string, ip_dst_addr,
> threatinteljoinbolt.joiner.ts, host, enrichmentjoinbolt.joiner.ts,
> adapter.hostfromjsonlistadapter.begin.ts,
> threatintelsplitterbolt.splitter.begin.ts, ip_src_addr, user_agent,
> timestamp, method, request_body_len, uri, tags, source.type,
> adapter.geoadapter.end.ts, referrer, threatintelsplitterbolt.splitter.end.ts,
> adapter.threatinteladapter.begin.ts, ip_src_port, guid, response_body_len]'
> org.apache.metron.stellar.dsl.ParseException: Unable to parse: STATS_INIT()
> due to: org/apache/http/conn/HttpClientConnectionManager
> at
> org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:166)
> at
> org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154)
> at
> org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.execute(DefaultStellarStatefulExecutor.java:160)
> at
> org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.assign(DefaultStellarStatefulExecutor.java:95)
> at
> org.apache.metron.profiler.DefaultProfileBuilder.assign(DefaultProfileBuilder.java:291)
> at
> org.apache.metron.profiler.DefaultProfileBuilder.apply(DefaultProfileBuilder.java:136)
> at
> org.apache.metron.profiler.DefaultMessageDistributor.distribute(DefaultMessageDistributor.java:177)
> at
> org.apache.metron.profiler.storm.ProfileBuilderBolt.handleMessage(ProfileBuilderBolt.java:411)
> at
> org.apache.metron.profiler.storm.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:337)
> at
> org.apache.storm.topology.WindowedBoltExecutor$1.onActivation(WindowedBoltExecutor.java:307)
> at org.apache.storm.windowing.WindowManager.onTrigger(WindowManager.java:145)
> at
> org.apache.storm.windowing.WatermarkTimeTriggerPolicy.handleWaterMarkEvent(WatermarkTimeTriggerPolicy.java:80)
> at
> org.apache.storm.windowing.WatermarkTimeTriggerPolicy.track(WatermarkTimeTriggerPolicy.java:48)
> at org.apache.storm.windowing.WindowManager.track(WindowManager.java:177)
> at org.apache.storm.windowing.WindowManager.add(WindowManager.java:110)
> at
> org.apache.storm.windowing.WaterMarkEventGenerator.run(WaterMarkEventGenerator.java:80)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/http/conn/HttpClientConnectionManager
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getConstructor0(Class.java:3075)
> at java.lang.Class.getConstructor(Class.java:1825)
> at
> org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.createFunction(BaseFunctionResolver.java:240)
> at
> org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.resolveFunction(BaseFunctionResolver.java:194)
> at
> org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.resolveFunctions(BaseFunctionResolver.java:166)
> at com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:109)
> at
> org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:149)
> at
> org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:49)
> at
> org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:688)
> at
> org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:656)
> at
> org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259)
> at
> org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
> ... 21 more{code}
> When this happens the Classpath Function Resolver should allow me to use
> whichever functions can be loaded correctly.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
