[
https://issues.apache.org/jira/browse/NIFI-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406676#comment-16406676
]
Joseph Witt commented on NIFI-4994:
-----------------------------------
[~ijokarumawak] first of all thanks for a great writeup to help clarify the
issue.
I'm focused on this line "For example, ReportLineageToAtlas reporting task uses
Atlas client library, and the library has static code blocks to initialize its
dependencies, such as Kafka client with parameters passed by NiFi reporting
task based on user inputs. Once the initialization is done, there is no way to
re-initialize those objects"
In my opinion we should not make the proposed change to NiFi. We should work
with the atlas community to enable their client to behave in a more friendly
manner. This static block initialization construct is not a favorable model in
general and is tuned toward a model where a given atlas client can bind all the
things it needs when an application starts up. That might make sense inside
the traditional hadoop environment today but that doesn't mean this is a good
design.
I dont have good data to suggest whether we should do what you propose anyway
and would be curious of any side effects in terms of garbage
collection/heap/etc.. However, I'd prefer we avoid altering this for the case
noted and rather we work with the atlas team to improve their client.
I'm not saying i'd vote against this as-is but I'd really prefer we avoid it
unless we have a more compelling/broader case for it.
> Reload instance class loader at every component restart
> -------------------------------------------------------
>
> Key: NIFI-4994
> URL: https://issues.apache.org/jira/browse/NIFI-4994
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
> Priority: Major
>
> h2. Current Class loader reload mechanism
> NiFi framework provides RequiresInstanceClassLoading to annotate components
> (Processor, Controller Service or Reporting Task) to use per instance class
> loader. Instance class loaders are reloaded when associated components are
> restarted, if any component property flagged as
> 'dynamicallyModifiesClasspath' is updated. Such component property is used to
> let user specify additional class pass resource URLs. E.g. PutHDFS
> 'Additional Classpath Resources'.
> h2. Limitation of current mechanism
> Current mechanism requires component properties to trigger reloading to
> represent class path resource URLs. That works well in scenarios for users to
> add/del/mod additional class path resources.
> However there is different needs for reloading instance class loader. That is
> re-initializing static state in dependencies.
> For example, ReportLineageToAtlas reporting task uses Atlas client library,
> and the library has static code blocks to initialize its dependencies, such
> as Kafka client with parameters passed by NiFi reporting task based on user
> inputs. Once the initialization is done, there is no way to re-initialize
> those objects, meaning even if NiFi user changes reporting task
> configuration, those will not be able to reflected.
> https://github.com/apache/atlas/blob/master/notification/src/main/java/org/apache/atlas/hook/AtlasHook.java#L63
> The only possible operations to reset such static state are, creating another
> instance, configure the instance again, remove the old one and start new one.
> Or restart NiFi process. This limitation affects UX negatively.
> h2. Improvement design
> Possible approaches are:
> # Enhance RequiresInstanceClassLoading annotation so that custom components
> can specify whether it needs to be reloaded on every component restart
> without condition.
> # Add new PropertyDescriptor instance field to denote that instance class
> loader needs to be reloaded if a property is changed. Similar to
> 'dynamicallyModifiesClasspath' but for configurations other than URI. Then,
> include those property value to
> AbstractConfiguredComponent.additionalResourcesFingerprint
> No.1 is more naive approach, but require less framework code change. No.2
> looks redundant, also could be error-prone, e.g. by forgetting to mark some
> properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
