Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

Plugins are platform extensions and provide a way to extend platform functionality, not an application extensions. It may be useful for others who participate in this discussion to provide a use case outside of Mesos/Kubertenes support that shows limitations of plugins and LIBRARY_JARS

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

The Apex platform system configuration is hardcoded in the Apex-core Java code. But the platform does not have a flexible system configuration. The plugins can only cover the run-time configuration of Apex applications. It means the plugins cannot be a part of the system configuration of the

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

Apex platform dependencies are already covered with the compile time check in place. Platform extensions are covered by plugins. Thank you, Vlad On 2/4/18 08:56, Sergey Golovko wrote: The usage of any Apex attributes is the generic configuration of Apex applications on the end-user level.

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

The usage of any Apex attributes is the generic configuration of Apex applications on the end-user level. But the subject of the discussion is to provide the system level configuration of Apex applications. I guess the having of the two different layers of the configuration (system and end-user)

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

Yes generic in the Attribute class > On Feb 3, 2018, at 10:00 AM, Vlad Rozov wrote: > > +1 assuming that support for merge/override will be generic for all > attributes that support list/set of values and not limited to LIBRARY_JARS > attribute only. > > Thank you, > >

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

+1 assuming that support for merge/override will be generic for all attributes that support list/set of values and not limited to LIBRARY_JARS attribute only. Thank you, Vlad On 2/3/18 09:13, Pramod Immaneni wrote: I too agree that the discussion has veered off from the original topic. Why

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

I too agree that the discussion has veered off from the original topic. Why can't LIBRARY_JARS be used for this, albeit with a minor improvement? Currently, our attribute layering is an override, so if you have an attribute that is specified as apex.application..attr. it overrides apex.attr. for

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

IMO, support for Kubernetes, Docker images, Mesos and anything outside of Yarn deployments is a topic by itself and design for such support needs to be discussed. I do not want to propose any specific design, but assume that logic to create proper execution environment would be coded into Apex

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

In cases where we have an "über" docker image containing support for multiple execution environments it might be useful for the Apex core to infer what kind of execution environment to use for a particular invocation (say based on configuration values/environment variables) and in that case the

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

Unfortunately the moving of .apa file to a docker image cannot resolve all problems with the dependencies. If we assume an Apex application should be run in different execution environments, the application docker image must contain all possible execution environment dependencies. I think the

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

The current dependencies are based on how Apex YARN client works. YARN depends on a DFS implementation for deployment (not necessarily HDFS). I think a better way to look at this is to consider that instead of an .apa file the application is a docker image, which would contain Apex and all

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

I agree. It can be implemented with usage of plugins. But if I need to enable and configurate the plugin I need to put this information into dt-site.xml. It means The plugin and its parameter must be documented and the list of the added specific jars will be visible and available for updates to

Re: [Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

There is a way to get the same end result by using plugins. It will be good to understand why plugin can't be used and can they be extended to provide the required functionality. Thank you, Vlad On 1/29/18 15:14, Sergey Golovko wrote: Hello All, In Apex there are two ways to deploy

[Proposal] Extension of the Apex configuration to add dependent jar files in runtime.

Hello All, In Apex there are two ways to deploy non-Hadoop jars to the deployed cluster. The first approach is static (hardcoded) and it is used by Apex platform developers only. There are several final static arrays of Java classes in StramClient.java that define which of the available jars