The usage of any Apex attributes is the generic configuration of Apex
applications on the end-user level. But the subject of the discussion is to
provide the system level configuration of Apex applications. I guess the
having of the two different layers of the configuration (system and
end-user) is a generic approach for all good designed tools.

Thanks,
Sergey


On Sat, Feb 3, 2018 at 10:02 AM, Pramod Immaneni <pra...@datatorrent.com>
wrote:

> Yes generic in the Attribute class
>
> > On Feb 3, 2018, at 10:00 AM, Vlad Rozov <vro...@apache.org> wrote:
> >
> > +1 assuming that support for merge/override will be generic for all
> attributes that support list/set of values and not limited to LIBRARY_JARS
> attribute only.
> >
> > Thank you,
> >
> > Vlad
> >
> > On 2/3/18 09:13, Pramod Immaneni wrote:
> >> I too agree that the discussion has veered off from the original topic.
> Why
> >> can't LIBRARY_JARS be used for this, albeit with a minor improvement?
> >> Currently, our attribute layering is an override, so if you have an
> >> attribute that is specified as apex.application.<appname>.
> attr.<attrname>
> >> it overrides apex.attr.<attrname> for that application. What if were to
> >> expand the attribute definition to allow for the specification of how
> the
> >> layering of attributes will be combined, override being one option,
> merge
> >> being another with these being implemented with a combiner interface?
> This
> >> way a set of common jars could be specified using dt.attr.LIBRARY_JARS
> and
> >> applications can still add extra jars on top.
> >>
> >> On Fri, Feb 2, 2018 at 6:32 PM, Vlad Rozov <vro...@apache.org> wrote:
> >>
> >>> IMO, support for Kubernetes, Docker images, Mesos and anything outside
> of
> >>> Yarn deployments is a topic by itself and design for such support
> needs to
> >>> be discussed. I do not want to propose any specific design, but assume
> that
> >>> logic to create proper execution environment would be coded into Apex
> >>> client. Whether it (hardcoded logic to create an execution
> environment) can
> >>> be expressed simply as a list of dependent classes or jars is at
> minimum
> >>> questionable. Until design is proposed and agreed upon, I'd prefer to
> use
> >>> plugins for the subject.
> >>>
> >>> Thank you,
> >>>
> >>> Vlad
> >>>
> >>>
> >>> On 2/2/18 13:17, Sanjay Pujare wrote:
> >>>
> >>>> In cases where we have an "├╝ber" docker image containing support for
> >>>> multiple execution environments it might be useful for the Apex core
> to
> >>>> infer what kind of execution environment to use for a particular
> >>>> invocation  (say based on configuration values/environment variables)
> and
> >>>> in that case the core will load the corresponding libraries. And I
> think
> >>>> this kind of flexibility or support would be difficult through the
> plugins
> >>>> hence I think Sergey's proposal will be useful.
> >>>>
> >>>> Sanjay
> >>>>
> >>>>
> >>>> On Fri, Feb 2, 2018 at 11:18 AM, Sergey Golovko <
> ser...@datatorrent.com>
> >>>> wrote:
> >>>>
> >>>> Unfortunately the moving of .apa file to a docker image cannot
> resolve all
> >>>>> problems with the dependencies. If we assume an Apex application
> should
> >>>>> be
> >>>>> run in different execution environments, the application docker image
> >>>>> must
> >>>>> contain all possible execution environment dependencies.
> >>>>>
> >>>>> I think the better way is to assume that the original application
> docker
> >>>>> image like the current .apa file should contain the application
> specific
> >>>>> dependencies only. And some smart client tool should create the
> >>>>> executable
> >>>>> application docker image form the original one and include the
> execution
> >>>>> specific environment dependencies into the target application docker
> >>>>> image.
> >>>>> It means anyway an smart client Apex tool should have an interface to
> >>>>> define different environment dependencies or combination of different
> >>>>> dimensions of the environment dependencies.
> >>>>>
> >>>>> Thanks,
> >>>>> Sergey
> >>>>>
> >>>>>
> >>>>> On Fri, Feb 2, 2018 at 10:23 AM, Thomas Weise <t...@apache.org>
> wrote:
> >>>>>
> >>>>> The current dependencies are based on how Apex YARN client works.
> YARN
> >>>>>> depends on a DFS implementation for deployment (not necessarily
> HDFS).
> >>>>>>
> >>>>>> I think a better way to look at this is to consider that instead of
> an
> >>>>>>
> >>>>> .apa
> >>>>>
> >>>>>> file the application is a docker image, which would contain Apex
> and all
> >>>>>> dependencies that the "StramClient"  today adds for YARN.
> >>>>>>
> >>>>>> In that world there would be no Apex CLI or Apex specific client.
> >>>>>>
> >>>>>> Thomas
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Feb 1, 2018 at 5:57 PM, Sergey Golovko <
> ser...@datatorrent.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>> I agree. It can be implemented with usage of plugins. But if I need
> to
> >>>>>>> enable and configurate the plugin I need to put this information
> into
> >>>>>>> dt-site.xml. It means The plugin and its parameter must be
> documented
> >>>>>>>
> >>>>>> and
> >>>>>> the list of the added specific jars will be visible and available
> for
> >>>>>>> updates to the end-user. The implementation via plugins is more
> dynamic
> >>>>>>> solution that is more convenient for the application developers.
> But
> >>>>>>>
> >>>>>> I'm
> >>>>>> talking about the static configuration of the Apex build or
> >>>>>> installation
> >>>>>> that relates more to the platform development.
> >>>>>>> The current Apex core implementation uses the static unchanged
> list of
> >>>>>>>
> >>>>>> jars
> >>>>>>
> >>>>>>> for long time, because the Apex implementation still contains
> several
> >>>>>>>
> >>>>>> basic
> >>>>>>
> >>>>>>> static assumptions (for instance, the usage of YARN, HDSF, etc.).
> And
> >>>>>>>
> >>>>>> the
> >>>>>> current Apex assumptions are hardcoded in the implementation. But
> if we
> >>>>>> are
> >>>>>>
> >>>>>>> going to improve Apex and use Java interfaces in generic Apex
> >>>>>>> implementation, the current static approach in Apex code to
> hardcode a
> >>>>>>>
> >>>>>> list
> >>>>>>
> >>>>>>> of dependent jars will not work anymore. It will require to
> include a
> >>>>>>>
> >>>>>> new
> >>>>>> solution to add/change jars in specific Apex builds/configurations.
> >>>>>> And I
> >>>>>> don't think the usage of the plugins will be good for that.
> >>>>>>> Thanks,
> >>>>>>> Sergey
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Feb 1, 2018 at 1:47 PM, Vlad Rozov <vro...@apache.org>
> wrote:
> >>>>>>>
> >>>>>>> There is a way to get the same end result by using plugins. It will
> >>>>>>> be
> >>>>>> good to understand why plugin can't be used and can they be extended
> >>>>>>> to
> >>>>>> provide the required functionality.
> >>>>>>>> Thank you,
> >>>>>>>>
> >>>>>>>> Vlad
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 1/29/18 15:14, Sergey Golovko wrote:
> >>>>>>>>
> >>>>>>>> Hello All,
> >>>>>>>>> In Apex there are two ways to deploy non-Hadoop jars to the
> deployed
> >>>>>>>>> cluster.
> >>>>>>>>>
> >>>>>>>>> The first approach is static (hardcoded) and it is used by Apex
> >>>>>>>>>
> >>>>>>>> platform
> >>>>>>> developers only. There are several final static arrays of Java
> >>>>>>>> classes
> >>>>>> in StramClient.java
> >>>>>>>>> that define which of the available jars should be included into
> >>>>>>>>>
> >>>>>>>> deployment
> >>>>>>>> for every Apex application.
> >>>>>>>>> The second approach is to add paths of all dependent jar-files to
> >>>>>>>>>
> >>>>>>>> the
> >>>>>> value
> >>>>>>>>> of the attribute LIB_JARS. The end-user can set/update the value
> of
> >>>>>>>>>
> >>>>>>>> the
> >>>>>>> attribute LIB_JARS via dt-site.xml files, command line parameters,
> >>>>>>>>> application properties and plugins. The usage of the
> >>>>>>>>> attribute LIB_JARS is the official documented way for all Apex
> users
> >>>>>>>>>
> >>>>>>>> to
> >>>>>>> manage by the deployment jars.
> >>>>>>>>> But some of the dependent jars (not from the Apex core) can be
> >>>>>>>>>
> >>>>>>>> common
> >>>>>> for
> >>>>>>>> all customer's applications for a specific installation and/or
> >>>>>>>> execution
> >>>>>>> environment. Unfortunately the Apex implementation does not contain
> >>>>>>>> the
> >>>>>>> middle solution that would allow the Apex developers and customer
> >>>>>>>> support
> >>>>>>>> to
> >>>>>>>>> define and add new dependent jar-files (jars that should not be
> >>>>>>>>> configurable/managed by the end-user) without the
> >>>>>>>>>
> >>>>>>>> updates/recompilation
> >>>>>>> of
> >>>>>>>
> >>>>>>>> the Apex Java code during the Apex building process and/or
> >>>>>>>>> installation/configuration.
> >>>>>>>>>
> >>>>>>>>> Also the having of such kind of flexibility would allow the Apex
> >>>>>>>>>
> >>>>>>>> core
> >>>>>> developers to use Java interfaces during the development to define
> >>>>>>>> an
> >>>>>> abstraction layer in Apex implementation and configurate Apex core
> >>>>>>>> to
> >>>>>> add
> >>>>>>>> some specific jars to all Apex applications without recompilation
> of
> >>>>>>>> the
> >>>>>>> Apex source code.
> >>>>>>>>> For instance, now the usage of HDFS is hardcoded in Apex platform
> >>>>>>>>>
> >>>>>>>> code
> >>>>>> but
> >>>>>>>> it can be replaced with any other distributed or cloud base file
> >>>>>>>> system.
> >>>>>>> The Apex core code can use an interface for all I/O operations but
> >>>>>>>> the
> >>>>>> supporting of a real specific file system implementation can be
> >>>>>>>> added
> >>>>>> as
> >>>>>>
> >>>>>>> an
> >>>>>>>>> independent jar-file. Or if the implementation of some of Apex
> >>>>>>>>>
> >>>>>>>> operators
> >>>>>>> depend on a specific service, and it is necessary to add some of
> the
> >>>>>>>>> service jars to every Apex application implicitly.
> >>>>>>>>>
> >>>>>>>>> The proposal:
> >>>>>>>>>
> >>>>>>>>> - add a predefined configuration text file (we can make any
> choice
> >>>>>>>>>
> >>>>>>>> for
> >>>>>> the
> >>>>>>>> file syntax: XML, JSON or Properties) to Apex engine resources
> with
> >>>>>>>>> predefined values of some of the Apex attributes (now we can
> include
> >>>>>>>>> LIB_JARS
> >>>>>>>>> attribute only);
> >>>>>>>>> - allow to have a configuration text file with the same
> >>>>>>>>>
> >>>>>>>> functionality
> >>>>>> in
> >>>>>>
> >>>>>>> the Apex installation folder "conf";
> >>>>>>>>> - read the content of the predefined configuration text files by
> the
> >>>>>>>>>
> >>>>>>>> stram
> >>>>>>>> client in runtime and add the jars to the list of the dependent
> >>>>>>>> jars;
> >>>>>> - allow to use paths to jars and Java classes to refer to the
> >>>>>>>> dependent
> >>>>>>> jars (the references can have the extensions: .class and .jar).
> >>>>>>>>> Thanks,
> >>>>>>>>> Sergey
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >
>
>

Reply via email to