“EDP internal” I meant current EDP specific code. And since job configs are
job-specific
I’d prefer to have urls containing /jobs or at least /edp for that.
Regards,
Alexander Ignatov
On 24 Jan 2014, at 23:20, Matthew Farrellee <m...@redhat.com> wrote:
> what do you consider "EDP internal", and how does it relate to the v1.1 or v2
> API?
>
> i'm ok with making it plugin independent. i'd just suggest moving it out of
> /jobs and to something like /extra/config-hints/{type}, maybe along with
> /extra/validations/config.
>
> best,
>
>
> matt
>
> On 01/22/2014 06:25 AM, Alexander Ignatov wrote:
>> Current EDP config-hints are not only plugin specific. Several types of jobs
>> must have certain key/values and without it job will fail. For instance,
>> MapReduce (former Jar) job type requires Mapper/Reducer classes parameters
>> to be set[1]. Moreover, for such kind of jobs we already have separated
>> configuration defaults [2]. Also initial versions of patch implementing
>> config-hints contained plugin-independent defaults for all each job types
>> [3].
>> I remember we postponed decision about which configs are commmon for all
>> plugins and agreed to show users all vanilla-specific defaults. That's why
>> now
>> we have several TODOs in the code about config-hints should be
>> plugin-specific.
>>
>> So I propose to leave config-hints REST call in EDP internal and make it
>> plugin-independent (or job-specific) by removing of parsing all
>> vanilla-specific
>> defaults and define small list of configs which is definitely common for
>> each type of jobs.
>> The first things come to mind:
>> - For MapReduce jobs it's already defined in [1]
>> - Configs like number of map and reduce tasks are common for all type of jobs
>> - At least user always has an ability to set any key/value(s) as
>> params/arguments for job
>>
>>
>> [1] http://docs.openstack.org/developer/savanna/userdoc/edp.html#workflow
>> [2]
>> https://github.com/openstack/savanna/blob/master/savanna/service/edp/resources/mapred-job-config.xml
>> [3] https://review.openstack.org/#/c/45419/10
>>
>> Regards,
>> Alexander Ignatov
>>
>>
>>
>> On 20 Jan 2014, at 22:04, Matthew Farrellee <m...@redhat.com> wrote:
>>
>>> On 01/20/2014 12:50 PM, Andrey Lazarev wrote:
>>>> Inlined.
>>>>
>>>>
>>>> On Mon, Jan 20, 2014 at 8:15 AM, Matthew Farrellee <m...@redhat.com
>>>> <mailto:m...@redhat.com>> wrote:
>>>>
>>>> (inline, trying to make this readable by a text-only mail client
>>>> that doesn't use tabs to indicate quoting)
>>>>
>>>> On 01/20/2014 02:50 AM, Andrey Lazarev wrote:
>>>>
>>>> ------
>>>> FIX - @rest.get('/jobs/config-hints/____<job_type>') -
>>>> should move to
>>>> GET /plugins/<plugin_name>/<____plugin_version>, similar to
>>>> get_node_processes
>>>> and get_required_image_tags
>>>> ------
>>>> Not sure if it should be plugin specific right now. EDP
>>>> uses it
>>>> to show some
>>>> configs to users in the dashboard. it's just a cosmetic
>>>> thing.
>>>> Also when user
>>>> starts define some configs for some job he might not define
>>>> cluster yet and
>>>> thus plugin to run this job. I think we should leave it
>>>> as is
>>>> and leave only
>>>> abstract configs like Mapper/Reducer class and allow
>>>> users to
>>>> apply any
>>>> key/value configs if needed.
>>>>
>>>>
>>>> FYI, the code contains comments suggesting it should be
>>>> plugin specific.
>>>>
>>>>
>>>> https://github.com/openstack/____savanna/blob/master/savanna/____service/edp/workflow_creator/____workflow_factory.py#L179
>>>>
>>>> <https://github.com/openstack/__savanna/blob/master/savanna/__service/edp/workflow_creator/__workflow_factory.py#L179>
>>>>
>>>>
>>>> <https://github.com/openstack/__savanna/blob/master/savanna/__service/edp/workflow_creator/__workflow_factory.py#L179
>>>>
>>>> <https://github.com/openstack/savanna/blob/master/savanna/service/edp/workflow_creator/workflow_factory.py#L179>>
>>>>
>>>> IMHO, the EDP should have no plugin specific dependencies.
>>>>
>>>> If it currently does, we should look into why and see if we
>>>> can't
>>>> eliminate this entirely.
>>>>
>>>> [AL] EDP uses plugins in two ways:
>>>> 1. for HDFS user
>>>> 2. for config hints
>>>> I think both items should not be plugin specific on EDP API
>>>> level. But
>>>> implementation should go to plugin and call plugin API for result.
>>>>
>>>>
>>>> In fact they are both plugin specific. The user is forced to click
>>>> through a plugin selection (when launching a job on transient
>>>> cluster) or the plugin selection has already occurred (when
>>>> launching a job on an existing cluster).
>>>>
>>>> Since the config is something that is plugin specific, you might not
>>>> have hbase hints from vanilla but you would from hdp, and you
>>>> already have plugin information whenever you ask for a hint, my view
>>>> that this be under the /plugins namespace is growing stronger.
>>>>
>>>>
>>>> [AL] Disagree. They are plugin specific, but EDP itself could have
>>>> additional plugin-independent logic inside. Now config hints return EDP
>>>> properties (like mapred.input.dir) as well as plugin-specific
>>>> properties. Placing it under /plugins namespace will give a vision that
>>>> it is fully plugin specific.
>>>>
>>>> I like to see EDP API fully plugin independent and in one workspace. If
>>>> core side needs some information internally it can easily go into the
>>>> plugin.
>>>
>>> I'm not sure if we're disagreeing. We may, in fact, be in violent agreement.
>>>
>>> The EDP API is fully plugin independent, and should stay that way as a
>>> project goal. config-hints is extra data that the horizon app can use to
>>> help give users suggestions about what config they may want to optionally
>>> add to their job. Those config options are independent of the job and
>>> specific to the cluster where the job will run, which is the purview of the
>>> plugin.
>>>
>>> Moving config-hints out of the EDP API will make this even more clear.
>>>
>>> Best,
>>>
>>>
>>> matt
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
